Data processing device and data processing method

Abstract

The embodiment of the present application discloses a data processing device and a data processing method, which relate to the field of image processing, and at least solve the problem of high requirements for bus bandwidth and on-chip storage space and low computing efficiency in the prior art. The specific scheme is: the input feature map to be processed is transferred from the off-chip storage to the on-chip storage unit, and initialized and updated on the computing array according to the calculation situation, and at the same time, the spatial multiplexing of the convolution kernel weight data is performed on the computing array for parallel convolution calculation. After completing all the convolution calculations of an input feature map, the convolution calculations of other input feature maps are calculated, and the results are accumulated in sequence to obtain the output feature map.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a data processing device and a data processing method. Background Art

[0002] At present, convolutional neural networks have been widely used in scenarios such as image classification, video processing, speech recognition, and data analysis. As the number of objects that need to be processed (such as the number of images that need to be processed) and the complexity (such as the number of feature map data of the images that need to be processed) increase, higher requirements are placed on the computing power and storage bandwidth of the hardware architecture (or computing platform) that executes the convolutional neural network model. Among them, based on traditional hardware architectures, such as the computing power of the central processing unit (CPU), it can no longer meet the requirements of system performance (such as computing power and storage bandwidth) in the process of large-scale convolutional neural network reasoning. As for the graphics processing unit (GPU) with higher computing power, due to its high power consumption, the scope of use of GPU in convolutional neural network calculations is limited, such as it cannot be used in low-power, resource-constrained hardware architectures.

[0003] In addition, in order to meet the needs of different scenarios, the types of convolutional neural network models have become very diverse. For example, different convolutional neural network models have convolution kernels of different sizes to implement corresponding convolution calculations. This also requires the design of different hardware architectures to implement the reasoning process of convolutional neural networks in different scenarios.

[0004] Therefore, it is necessary to design a hardware architecture that is suitable for the reasoning of convolutional neural networks in different scenarios. Summary of the invention

[0005] The embodiments of the present application provide a data processing device and a data processing method, which at least solve the problems in the prior art of high requirements for bus bandwidth and on-chip storage space and low computing efficiency, and can adapt to convolution kernels of different sizes.

[0006] To achieve the above objectives, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a communication method and a data processing device, the device comprising: an off-chip storage unit, a first cache unit connected to the off-chip storage unit, a second cache unit connected to the off-chip storage unit, a computing array, a third cache unit, an arithmetic logic unit and a scheduling unit. The computing array comprises a plurality of computing units, the plurality of computing units comprises R rows and C columns, the computing units of R rows and C columns are sequentially connected in the row direction and the column direction, and R and C are both integers greater than 1. The first cache unit is also connected to the input end of the computing unit of the first row of the computing array. The second cache unit is also connected to the input end of all computing units of the computing array. The output end of the computing unit of the Rth row of the computing array is connected to the third cache unit. The third cache unit is also connected to the input end of the arithmetic logic unit. The scheduling unit is respectively connected to the off-chip storage unit, the first cache unit, the second cache unit, the computing array, the third cache unit and the arithmetic logic unit. The off-chip storage unit is used to store N input feature maps and M convolution kernels. N and M are both integers greater than or equal to 1. For each input feature map among the N input feature maps, the scheduling unit performs the following operations to obtain the convolution result of each input feature map and the corresponding weight data: the scheduling unit is used to store the feature map data of the first input feature map into the first cache unit, and store the weight data corresponding to the first input feature map in the M convolution kernels into the second cache unit, where the first input feature map is any feature map among the N input feature maps. The scheduling unit is also used to transmit the feature map data of the first input feature map stored in the first cache unit to the calculation array in units of 1 row and C columns in sequence, and map the weight data stored in the second cache unit to the calculation array, so as to perform convolution calculation on the calculation units in the calculation array, obtain the convolution result and store it in the third cache unit. The arithmetic logic unit is used to accumulate the convolution results corresponding to different channels of the same convolution kernel in the third cache unit to obtain M intermediate output feature maps corresponding to the M convolution kernels.

[0008] By adopting the above technical solution, the on-chip storage unit only needs to store a set of feature map data to be processed and the corresponding weight data at one time, which significantly reduces the requirements for on-chip storage space during the input feature map processing. By fixing the feature map data on the computing unit of the computing array and sliding the convolution window on the computing array to realize the convolution calculation method (or the method of fixing the feature map data), it is possible to support convolution calculations of convolution kernel sizes of any size, that is, it can adapt to convolution kernels of different sizes. In addition, by updating the feature map data to the computing array in units of 1 row and C columns, the calculation of an input feature map and a channel is uninterrupted, which significantly shortens the time of the entire calculation process.

[0009] In a possible design, the scheduling unit is used to transfer the feature map data of the first R rows and C columns in the first input feature map stored in the first cache unit, starting from the feature map data of the 1st column to the Cth column in the 1st row, from the input end of the 1st row of the computing unit of the computing array, in units of 1 row and C columns, in a cascade manner, so as to sequentially transfer the feature map data of the first R rows and C columns of the first input feature map to the computing units of the Rth row to the 1st row of the computing array for convolution calculation. The step size of the convolution kernel used for convolution calculation is S, and S is an integer greater than or equal to 1. When the convolution calculation of the feature map data in the first S rows from the 1st column to the Cth column in the first input feature map is completed, the scheduling unit is also used to control the feature map data in the 1st row to the RSth row in the calculation array to be transferred downward by S rows in a cascade manner, and the feature map data in the R+1th row to the R+S+1th row of the first input feature map stored in the first cache unit from the 1st column to the Cth column is transmitted to the first S rows of the calculation unit in the calculation array through the input end of the 1st row of the calculation unit, and so on, until the feature map data in the H-R+1th row from the 1st column to the Cth column in the first input feature map is transmitted to the 1st row of the calculation unit in the calculation array, where H is the number of rows of feature map data included in the first input feature map, and H is an integer greater than R. When the convolution calculation of the feature map data in the H-R+1th row to the Cth column in the first input feature map is completed, the scheduling unit is also used to control the feature map data in the 1st row to the RSth row in the calculation array to be transferred downward by S rows in a cascade manner. If Then the first input feature map stored in the first cache unit in the first row Column to The characteristic graph data of the column is transmitted to the first row of the computing unit of the computing array through the input end of the first row of the computing unit of the computing array; if Then the first input feature map stored in the first cache unit in the first row Column to The feature map data of the column is transmitted to the first row of the computing unit of the computing array through the input end of the first row of the computing unit of the computing array. And so on, until the feature map data of the L-C+1th column to the Lth column in the H-R+1th row in the first input feature map is transmitted to the first row of the computing unit of the computing array, where L is the number of columns of the feature map data included in the first input feature map, K2 is the number of columns of the weight data corresponding to the first input feature map in one of the M convolution kernels, L is an integer greater than C, and K2 is an integer greater than or equal to 1. Based on this scheme, in the convolution calculation of an input feature map and a channel, the feature map data of the input feature map only needs to be transmitted once through the bus, and the other transmission processes are all realized by cascading between computing units, which reduces the requirements for bus bandwidth during the calculation process.

[0010] In a possible design, the scheduling unit is used to perform the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map corresponds to the first channels of the M convolution kernels, and the first channel corresponds to the first input feature map: the scheduling unit is used to map the first weight data stored in the second cache unit to the computing units of the R-K1+1th row to the first K2 columns of the Rth row of the computing array, so that the computing units of the R-K1+1th row to the first K2 columns of the Rth row perform convolution calculations according to the first weight data. When the convolution calculations of the computing units of the R-K1+1th row to the first K2 columns of the Rth row are completed, the first weight data is controlled to be transferred to the right by S columns in a cascade manner, so that the computing units of the R-K1+1th row to the Sth column of the Rth row to the K2+S-1th column perform convolution calculations according to the first weight data, and so on, until the first weight data is transferred to the computing units of the C-K2+1th column to the Cth column of the R-K1+1th row to the Rth row. The first weight data is the weight data of the first channel of any one of the M convolution kernels, wherein K1 is the number of rows of weight data included in the first channel of one of the M convolution kernels, and K1 is an integer greater than or equal to 1. When the convolution calculation of the computing units from the R-K1+1th row to the C-K2+1th column to the Cth column of the R-K1+1th row to the R-th row is completed, the scheduling unit remaps the first weight data to the computing units from the R-K1+1th row to the first K2 columns of the R-K1+1th row of the computing array until the convolution calculation of all feature map data of the first input feature map and the first weight data is completed. Based on this scheme, by fixing the feature map data on the computing units of the computing array and sliding the convolution window on the computing array to realize the convolution calculation method (or the method of fixing the feature map data), it is possible to support convolution calculations of convolution kernel sizes of any size, that is, it can adapt to convolution kernels of different sizes.

[0011] In one possible design, the scheduling unit is used to perform the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map corresponds to the first channels of the M convolution kernels, and the first channel corresponds to the first input feature map: the scheduling unit is used to repeatedly map the first weight data stored in the second cache unit to the first channel of the calculation array; Row to row R, columns 1 to 1 Column, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data. When the convolution calculation is completed, the first weight data repeatedly mapped to the calculation array is controlled to be transferred to the right S columns in a cascade manner, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data, and so on, until the first weight data is passed to the calculation units from the R-K1+1th row to the Rth row, from the C-K2+1th column to the Cth column. The first weight data is the weight data of the first channel of any one of the M convolution kernels, where K1 is the number of rows of the weight data of the first channel, K2 is the number of columns of the weight data of the first channel, and K1 and K2 are both integers greater than or equal to 1. When the convolution calculation of the calculation units from the R-K1+1th row to the C-K2+1th column to the Cth column of the calculation array is completed, the scheduling unit re-maps the first weight data to the Row to row R, columns 1 to 1 In the calculation units of the columns, the convolution calculation of all feature map data of the first input feature map and the first weight data is completed. Based on this solution, by using the convolution window to perform convolution calculation in the calculation array for spatial multiplexing, the number of times the convolution window slides in the calculation array during the convolution calculation process is reduced, and the parallel calculation efficiency of the calculation units in the calculation array is improved.

[0012] In a possible design, the output end of the arithmetic logic unit is connected to the off-chip storage unit. The arithmetic logic unit is also used to perform pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit. The scheduling unit is also used to store the processed intermediate output feature maps in the off-chip storage unit. Based on this solution, after completing the convolution calculation, pooling and / or activation processing can also be performed.

[0013] In a possible design, the data processing device also includes: a fourth cache unit connected to the output end of the arithmetic logic unit, and the fourth cache unit is also connected to the scheduling unit. The arithmetic logic unit is also used to perform pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit, and store the processed intermediate output feature maps in the fourth cache unit. The scheduling unit is also used to store the processed intermediate output feature maps stored in the fourth cache unit in an off-chip storage unit. Based on this solution, when the bus resources are insufficient or the off-chip storage unit is busy so that the processed result cannot be directly transmitted to the off-chip storage unit, the result can be first stored in the fourth cache unit, and the result can be transmitted to the off-chip storage unit when appropriate.

[0014] In a possible design, the third cache unit is also connected to the off-chip storage unit. The scheduling unit is also used to store the N input feature maps stored in the off-chip storage unit to the third cache unit. The arithmetic logic unit is also used to activate the N input feature maps stored in the third cache unit. The scheduling unit is also used to store the activated N input feature maps to the off-chip storage unit. Based on this solution, it is possible to support the operation of activating data first and then convolving it.

[0015] In a second aspect, an embodiment of the present application also provides a data processing method, which is applied to a data processing device, and the data processing device includes: an off-chip storage unit, a first cache unit connected to the off-chip storage unit, a second cache unit connected to the off-chip storage unit, a computing array, a third cache unit, an arithmetic logic unit and a scheduling unit. Among them, the computing array includes multiple computing units, and the multiple computing units include R rows and C columns. The computing units of R rows and C columns are connected in sequence in the row direction and the column direction, and R and C are both integers greater than 1. The first cache unit is also connected to the input end of the computing unit of the first row of the computing array. The second cache unit is also connected to the input end of all computing units of the computing array. The output end of the computing unit of the Rth row of the computing array is connected to the third cache unit. The third cache unit is also connected to the input end of the arithmetic logic unit. The scheduling unit is respectively connected to the off-chip storage unit, the first cache unit, the second cache unit, the computing array, the third cache unit and the arithmetic logic unit. The method may include: the off-chip storage unit stores N input feature maps and M convolution kernels. N and M are both integers greater than or equal to 1. For each of the N input feature maps, the scheduling unit performs the following operations to obtain the convolution result of each input feature map and the weight data of the corresponding channel: the scheduling unit stores the feature map data of the first input feature map in the first cache unit, and stores the weight data corresponding to the first input feature map in the M convolution kernels in the second cache unit, where the first input feature map is any feature map of the N input feature maps. The scheduling unit transmits the feature map data of the first input feature map stored in the first cache unit to the calculation array in units of 1 row and C columns in sequence, maps the weight data stored in the second cache unit to the calculation array, so that the calculation units in the calculation array perform convolution calculations, obtain the convolution results and store them in the third cache unit. The arithmetic logic unit accumulates the convolution results corresponding to different channels of the same convolution kernel in the third cache unit to obtain M intermediate output feature maps corresponding to the M convolution kernels.

[0016] In a possible design, the scheduling unit sequentially transmits all feature map data of the first input feature map stored in the first cache unit to the computing array in units of 1 row and C columns, including: the scheduling unit sequentially transmits the feature map data of the first R rows and C columns of the first input feature map stored in the first cache unit, starting from the feature map data of the 1st column to the Cth column in the 1st row, from the input end of the computing unit of the 1st row of the computing array, in units of 1 row and C columns in a cascade manner, so as to sequentially transmit the feature map data of the first R rows and C columns of the first input feature map to the computing units of the Rth row to the 1st row of the computing array for convolution calculation. The step size of the convolution kernel used for convolution calculation is S, and S is an integer greater than or equal to 1. When the convolution calculation of the feature map data in the first S rows from the 1st column to the Cth column in the first input feature map is completed, the scheduling unit controls the feature map data in the 1st row to the RSth row in the calculation array to be transferred downward by S rows in a cascade manner, and transmits the feature map data in the R+1th row to the R+S+1th row of the first input feature map stored in the first cache unit to the first S rows of the calculation units in the calculation array through the input end of the 1st row of the calculation unit, and so on, until the feature map data in the 1st column to the Cth column in the H-R+1th row of the first input feature map is transmitted to the 1st row of the calculation unit of the calculation array, where H is the number of rows of feature map data included in the first input feature map, and H is an integer greater than R. When the convolution calculation of the feature map data in the H-R+1th row to the Cth column in the first input feature map is completed, the scheduling unit controls the feature map data in the 1st row to the RSth row in the calculation array to be transferred downward by S rows in a cascade manner. If Then the first input feature map stored in the first cache unit in the first row Column to The characteristic graph data of the column is transmitted to the first row of the computing unit of the computing array through the input end of the first row of the computing unit of the computing array; if Then the first input feature map stored in the first cache unit in the first row Column to The feature map data of the column is transmitted to the first row of the computing unit of the computing array through the input end of the first row of the computing unit of the computing array. And so on, until the feature map data of the L-C+1th column to the Lth column in the H-R+1th row in the first input feature map is transmitted to the first row of the computing unit of the computing array, where L is the number of columns of feature map data included in the first input feature map, K2 is the number of columns of weight data corresponding to the first input feature map in one of the M convolution kernels, L is an integer greater than C, and K2 is an integer greater than or equal to 1.

[0017] In a possible design, the scheduling unit maps the weight data stored in the second cache unit to the computing array so that the computing units in the computing array perform convolution calculations to obtain convolution results, including:

[0018] The scheduling unit performs the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map corresponds to the first channels of the M convolution kernels, and the first channel corresponds to the first input feature map: the scheduling unit maps the first weight data stored in the second cache unit to the computing units of the R-K1+1th row to the first K2 columns of the Rth row of the computing array, so that the computing units of the R-K1+1th row to the first K2 columns of the Rth row perform convolution calculations according to the first weight data. When the convolution calculations of the computing units of the R-K1+1th row to the first K2 columns of the Rth row are completed, the first weight data is controlled to be transferred to the right by S columns in a cascade manner, so that the computing units of the R-K1+1th row to the Sth column to the K2+S-1th column of the Rth row perform convolution calculations according to the first weight data, and so on, until the first weight data is transferred to the computing units of the C-K2+1th column to the Cth column of the R-K1+1th row to the Rth row. The first weight data is the weight data of the first channel of any one of the M convolution kernels, where K1 is the number of rows of weight data included in the first channel of one of the M convolution kernels, and K1 is an integer greater than or equal to 1. When the convolution calculation of the calculation units from the R-K1+1th row to the C-K2+1th column to the Cth column of the Rth row is completed, the scheduling unit remaps the first weight data to the calculation units from the R-K1+1th row to the first K2 columns of the Rth row of the calculation array until the convolution calculation of all feature map data of the first input feature map and the first weight data is completed.

[0019] In one possible design, the scheduling unit maps the weight data stored in the second cache unit to the computing array so that the computing units in the computing array perform convolution calculations to obtain convolution results, including: the scheduling unit performs the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map corresponds to the first channels of the M convolution kernels, and the first channel corresponds to the first input feature map: the scheduling unit repeatedly maps the first weight data stored in the second cache unit to the first channels of the computing array; Row to row R, columns 1 to 1 Column, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data. When the convolution calculation is completed, the first weight data repeatedly mapped to the calculation array is controlled to be transferred to the right S columns in a cascade manner, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data, and so on, until the first weight data is passed to the calculation units from the R-K1+1th row to the Rth row, from the C-K2+1th column to the Cth column. The first weight data is the weight data of the first channel of any one of the M convolution kernels, where K1 is the number of rows of the weight data of the first channel, K2 is the number of columns of the weight data of the first channel, and K1 and K2 are both integers greater than or equal to 1. When the convolution calculation of the calculation units from the R-K1+1th row to the C-K2+1th column to the Cth column of the calculation array is completed, the scheduling unit re-maps the first weight data to the Row to row R, columns 1 to 1 The convolution calculation of all feature map data and the first weight data in the calculation units of the column up to the first input feature map is completed.

[0020] In one possible design, the output end of the arithmetic logic unit is connected to the off-chip storage unit. The method also includes: the arithmetic logic unit performs pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit. The scheduling unit stores the processed intermediate output feature maps in the off-chip storage unit.

[0021] In one possible design, the data processing device further includes: a fourth cache unit connected to the output end of the arithmetic logic unit, and the fourth cache unit is also connected to the scheduling unit. The method also includes: the arithmetic logic unit performs pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit, and stores the processed intermediate output feature maps in the fourth cache unit. The scheduling unit stores the processed intermediate output feature maps stored in the fourth cache unit in an off-chip storage unit.

[0022] In one possible design, the third cache unit is also connected to the off-chip storage unit. The method further includes: the scheduling unit stores the N input feature maps stored in the off-chip storage unit to the third cache unit. The arithmetic logic unit activates the N input feature maps stored in the third cache unit. The scheduling unit stores the activated N input feature maps to the off-chip storage unit.

[0023] It should be noted that the second aspect and its possible design can correspond one-to-one to the above-mentioned first aspect and its possible design, and the beneficial effects brought about by them also correspond one-to-one, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A logical composition diagram of a convolutional neural network provided in an embodiment of the present application;

[0025] Figure 2 A logical schematic diagram of performing convolution calculation on an input feature map in the convolution layer provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a process of performing convolution calculation on N input feature maps and convolution kernel 1 provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the hardware architecture of a convolutional neural network provided by the prior art;

[0028] Figure 5 A schematic diagram of the architecture of a data processing device provided in an embodiment of the present application;

[0029] Figure 6 A logical schematic diagram of a convolution calculation provided in an embodiment of the present application;

[0030] Figure 7 A logical schematic diagram of another convolution calculation provided in an embodiment of the present application;

[0031] Figure 8 A logical schematic diagram of another convolution calculation provided in an embodiment of the present application;

[0032] Fig. 9 A schematic diagram of data initialization provided in an embodiment of the present application;

[0033] Fig.10 A schematic diagram of a convolution calculation method provided in an embodiment of the present application;

[0034] Fig.11 A sliding schematic diagram of a convolution window provided in an embodiment of the present application;

[0035] Fig.12 A schematic diagram of a method for spatial multiplexing of weighted data provided in an embodiment of the present application;

[0036] Fig.13 A schematic diagram of a data updating method provided in an embodiment of the present application;

[0037] Fig.14 A schematic diagram of another data updating method provided in an embodiment of the present application;

[0038] Fig.15 A flowchart of a data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] Convolutional neural networks can process various types of data based on convolutional neural network model parameters. Generally speaking, in convolutional neural networks, data processing can include operations such as convolution, pooling, and activation.

[0040] For example, please refer to Figure 1 , is a logical composition diagram of a convolutional neural network. Figure 1 As shown, the convolutional neural network may include multiple logical layers, namely, an input layer, a convolutional layer, a pooling and / or activation layer, and an output layer. The input layer may be used to store input data. The input data stored in the input layer may be processed by convolution calculation of the convolutional layer, pooling and / or activation of the pooling and / or activation layer, and then the output data processed by the convolutional neural network may be obtained in the output layer. The convolution calculation and pooling and / or activation processing of the input data may be performed once or multiple times.

[0041] Here, the input data is one or more input feature maps. The convolution layer may include one or more convolution kernels, each convolution kernel may include one or more channels, and each channel may include one or more weight data. The channels included in the convolution kernel are similar to the input feature maps stored in the input layer. Figure 1 One to one correspondence. When the input feature map is convolved in the convolution layer, the feature map data on each input feature map can be convolved with the weight data on the channel corresponding to the input feature map in each convolution kernel. For ease of description, in the embodiment of the present application, this process may also be referred to as the convolution calculation of the input feature map and the corresponding channel. After all channels of a convolution kernel are convolved with the corresponding input feature map, the same number of convolution results as the number of channels of the convolution kernel can be obtained. These convolution results are accumulated to obtain the result of the convolution calculation of the input feature map using the convolution kernel (or referred to as the output feature map). Similar calculations can be performed on other convolution kernels in the convolution layer, and finally the same number of output feature maps as the number of convolution kernels are obtained.

[0042] For example, assume that the number of input feature maps is N, the number of convolution kernels in the convolution layer is M, and each convolution kernel includes N input feature maps. Figure 1 One corresponds to N channels. Please refer to Figure 2 , which shows the logical diagram of the convolution calculation of the input feature map in this convolution layer. Figure 2 As shown, the M convolution kernels included in the convolution layer are used to perform convolution calculations with the N input feature maps to obtain M output feature maps corresponding to the convolution kernels. For each of the M convolution kernels, please refer to Figure 3 , shows the process of convolution calculation of N input feature maps and a convolution kernel (such as convolution kernel 1). Figure 3 As shown, the input features Figure 1Convolution is performed with channel 1 in convolution kernel 1 to obtain convolution result 1-1. Input features Figure 2 Convolution calculation is performed with channel 2 in convolution kernel 1 to obtain convolution result 2-1; and by analogy, input feature map N is convolved with channel N in convolution kernel 1 to obtain convolution result N-1. By adding the above convolution results 1-1 to N-1, the result of convolution of N input feature maps with convolution kernel 1 can be obtained, that is, the output feature map is obtained. Figure 1 Similarly, the N input feature maps can also be convolved with other convolution kernels to obtain other output feature maps, that is, to obtain output feature maps. Figure 2 , output features Figure 3 ,…, output feature map M.

[0043] It is understandable that a large number of multiplication and addition operations are required during the convolution calculation process. As the amount of feature map data included in the input feature map increases, the amount of calculation and the amount of intermediate data will also increase significantly, which places higher demands on the computing power and on-chip storage capacity of the hardware architecture that carries the convolution calculation. At the same time, the sizes of convolution kernels of different convolutional neural network models are also different. Therefore, in order to be suitable for the calculation of different types of convolutional neural network models, the hardware architecture also needs to be able to adapt to convolution kernels of different sizes.

[0044] In order to solve the above problem, in the prior art, a convolution kernel of any size can be divided into 1×1 convolution templates, and the convolution result corresponding to the convolution kernel can be obtained by accumulating the results calculated using the divided convolution templates. Figure 4 As shown, the multiplication and addition operation of the convolution calculation can be realized by a basic calculation unit array composed of multiple calculation units. The interconnection structure of the basic unit array can be composed of a multi-stage pipeline: in the row direction, the basic calculation unit array shares the input feature map data and realizes parallel calculation by using different weight parameters; in the column direction, the calculation results of the basic calculation unit array are passed row by row and participate in the calculation as the input of the next row. In this way, all the convolution calculations of a channel in the convolution kernel are completed in each calculation unit in turn, and the convolution calculation results of the channel are passed downward. Each column of basic calculation units can calculate and output an output feature map respectively, realizing the parallel processing of the convolution calculations of a feature map and multiple different convolution kernels.

[0045] However, during the calculation process, a large amount of intermediate data will be generated, which needs to be stored in the on-chip storage space (such as Figure 4At the same time, the feature map data needs to be repeatedly transmitted once in each calculation cycle, which requires a large on-chip storage space. Since the amount of input feature map data is generally very large, when the on-chip storage space cannot store all of it, it is necessary to use off-chip storage space (such as Figure 4 The data and parameters shown in the figure are taken from the off-chip cache) and the process is completed through the system bus, which will place very high requirements on the bus bandwidth. In addition, since all convolution calculations of each channel are performed in one computing unit, it will lead to problems such as long calculation time and low calculation efficiency.

[0046] The data processing device provided in the embodiment of the present application can effectively reduce the demand for on-chip storage space and bus bandwidth, improve the computational efficiency of convolution calculations, and can adapt to convolution kernels of different sizes.

[0047] Please refer to Figure 5 , is a schematic diagram of the architecture of a data processing device provided in an embodiment of the present application. Figure 5 As shown, the data processing device may include an off-chip storage unit, a first cache unit, a second cache unit, a computing array, a third cache unit, an arithmetic logic unit, and a scheduling unit.

[0048] Wherein, the computing array includes a plurality of computing units, such as the computing array includes R rows and C columns of computing units. Inside the computing array, the R rows and C columns of computing units can be interconnected in the form of a two-dimensional cascade. R and C are both integers greater than or equal to 1. Wherein, the first cache unit, the second cache unit and the third cache unit are all storage units arranged on the chip. The hardware architecture can be implemented in one or more ways of a field programmable gate array (FPGA), a programmable array logic (PLA) and / or an integrated circuit (such as an integrated circuit for a specific application (ASIC)).

[0049] In this architecture, Figure 5As shown, the off-chip storage unit can be connected to the first cache unit and the second cache unit respectively. The first cache unit can also be connected to the input end of the 1st row of computing units in the computing array. The second cache unit can also be connected to each computing unit in the computing array. The output end of the Rth row (i.e., the last row) in the computing array can also be connected to the third cache unit. The third cache unit can also be connected to the input end of one or more arithmetic logic units. The above-mentioned off-chip storage unit, the first cache unit, the second cache unit, the computing array including R rows and C columns of computing units, the third cache unit and the arithmetic logic unit are all connected to the scheduling unit.

[0050] Further, such as Figure 5 As shown, the data processing device may further include a fourth cache unit, and the output end of one or more arithmetic logic units may be connected to the off-chip storage unit through the fourth cache unit. Alternatively, the output end of one or more arithmetic logic units may be connected to the off-chip storage unit ( Figure 5 not shown).

[0051] Based on the above Figure 5 The data processing device shown can perform neural network reasoning. Specifically: the off-chip storage unit stores convolutional neural network model parameters (such as weight data and feature map data, etc.). The scheduling unit can store the feature map data of the input feature map to be calculated in the first cache unit, and store the weight data of different convolution kernels corresponding to the feature map data stored in the first cache unit in the second cache unit. The scheduling unit can input the feature map data in the first cache unit and the weight data of different convolution kernels in the second cache unit into the calculation array. The calculation units of the calculation array can transfer data (such as calculation results, feature map data or weight data) and perform convolution calculations in the row direction and / or column direction in a cascade manner to obtain the convolution results of different channels of different convolution kernels and store them in the third cache unit. The arithmetic logic unit can accumulate the convolution results corresponding to different channels of the same convolution kernel in the third cache unit to obtain the intermediate output feature map corresponding to different convolution kernels.

[0052] Furthermore, the arithmetic logic unit can also perform operations such as pooling and activation on the intermediate output feature map in the third cache unit according to the parameters of the current convolutional neural network model. The scheduling unit can also transmit the calculation result processed by the arithmetic logic unit to the off-chip storage unit or transmit it to the off-chip storage unit through the fourth cache unit.

[0053] The above-mentioned off-chip storage unit is a device or equipment with storage function. It should be noted that: Figure 5The data processing device shown is shown as an example in which the off-chip storage unit is included in the data processing device. In some other embodiments, the off-chip storage unit can also be arranged outside the data processing device, and it can exchange data with the data processing device through an interface, which is not limited in the embodiment of the present application.

[0054] The following combination Figure 5 The data processing device shown in the figure describes the implementation process of the present application in detail.

[0055] In order to more clearly illustrate the process of the embodiment of the present application, the following assumes that there are N input feature maps, each of which includes H rows and L columns of feature map data, and M convolution kernels used for convolution calculation. Among them, each convolution kernel has N channels, and the N channels of each convolution kernel are related to the N input feature maps. Figure 1 One-to-one, each channel includes K1 rows and K2 columns of weight data, the step size of each convolution kernel is S, and the calculation array includes R rows and C columns of calculation units. Among them, R, C, N, M, K1, K2 and S are all integers greater than 1, H is an integer greater than R, and L is an integer greater than C.

[0056] It should be noted that the number of rows and / or columns of different input feature maps in the N input feature maps may be the same or different. The size and / or step size of different convolution kernels in the M convolution kernels may be the same or different. In this embodiment, the size of the convolution kernel may be represented by the number of rows and columns of the weight data of the channel included in the convolution kernel.

[0057] In adopting Figure 5 When the data processing device shown in the figure performs convolutional neural network calculation, the result of the convolution calculation can be obtained through the following process. Figure 6 As shown, the data processing device can process any one of the N input feature graphs (such as the input feature Figure 1 ), respectively convolve with the corresponding channel of each of the M convolution kernels (such as channel 1 of each convolution kernel), and obtain M convolution results of convolution with channel 1 of each convolution kernel. For example, obtain convolution result 1-1 of convolution with channel 1 of convolution kernel 1, convolution result 1-2 of convolution with channel 1 of convolution kernel 2, and convolution result 1-N of convolution with channel 1 of convolution kernel N. Then, as Figure 7 As shown, the data processing device can process any other one of the N input feature graphs (such as the input feature Figure 2), respectively, convolve with the corresponding channel of each of the M convolution kernels (such as channel 2 of each convolution kernel), and obtain M convolution results of convolution with channel 2 of each convolution kernel. For example, obtain convolution result 2-1 of convolution with channel 2 of convolution kernel 1, convolution result 2-2 of convolution with channel 2 of convolution kernel 2, and convolution result 2-N of convolution with channel 2 of convolution kernel N. And so on, until the data processing device completes the corresponding convolution calculation of each input feature map of all N input feature maps, and obtains M*N convolution results. Figure 8 As shown, the data processing device can then accumulate the convolution results obtained by convolving different channels of the same convolution kernel among these convolution results to obtain an intermediate output feature map corresponding to the convolution kernel. For example, the data processing device can accumulate the convolution results 1-1, 2-1, and finally N-1 obtained by convolution calculations on different channels in convolution kernel 1 to obtain an intermediate output feature map corresponding to convolution kernel 1. Figure 1 ; Convolution calculations are performed on different channels in convolution kernel 2 to obtain convolution results 1-2, convolution results 2-2, and finally convolution results N-2, and the intermediate output features corresponding to convolution kernel 2 are obtained. Figure 2 ; Convolution calculations are performed on different channels in the convolution kernel M to obtain the convolution results 1-M, convolution results 2-M, and finally convolution results NM, which are accumulated to obtain the intermediate output feature map M corresponding to the convolution kernel M.

[0058] The above process is described in detail below with reference to the accompanying drawings.

[0059] In the data processing device, the off-chip storage unit is used to store the above-mentioned N input feature maps and M convolution kernels.

[0060] For each of the N input feature maps stored in the off-chip storage unit, the scheduling unit may obtain the convolution result of each input feature map and the weight data of the corresponding channel by performing operations 1 to 4.

[0061] In this embodiment, the scheduling unit can serve as the control center of the data processing device to control each step in the convolutional neural network model calculation process. The following takes the first input feature map of N input feature maps as an example to illustrate the process in which the scheduling unit obtains the convolution result of the input feature map and the weight data of the corresponding channel by scheduling each unit module of the data processing device. The first input feature map is any one of the N input feature maps, and the channel corresponding to the first input feature map is called the first channel.

[0062] Operation 1: The scheduling unit is used to store the feature map data of the first input feature map into the first cache unit, and store the weight data of the first channel of each convolution kernel in the M convolution kernels into the second cache unit.

[0063] Combination Figure 5 In the architecture shown, all data required for convolutional neural network inference (such as parameters, feature map data, and weight data of the convolutional neural network model) are stored in an off-chip storage unit with high storage capacity. The scheduling unit can store the feature map data to be calculated, such as the feature map data of the first input feature map, and the weight data of the channel corresponding to the first input feature map, such as the first channel, from the off-chip storage unit to the on-chip storage unit. For example, the feature map data of the first input feature map is stored in the first cache unit, and the weight data of the channel corresponding to the first input feature map is stored in the second cache unit.

[0064] The scheduling unit may transmit the data in the first cache unit and the second cache unit to the computing array so that the computing units of the computing array perform convolution calculations to obtain M convolution results corresponding to all feature map data in the first input feature map and the first channels of the M convolution kernels. Specifically, the convolution process of the first channel of one of the M convolution kernels with the first input feature map may include the following operations 2 and 3.

[0065] Operation 2: The scheduling unit is used to transfer all feature map data of the first input feature map stored in the first cache unit in units of 1 row and C columns from the first cache unit to the computing array in sequence, for performing convolution calculations on the computing units in the computing array.

[0066] Operation 3: The scheduling unit is used to map the weight data of the first channel of each of the M convolution kernels stored in the second cache unit to the computing unit of the computing array, and perform convolution calculation.

[0067] Among them, the scheduling unit can transmit the feature map data of the first input feature map stored on the chip to the computing array in units of 1 row and C columns.

[0068] Exemplarily, the scheduling unit may pass the feature map data of the first R rows and C columns of the first input feature map stored in the first cache unit, starting from the feature map data of the 1st column to the Cth column in the 1st row, from the input end of the 1st row of the computing unit of the computing array, in units of 1 row and C columns, in a cascade manner, so as to fill the computing array with the feature map data of the first R rows and C columns of the input feature map for convolution calculation of the computing array. In the embodiment of the present application, the process of filling the computing array with the feature map data of the first R rows and C columns of the input feature map is called data initialization. It should be noted that when initialization is completed, the feature map data of the first R rows and C columns of the first input feature map are sequentially transmitted to the computing units from the Rth row to the 1st row of the computing array. That is to say, the feature map data on the computing units of the Rth row of the computing array are the feature map data from the 1st column to the Cth column in the 1st row of the first input feature map, and the feature map data on the computing units of the R-1th row of the computing array are the feature map data from the 1st column to the Cth column in the 2nd row of the first input feature map, and so on. The feature map data on the computing units of the 1st row of the computing array are the feature map data from the 1st column to the Cth column in the Rth row of the first input feature map.

[0069] For example, see Fig. 9 , which is a schematic diagram of data initialization provided in an embodiment of the present application. Taking C=4 as an example, the scheduling unit can Fig. 9 The feature map data of the first column to the fourth column in the first row of the input feature map shown in (a) is input into the first row of the calculation unit of the calculation array. The result after input into the calculation array is as follows Fig. 9 The feature map data from columns 1 to 4 in row 1 of the input feature map will be passed down one row in the calculation array in a cascade manner, and the result is as follows: Fig. 9 As shown in (c) in the figure, the feature map data from the 1st column to the 4th column in the 1st row of the input feature map will be transferred to the 2nd row of the calculation unit of the calculation array. The above process is repeated until these feature map data are transferred to the last row of the calculation array, that is, the Rth row. The result is as follows Fig. 9 As shown in (d) in .

[0070] In the process of transferring the feature map data of the 1st to 4th columns in the 1st row of the input feature map downward row by row in a cascade manner, the scheduling unit may transfer the feature map data of the 1st to 4th columns in the 2nd row of the input feature map to the computing array in the same manner as transferring the feature data of the 1st row, and transfer it downward row by row in a cascade manner in the computing array until the feature map data of the 1st to 4th columns in the 2nd row of the input feature map is transferred to the second to last row (i.e., the R-1th row) of the computing array. This process is repeated until the feature map data of the first R rows and 4 columns of the input feature map fill the computing units of the computing array, and the result is as follows: Fig. 9As shown in (e) in . That is, the feature map data on the calculation unit of the Rth row of the calculation array is the feature map data of the 1st to 4th columns in the 1st row of the first input feature map, the feature map data on the calculation unit of the R-1th row of the calculation array is the feature map data of the 1st to 4th columns in the 2nd row of the first input feature map, and so on, the feature map data on the calculation unit of the 1st row of the calculation array is the feature map data of the 1st to 4th columns in the Rth row of the first input feature map.

[0071] In the above data initialization process, the scheduling unit may also map the weight data of the first channel of each convolution kernel in the M convolution kernels, which corresponds to the first input feature map, to the calculation array, that is, perform the above operation 3. Afterwards, the feature map data in the calculation array may be convolved by moving the weight data mapped to the calculation array. In this embodiment, the process of moving the weight data mapped to the calculation array may be referred to as a sliding convolution window.

[0072] Taking the example of performing convolution calculation using the first channel of one of the M convolution kernels and the feature map data in the computing array (i.e., the feature map data of the first R rows and C columns of the first input feature map), the scheduling unit can map the weight data (such as first weight data) of the first channel of the convolution kernel stored in the second cache unit to the computing units from the R-K1+1th row to the first K2 columns of the Rth row of the computing array, so that the computing units from the R-K1+1th row to the first K2 columns of the Rth row perform convolution calculation according to the first weight data. When the convolution calculation of the computing units from the R-K1+1th row to the first K2 columns of the Rth row is completed, the scheduling unit can control the first weight data to be transferred S columns to the right in a cascade manner, so that the computing units from the Sth column to the K2+S-1th column from the R-K1+1th row to the Rth row perform convolution calculation according to the first weight data, and so on, until the first weight data is transferred to the computing units from the C-K2+1th column to the Cth column from the R-K1+1th row to the Rth row, so that the corresponding position calculation units perform convolution calculation according to the first weight data.

[0073] For example, with C = 4, K1 = K2 = 3, S = 1, and X ijk Represents the feature map data of the jth row and kth column in the i-th input feature map, expressed as W ijk Take the weight data of the jth row and kth column of the ith channel of a convolution kernel as an example. After data initialization, the feature map data distribution on the last three rows of computing units on the computing array is as follows: Fig.10 The scheduling unit can map the first weight data to the last three rows in the calculation array, that is, the calculation units from the 1st column to the 3rd column in the R-2nd row to the Rth row. The mapping result of the first weight data is as follows: Fig.10 As shown in (b) in the figure. 111The computing unit where it is located can multiply the feature map data on the computing unit with the corresponding weight data, that is, the calculation performed by the computing unit is X 111 *W 111 , using R 111 Represents X 111 *W 111 The multiplication result of . Similarly, other calculation units in the convolution window can also perform multiplication operations on the corresponding data. In this way, the multiplication operation in the convolution window can obtain 9 calculation results, which are R 111 , R 112 , R 113 , R 121 , R 122 , R 123 , R 131 , R 132 and R 133 The calculation results are distributed as follows Fig.10 After obtaining the results of these multiplication operations, the convolution calculation results corresponding to the convolution window can be obtained by accumulating them.

[0074] In the embodiment of the present application, each computing unit is connected in a two-dimensional cascade. Therefore, each calculation result in the convolution window can be passed to the next row / column computing unit through cascade connection as input data of the next row / column computing unit. Finally, the convolution result corresponding to the convolution window can be obtained. For example, the transfer of the calculation result of the multiplication operation can be as follows: Fig.10 As shown in (d) in Fig.10 As shown in (d) in the convolution window, the calculation results can be passed backward in the row direction and then passed downward in the last column. 131 It can be passed to the calculation unit in the first row and second column of the convolution window, which calculates the R 132 With the received R 131 Add up and get R 132 +R 131 Then it is passed to the calculation unit in the first row and third column of the convolution window. Similarly, R 131 +R 132 +R 133 , R 121 +R 122 +R 123 and R 111 +R 112 +R 113 The calculation units in the 3rd column, 1st row and 2nd row can pass the corresponding accumulated results downward through the cascade and accumulate them by the next level of calculation units. Finally, R can be obtained on the calculation unit in the 3rd row and 3rd column of the convolution window.111 +R 112 +R 113 +R 121 +R 122 +R 123 +R 131 +R 132 +R 133 The accumulated result is the convolution result corresponding to the convolution window.

[0075] When the convolution calculation of the convolution window on the last three rows in the calculation array, that is, the calculation units from the 1st column to the 3rd column in the R-2nd row to the Rth row, is completed, the first weight data can be transferred to the right by 1 column in a cascade manner and the convolution calculation is performed. For example, please refer to Fig.11 When the convolution window is Fig.11 After the convolution calculation of the position shown in (a) is completed, each weight data in the convolution window can be transferred to the right by cascading one column. The position of the convolution window and the first weight data after the transfer is shown in Fig.11 The calculation unit at the corresponding position of the convolution window can perform convolution calculation on the feature map data at the corresponding position. The calculation process is the same as Fig.10 The calculation process in the example shown is similar. Repeating the above operation, the convolution window can slide in the row direction in units of step size and perform convolution kernel calculation until the convolution window slides to the rightmost end of the calculation array and completes the convolution calculation at this position.

[0076] The above example is to illustrate the case where only one convolution window is mapped to the computing array at one time. In other embodiments of the present application, the weight data can also be mapped in a spatial multiplexing manner. Exemplarily, the scheduling unit is used to repeatedly map the first weight data stored in the second cache unit to the first weight data stored in the computing array. Row to row R, columns 1 to 1 Columns in order to calculate the Row to row R, columns 1 to 1 The corresponding calculation unit in the column performs convolution calculation according to the first weight data; when the convolution calculation is completed, the first weight data is controlled to be transmitted to the right S columns in a cascade manner so that the first From row to row R, from column S+1 to column S+2 The calculation unit of the column performs convolution calculation according to the first weight data, and so on, until the first weight data is passed to the calculation unit of the C-K2+1 column to the C column from the R-K1+1 row to the R row. In other words, the scheduling unit can map a convolution window to different positions on the calculation array multiple times at one time. For example, please refer to Fig.12, taking C=11, K1=K2=3, S=1 as an example. The first weight data can be mapped three times at different positions of the last three rows of the calculation array at one time. The convolution window and the corresponding weight data after mapping three times are as follows: Fig.12 As shown in (a) in FIG. 1 . That is, the last three rows of the computation array contain three convolution windows at the same time, so that parallel computation can be performed in the computation array, and the computation results of the three convolution windows can be obtained at the same time. After obtaining the computation results corresponding to each convolution window, the weight data in the corresponding convolution window can be transferred to the right by one column through cascading to realize the sliding of the convolution window. The convolution window after sliding and the corresponding weight data are shown in FIG. Fig.12 As shown in (b) in the figure, the computing unit at the corresponding position continues to perform convolution calculations on other feature map data on the last three rows of computing units. Until the rightmost convolution window slides to the rightmost end of the computing array and completes the convolution calculation at that position.

[0077] After the above calculation process, the convolution calculation of the feature map data from the 1st column to the Cth column in the first S rows of the first input feature map is completed. Figure 10-12 In the example shown, the convolution calculation is completed for the pixel data from the 1st column to the Cth column in the 1st row in the first input feature map.

[0078] When the convolution calculation of the pixel data from the 1st column to the Cth column in the first S rows in the first input feature map is completed, the scheduling unit can be used to control the feature map data from the 1st row to the RSth row in the calculation array to be passed down by S rows in a cascade manner, and the feature map data from the 1st column to the Cth column in the R+1th row to the R+S+1th row of the first input feature map stored in the first cache unit is transmitted to the first S rows of the calculation units in the calculation array through the input end of the 1st row of the calculation unit of the calculation array, and so on, until the feature map data from the 1st column to the Cth column in the H-R+1th row of the first input feature map is transmitted to the 1st row of the calculation unit of the calculation array.

[0079] When the convolution calculation involving the feature map data in columns 1 to C in the first S rows of the first input feature map is completed, the scheduling unit can transfer the other feature map data of the first input feature map to the computing array. In the embodiment of the present application, this process can also be referred to as updating the feature map data on the computing array. For example, please refer to Fig.13 , take C = 4, the size of the convolution kernel K1 = K2 = 3, and the step size of the convolution kernel S = 1 as an example. When the data is initialized, the scheduling unit schedules, such as Fig.13 The feature map data of the first R rows and 4 columns of the input feature map shown in (a) are filled with the calculation array, and the result is as follows Fig.13As shown in (b) in FIG. 1 . When the convolution calculation of the feature map data of the first to fourth columns in the first row of the first input feature map is completed, the scheduling unit can control the feature map data in the computing units of the first to the R-1th rows of the computing array to be transferred downward by one row through cascading, and the feature map data in the computing unit of the Rth row (i.e., the feature map data of the first four columns of the first row of the input feature map) is eliminated. For example, Fig.13 As shown in (b) of FIG. 1 , the feature map data on the first row of the original calculation array is transferred to the second row of the calculation unit in the direction shown by arrow 1301, and so on, the feature map data on the R-1th row of the original calculation array is transferred to the Rth row of the calculation unit in the direction shown by arrow 1302. The scheduling unit can also take 1 row and 4 columns as a unit, and input the feature map data of the R+1th row and 4th column of the first input feature map into the first row of the calculation array, and the result is shown in FIG. Fig.13 After completing the update of the feature map data in the calculation array, the calculation units in the last three rows of the calculation array can continue to perform convolution calculations on the updated feature map data in the calculation array. The convolution calculation process is similar to Figure 10-12 The convolution calculation process for the feature map data from the 1st column to the Cth column in the 1st row of the first input feature map in the example shown is similar and will not be described in detail here.

[0080] In this way, the convolution calculation of the feature map data from the 1st column to the Cth column in the first 2S rows in the first input feature map is completed. By repeating the above process, the calculation array can continuously perform convolution calculation on the feature data from the 1st row to the first C columns of the H-R+1th row in the first input feature map.

[0081] When the convolution calculation of the feature map data from the 1st column to the Cth column in the H-R+1th row in the first input feature map is completed, the scheduling unit is also used to control the feature map data on the calculation units in the 1st row to the RSth row in the calculation array to be passed down S rows in a cascade manner, and transfer other feature map data of the first input feature map stored in the first cache unit to the calculation array depending on the situation.

[0082] For example, if Then the first input feature map stored in the first cache unit in the first row Column to The characteristic graph data of the column is transmitted to the first row of the computing unit of the computing array through the input end of the first row of the computing unit of the computing array; if Then the first input feature map stored in the first cache unit in the first row To The feature map data of the column is transmitted to the 1st row of the computing unit of the computing array through the input end of the 1st row of the computing unit of the computing array, and so on, until the feature map data of the L-C+1th column to the Lth column in the H-R+1th row in the first input feature map is transmitted to the 1st row of the computing unit of the computing array.

[0083] For example, please refer to Fig.14 , take C = 4, the size of the convolution kernel K1 = K2 = 3, and the step size of the convolution kernel S = 1 as an example. When the convolution calculation involving the feature map data from the 1st column to the 4th column in the H-R+1th row is completed, the scheduling unit schedules, such as Fig.14 The feature map data of the last R rows from the 1st column to the 4th column in the input feature map shown in (a) are filled with the calculation array, and the result is as follows Fig.14 As shown in (b) in FIG. 1 . When the convolution calculation of the feature map data in the 1st to 4th columns in the H-R+1th row of the input feature map is completed, the scheduling unit can control the feature map data in the calculation units in the 1st to R-1th rows of the calculation array to be transferred downward by 1 row respectively through cascading, and the feature map data in the calculation unit in the Rth row (i.e., the feature map data in the first 4 columns of the H-R+1th row of the input feature map) is eliminated. For example, Fig.14 As shown in (b), the feature map data on the first row of the original calculation array is transferred to the second row of the calculation unit in the direction shown by arrow 1401, and so on, the feature map data on the R-1th row of the original calculation array is transferred to the Rth row of the calculation unit in the direction shown by arrow 1402. The scheduling unit can also input the feature map data of the third to sixth columns in the first row of the input feature map into the first row of the calculation array in units of 1 row and 4 columns. Please refer to Fig.14 (c) in FIG. 1 shows a portion of the feature map data including the feature map data of the 3rd to 6th columns in the 1st row in the input feature map. The result of performing the above input operation is as follows: Fig.14 After completing the update of the feature map data in the calculation array, the calculation units in the last three rows of the calculation array can continue to perform convolution calculations on the updated feature map data in the calculation array. The convolution calculation process is similar to Figure 10-12The convolution calculation process for the feature map data from the 1st column to the Cth column in the 1st row of the first input feature map in the example shown is similar and will not be described in detail here. And so on, until the feature map data from the L-C+1th column to the Lth column in the H-R+1th row of the first input feature map is transmitted to the 1st row calculation unit of the calculation array, and the convolution calculation is completed. In this way, the convolution calculation of the first input feature map and the first channel of a convolution kernel is completed to obtain a convolution result corresponding to the first channel of all feature map data in the first input feature map and a convolution kernel. Afterwards, the above operations 2 and 3 can be re-executed so that the first input feature map is convolved with the first channels of other convolution kernels to obtain M convolution results corresponding one-to-one to all feature map data in the first input feature map and the first channels of M convolution kernels.

[0084] It can be seen that as the convolution calculation proceeds, the data in the calculation array is always filled. Therefore, during the convolution calculation process of the first input feature map and a channel (such as the first channel of a convolution kernel), there is no need to initialize the data again, which improves the calculation efficiency.

[0085] Operation 4: After obtaining M convolution results that correspond one-to-one to all feature map data in the first input feature map and the first channels of the M convolution kernels, the scheduling unit controls these convolution results to be transmitted to the third cache unit through the output end of the Rth row computing unit of the computing array.

[0086] It can be understood that a convolution result is composed of multiple calculation results. Exemplarily, each time a calculation result corresponding to a convolution window is obtained, the scheduling unit can control the calculation unit that finally accumulates and obtains the calculation result to pass the calculation result downward in a cascade manner until it is output to the third cache unit through the output end of the Rth row of calculation units in the calculation array. In this way, after completing all calculations of the first input feature map and the first channel in a convolution kernel, a complete convolution result corresponding to the convolution process is stored in the third cache unit.

[0087] Similarly, after completing the convolution calculation of the first input feature map and the first channel of all M convolution kernels, the above steps 1 to 4 can be performed on other feature maps in the N input feature maps, and M convolution results can be obtained for each input feature map and stored in the third cache unit.

[0088] After storing the M convolution results corresponding to each of the N input feature maps in the third cache unit, the arithmetic logic unit can be used to accumulate the convolution results corresponding to different channels of the same convolution kernel in the third cache unit to obtain M intermediate output feature maps corresponding to the M convolution kernels.

[0089] According to the above description, M convolution results corresponding to each input feature map in all N input feature maps are stored in the third cache unit. Then, the scheduling unit can control the arithmetic logic unit to accumulate all the convolution results obtained by convolution of different channels of the same convolution kernel in the M convolution kernels to obtain the intermediate output feature map corresponding to the convolution kernel.

[0090] For example, N=3, M=4, that is, the number of input feature maps is 3, respectively: input feature Figure 1 , input features Figure 2 and input features Figure 3 The number of convolution kernels is 4, namely: convolution kernel 1, convolution kernel 2, convolution kernel 3 and convolution kernel 4. Each convolution kernel includes 3 channels, namely channel 1, channel 2 and channel 3. After completing the convolution calculation of the three input feature maps (i.e., executing the above steps 1 to 4 respectively), the third cache unit may include the following convolution results: Input feature Figure 1 The convolution result obtained by convolution with channel 1 of convolution kernel 1 is 1-1, input feature Figure 1 Convolution result 1-2 obtained by convolution with channel 1 of convolution kernel 2, input feature Figure 1 Convolution result 1-3 obtained by convolution with channel 1 of convolution kernel 3, input feature Figure 1 Convolution results 1-4 obtained by convolution with channel 1 of convolution kernel 4; input features Figure 2 Convolution result 2-1 obtained by convolution with channel 2 of convolution kernel 1, input feature Figure 2 The convolution result 2-2 obtained by convolution with channel 2 of convolution kernel 2, input feature Figure 2 Convolution result 2-3 obtained by convolution with channel 2 of convolution kernel 3, input feature Figure 2 Convolution result 2-4 obtained by convolution with channel 2 of convolution kernel 4; input features Figure 3 Convolution result 3-1 obtained by convolution with channel 3 of convolution kernel 1, input feature Figure 3 Convolution result 3-2 obtained by convolution with channel 3 of convolution kernel 2, input feature Figure 3 Convolution result 3-3 obtained by convolution with channel 3 of convolution kernel 3, input feature Figure 3 Convolution result 3 is obtained by convolving with channel 3 of convolution kernel 4.

[0091] Under the control of the scheduling unit, the arithmetic logic unit can accumulate the convolution results 1-1, 2-1 and 3-1 of channel 1, channel 2 and channel 3 of convolution kernel 1 to obtain the intermediate output feature Figure 1 The intermediate output features Figure 1It is the intermediate output feature map obtained after all three input feature maps are convolved by convolution kernel 1. Similarly, the arithmetic logic unit can accumulate the convolution results 1-2, 2-2, and 3-2 of channel 1, channel 2, and channel 3 of convolution kernel 2 to obtain the intermediate output feature map. Figure 2 The intermediate output features Figure 2 It is the intermediate output feature map obtained after all three input feature maps are convolved by convolution kernel 2. The arithmetic logic unit can also accumulate the convolution results 1-1, 2-1 and 3-1 of channel 1, channel 2 and channel 3 of convolution kernel 3 to obtain the intermediate feature map. Figure 3 The intermediate feature Figure 3 It is the intermediate output feature map obtained after all three input feature maps are convolved by convolution kernel 3.

[0092] In the above embodiment, the M convolution results corresponding to each input feature map in the N input feature maps are stored in the third cache unit, and then accumulated to obtain M intermediate output feature maps. In some other embodiments, since the convolution results of different input feature maps are input into the third cache unit in sequence, that is, the third cache unit will first store the convolution results of one input feature map and the corresponding channels of all convolution kernels, and then store the convolution results of another input feature map and other channels corresponding to all convolution kernels. Therefore, in the embodiment of the present application, after the first input feature map and the first channel of each convolution kernel of the M convolution kernels complete the convolution calculation to obtain the corresponding M convolution results and store them in the third cache unit, when the third cache unit receives the convolution results of any other input feature map of the N input feature maps and the corresponding channels, the arithmetic logic unit can accumulate the stored convolution results of the first input feature map with the received convolution results, that is, the accumulation process of the convolution results of different channels in the same convolution kernel can also be synchronized with the process of storing the convolution results in the third cache unit. For example, let's take N=3 and M=4 as an example. Figure 1 After the convolution calculations are completed, the third cache unit can store the convolution results corresponding to channel 1 of all the convolution kernels in the four convolution kernels. Figure 2 When the convolution result of channel 2 of convolution kernel 1 is transmitted to the third cache unit, the arithmetic logic unit can combine the convolution result with the input feature vector stored in the third cache unit. Figure 1 Accumulate the convolution result with channel 1 of convolution kernel 1 and replace the input feature with the accumulated result Figure 1 Convolution with convolution kernel 1 channel 1. In this way, the input feature Figure 2After the convolution calculations involved are completed, the third cache unit stores the four convolution results obtained by adding the channel 1 of each of the four convolution kernels and the channel 2 of the convolution kernel. In other words, in the third cache unit, only the M intermediate output feature maps, which are the same as the number of convolution kernels, need to be saved at the same time. This can significantly reduce the capacity requirements for the third cache unit, and thus reduce the requirements for on-chip storage space. After the accumulation of all convolution results is completed, the M intermediate output feature maps obtained after the convolution calculation of N input feature maps and M convolution kernels are stored in the third cache unit.

[0093] When it is necessary to perform operations such as activation and / or pooling on the M intermediate output feature maps stored in the above-mentioned third cache unit, the arithmetic logic unit can also be used to perform corresponding activation and / or pooling operations on the M intermediate output feature maps stored in the third cache unit under the control of the scheduling unit, and transmit the output result to the off-chip storage unit, or transmit it to the off-chip storage unit through the fourth cache unit.

[0094] In the above examples, the calculation of the convolutional neural network is described by taking the convolution first and then the activation as an example. In other embodiments, Figure 5 The data processing device shown can also be used to support convolutional neural network calculations that are activated first and then convolved. Exemplarily, the scheduling unit can store the input feature map from the off-chip storage unit into the third cache unit. The arithmetic logic unit can perform an activation operation on the input feature map stored in the third cache unit under the control of the scheduling unit. For example, the activation operation can be a nonlinear calculation using a Relu function. After obtaining the result after the activation operation (such as the activated input feature map), the arithmetic logic unit can transfer these activated input feature maps to the off-chip storage unit, or to the off-chip storage unit through the fourth cache unit. And these activated input feature maps stored in the off-chip storage unit can be transferred to the on-chip storage space (such as the first cache unit) under the control of the scheduling unit to perform the convolution calculation in the above example.

[0095] It should be noted that, in the above example, the input feature map stored by the scheduling unit in the first cache unit is taken as a complete input feature map. In other embodiments, the scheduling unit may also divide the N input feature maps according to the resource situation of the computing array, and then store the divided feature map data to be calculated in the first cache unit. For example, when the data volume of the first input feature map is large (such as greater than the storage capacity of the first cache unit), the scheduling unit may divide the first input feature map to obtain a plurality of feature maps with relatively small data volumes, and each of these relatively small feature maps is used as a feature map to be calculated, and is stored in the first cache unit in turn, and the above calculations are performed respectively. For another example, when the data volume of the input feature map is small, the scheduling unit may store a plurality of input feature maps in the first cache unit at one time, so as to reduce the number of data interactions between the on-chip storage unit and the off-chip storage unit.

[0096] By adopting the above technical solution, the on-chip storage unit only needs to store a set of feature map data to be processed and the corresponding weight data at one time, which significantly reduces the requirements for on-chip storage space during the input feature map processing. In the convolution calculation of an input feature map and a channel, the feature map data of the input feature map only needs to be transmitted once through the bus, and the other transmission processes are all realized by cascading between the computing units, which reduces the requirements for bus bandwidth during the calculation process. By fixing the feature map data on the computing unit of the computing array and sliding the convolution window on the computing array to realize the convolution calculation method (or the method of fixing the feature map data), it is possible to support convolution calculations of convolution kernel sizes of any size, that is, it can adapt to convolution kernels of different sizes. In addition, by updating the feature map data to the computing array in units of 1 row and C columns, the calculation of an input feature map and a channel is uninterrupted, which significantly shortens the time consumption of the entire calculation process.

[0097] In addition, only the same number of convolution results as the number of convolution kernels (i.e., the number of output feature maps) need to be stored at one time, further reducing the requirement for on-chip storage space during the processing of input feature maps. By using convolution windows to perform spatial multiplexing in the computing array for convolution calculations, the number of times the convolution window slides in the computing array during the convolution calculation process is reduced, thereby improving the parallel computing efficiency of the computing units in the computing array.

[0098] The present application also provides a data processing method, which can be applied to the above Figure 5 Data processing device. Based on the data processing method, the demand for on-chip storage space and bus bandwidth can be effectively reduced, the computational efficiency of convolution calculations can be improved, and convolution kernels of different sizes can be adapted.

[0099] In order to more clearly illustrate the process of the embodiment of the present application, the following assumes that there are N input feature maps, each of which includes H rows and L columns of feature map data, and M convolution kernels used for convolution calculation. Among them, each convolution kernel has N channels, and the N channels of each convolution kernel are related to the N input feature maps. Figure 1 One-to-one, each channel includes K1 rows and K2 columns of weight data, the step size of each convolution kernel is S, and the calculation array includes R rows and C columns of calculation units. Among them, R, C, N, M, K1, K2 and S are all integers greater than 1, H is an integer greater than R, and L is an integer greater than C. Fig.15 As shown, the data processing method may include S1501-S1503.

[0100] S1501. The scheduling unit stores feature map data of a first input feature map in a first cache unit, and stores weight data corresponding to the first input feature map in M ​​convolution kernels in a second cache unit, where the first input feature map is any feature map among the N input feature maps.

[0101] Exemplarily, the scheduling unit may store weight data of a first channel of each of the M convolution kernels in a second cache unit, where the first channel corresponds to the first input feature map.

[0102] S1502, the scheduling unit transmits the feature map data of the first input feature map stored in the first cache unit to the computing array in units of 1 row and C columns, and maps the weight data stored in the second cache unit to the computing array so that the computing units in the computing array perform convolution calculations, obtain convolution results and store them in the third cache unit.

[0103] Exemplarily, the scheduling unit transmits the feature map data of the first R rows and C columns of the first input feature map stored in the first cache unit, starting from the feature map data of the 1st column to the Cth column in the 1st row, from the input end of the 1st row of the computing unit of the computing array, in units of 1 row and C columns, in a cascade manner, so as to sequentially transmit the feature map data of the first R rows and C columns of the first input feature map to the computing units of the Rth row to the 1st row of the computing array for convolution calculation. The step size of the convolution kernel used for convolution calculation is S, and S is an integer greater than or equal to 1.

[0104] When the convolution calculation of the feature map data from the 1st to the Cth column in the first S rows of the first input feature map is completed, the scheduling unit controls the feature map data from the 1st to the RSth rows in the calculation array to be passed down by S rows in a cascade manner, and transmits the feature map data from the 1st to the Cth columns in the R+1th to the R+S+1th rows of the first input feature map stored in the first cache unit to the first S rows of the calculation units in the calculation array through the input end of the 1st row of the calculation unit of the calculation array, and so on, until the feature map data from the 1st to the Cth column in the H-R+1th row of the first input feature map is transmitted to the 1st row of the calculation unit of the calculation array, where H is the number of rows of feature map data included in the first input feature map, and H is an integer greater than R.

[0105] When the convolution calculation of the feature map data from the 1st column to the Cth column in the H-R+1th row in the first input feature map is completed, the scheduling unit controls the feature map data from the 1st row to the RSth row in the calculation array to be passed down S rows through cascading.

[0106] if Then the first input feature map stored in the first cache unit in the first row Column to The feature map data of the column is transmitted to the computing unit in the first row of the computing array through the input end of the computing unit in the first row of the computing array.

[0107] if Then the first input feature map stored in the first cache unit in the first row Column to The feature map data of the column is transmitted to the computing unit in the first row of the computing array through the input end of the computing unit in the first row of the computing array.

[0108] And so on, until the feature map data from the L-C+1th column to the Lth column in the H-R+1th row in the first input feature map is transmitted to the 1st row computing unit of the computing array, where L is the number of columns of feature map data included in the first input feature map, K2 is the number of columns of weight data corresponding to the first input feature map in one of the M convolution kernels, L is an integer greater than C, and K2 is an integer greater than or equal to 1.

[0109] The scheduling unit performs the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map correspond to the first channels of the M convolution kernels one by one, and the first channel corresponds to the first input feature map:

[0110] The scheduling unit maps the first weight data stored in the second cache unit to the calculation units of the R-K1+1th row to the first K2 columns of the Rth row of the calculation array, so that the calculation units of the R-K1+1th row to the first K2 columns of the Rth row perform convolution calculations according to the first weight data. When the convolution calculations of the calculation units of the R-K1+1th row to the first K2 columns of the Rth row are completed, the first weight data is controlled to be transferred to the right by S columns in a cascade manner, so that the calculation units of the Sth column to the K2+S-1th column of the Rth row to the Rth row perform convolution calculations according to the first weight data, and so on, until the first weight data is transferred to the calculation units of the C-K2+1th column to the Cth column of the R-K1+1th row to the Rth row. The first weight data is the weight data of the first channel of any one of the M convolution kernels, wherein K1 is the number of rows of weight data included in the first channel of one of the M convolution kernels, and K1 is an integer greater than or equal to 1.

[0111] When the convolution calculation of the computing units from the R-K1+1th row to the Rth row and the C-K2+1th column to the Cth column is completed, the scheduling unit remaps the first weight data to the computing units from the R-K1+1th row to the first K2 columns of the Rth row of the computing array until the convolution calculation of all feature map data of the first input feature map and the first weight data is completed.

[0112] In the above examples, mapping one convolution window on the computing array at a time is used as an example for explanation. In other embodiments of the present application, multiple convolution windows can also be mapped on the computing array at a time so that the computing array can perform convolution calculations on multiple convolution windows at the same time.

[0113] Exemplarily, the scheduling unit performs the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map correspond to the first channels of the M convolution kernels one by one, and the first channel corresponds to the first input feature map:

[0114] The scheduling unit repeatedly maps the first weight data stored in the second cache unit to the first Row to row R, columns 1 to 1 Column, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data. When the convolution calculation is completed, the first weight data repeatedly mapped to the calculation array is controlled to be transferred to the right by S columns in a cascade manner, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data, and so on, until the first weight data is transferred to the calculation unit from the C-K2+1th column to the Cth column from the R-K1+1th row to the Rth row. The first weight data is the weight data of the first channel of any convolution kernel among the M convolution kernels, where K1 is the number of rows of the weight data of the first channel, K2 is the number of columns of the weight data of the first channel, and K1 and K2 are both integers greater than or equal to 1.

[0115] When the convolution calculation of the computing units from the R-K1+1th row to the Rth row, from the C-K2+1th column to the Cth column is completed, the scheduling unit remaps the first weight data to the Row to row R, columns 1 to 1 In the calculation units of the column, until the convolution calculation of all feature map data of the first input feature map and the first weight data is completed.

[0116] S1503. The arithmetic logic unit accumulates the convolution results corresponding to different channels of the same convolution kernel in the third cache unit to obtain M intermediate output feature maps corresponding to the M convolution kernels.

[0117] In the embodiment of the present application, when other processing (such as pooling and / or activation processing) is required for the above-mentioned intermediate output feature map, the arithmetic logic unit can perform pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit. The scheduling unit can store the processed intermediate output feature map in an off-chip storage unit.

[0118] It should be noted that, in the process of the scheduling unit storing the processed intermediate output feature map to the off-chip storage unit through the bus, the storage process may be blocked due to insufficient bus resources or the busy off-chip storage unit. Therefore, in the embodiment of the present application, the scheduling unit can also cache the processed intermediate output feature map through the fourth cache unit set in the data processing device, and store the processed intermediate output feature map in the off-chip storage unit when the bus resources and the off-chip storage unit allow. That is, the arithmetic logic unit performs pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit respectively. The scheduling unit stores the processed intermediate output feature map to the off-chip storage unit through the fourth cache unit.

[0119] In the above examples, the calculation of the convolutional neural network is explained by taking the example of convolution first and then activation. In other embodiments, the data processing method can also be used to support the calculation of the convolutional neural network with activation first and then convolution. Exemplarily, the scheduling unit can store the input feature map from the off-chip storage unit into the third cache unit. The arithmetic logic unit can perform an activation operation on the input feature map stored in the third cache unit under the control of the scheduling unit. After obtaining the result after the activation operation (such as the activated input feature map), the arithmetic logic unit can transfer these activated input feature maps to the off-chip storage unit, or to the off-chip storage unit through the fourth cache unit. And these activated input feature maps stored in the off-chip storage unit can be transferred to the on-chip storage space (such as the first cache unit) under the control of the scheduling unit to perform the convolution calculation in the above example.

[0120] It is understandable that the above Fig.15 Each step of the data processing method shown can correspond to the functional description process of the data processing device mentioned above, and its beneficial effects are similar, which will not be repeated here.

[0121] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing device, characterized in that: include: An off-chip storage unit, a first cache unit connected to the off-chip storage unit, a second cache unit connected to the off-chip storage unit, a computing array, a third cache unit, an arithmetic logic unit and a scheduling unit; Wherein, the computing array includes a plurality of computing units, the plurality of computing units include R rows and C columns, the computing units of the R rows and C columns are sequentially connected in the row direction and the column direction, R, C are both integers greater than 1; the first cache unit is also connected to the input end of the computing unit of the first row of the computing array; the second cache unit is also connected to the input ends of all computing units of the computing array; the output end of the computing unit of the Rth row of the computing array is connected to the third cache unit; the third cache unit is also connected to the input end of the arithmetic logic unit; the scheduling unit is respectively connected to the off-chip storage unit, the first cache unit, the second cache unit, the computing array, the third cache unit and the arithmetic logic unit; The off-chip storage unit is used to store N input feature maps and M convolution kernels; N and M are both integers greater than or equal to 1; For each of the N input feature maps, the scheduling unit performs the following operations to obtain a convolution result of each input feature map and corresponding weight data: The scheduling unit is used to store feature map data of a first input feature map in the first cache unit, and store weight data corresponding to the first input feature map in the M convolution kernels in the second cache unit, wherein the first input feature map is any one of the N input feature maps; The scheduling unit is further used to transmit the feature map data of the first input feature map stored in the first cache unit to the computing array in units of 1 row and C columns in sequence, and map the weight data stored in the second cache unit to the computing array for the computing units in the computing array to perform convolution calculations, obtain convolution results and store them in the third cache unit; The arithmetic logic unit is used to accumulate the convolution results corresponding to different channels of the same convolution kernel in the third cache unit to obtain M intermediate output feature maps corresponding to the M convolution kernels.

2. The data processing device according to claim 1, characterized in that: The scheduling unit is used to sequentially transmit the feature map data of the first R rows and C columns of the first input feature map stored in the first cache unit, starting from the feature map data of the 1st column to the Cth column in the 1st row, from the input end of the 1st row of the computing unit of the computing array, in units of 1 row and C columns, in a cascade manner, so as to sequentially transmit the feature map data of the first R rows and C columns of the first input feature map to the computing units of the Rth row to the 1st row of the computing array for convolution calculation; wherein the step size of the convolution kernel used for the convolution calculation is S, and S is an integer greater than or equal to 1; When the convolution calculation of the feature map data of the 1st column to the Cth column in the first S rows of the first input feature map is completed, the scheduling unit is further used to control the feature map data of the 1st row to the RSth row in the calculation array to be transferred downward by S rows in a cascade manner, and the feature map data of the 1st column to the Cth column in the R+1th row to the R+S+1th row of the first input feature map stored in the first cache unit is transmitted to the first S rows of the calculation units of the calculation array through the input end of the 1st row of the calculation unit of the calculation array, and so on, until the feature map data of the 1st column to the Cth column in the H-R+1th row of the first input feature map is transmitted to the 1st row of the calculation unit of the calculation array, wherein H is the number of rows of feature map data included in the first input feature map, and H is an integer greater than R; When the convolution calculation of the feature map data from the 1st column to the Cth column in the H-R+1th row in the first input feature map is completed, the scheduling unit is further used to control the feature map data from the 1st row to the RSth row in the calculation array to be transferred downward by S rows in a cascade manner; if Then the first input feature map stored in the first cache unit is Column to The characteristic graph data of the column is transmitted to the computing unit in the first row of the computing array through the input end of the computing unit in the first row of the computing array; if Then the first input feature map stored in the first cache unit is Column to The characteristic graph data of the column is transmitted to the computing unit in the first row of the computing array through the input end of the computing unit in the first row of the computing array; And so on, until the feature map data from the L-C+1th column to the Lth column in the H-R+1th row in the first input feature map is transmitted to the 1st row computing unit of the computing array, where L is the number of columns of feature map data included in the first input feature map, K2 is the number of columns of weight data corresponding to the first input feature map in one of the M convolution kernels, L is an integer greater than C, and K2 is an integer greater than or equal to 1.

3. The data processing device according to claim 1, characterized in that: The scheduling unit is configured to perform the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map corresponds to the first channels of the M convolution kernels, where the first channels correspond to the first input feature map: The scheduling unit is used to map the first weight data stored in the second cache unit to the calculation units from the R-K1+1th row to the first K2 columns of the Rth row of the calculation array, so that the calculation units from the R-K1+1th row to the first K2 columns of the Rth row perform convolution calculation according to the first weight data; when the convolution calculation of the calculation units from the R-K1+1th row to the first K2 columns of the Rth row is completed, control the first weight data to be transferred to the right by S columns in a cascade manner, so that the calculation units from the Sth column to the K2+S-1th column of the R-K1+1th row to the Rth row perform convolution calculation according to the first weight data, and so on, until the first weight data is transferred to the calculation units from the C-K2+1th column to the Cth column of the R-K1+1th row to the Rth row; the first weight data is the weight data of the first channel of any one of the M convolution kernels, wherein K1 is the number of rows of weight data included in the first channel of one of the M convolution kernels, and K1 is an integer greater than or equal to 1; When the convolution calculation of the computing units from the R-K1+1th row to the Rth row and the C-K2+1th column to the Cth column is completed, the scheduling unit remaps the first weight data to the computing units from the R-K1+1th row to the first K2 columns of the Rth row of the computing array until the convolution calculation of all feature map data of the first input feature map and the first weight data is completed.

4. The data processing device according to claim 1, characterized in that: The scheduling unit is configured to perform the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map corresponds to the first channels of the M convolution kernels, where the first channels correspond to the first input feature map: The scheduling unit is used to repeatedly map the first weight data stored in the second cache unit to the first Row to row R, columns 1 to 1 Column, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data; when the convolution calculation is completed, the first weight data repeatedly mapped to the calculation array is controlled to be transferred to the right by S columns in a cascade manner, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data, and so on, until the first weight data is transferred to the calculation unit from the C-K2+1th column to the Cth column from the R-K1+1th row to the Rth row; the first weight data is the weight data of the first channel of any convolution kernel of the M convolution kernels, wherein K1 is the number of rows of the weight data of the first channel, K2 is the number of columns of the weight data of the first channel, and K1 and K2 are both integers greater than or equal to 1; When the convolution calculation of the computing units from the R-K1+1th row to the Rth row, from the C-K2+1th column to the Cth column is completed, the scheduling unit repeatedly maps the first weight data to the Row to row R, columns 1 to 1 In the calculation units of the column, until the convolution calculation of all feature map data of the first input feature map and the first weight data is completed.

5. The data processing device according to any one of claims 1 to 4, characterized in that: The output end of the arithmetic logic unit is connected to the off-chip storage unit; The arithmetic logic unit is further used to perform pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit respectively; The scheduling unit is also used to store the processed intermediate output feature map in the off-chip storage unit.

6. The data processing device according to any one of claims 1 to 4, characterized in that: The data processing device further comprises: a fourth cache unit connected to the output end of the arithmetic logic unit, the fourth cache unit is also connected to the scheduling unit; The arithmetic logic unit is further used to perform pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit, and store the processed intermediate output feature maps in the fourth cache unit; The scheduling unit is further used to store the processed intermediate output feature map stored in the fourth cache unit into the off-chip storage unit.

7. The data processing device according to any one of claims 1 to 4, characterized in that: The third cache unit is also connected to the off-chip storage unit; The scheduling unit is further configured to store the N input feature maps stored in the off-chip storage unit into the third cache unit; The arithmetic logic unit is further used to perform activation processing on the N input feature maps stored in the third cache unit; The scheduling unit is also used to store the N activated input feature maps in the off-chip storage unit.

8. A data processing method, characterized in that: Applied to a data processing device, the data processing device comprises: an off-chip storage unit, a first cache unit connected to the off-chip storage unit, a second cache unit connected to the off-chip storage unit, a computing array, a third cache unit, an arithmetic logic unit and a scheduling unit; Wherein, the computing array includes a plurality of computing units, the plurality of computing units include R rows and C columns, the computing units of the R rows and C columns are sequentially connected in the row direction and the column direction, R, C are both integers greater than 1; the first cache unit is also connected to the input end of the computing unit of the first row of the computing array; the second cache unit is also connected to the input ends of all computing units of the computing array; the output end of the computing unit of the Rth row of the computing array is connected to the third cache unit; the third cache unit is also connected to the input end of the arithmetic logic unit; the scheduling unit is respectively connected to the off-chip storage unit, the first cache unit, the second cache unit, the computing array, the third cache unit and the arithmetic logic unit; The method comprises: The off-chip storage unit stores N input feature maps and M convolution kernels; N and M are both integers greater than or equal to 1; For each of the N input feature maps, the scheduling unit performs the following operations to obtain a convolution result of each input feature map with the weight data of the corresponding channel: The scheduling unit stores feature map data of a first input feature map in the first cache unit, and stores weight data corresponding to the first input feature map in the M convolution kernels in the second cache unit, where the first input feature map is any one of the N input feature maps; The scheduling unit sequentially transmits the feature map data of the first input feature map stored in the first cache unit to the computing array in units of 1 row and C columns, maps the weight data stored in the second cache unit to the computing array, so that the computing units in the computing array perform convolution calculations, obtain convolution results and store them in the third cache unit; The arithmetic logic unit accumulates the convolution results corresponding to different channels of the same convolution kernel in the third cache unit to obtain M intermediate output feature maps corresponding to the M convolution kernels.

9. The data processing method according to claim 8, characterized in that: The scheduling unit sequentially transmits all feature map data of the first input feature map stored in the first cache unit to the computing array in units of 1 row and C columns, including: The scheduling unit transmits the feature map data of the first R rows and C columns of the first input feature map stored in the first cache unit, starting from the feature map data of the 1st column to the Cth column in the 1st row, from the input end of the 1st row of the computing unit of the computing array, in units of 1 row and C columns, in a cascade manner, so as to transmit the feature map data of the first R rows and C columns of the first input feature map to the computing units of the Rth row to the 1st row of the computing array in sequence for convolution calculation; wherein the step size of the convolution kernel used for convolution calculation is S, and S is an integer greater than or equal to 1; When the convolution calculation of the feature map data of the 1st column to the Cth column in the first S rows of the first input feature map is completed, the scheduling unit controls the feature map data of the 1st row to the RSth row in the calculation array to be transferred downward by S rows in a cascade manner, and transmits the feature map data of the 1st column to the Cth column in the R+1th row to the R+S+1th row of the first input feature map stored in the first cache unit to the first S rows of the calculation unit of the calculation array through the input end of the 1st row of the calculation unit of the calculation array, and so on, until the feature map data of the 1st column to the Cth column in the H-R+1th row of the first input feature map is transmitted to the 1st row of the calculation unit of the calculation array, wherein H is the number of rows of feature map data included in the first input feature map, and H is an integer greater than R; When the convolution calculation of the feature map data from the 1st column to the Cth column in the H-R+1th row in the first input feature map is completed, the scheduling unit controls the feature map data from the 1st row to the RSth row in the calculation array to be transferred downward by S rows in a cascade manner; if Then the first input feature map stored in the first cache unit is Column to The characteristic graph data of the column is transmitted to the computing unit in the first row of the computing array through the input end of the computing unit in the first row of the computing array; if Then the first input feature map stored in the first cache unit is Column to The characteristic graph data of the column is transmitted to the computing unit in the first row of the computing array through the input end of the computing unit in the first row of the computing array; And so on, until the feature map data from the L-C+1th column to the Lth column in the H-R+1th row in the first input feature map is transmitted to the 1st row computing unit of the computing array, where L is the number of columns of feature map data included in the first input feature map, K2 is the number of columns of weight data corresponding to the first input feature map in one of the M convolution kernels, L is an integer greater than C, and K2 is an integer greater than or equal to 1.

10. The data processing method according to claim 8, characterized in that: The scheduling unit maps the weight data stored in the second cache unit to the computing array so that the computing units in the computing array perform convolution calculations to obtain convolution results, including: The scheduling unit performs the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map correspond to the first channels of the M convolution kernels one by one, where the first channels correspond to the first input feature map: The scheduling unit maps the first weight data stored in the second cache unit to the calculation units from the R-K1+1th row to the first K2 columns of the Rth row of the calculation array, so that the calculation units from the R-K1+1th row to the first K2 columns of the Rth row perform convolution calculation according to the first weight data; when the convolution calculation of the calculation units from the R-K1+1th row to the first K2 columns of the Rth row is completed, the first weight data is controlled to be transferred to the right by S columns in a cascade manner, so that the calculation units from the Sth column to the K2+S-1th column of the R-K1+1th row to the Rth row perform convolution calculation according to the first weight data, and so on, until the first weight data is transferred to the calculation units from the C-K2+1th column to the Cth column of the R-K1+1th row to the Rth row; the first weight data is the weight data of the first channel of any one of the M convolution kernels, wherein K1 is the number of rows of weight data included in the first channel of one of the M convolution kernels, and K1 is an integer greater than or equal to 1; When the convolution calculation of the computing units from the R-K1+1th row to the Rth row and the C-K2+1th column to the Cth column is completed, the scheduling unit remaps the first weight data to the computing units from the R-K1+1th row to the first K2 columns of the Rth row of the computing array until the convolution calculation of all feature map data of the first input feature map and the first weight data is completed.

11. The data processing method according to claim 8, characterized in that: The scheduling unit maps the weight data stored in the second cache unit to the computing array so that the computing units in the computing array perform convolution calculations to obtain convolution results, including: The scheduling unit performs the following operations for each of the M convolution kernels to obtain M convolution results in which all feature map data in the first input feature map correspond to the first channels of the M convolution kernels one by one, where the first channels correspond to the first input feature map: The scheduling unit repeatedly maps the first weight data stored in the second cache unit to the first Row to row R, columns 1 to 1 Column, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data; when the convolution calculation is completed, the first weight data repeatedly mapped to the calculation array is controlled to be transferred to the right by S columns in a cascade manner, so that the calculation unit at the corresponding position performs convolution calculation according to the first weight data, and so on, until the first weight data is transferred to the calculation unit from the C-K2+1th column to the Cth column from the R-K1+1th row to the Rth row; the first weight data is the weight data of the first channel of any convolution kernel of the M convolution kernels, wherein K1 is the number of rows of the weight data of the first channel, K2 is the number of columns of the weight data of the first channel, and K1 and K2 are both integers greater than or equal to 1; When the convolution calculation of the computing units from the R-K1+1th row to the Rth row, from the C-K2+1th column to the Cth column is completed, the scheduling unit repeatedly maps the first weight data to the Row to row R, columns 1 to 1 In the calculation units of the column, until the convolution calculation of all feature map data of the first input feature map and the first weight data is completed.

12. The data processing method according to any one of claims 8 to 11, characterized in that: The output end of the arithmetic logic unit is connected to the off-chip storage unit; The method further comprises: The arithmetic logic unit performs pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit respectively; The scheduling unit stores the processed intermediate output feature map in the off-chip storage unit.

13. The data processing method according to any one of claims 8 to 11, characterized in that: The data processing device further comprises: a fourth cache unit connected to the output end of the arithmetic logic unit, the fourth cache unit is also connected to the scheduling unit; The method further comprises: The arithmetic logic unit performs pooling and / or activation processing on the M intermediate output feature maps stored in the third cache unit, and stores the processed intermediate output feature maps in the fourth cache unit; The scheduling unit stores the processed intermediate output feature map stored in the fourth cache unit to the off-chip storage unit.

14. The data processing method according to any one of claims 8 to 11, characterized in that: The third cache unit is also connected to the off-chip storage unit; The method further comprises: The scheduling unit stores the N input feature maps stored in the off-chip storage unit into the third cache unit; The arithmetic logic unit performs activation processing on the N input feature maps stored in the third cache unit; The scheduling unit stores the activated N input feature maps in the off-chip storage unit.

Images