Embedded device configurable neural network operator and embedded device

Abstract

The application discloses an embedded device and a neural network operation device, and belongs to the technical field of convolution operation of embedded devices. The operation device comprises an operation control module, a convolution operation module, a global average pooling module and a full connection calculation module. The convolution operation module comprises a multi-channel convolution calculation circuit, which is used for sequentially executing second-layer to last-layer convolution calculation. In each layer of convolution calculation, a convolution kernel {c ij} m*m The convolution calculation process of image data {p ij} m*m under the convolution kernel sliding window of each channel comprises the following steps: the operation control module controls the intermediate result calculation circuit to continuously execute the sub-process of different pixels in the sliding window and continuously output different intermediate results in adjacent time sequences; and the accumulation circuit accumulates the newly input intermediate results at each time sequence to obtain the pixel corresponding to the sliding window position in the convolution result. The framework can realize the convolution neural network operation on the embedded device, and can improve the operation speed while reducing the circuit complexity.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of convolution operation of embedded devices, and more particularly relates to an embedded device configurable neural network operation device and an embedded device. BACKGROUND

[0002] With the development of the image processing field, convolution neural networks have been widely used in image processing fields such as face recognition and object recognition. At present, the mainstream image processing calculation mode is still through a computer, that is, a CPU+GPU mode. Since the power consumption, complexity and cost of the traditional CPU+GPU mode have greatly exceeded the range accepted by embedded devices, there is currently no embedded device designed with image convolution processing capability. Generally, embedded devices upload the collected data to the cloud, and the server of the cloud processes the data and returns the results to the embedded device.

[0003] However, the processing mode through the cloud causes great pressure on the communication system due to the large amount of data information sent and received. At the same time, for some aspects such as intelligent driving, the cloud processing mode may cause danger due to the delay or failure of the communication system. The processing in these fields must be completed locally and cannot be entrusted to the cloud.

[0004] Therefore, it is necessary to provide an implementation scheme capable of performing convolution calculation on an embedded device. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides an embedded device configurable neural network operation device and an embedded device, which aims to perform convolution neural network processing on the embedded device to complete the image processing function.

[0006] To achieve the above-mentioned purpose, according to one aspect of the present application, an embedded device configurable neural network operation device is provided, which comprises an operation control module, a convolution operation module, a global average pooling module and a full connection calculation module controlled by the operation control module, the convolution operation module comprising a convolution parameter storage structure for storing convolution parameters, a convolution result storage structure for storing convolution results of each step, a convolution preprocessing circuit and a multi-channel convolution calculation circuit; wherein,

[0007] The convolution preprocessing circuit is used to read the original image data from the external memory and perform first layer convolution calculation with the corresponding convolution parameters to generate first convolution image data with multiple channels;

[0008] The multi-channel convolution calculation circuit comprises a multiply-add circuit, the multiply-add circuit comprises an intermediate result calculation circuit and an accumulation circuit, the intermediate result calculation circuit comprises a multi-channel multiplication circuit and a multi-stage addition circuit; the multi-channel convolution calculation circuit is used for sequentially performing second-layer-to-last-layer convolution calculation and generating a convolution image data with multiple channels after each layer of convolution calculation; wherein, in each layer of convolution calculation, a convolution kernel {c ij} m*m is used for performing convolution calculation on image data {p ij} m*m under the convolution kernel sliding window; the process of the convolution calculation comprises: the operation control module controls the intermediate result calculation circuit to continuously perform a convolution sub-process of different pixels in the sliding window and continuously output corresponding intermediate results in adjacent time sequences, and the accumulation circuit accumulates newly input intermediate results in each time sequence to obtain a pixel at a corresponding sliding window position in the convolution result; the convolution sub-process comprises controlling the multi-channel multiplication circuit to perform multiplication of each channel pixel p ij and corresponding convolution parameter c ij to obtain a plurality of products, and the multi-stage addition circuit performs hierarchical summation of all products at different time sequences to obtain an intermediate result and input the accumulation circuit; wherein, {c m*m represents an m*m matrix, c ij and p ij respectively represent the convolution parameter and the pixel in the ith row and the jth column.

[0009] The global average pooling module is used for averaging all pixels of the convolution image data of each channel obtained by the last layer of convolution calculation to obtain an average pooling result of the corresponding channel.

[0010] The full connection calculation module is used for performing full connection calculation according to the average pooling result of each channel to obtain a classification result of the original image data.

[0011] In one embodiment, in the multiply-add circuit, the multi-channel multiplication circuit obtains a plurality of products as summation factors to input a first-stage addition circuit, the previous-stage addition circuit performs addition operation on the input summation factors to obtain a plurality of addition results as summation factors to input a next-stage addition circuit in the same time sequence, the number of summation factors is gradually reduced through the multi-stage addition circuit until an addition result is output by the last-stage addition circuit and input into the accumulation circuit as an intermediate result; the accumulation circuit accumulates newly input intermediate results in each time sequence, and obtains a pixel at a corresponding sliding position after accumulation of m 2 intermediate results.

[0012] In one embodiment, in the multi-channel multiplication circuit, each addition component sums input α summation factors in a group of two to obtain α / 2 summation results and input the new summation factors into a next-stage circuit.

[0013] In one embodiment, when the multiply-add circuit completes the convolution calculation of the image data {p ij} m*m with the convolution kernel {c ij} m*m in the current sliding window, the convolution calculation of the image data {p ij} m*m with the convolution kernel {c ij} m*m in the next sliding window is continued in the same mode until the sliding window ends, obtaining the convolution image data corresponding to the channel.

[0014] In one embodiment, the multi-channel convolution calculation circuit has a plurality of multiply-add circuits, and the plurality of multiply-add circuits operate in parallel to realize a plurality of convolution kernels {c ij} m*m in parallel. ij} m*m

[0015] In one embodiment, the size of the convolution kernel used by the multi-channel convolution calculation circuit when performing each layer of convolution calculation is 3*3, the sliding step of the sliding window is 2 when the convolution image data to be convolved exceeds 20*20, and the sliding step of the sliding window is 1 when the convolution image data to be convolved does not exceed 20*20.

[0016] In one embodiment, the neural network operator further includes an address generation module, the address generation module includes an address reading circuit and an address storage circuit, the address reading circuit generates the reading address of the image data and the reading address of the corresponding convolution parameter for each step of convolution operation, and the address storage circuit generates the storage address of the convolution image data after each step of convolution operation; wherein,

[0017] The rule for the address storage circuit to generate the storage address of each step of convolution image data includes: the convolution image data of different channels in the same layer is stored in different block storage units in the convolution result storage structure, and each pixel of the convolution image data of the same channel is stored in the corresponding block storage unit according to the position order of the pixel in the image, forming a mapping relationship between the storage address and the pixel position: if the position of the pixel in the image is the xth row and the yth column, the address stored in the block storage unit is a*(x-1)+y or x+b*(y-1), wherein a is the number of pixels in a row of the image, and b is the number of pixels in a column of the image.

[0018] The address reading circuit generates the reading address of the image data {p ij} m*m ​The rule of reading address of each pixel includes: locating the position of the middle pixel in the current sliding window in the channel image data, determining the positions of other pixels in the channel image data according to the positional relationship between the middle pixel and other pixels in the current sliding window, and calculating the address of each pixel in the current sliding window based on the mapping relationship between the storage address and the pixel position.

[0019] In one of the embodiments, the convolution result storage structure has two separate storage spaces, and each storage space has a plurality of storage blocks.

[0020] The rule of the multi-channel convolution calculation circuit storing the convolution result in the convolution result storage structure includes:

[0021] The adjacent two layers of convolution image data are stored in two different storage spaces; in the same storage space, the beta-th convolution image data is stored in the storage block of the beta-2-th convolution image data, beta is the convolution layer serial number, beta>=3, and the beta-th convolution image data gradually covers the pixels at the same position of the beta-2-th convolution image data according to the position of the pixels.

[0022] In one of the embodiments, the full connection calculation module is used to multiply and add the average pooling result of each channel and the parameter of the same full connection layer channel, compare the multiplication and addition results of different full connection layer channels, and take the maximum result as the classification result of the image data.

[0023] According to another aspect of the present application, an embedded device is provided, which comprises the above-mentioned embedded device configurable neural network operator.

[0024] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0025] (1) The present application builds a neural network operator applied to an embedded device, which includes an operation control module and a convolution operation module, a global average pooling module and a full connection calculation module controlled by the operation control module. The framework can realize convolution neural network operation on an embedded device.

[0026] (2) Compared with directly using a GPU to realize the convolution operation of a neural network, the speed of executing the convolution operation of a neural network on an embedded device is slower. In order to improve the speed, the present application designs a convolution operation module, which includes a convolution parameter storage structure, a convolution result storage structure, a convolution preprocessing circuit and a multi-channel convolution calculation circuit. Under the control of the operation control module, the first layer convolution operation is first executed by using the convolution preprocessing circuit to obtain multi-channel first convolution image data, and then the subsequent convolution operation is executed by using the multi-channel convolution calculation circuit to finally obtain multi-channel convolution image data. In order to realize the multi-channel convolution calculation circuit to perform convolution kernel {c ij}m*m convolution calculation of the image data {p ij} m*m The convolution calculation of the image data {p

[0027] (3) The convolution parameter storage structure is included in the convolution operation module of the application, and the designed convolution parameter can be written into the convolution parameter storage structure in advance, and the convolution parameter in the convolution parameter storage structure can be updated as needed. When the neural network operator performs convolution operation, the convolution parameter filled in the convolution parameter storage structure can be directly read for convolution calculation, and flexible configuration of the convolution parameter is realized.

[0028] (4) The application adopts a global average pooling module, which performs a global average pooling module again after all convolutions are completed. Compared with performing a pooling after each convolution, the structure design is simpler, the power consumption is lower, and the processing speed is faster.

[0029] Further, by setting specific storage rules and reading rules through the address storage circuit, the responsible addressing process can be avoided, so as to quickly obtain the storage addresses of each pixel point in the current sliding window, simplify the circuit design and improve the operation speed.

[0030] Further, by using different space alternately storage, the storage space can be saved, and the convolution calculation data error can be avoided.

[0031] Further, when the image data is large, the sliding step is 2, which can replace the traditional pooling layer with the sliding step. Not only reduces the loss of information in the pooling process, but also reduces the parameters and the pooling layer has the same effect, and the design is simple, reduces the redundant module, and low power consumption. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The structure block diagram of the configurable neural network operator of the embedded device of an embodiment;

[0033] Figure 2 The structure schematic diagram of the multiply-add circuit of an embodiment;

[0034] Figure 3 The flowchart of generating an intermediate result by the intermediate result calculation circuit of an embodiment;

[0035] Figure 4 The timing control diagram of quickly realizing the convolution calculation of the convolution kernel and the image data on the same multiply-add circuit of an embodiment;

[0036] Figure 5 An operation schematic diagram of a full connection layer of an embodiment;

[0037] Figure 6 An operation schematic diagram of a 3*3 sliding window and other pixels of an embodiment;

[0038] Figure 7 A storage structure division schematic diagram of a convolution result of an embodiment. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0040] As Figure 1 shown is a structure block diagram of a configurable neural network operation device of an embodiment, which includes an operation control module and a convolution operation module, a global average pooling module and a full connection calculation module controlled by the operation control module, the convolution, pooling and full connection are all associated structures in a neural network, and the above framework can realize the calculation of a convolutional neural network.

[0041] The convolution operation module includes a convolution parameter storage structure, a convolution result storage structure, a convolution preprocessing circuit and a multi-channel convolution calculation circuit.

[0042] The convolution parameter storage structure is used to store the designed convolution parameters in advance, and the device can directly read the convolution parameters corresponding to the convolution parameter storage structure when performing each step of convolution operation. The flexible configuration of the convolution operation device can be realized by updating the parameters of the convolution parameter storage structure.

[0043] The convolution result storage structure is used to store the convolution calculation result of each layer obtained by the device performing each step of convolution operation, and the convolution calculation result of the previous layer is read from the convolution result storage structure when performing the next layer of convolution.

[0044] Under the control of the operation control module, the convolution preprocessing circuit reads the original image data from the external memory and reads the corresponding convolution parameters from the convolution parameter storage structure to perform the first layer of convolution calculation. The first layer of convolution calculation generates a multi-channel first convolution image data through a plurality of convolution kernels and stores it in the convolution result storage structure, and the number of channels is equal to the number of convolution kernels used.

[0045] Under the control of the operation control module, the multi-channel convolution calculation circuit is used for sequentially executing second layer to last layer convolution calculation, and the calculation logic of each layer of convolution calculation is same. Wherein, the process of executing the i-th layer of convolution calculation includes: reading the (i-1)-th convolution image data and corresponding convolution parameters to perform the i-th layer of convolution calculation, the i-th layer of convolution calculation generates the i-th convolution image data with multiple channels through multiple convolution kernels and stores in the convolution result storage structure, and i=2, 3, 4,..., n, n is the number of convolution layers. In other words, the convolution result of the previous layer is taken as the convolution input image of the next layer to perform the convolution operation again, and the updated convolution result is obtained.

[0046] Since the multi-channel convolution calculation circuit uses multiple convolution kernels when performing each layer of convolution, the input image data has multiple channels, and the multi-channel image data is generated, which involves a large amount of calculation. If each step of calculation corresponds to an independent hardware circuit, a large circuit structure needs to be designed, thereby occupying a large space position of the embedded device. If multiple calculation steps of the same calculation share a set of hardware circuits, and each step of calculation is sequentially executed in the order, the calculation speed of the whole device will be too slow. Based on this, the characteristics of each layer of convolution calculation process are analyzed, the calculation process of each convolution kernel is optimized, and the calculation of each convolution kernel is quickly realized through the same hardware circuit.

[0047] In order to facilitate understanding, the relationship between convolution calculation and convolution kernel calculation is simply explained as follows, which is the conventional relationship of traditional convolution operation.

[0048] The number of channels of the (i-1)-th convolution image data output after the (i-1)-th layer of convolution calculation is set as k β-1 , that is, the number of channels of the input image data of the i-th layer of convolution is k β-1 , respectively, channel 1, channel 2,..., channel k β-1 , through the convolution operation of k β convolution kernels, the number of channels of the i-th convolution image data output is k β , and the convolution result of one convolution kernel is taken as the output result of one channel. The size of each convolution kernel is same but the convolution parameters are different. The convolution operation rules of each convolution kernel are same. Taking one convolution kernel to calculate the convolution result of one channel as an example, the convolution kernel is set as a m*m matrix {c ij} m*m , c ij is the convolution parameter of the i-th row and j-th column in the convolution kernel, and k β-1 channel (i-1)-th convolution image data needs to be calculated. By sliding the sliding window on each channel of the convolution image data, the m*m matrix image data {p ij} m*m , c ijFor the pixel in the i-th row and the j-th column in the matrix image data, the convolution kernel {c ij} m*m The matrix image data {p ij} m*m After the point multiplication, all the point multiplication results p ij *c ij are accumulated, that is, the sum of all the point multiplication results p is obtained as the calculation result of the current sliding window position, which is the pixel in the i-th convolution image data corresponding to the current sliding window position. If the current sliding window position is located at the top left corner of the β-1-th convolution image data, the finally obtained pixel is the pixel in the first row and the first column of the β-th convolution image data. By moving the sliding window, the pixels in other positions of the β-th convolution image data are obtained through the same process, so that the channel data of the β-th convolution image data corresponding to the current convolution kernel is obtained. By updating the convolution kernel, different channel data of the β-th convolution image data can be obtained.

[0049] Through the analysis of the above process, it can be known that, for the same convolution kernel, the point multiplication and accumulation calculation between the convolution kernel and the image data in different positions are realized by moving the sliding window, so that the pixels in different positions in the convolution result are obtained. In one sliding window, the convolution kernel {c ij} m*m and the matrix image data {p ij} m*m of each channel are multiplied, and all the point multiplication results p ij *c ij are accumulated to obtain The calculation amount of this process is large. The present application decomposes the process, and obtains an intermediate result through the convolution calculation of each convolution parameter c ij and the pixel p ij of each channel, that is, the intermediate result is The pixel position and the corresponding convolution parameter are updated to obtain different intermediate results, therefore, m*m sub-processes are executed to obtain m*m intermediate results, and all the intermediate results are summed to obtain the convolution result of the convolution kernel {c ij} m*m and the matrix image data {p ij} m*m of each channel.

[0050] Based on this, the multi-channel convolution calculation circuit proposed by the present application includes a multiplication-addition circuit, as shown in Figure 2 , the multiplication-addition circuit includes an intermediate result calculation circuit and an accumulation circuit, the intermediate result calculation circuit includes a multi-channel multiplication circuit and a multi-stage addition circuit. When it is necessary to execute the convolution kernel {c ij} m*m and the image data {pij} m*m During convolution calculations, the multi-channel multiplication circuit executes the multiplication of each channel pixel p simultaneously. ij With the corresponding convolution parameters c ij The multiplication operation yields multiple products. A multi-stage adder circuit sums all products at different timing intervals to obtain an intermediate result, which is then input into the accumulator circuit. For example... Figure 3 The diagram shows a flowchart of an embodiment where an intermediate result is generated by an intermediate result calculation circuit. The multi-channel multiplication circuit executes the calculation of each channel pixel p in the first time sequence. ij With the corresponding convolution parameters c ij The multiplication operation yields multiple products, which are then input as summation factors into the first-stage adder circuit. In the second timing sequence, the input summation factors are grouped and added together, resulting in multiple sums, which are then input as summation factors into the third-stage adder circuit. This process continues, gradually reducing the number of summation factors through multiple adder circuits until the final adder circuit outputs a single sum, which is then input as an intermediate result into the accumulator circuit. Specifically, each adder component sums the input α summation factors in pairs, obtaining α / 2 sums, which are then input as new summation factors into the next stage circuit.

[0051] Based on the design of the multi-channel convolution calculation circuit described above, and combined with the control logic of the operation control module, the calculation speed is improved. For example... Figure 4 The diagram shows a timing control diagram for rapidly implementing convolution calculation of convolution kernel and image data on the same multiply-accumulate circuit in one embodiment. The operation control module controls the intermediate result calculation circuit to continuously perform convolution calculation of different pixels with corresponding convolution parameters in adjacent timing sequences and continuously output different intermediate results. The accumulation circuit accumulates the newly input intermediate results in each timing sequence to obtain the pixel corresponding to the sliding window position in the convolution result and stores it in the convolution result storage structure.

[0052] In one embodiment, when the multiply-accumulate circuit completes the convolution kernel {c} under the control of the arithmetic control module ij} m*m Image data of each channel under the current sliding window {p ij} m*m After the convolution calculation, continue to execute the convolution kernel {c} in the same pattern. ij} m*m Image data of each channel in the next sliding window {p ij} m*m The convolution calculation continues until the sliding window finishes sliding, obtaining the convolutional image data for the corresponding channel.

[0053] In an embodiment, the multi-channel convolution calculation circuit has a plurality of multiply-accumulate circuits, and the plurality of multiply-accumulate circuits operate in parallel to implement a plurality of convolution kernels {c ij} m*m The convolution calculation of the image data {p ij} m*m of each channel under the sliding window of the convolution kernel, that is, the convolution image data of each channel is generated synchronously.

[0054] In an embodiment, the size of the convolution kernel used by the multi-channel convolution calculation circuit when performing each layer of convolution calculation is 3*3, the sliding step of the sliding window is 2 when the convolution image data to be convolved exceeds 20*20, and the sliding step of the sliding window is 1 when the convolution image data to be convolved does not exceed 20*20. In this embodiment, when the image data is large, the sliding step of 2 can be used instead of the traditional pooling layer, which not only reduces the loss of information in the pooling process, but also reduces the parameters and has the same effect as the pooling layer, and the design is simple, reduces redundant modules, and has low power consumption.

[0055] The global average pooling module is used to read the convolution image data of each channel after completing the last layer of convolution calculation and to average all pixels of the convolution image data in each channel to obtain the average pooling result of the corresponding channel. Unlike the pooling layer of general convolution operation, the global average pooling module replaces a layer of fully connected layer, reduces the parameters entering the fully connected layer, reduces the calculation power consumption, and improves the processing speed.

[0056] The fully connected calculation module is used to perform fully connected calculation according to the average pooling results of each channel to obtain the classification result of the original image data. As shown in Figure 5 , all average pooling results are multiplied by and added to the corresponding fully connected parameters, and the results are compared in size, and the maximum calculation result is selected as the final result to obtain the result of neural network processing.

[0057] In an embodiment, the neural network operator further includes an address generation module, the address generation module includes an address reading circuit and an address storage circuit, the address reading circuit generates a reading address of image data and a reading address of corresponding convolution parameters for each convolution operation, the address storage circuit generates a storage address of convolution image data after each convolution operation, and the convolution operation module reads image data from the convolution result storage structure based on the image data reading address generated by the address reading circuit at each convolution operation, reads convolution parameters from the convolution parameter storage structure based on the reading address of the convolution parameters, then performs convolution operation, and stores the convolution result in the corresponding position in the convolution result storage structure according to the storage address generated by the address storage circuit.

[0058] The rules for generating the storage address of each convolutional image data step by the address storage circuit include: convolutional image data from different channels of the same layer are stored in different block storage units in the convolution result storage structure, and each pixel of the convolutional image data of the same channel is stored in the corresponding block storage unit according to the pixel's position in the image, forming a mapping relationship between storage address and pixel position: if the pixel's position in the image is x-th row and y-th column, the address stored in the block storage unit is a*(x-1)+y or x+b*(y-1), where a is the number of pixels in a row of the image and b is the number of pixels in a column of the image. For example, a=10, b=10, if the pixel's position in the image data is (1,1), then the storage address of the storage block is 1; if the pixel's position in the image data is (10,10), then the storage address of the storage block is 100; if the pixel's position in the image data is (4,5), then the storage address of the storage block is 35 or 44.

[0059] When executing convolution kernel {c ij} m*m Image data with each channel located under the convolution kernel sliding window {p ij} m*m During convolution calculations, if pixels are stored sequentially according to the rules described above, the address read circuit generates the image data {p} for each convolution operation. ij} m*m The rule for reading the address of each pixel is as follows: locate the position of the middle pixel in the current sliding window in the channel image data, then determine the position of other pixels in the channel image data according to the positional relationship between the middle pixel and other pixels in the current sliding window, and calculate the address of each pixel in the current sliding window based on the mapping relationship between storage address and pixel position. For example... Figure 6 The diagram shows the positional relationship (x, y) between other pixels and the middle pixel within a 3x3 sliding window. Once the pixel's position in the channel image data is determined, its storage address within the storage block can be directly calculated based on the mapping between storage address and pixel position. It's understandable that image data for each channel is stored in different storage blocks according to the same rules; therefore, when performing a convolution kernel {c ij} m*m Image data with each channel located under the convolution kernel sliding window {p ij} m*m When performing convolution calculations, for the same position p in different channel images... ij The storage address of each pixel is consistent across different storage blocks. In this embodiment, by setting specific storage and retrieval rules through the address storage circuit, the complex addressing process can be avoided, thereby quickly obtaining the storage address of each pixel within the current sliding window, simplifying circuit design and improving computational speed.

[0060] In an embodiment, in order to save storage space, an overwriting storage manner is adopted, that is, the result of the later convolution is stored in the storage block of the result of the previous convolution and overwrites the data at the same address in the storage block. Specifically, the convolution result storage structure has two separate storage spaces, each of which has a plurality of storage blocks, and the rule of the multi-channel convolution calculation circuit for storing the convolution result in the convolution result storage structure includes: adjacent two layers of convolution image data are respectively stored in two different storage spaces; in the same storage space, the βth convolution image data is stored in the storage block of the β-2th convolution image data, β is the convolution layer number, β≥3, and the βth convolution image data gradually overwrites the pixels at the same position in the β-2th convolution image data according to the position of the pixels. As shown in Figure 7 The convolution result storage structure has an upper storage space and a lower storage space, each of which has a plurality of storage blocks, and i-j in the figure identifies the jth channel image data in the ith layer of convolution result. The channel image data of the first layer of convolution result is all stored in different storage blocks of the upper storage space, then the channel image data of the second layer of convolution result is all stored in different storage blocks of the lower storage space, the channel image data of the third layer of convolution result is all stored in different storage blocks of the upper storage space and gradually overwrites the first layer of convolution result, and so on. For example, for the storage block 1 of the upper storage space, the 1-1 image data has been stored, when the 3-1 image data is gradually stored, the pixel at the same position in the 1-1 image data will be gradually overwritten by the pixel in the 3-1 image data, for example, the pixel at (x, y) in the 3-1 image data will be directly stored in the pixel address space of (x, y) in the 1-1 image data and overwrite it. In this embodiment, the data is stored in the overwriting manner, which can save storage space, but if the conventional overwriting manner is adopted, that is, the result of the later layer of convolution gradually overwrites the convolution image data of the previous layer, since the convolution data needs to be gradually obtained by sliding the sliding window on the convolution image data of the previous layer when the convolution calculation is performed, if the convolution image data of the previous layer is gradually overwritten, the convolution calculation data may be wrong. Therefore, the different space alternating storage manner is further adopted in this embodiment, which can save storage space while avoiding the error of the convolution calculation data.

[0061] Correspondingly, the application also relates to an embedded device, which comprises the neural network operator configurable by any of the above-mentioned embedded devices, for example, the embedded device can be a mobile phone, a bracelet, a smart camera, etc. By configuring the neural network operator on the embedded device, the convolution neural network calculation can be directly realized on the embedded device, thereby omitting the complex process of uploading the collected data to the cloud, processing the data by the server of the cloud, and returning the result to the embedded device.

[0062] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A configurable neural network arithmetic unit for embedded devices, characterized in that, It includes a computation control module and a convolution operation module, a global average pooling module, and a fully connected computation module controlled by the computation control module. The convolution operation module includes a convolution parameter storage structure for storing convolution parameters, a convolution result storage structure for storing the convolution results of each step, a convolution preprocessing circuit, and a multi-channel convolution computation circuit; wherein, The convolution preprocessing circuit is used to read the raw image data from the external memory and perform the first layer convolution calculation with the corresponding convolution parameters to generate first convolution image data with multiple channels. The multi-channel convolution calculation circuit includes a multiply-accumulate circuit, which further includes an intermediate result calculation circuit and an accumulation circuit. The intermediate result calculation circuit includes a multi-channel multiplication circuit and a multi-level addition circuit. The multi-channel convolution calculation circuit is used to sequentially perform convolution calculations from the second to the last layer, generating multi-channel convolutional image data after each layer's calculation. In each convolution calculation, a convolution kernel {c} is executed. ij } m*m Image data with each channel located under the convolution kernel sliding window {p ij } m*m The convolution calculation process includes: the operation control module controls the intermediate result calculation circuit to continuously execute the convolution sub-process of different pixels within the sliding window in adjacent time sequences and continuously output the corresponding intermediate results; the accumulation circuit accumulates the newly input intermediate results in each time sequence to obtain the pixel at the corresponding sliding window position in the convolution result; the convolution sub-process includes controlling the multi-channel multiplication circuit to execute the convolution sub-process of each channel pixel p in parallel. ij With the corresponding convolution parameters c ij Multiplication yields multiple products. A multi-stage adder circuit sums all products at different timing intervals to obtain an intermediate result, which is then input into the accumulator circuit; where, {*} m*m Let c represent an m*m matrix. ij and p ij Let represent the convolution parameters and pixels in the i-th row and j-th column, respectively; The global average pooling module is used to average all pixels of the convolutional image data of each channel obtained from the last convolutional layer to obtain the average pooling result of the corresponding channel. The fully connected computation module is used to perform fully connected computation based on the average pooling results of each channel to obtain the classification results of the original image data.

2. The configurable neural network arithmetic unit for embedded devices as described in claim 1, characterized in that, In the multiply-accumulate circuit, the multi-channel multiplier circuit obtains multiple products as summation factors, which are then input into the first-stage adder circuit. The preceding adder circuit performs addition operations on the input summation factors in groups within the same time sequence, obtaining multiple summed results, which are then input into the next-stage adder circuit as summation factors. Through multiple adder circuits, the number of summation factors is gradually reduced until the last adder circuit outputs a single summed result, which is then input into the accumulator circuit as an intermediate result. The accumulator circuit accumulates the newly input intermediate results in each time sequence, completing m... 2 The pixel corresponding to the sliding position is obtained by accumulating the intermediate results.

3. The configurable neural network arithmetic unit for embedded devices as described in claim 2, characterized in that, In a multi-channel multiplication circuit, each adder component sums the input α summation factors in pairs to obtain α / 2 summation results, which are then used as new summation factors to input into the next stage circuit.

4. The configurable neural network arithmetic unit for embedded devices as described in claim 1, characterized in that, When the multiply-accumulate circuit completes the convolution kernel {c under the control of the arithmetic control module ij } m*m Image data of each channel under the current sliding window {p ij } m*m After the convolution calculation, continue to execute the convolution kernel {c} in the same pattern. ij } m*m Image data of each channel in the next sliding window {p ij } m*m The convolution calculation continues until the sliding window finishes sliding, obtaining the convolutional image data for the corresponding channel.

5. The configurable neural network arithmetic unit for embedded devices as described in claim 1, characterized in that, The multi-channel convolution calculation circuit has multiple multiply-accumulate circuits, which operate in parallel to implement multiple convolution kernels {c}. ij } m*m Image data of each channel under the sliding window of the convolution kernel {p ij } m*m Convolution calculation.

6. The configurable neural network arithmetic unit for embedded devices as described in claim 1, characterized in that, The multi-channel convolutional computation circuit uses a 3*3 kernel size when performing convolutional computation for each layer. When the convolutional image data to be convolved exceeds 20*20, the sliding window stride is 2, and when the convolutional image data to be convolved does not exceed 20*20, the sliding window stride is 1.

7. The configurable neural network arithmetic unit for embedded devices as described in claim 1, characterized in that, The neural network arithmetic unit further includes an address generation module, which comprises an address reading circuit and an address storage circuit. The address reading circuit generates the reading address of the image data for each convolution operation and the corresponding reading address of the convolution parameters. The address storage circuit generates the storage address of the convolutional image data after each convolution operation. The rules for generating the storage address of each convolutional image data step by the address storage circuit include: convolutional image data from different channels of the same layer are stored in different block storage units in the convolution result storage structure, and each pixel of the convolutional image data of the same channel is stored in the corresponding block storage unit according to the position of the pixel in the image, forming a mapping relationship between storage address and pixel position: if the pixel position in the image is the xth row and yth column, the address stored in the block storage unit is a*(x-1)+y or x+b*(y-1), where a is the number of pixels in a row of the image and b is the number of pixels in a column of the image; The address read circuit generates image data for each convolution operation {p ij } m*m The rules for reading the address of each pixel include: locating the position of the middle pixel in the current sliding window in the channel image data, determining the position of other pixels in the channel image data based on the positional relationship between other pixels in the current sliding window and the middle pixel, and calculating the address of each pixel in the current sliding window based on the mapping relationship between storage address and pixel position.

8. The configurable neural network arithmetic unit for embedded devices as described in claim 7, characterized in that, The convolution result storage structure has two separate storage spaces, each containing multiple storage blocks; The rules for storing convolution results in a multi-channel convolution computation circuit include: Image data from two adjacent convolutional layers are stored in two different storage spaces. In the same storage space, the β-th convolutional image data is stored in the storage block of the β-2-th convolutional image data, where β is the convolutional layer number, β≥3, and the β-th convolutional image data gradually covers the pixels of the β-2-th convolutional image data located at the same position according to the pixel position.

9. The configurable neural network arithmetic unit for embedded devices as described in claim 1, characterized in that, The fully connected computation module is used to perform multiplication and addition operations on the average pooling results of each channel and the parameters of the same fully connected layer channel, compare the multiplication and addition results of different fully connected layer channels, and take the largest result as the classification result of the image data.

10. An embedded device, characterized in that, The embedded device configurable neural network processor includes any one of claims 1 to 9.

Images