Image convolution circuit, method, image recognition method, device and equipment

Abstract

The present invention provides an image convolution circuit, method, image recognition method, device and equipment, wherein the image convolution circuit includes M cache units and a convolution operation circuit, wherein the convolution operation circuit includes a convolution center operation unit and M 2 ‑1 non-convolution center operation unit, convolution center operation unit and M 2 ‑1 non-convolution center operation unit constitutes an M*M operation unit matrix; the convolution center operation unit includes a convolution center multiplier, a convolution center weight register, and a center pixel register. The first input end of the convolution center multiplier is used to connect to the convolution center weight register to input the weight of the convolution kernel center, and the second input end is used to connect to the center pixel register to input the pixel value of the convolution kernel center pixel point corresponding to the target image; the non-convolution center operation unit includes a non-convolution center multiplier, a non-convolution center weight register, a non-center pixel register, an adder, and a gate.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to image convolution circuits, methods, image recognition methods, devices and equipment. Background Art

[0002] Deep learning provides a highly efficient method for machines to solve complex problems. In recent years, computer vision has become integrated into every aspect of our lives, from autonomous driving in transportation to medical image processing in healthcare to ubiquitous facial recognition applications. Computer vision plays a vital role in every industry, not only providing strong security for our lives but also meeting the public's desire for convenience. Image feature extraction and classification, a fundamental research area within deep learning neural networks, has been widely pursued by industry scholars and has played a significant role in driving the advancement of machine vision technology.

[0003] Although the convolutional neural networks included in machine learning have powerful performance, as the network depth continues to expand, problems such as complex network layers, huge data volumes, and intensive data storage have emerged. These problems have put forward more stringent requirements on the operating equipment and restricted the application of convolutional neural networks in embedded systems. Therefore, it is necessary to build a hardware-friendly convolutional neural network so that it can be implemented in hardware on low-cost devices with low computing power.

[0004] The convolution layer in a convolutional neural network contains a large number of convolution operation units. The convolution operation unit is a functional module that performs a series of multiplication and addition operations on the image pixel values ​​and training weights within the convolution window. Currently, shift registers are usually used to implement convolution operations in the convolution layer (also known as convolution window operations). Figure 1This diagram illustrates a convolution operation using shift registers. The shift registers act as line buffers (Line Buffer 0-Line Buffer 2), sequentially loading serial image data (data_in) with the clock tick. These are then shifted right at each clock edge. The data at the end of each line is output as the data_out vector. After multiple shifts, the output data_out vectors (data_out0-data_out2) form a window, which serves as the convolution operation window. This window is then fed into the convolution operation unit (d0-d1). Specifically, after all line buffers are filled, the output data_out vectors form the first 3×3 window after two clock cycles. Thereafter, the convolution window shifts right by one unit with each clock edge. If the input feature map is N×N and the convolution window is K×K, a total of K line buffers with N memory cells are required, resulting in excessive memory consumption. Furthermore, when the convolution window implemented using this method slides to the end of a line, invalid window data is generated. Figure 2 The figure shows a case of invalid data in the image convolution process. The gray square is the current location of the convolution window. Since the convolution window is no longer a complete window in space, the result of convolution based on the gray square is invalid, that is, the gray square is an invalid pixel. Summary of the Invention

[0005] Based on the above situation, the main purpose of the present invention is to provide an image convolution circuit, method, image recognition method, device and equipment, so as to improve the convolution efficiency with a simpler circuit structure.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] An image convolution circuit includes M cache units and a convolution operation circuit, wherein the convolution operation circuit includes a convolution center operation unit and M 2 -1 non-convolution center operation unit, M cache units and the convolution center operation unit and M 2-1 non-convolution center operation unit is connected in such a way that these operation units constitute an M*M operation unit matrix; the non-convolution center operation unit connected to the convolution center operation unit is the first non-convolution center operation unit; the convolution center operation unit includes a convolution center multiplier, a convolution center weight register and a center pixel register, the first input end of the convolution center multiplier is connected to the convolution center weight register to input the weight of the convolution kernel center, and the second input end is connected to the center pixel register to input the pixel value of the convolution kernel center pixel point corresponding to the target image; the non-convolution center operation unit includes a non-convolution center multiplier, a non-convolution center weight register, a non-center pixel register, an adder and a gate, the first input end of the non-convolution center multiplier is connected to a non-convolution center weight register to input the weight corresponding to the non-convolution kernel center, and the second input end is connected to the center pixel register to input the pixel value of the convolution kernel center pixel point corresponding to the target image The center pixel register inputs the pixel value of the non-convolution kernel center pixel point of the target image, the output end of the non-convolution center multiplier is connected to the first input end of the adder, the output end of the adder is connected to the first input end of the gate, and the gate control end of the gate inputs a gate control signal to control the first input end of the gate corresponding to the non-convolution kernel center that coincides with the pixel point in the target image to be gated, and to control the second input ends of the remaining gates to be gated; each cache unit is used to cache the pixel values ​​of a certain row of pixel points from the target image, and provide the cached pixel values ​​to each center pixel register or non-center pixel register of the corresponding row of the operation unit matrix; in the first non-convolution center operation unit, the second input end of the adder is connected to the output end of the convolution center multiplier, and the second input end of the gate is connected to the output end of the convolution center multiplier; from the 1st to the Mth 2 -2 non-convolution center operation units, the output end of the gate is connected to the second input end of the adder in the next non-convolution center operation unit and the second input end of the gate; 2 The output end of the gate in the -1 non-convolution center operation unit is used to output the image convolution result; wherein M is an odd number greater than or equal to 3.

[0008] Preferably, the center pixel register, the 1st non-center pixel register to the M-1th non-center pixel register sequentially constitute a shift register group including M registers, wherein the 1st register to the 1+(M-1) / 2th register sequentially store the pixel values ​​of the center pixel point to the (M-1) / 2th pixel point to the right of the center pixel point in the target image, and the 2nd+(M-1) / 2th register to the Mth register store the pixel values ​​of the (M-1) / 2th pixel point to the left of the center pixel point in the target image; the corresponding cache unit inputs the pixel values ​​of the pixel points in the row where the center pixel point is located from the 1+(M-1) / 2th register, and each input is a When there are pixel values: the pixel values ​​stored in the 2nd+(M-1) / 2th register to the M-1th register are respectively moved to the right to the next register, the pixel value of the 1st register is moved to the 2nd+(M-1) / 2th register, and the pixel values ​​stored in the 2nd register to the 1st+(M-1) / 2th register are respectively moved to the left to the previous register, or: the pixel values ​​stored in the 3rd+(M-1) / 2th register to the Mth register are respectively moved to the left to the previous register, the pixel value of the 1st register is moved to the Mth register, and the pixel values ​​stored in the 2nd register to the 1st+(M-1) / 2th register are respectively moved to the left to the previous register.

[0009] Preferably, the image convolution circuit further comprises M gate switches, wherein the input end of each gate switch is connected to the output end of the target image memory, the output end of each gate switch is connected to the input end of the corresponding cache unit, and the output end of each cache unit is connected to a non-center pixel register of the corresponding row of the operation unit matrix;

[0010] Each strobe switch further includes a strobe terminal, which selectively controls the strobe switch of the corresponding row to be turned on and controls the strobe switches of the remaining rows to be turned off according to the value of the row where the current pixel point is located output by the target image memory.

[0011] Preferably, when the center pixel point is located at the upper edge row of the target image, (M-1) / 2 corresponding selection switches are controlled to remain off, and at least the second input ends of the selection switches of (M-1) / 2 corresponding rows in the control operation unit matrix are controlled to be selected.

[0012] Preferably, the image convolution circuit further comprises a target image memory, which takes the pixel point at the upper left corner of the target image as a starting point and numbers the pixels in the target image in increasing order from left to right and from top to bottom; takes the address of the pixel point at the upper left corner of the target image in the target image memory as a starting address, determines the offset address of the corresponding pixel point in the target image according to the pixel number, and sequentially stores the pixel values ​​of the pixel points in the target image in the target image memory;

[0013] The position of the pixel point in the target image is determined according to the offset address of the pixel point input into the central pixel register, and the gating control signal is determined according to the position.

[0014] Preferably, the image convolution circuit also includes an activation function circuit, which includes a comparator and an activation gate. The first input end of the comparator inputs the image convolution result output by the convolution operation circuit, the second input end inputs a low level, and the output end is connected to the gate control end of the activation gate; the first input end of the activation gate inputs the image convolution result output by the operation circuit, the second input end inputs a low level, and the output signal of the output end serves as the final image convolution result of the image convolution circuit.

[0015] The present invention also provides an image convolution method, which adopts the image convolution circuit, including the following steps: each cache unit caches the pixel value of a pixel point in a certain row of the target image, and provides the cached pixel value to each center pixel register or non-center pixel register of the corresponding row of the operation unit matrix; the convolution center multiplier multiplies the weight of the convolution kernel center with the pixel value of the convolution kernel center pixel point, and outputs the multiplication result to the second input end of the adder in the first non-convolution center operation unit and the second input end of the gate; the gate control signal of the gate control end of the gate in each non-convolution center operation unit controls the first input end of the gate corresponding to the non-convolution kernel center coinciding with the pixel point in the target image to be gated, and controls the second input ends of the remaining gates to be gated; the first to Mth 2 -2 non-convolution center operation units, the gate outputs the gating result to the second input end of the adder in the next non-convolution center operation unit and the second input end of the gate according to the control of the gating control signal; 2 -1 non-convolution center operation unit in the gate output image convolution result.

[0016] Preferably, the convolution method adopts the image convolution circuit, and the corresponding cache unit inputs the pixel value of the pixel point in the row where the center pixel point is located from the 1+(M-1) / 2th register. Each time a pixel value is input: the pixel values ​​stored in the 2+(M-1) / 2th register to the M-1th register are respectively moved to the right to the next register, the pixel value of the 1st register is moved to the 2+(M-1) / 2th register, and the pixel values ​​stored in the 2nd register to the 1+(M-1) / 2th register are respectively moved to the left to the previous register, or: the pixel values ​​stored in the 3+(M-1) / 2th register to the Mth register are respectively moved to the left to the previous register, the pixel value of the 1st register is moved to the Mth register, and the pixel values ​​stored in the 2nd register to the 1+(M-1) / 2th register are respectively moved to the left to the previous register.

[0017] Preferably, the convolution method adopts the image convolution circuit, and the selection end of each selection switch selects to control the selection switch of the corresponding row to be turned on according to the value of the row where the current pixel point is located output by the target image memory, and controls the selection switches corresponding to the remaining rows to be turned off.

[0018] Preferably, when the center pixel point is located at the upper edge row of the target image, the (M-1) / 2 corresponding selection switches are controlled to remain off, and the second input ends of the non-center pixel registers of the (M-1) / 2 corresponding rows in the control operation unit matrix are controlled to be selected.

[0019] Preferably, the convolution method adopts the image convolution circuit to determine the position of the pixel point in the target image according to the offset address of the pixel point input into the central pixel register, and determines the selection control signal according to the position.

[0020] Preferably, the convolution method adopts the image convolution circuit. When the image convolution result is greater than 0, the first input end of the activation gate is selected, and the final image convolution result is the same as the image convolution result; when the image convolution result is less than 0, the second input end of the activation gate is selected, and the final image convolution result is 0.

[0021] The present invention also provides an image recognition method, comprising the following steps: S100, sequentially subjecting a target image to convolution processing through a plurality of first image convolution circuits and a plurality of second image convolution circuits to obtain a first convolution image; S200, performing pooling processing on the first convolution image to obtain a first pooled image; S300, sequentially subjecting the first pooled image to convolution processing through a plurality of third image convolution circuits and a plurality of fourth image convolution circuits to obtain a second convolution image; S400, performing pooling processing on the second convolution image to obtain a second pooled image; S50 0, convolution processing is performed on the second pooled image through multiple fifth image convolution circuits and multiple sixth image convolution circuits in sequence to obtain a third convolution image; S600, maximum pooling processing is performed on the third convolution image to obtain a third pooled image; S700, full connection processing is performed on the third pooled image to obtain a category result of image recognition; wherein, the first image convolution circuit, the second image convolution circuit, the third image convolution circuit, the fourth image convolution circuit, the fifth image convolution circuit and the sixth image convolution circuit all adopt any one of the image convolution circuits.

[0022] Preferably, the pooling process in steps S200 and S400 is maximum pooling process.

[0023] Preferably, the image recognition method is applied to handwritten letter recognition.

[0024] The present invention also provides an image recognition device, comprising: a first convolution unit, for subjecting a target image to convolution processing through a plurality of first image convolution circuits and a plurality of second image convolution circuits in sequence to obtain a first convolution image; a first pooling unit, for subjecting the first convolution image to pooling processing to obtain a first pooled image; a second convolution unit, for subjecting the first pooled image to convolution processing through a plurality of third image convolution circuits and a plurality of fourth image convolution circuits in sequence to obtain a second convolution image; a second pooling unit, for subjecting the second convolution image to pooling processing to obtain a second pooled image; and a third A convolution unit is used to convolve the second pooled image through multiple fifth image convolution circuits and multiple sixth image convolution circuits in sequence to obtain a third convolution image; a third pooling unit is used to perform maximum pooling processing on the third convolution image to obtain a third pooled image; a fully connected unit is used to perform fully connected processing on the third pooled image to obtain a category result of image recognition; wherein the first image convolution circuit, the second image convolution circuit, the third image convolution circuit, the fourth image convolution circuit, the fifth image convolution circuit and the sixth image convolution circuit all adopt any of the above-mentioned image convolution circuits.

[0025] The present invention also provides an image recognition chip, comprising any one of the above-mentioned image convolution circuits.

[0026] The present invention also provides an embedded image recognition device, comprising the image recognition chip.

[0027] Preferably, the image convolution circuit may include multiple different convolution kernel (i.e., convolution operation circuit) groups; each convolution kernel group includes multiple identical convolution kernels, and each convolution kernel can be considered a convolution kernel on a channel. A group gating signal may be set for each convolution kernel group, and a channel gating signal may be set for each convolution kernel within each convolution kernel group. When a convolution kernel within a convolution kernel group is to be used, the convolution kernel group is selected by controlling the group gating signal, and the convolution kernel is selected by controlling the channel gating signal.

[0028] Beneficial effects

[0029] Through the above circuit structure, the convolution center weight register and the center pixel register corresponding to the convolution kernel are placed in the first operation unit of the M*M operation unit matrix, which can not only ensure that the result of multiplying the convolution center weight of the convolution center and the pixel value of the corresponding center pixel is added to the product generated by other non-convolution center operation units, but also realize the gating of non-convolution center operation units. The convolution operation efficiency is high, and there is no need to set the corresponding adder and gate in the convolution center operation unit, so the circuit structure is relatively simple. In addition, by inputting the gating control signal determined according to the position of the center pixel point corresponding to the center pixel register in the target image, the first input end or the second input end of the gate is controlled to select the output result of the adder of the non-convolution center operation unit corresponding to the convolution kernel can be selected, while the output result of the adder of the non-convolution center operation unit at the rest of the convolution kernel position is not selected, thereby effectively avoiding the result of the invalid position affecting the convolution result.

[0030] Other beneficial effects of the present invention will be explained through the introduction of specific technical features and technical solutions in the specific implementation methods. Those skilled in the art should be able to understand the beneficial technical effects brought about by the introduction of these technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

[0032] Figure 1 Schematic diagram of image convolution circuit in the prior art;

[0033] Figure 2 Schematic diagram of the image convolution process;

[0034] Figure 3 A schematic diagram of an image convolution circuit according to a preferred embodiment of the present invention;

[0035] Figure 4A schematic diagram of an image convolution process according to a preferred embodiment of the present invention;

[0036] Figure 5 Schematic diagram of a convolution kernel according to a preferred embodiment of the present invention;

[0037] Figure 6 A schematic diagram of storage of weights of a convolution operation circuit according to a preferred embodiment of the present invention;

[0038] Figure 7 This is a schematic diagram of an activation function circuit in an image convolution circuit according to a preferred embodiment of the present invention;

[0039] Figure 8 A schematic diagram of the structure of a convolutional neural network according to a preferred embodiment of the present invention;

[0040] Figure 9 This is a partial picture of a data set for training a convolutional neural network according to a preferred embodiment of the present invention;

[0041] Figure 10 A flowchart of an image recognition method according to a preferred embodiment of the present invention;

[0042] Figure 11 Schematic diagram of an image recognition system according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described below based on the following embodiments, but the present invention is not limited to these embodiments. In the following detailed description of the present invention, some specific details are described in detail. In order to avoid obscuring the essence of the present invention, well-known methods, processes, procedures, and components are not described in detail.

[0044] Furthermore, persons of ordinary skill in the art will appreciate that the figures provided herein are for illustration purposes only and are not necessarily drawn to scale.

[0045] Unless the context clearly requires otherwise, throughout the specification and claims, the words "include," "comprising," and similar words should be construed in an inclusive sense rather than an exclusive or exhaustive sense; that is, in the sense of "including but not limited to."

[0046] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0047] Figure 3: This is a schematic diagram of an image convolution circuit according to an embodiment of the present invention. The image convolution circuit is used to perform a convolution operation on an input target image and output an image convolution result. The target image is the object of the convolution processing of the image convolution circuit. The target image can be a source image or a feature image. The image convolution circuit can be located at different convolution layers of a convolutional neural network circuit. A plurality of such image convolution circuits can form a convolution circuit with a depth of n. The image convolution circuit includes M cache units (which use registers) and a convolution operation circuit. The convolution operation circuit includes a convolution center operation unit 100 and M 2 -1 non-convolution center operation unit 200, M cache units and convolution center operation unit 100 and M 2 -1 non-convolution center operation unit is connected in such a way that these operation units constitute an M*M operation unit matrix, the non-convolution center operation unit connected to the convolution center operation unit is the first non-convolution center operation unit, and each cache unit corresponds to a row of operation units.

[0048] The convolution center operation unit includes a convolution center multiplier, a convolution center weight register, and a center pixel register. The first input end of the convolution center multiplier is used to connect to the convolution center weight register to input the weight of the convolution kernel center, and the second input end is used to connect to the center pixel register to input the pixel value of the convolution kernel center pixel point corresponding to the target image.

[0049] The non-convolution center operation unit includes a non-convolution center multiplier, a non-convolution center weight register, a non-center pixel register, an adder, and a gate. The first input end of the non-convolution center multiplier is used to connect to a non-convolution center weight register to input the weight of the corresponding non-convolution kernel center, and the second input end is used to connect to the corresponding non-center pixel register to input the pixel value of the non-convolution kernel center pixel point of the target image. The output end of the non-convolution center multiplier is connected to the first input end of the adder, and the output end of the adder is connected to the first input end of the gate. The gate control end of the gate inputs a gate control signal to control the first input end of the gate corresponding to the non-convolution kernel center that coincides with the pixel point in the target image to be gated, and to control the second input ends of the remaining gates to be gated.

[0050] Each cache unit is used to cache pixel values ​​from a row of pixels in the target image and provide the cached pixel values ​​to the center pixel register or non-center pixel register of the corresponding row of the arithmetic unit matrix. The cache unit only needs to cache the pixel value of a single pixel in the target image. Compared to the prior art, which requires storage space equal to the pixel width of the target image, the cache unit of this embodiment requires very little storage space, significantly saving storage resources.

[0051] In the first non-convolution center operation unit, the second input end of the adder is connected to the output end of the convolution center multiplier, and the second input end of the gate is connected to the output end of the convolution center multiplier; 2 -2 non-convolution center operation units, the output end of the gate is connected to the second input end of the adder in the next non-convolution center operation unit and the second input end of the gate; 2 The output end of the gate in the -1 non-convolution center operation unit is used to output the image convolution result; wherein M is an odd number greater than or equal to 3.

[0052] In some embodiments, when the selection control signal is at a high level, the first input end of the selection gate is controlled, and when the selection control signal is at a low level, the second input end of the selection gate is controlled. For the first non-convolution center operation unit, for example, when the selection control signal of the first non-convolution center operation unit is at a high level, the first input end of the selection gate is controlled, so that the result output from the output end of the adder of the first non-convolution center operation unit is selected and output; when the selection control signal of the first non-convolution center operation unit is at a low level, the second input end of the selection gate is controlled, so that the result output from the output end of the multiplier of the convolution center operation unit is selected and output. For the second non-convolution center operation unit to the Mth non-convolution center operation unit, 2 -1 non-convolution central operation unit, for example, when the selection control signal of a non-convolution central operation unit is at a high level, the first input end of the gate is controlled to be selected, so that the result output from the output end of the adder of the non-convolution central operation unit is selected and output; when the selection control signal of the non-convolution central operation unit is at a low level, the second input end of the gate is controlled to be selected, so that the result output from the output end of the gate of the previous non-convolution central operation unit is selected and output.

[0053] Through the above circuit structure, the convolution center weight register and the center pixel register corresponding to the convolution kernel (or called convolution window) are placed in the first operation unit of the M*M operation unit matrix, which can not only ensure that the result of multiplying the convolution center weight of the convolution center and the pixel value of the corresponding center pixel is added to the product generated by other non-convolution center operation units, but also realize the gating of non-convolution center operation units, and the convolution operation efficiency is high.

[0054] In some embodiments, to reduce data call time, the nine convolution weights in an image convolution circuit (including the weights at the center of the convolution kernel and the weights outside the convolution kernel) are grouped into a data block and stored in a non-volatile memory (such as FLASH). When performing a convolution operation, the group of convolution weights can be read out and input into the corresponding convolution weight register in just one clock cycle. In some embodiments, the bit width of each convolution weight is 13 bits, so the total bit width of a group of convolution weights is 117 bits. Figure 6 shown.

[0055] In order to ensure that the pixel point input to the first operation unit in the M*M operation unit matrix is ​​the center pixel point corresponding to the convolution kernel, in the first row of operation units (i.e., the convolution center operation unit, the first non-convolution center operation unit to the M-1th non-convolution center operation unit), the center pixel register, the first non-center pixel register to the M-1th non-center pixel register constitute a shift register group including M registers.

[0056] Among them, the 1st register to the 1+(M-1) / 2th register sequentially store the pixel values ​​of the center pixel point in the target image to the (M-1) / 2th pixel point to the right of the center pixel point, and the 2nd+(M-1) / 2th register to the Mth register store the pixel values ​​of the (M-1) / 2th pixel point to the left of the center pixel point in the target image; the corresponding cache unit inputs the pixel values ​​of the pixels in the row where the center pixel point is located from the 1+(M-1) / 2th register, and each time a pixel value is input:

[0057] The pixel values ​​stored in the 2nd+(M-1) / 2th register to the M-1th register are respectively shifted to the right to the next register (for the Mth register, after the pixel value stored in the M-1th register is input, it is equivalent to the pixel value originally stored in the Mth register being shifted out of the shift register group), the pixel value of the 1st register is moved to the 2nd+(M-1) / 2th register, and the pixel values ​​stored in the 2nd register to the 1st+(M-1) / 2th register are respectively shifted to the left to the previous register, or:

[0058] The pixel values ​​stored in the 3rd + (M-1) / 2nd register to the Mth register are respectively shifted to the left to the previous register (for the 2nd + (M-1) / 2nd register, after the pixel value stored in the 3rd + (M-1) / 2nd register is input, it is equivalent to the pixel value stored in the 2nd + (M-1) / 2nd register being shifted out of the shift register group), the pixel value of the 1st register is moved to the Mth register, and the pixel values ​​stored in the 2nd register to the 1st + (M-1) / 2nd register are respectively shifted to the left to the previous register.

[0059] Take M=3 and the current convolution center is located at pixel point P1 (i.e., the center pixel point is P1) as an example. The center pixel register, the first and second non-center pixel registers constitute a shift register group including 3 registers, wherein the first register p5, the second register p6 (i.e., the first non-center pixel register), and the third register p4 (i.e., the second non-center pixel register) sequentially store the pixel value of the center pixel point P1 in the target image, the pixel value of the first pixel point P2 to the right of the center pixel point, and the pixel value of the first pixel point to the left of the center pixel point (since the current center pixel point is P1, the pixels to the left of the center pixel point P1 are empty); the third non-center pixel registers p1, p6, and p4 are used to store the pixel value of the center pixel point P1, the first pixel point P2 to the right of the center pixel point, and the first pixel point P2 ... The first non-center pixel register p2 and the fifth non-center pixel register p3 sequentially store the pixel values ​​of the corresponding three pixels in the row above the center pixel (since the current center pixel is P1, the corresponding three pixels in the row above the center pixel are empty); the sixth non-center pixel register p7, the seventh non-center pixel register p8, and the eighth non-center pixel register p9 sequentially store the pixel values ​​of the corresponding three pixels in the row below the center pixel (since the current center pixel is P1, the leftmost pixel in the row below the center pixel is empty, and the other two pixels are P5 and P6 respectively). During the convolution operation, the selection control signals EN0, EN2 and EN3 control the signal at the first end of the corresponding gate (that is, the signal output by the adder of the operation unit where the gate is located), so that the value in the corresponding non-center pixel register is multiplied by the weight in the non-convolution center weight register. The selection control signals EN1, EN4, EN5, EN6 and EN7 control the signal at the second end of the corresponding gate (that is, the signal output by the previous gate) so that the value in the corresponding non-center pixel register is multiplied by the weight in the non-convolution center weight register is not added, thereby realizing the convolution operation when the current convolution center is located at the pixel point P1.

[0060] The cache unit Buffer1 corresponding to the first row of operation units inputs the pixel values ​​of the pixels in the row where the center pixel point is located (that is, the row where points P1, P2, P3, and P4 are located) from the second register p6. The process of inputting each pixel value is as follows: the pixel value stored in the first register p5 is moved to the third register p4, thereby overwriting the pixel value originally stored in the third register p4; the pixel value stored in the second register p6 is moved to the left to the previous register (that is, the first register p5), and the cache unit Buffer1 inputs the pixel value of the new pixel point into the second register p6.

[0061] Regarding the cache unit Buffer0 corresponding to the second row of operation units, it can input the pixel points in the row above the center pixel point from the third non-center pixel register p1. The process of inputting each pixel value is as follows: the pixel values ​​stored in the fourth non-center pixel register p2 and the third non-center pixel register p1 are moved to the right to the next non-center pixel register in turn. Therefore, the pixel value originally stored in the fifth non-center pixel register p3 is overwritten, and the cache unit Buffer0 inputs the pixel value of the new pixel point to the third non-center pixel register p1. Of course, the cache unit Buffer0 can also input the pixel points in the row above the center pixel point from the fifth non-center pixel register p3, and the pixel values ​​stored in the fifth non-center pixel register p3 and the fourth non-center pixel register p2 are moved to the left to the previous non-center pixel register in turn.

[0062] Regarding the cache unit Buffer2 corresponding to the third row of operation units, it can input the pixel points in the row below the center pixel point from the 6th non-center pixel register p7. The process of inputting each pixel value is as follows: the pixel values ​​stored in the 7th non-center pixel register p8 and the 6th non-center pixel register p7 are moved to the right to the next non-center pixel register in turn. Therefore, the pixel value originally stored in the 8th non-center pixel register p9 is overwritten, and the cache unit Buffer2 inputs the pixel value of the new pixel point to the 6th non-center pixel register p7. Of course, the cache unit Buffer2 can also input the pixel points in the row below the center pixel point from the 8th non-center pixel register p9, and the pixel values ​​stored in the 7th non-center pixel register p8 and the 6th non-center pixel register p7 are moved to the left to the previous non-center pixel register in turn.

[0063] In some embodiments, the image convolution circuit further comprises M gating switches, each gating switch having an input connected to an output of a target image memory, an output connected to an input of a corresponding cache unit, and an output connected to a non-center pixel register of a corresponding row of the arithmetic unit matrix. Each gating switch further comprises a gating terminal that, based on the value of the row in which the current pixel point is output from the target image memory, selects and controls the gating switch of the corresponding row to be turned on, and controls the gating switches corresponding to the remaining rows to be turned off. In some embodiments, the output of the target image memory outputs the pixel value of the next pixel point after each clock signal. In some embodiments, the bit width of each cache unit is 13 bits.

[0064] Still taking M=3 as an example for explanation. Cache unit Buffer1, cache unit Buffer0 and cache unit Buffer2 respectively have corresponding selection switches, and the selection terminals of these selection switches are sel1, sel0 and sel2 respectively. The process of inputting the pixel values ​​of pixel point P1, pixel point P2, pixel point P5 and pixel point P6 into the corresponding registers p5, register p6, register p1 and register p2 respectively is as follows: (1) Selection terminal sel1 is selected, selection terminals sel0 and sel2 are turned off, the pixel value of pixel point P1 is output from the output terminal of the target image memory to cache unit Buffer1, and cache unit Buffer1 inputs the pixel value of pixel point P1 into register p6. (2) Selection terminal sel2 is selected, selection terminals sel1 and sel0 are turned off, the pixel value of pixel point P5 is output from the output terminal of the target image memory to cache unit Buffer2, and cache unit Buffer2 inputs the pixel value of pixel point P5 into register p7. (3) The strobe terminal sel1 is turned on, the strobe terminals sel0 and sel2 are turned off, the pixel value of pixel point P2 is output from the output terminal of the target image memory to the buffer unit Buffer1, the pixel value of pixel point P1 stored in register p6 is shifted to the left to the previous register p5, and the buffer unit Buffer1 inputs the pixel value of pixel point P2 to register p6. (4) The strobe terminal sel2 is turned on, the strobe terminals sel1 and sel0 are turned off, the pixel value of pixel point P5 is output from the output terminal of the target image memory to the buffer unit Buffer2, the pixel value of pixel point P5 stored in register p7 is shifted to the right to the next register p8, and the buffer unit Buffer2 inputs the pixel value of pixel point P6 to register p7. At this point, the convolution operation can be performed when the current convolution center is located at pixel point P1. For the specific process, please refer to the description of the previous embodiment.

[0065] After the convolution operation is completed when the current convolution center is located at pixel point P5, the specific process of the pixel value of the buffer unit and register inputting the pixel point is as follows. (1) The strobe terminal sel1 is strobed, the strobe terminals sel0 and sel2 are turned off, the pixel value of pixel point P3 is output from the output end of the target image memory to the buffer unit Buffer1, the pixel value of pixel point P5 stored in register p5 is moved to register p4, the pixel value of pixel point P6 stored in register p6 is moved to the left to the previous register p5, and the buffer unit Buffer1 inputs the pixel value of pixel point P7 to register p6. (2) The strobe terminal sel0 is strobed, the strobe terminals sel1 and sel0 are turned off, the pixel value of pixel point P1 stored in register p2 is moved to the right to the next register p3, the pixel value of pixel point P2 stored in register p1 is moved to the right to the next register p2, the pixel value of pixel point P3 is output from the output end of the target image memory to the buffer unit Buffer0, and the buffer unit Buffer0 inputs the pixel value of pixel point P3 to register p1. (3) The strobe terminal sel2 is turned on, and the strobe terminals sel1 and sel0 are turned off. The pixel value of pixel point P9 stored in register p8 is shifted rightward to the next register p9. The pixel value of pixel point P10 stored in register p7 is shifted rightward to the next register p8. The pixel value of pixel point P11 is output from the output terminal of the target image memory to the buffer unit Buffer2. The buffer unit Buffer2 inputs the pixel value of pixel point P11 to register p7. At this point, the convolution operation can be performed when the current convolution center is located at pixel point P6. The specific process can be referred to the description of the previous embodiment.

[0066] In some embodiments, when the center pixel is located at the upper edge row of the target image, (M-1) / 2 corresponding selection switches are controlled to remain off, and the second input terminals of the non-center pixel registers of (M-1) / 2 corresponding rows in the control operation unit matrix are controlled to be selected. Still taking M=3 as an example for explanation. When the center pixel is P1, since the pixel points in the row above the center pixel P1 are empty, that is, the corresponding registers p1, p2, and p3 do not need to store the corresponding pixel points, the corresponding selection switches (3-1) / 2=1) (the selection switches corresponding to sel0) are controlled to remain off, and the second input terminals of at least (3-1) / 2=1 corresponding rows of the selectors (the selectors corresponding to EN2, EN3, and EN4) in the control operation unit matrix are controlled to be selected.

[0067] As mentioned above, when the center of the convolution window (i.e., the convolution kernel) is located at different pixels in the target image (i.e., the pixel value of the pixel is input into the center pixel register p5), the position of the convolution window participating in the convolution operation (which corresponds to the corresponding weight) is different. Figure 5It is a schematic diagram of a 3*3 convolution kernel (window). The convolution window table 1 shows the correspondence between the pixel position of the convolution window center and the position of the convolution window involved in the operation. Figure 3 And its description, it is easy to understand the position of the convolution window involved in the convolution operation and its convolution process.

[0068] Table 1 Correspondence between the pixel position of the convolution window center and the position involved in the convolution window operation

[0069] The location of the pixel where the convolution window center is located The position of the convolution window involved in the operation The upper left vertex of the target image m5, m6, m8, m9 The upper right vertex of the target image m4, m5, m7, m8 Lower left vertex of the target image m2, m3, m5, m6 Lower right vertex of the target image m1, m2, m4, m5 Left edge of the target image (excluding vertices) m2, m3, m5, m6, m8, m9 Right edge of the target image (excluding vertices) m1, m2, m4, m5, m7, m8 Upper edge of the target image (excluding vertices) m4, m5, m6, m7, m8, m9 Lower edge of the target image (excluding vertices) m1, m2, m3, m4, m5, m6 Other locations m1, m2, m3, m4, m5, m6, m7, m8, m9

[0070] In some embodiments, the image convolution circuit further includes a target image memory, which takes the pixel point in the upper left corner of the target image as the starting point, and numbers the pixel points in the target image in order from left to right and from top to bottom as pixel numbers that increase in sequence; takes the address of the pixel point in the upper left corner of the target image in the target image memory as the starting address, determines the offset address of the corresponding pixel point in the target image according to the pixel number, and stores the pixel values ​​of the pixel points in the target image in sequence in the target image memory; determines the position of the pixel point in the target image according to the offset address of the pixel point input to the central pixel register, and determines the selection control signal according to the position. Figure 4 As shown, the pixel P1 in the upper left corner of the target image is the starting point. From left to right and from top to bottom, the pixels in the target image are numbered in ascending order: P1, P2, ..., P16. Assuming that the address of pixel P1 in the target image memory is addr, the offset addresses of the addresses of pixels P1, P2, P3, ..., P16 in the target image memory relative to the address addr of pixel P1 are: (1-1=0), (2-1=0), (3-1=2) ... (16-1=15), that is, the addresses of pixels P1, P2, P3, ..., P16 in the target image memory are: addr, addr+1, addr+2 ..., addr+15, respectively. For another example, the address of the pixel below pixel P1 is addr+width, where width is the width of the target image. The position of the pixel point in the target image can be determined according to the offset address of the pixel point input to the central pixel register, and the selection control signal can be determined according to the position. Once the pixel point input to the central pixel register is determined, the position of the pixel point where the center of the convolution window is located is determined, as shown in Table 1. It can be determined which positions in the convolution window need to participate in the convolution calculation. The selection control signal of the gate is also determined to select the first input end or the second input end of the gate.

[0071] like Figure 7As shown, in some embodiments, the image convolution circuit further includes an activation function circuit to activate and correct the convolution result output by the convolution operation circuit. The activation function circuit includes a comparator 10 and an activation gate 20. The first input of the comparator 10 inputs the image convolution result Out output by the convolution operation circuit, the second input inputs a low level, and the output is connected to the gate control terminal of the activation gate 20. The first input of the activation gate inputs the image convolution result Out output by the operation circuit, the second input inputs a low level, and the output signal of the output terminal serves as the final image convolution result Out_f of the image convolution circuit. This activation function circuit implements the function of the ReLU function (the ReLU function is essentially a function that takes a maximum value compared to 0). When the image convolution result Out is greater than 0, the first input of the activation gate 20 is gated, that is, the image convolution result Out is gated for output. In this case, the final image convolution result Out_f is the same as the image convolution result Out. When the image convolution result Out is less than 0, the second input of the activation gate 20 is gated, that is, the low level is gated for output. In this case, the final image convolution result Out_f is 0.

[0072] When the above-mentioned image convolution circuit is applied to the convolution layer of a convolutional neural network, the stage control signal "stage" can be used to control the stage in which the convolutional neural network executes its tasks. Before the convolution layer begins operation, the storage controller generates an address signal to store the input image in the storage module. After storage is completed, the stage control signal "stage" is incremented by 1, entering the next stage. The convolution layer enable signal "conv_en" becomes valid (e.g., pulled high), and the convolution layer begins operation. At this point, the convolution operation enable signal "GO" is invalid (e.g., at a low level). After the pixel values ​​and corresponding weights of the target image pixels used by the convolution window are loaded into the corresponding registers, the convolution operation enable signal "GO" becomes valid (e.g., pulled high), and the convolution operation circuit begins to perform the convolution operation. After the convolution operation is completed, the convolution operation enable signal "GO" becomes invalid (e.g., pulled low), and the image convolution result is stored in the storage module. This process is repeated until all pixels of the input target image have been traversed. In some embodiments, the image convolution circuit may include multiple different convolution kernel (i.e., convolution operation circuit) groups; each convolution kernel group includes multiple identical convolution kernels, and each convolution kernel can be considered a convolution kernel on a channel. A group selection signal kel may be set for each convolution kernel group, and a channel selection signal chl may be set for each convolution kernel within each convolution kernel group. When a convolution kernel within a convolution kernel group is to be used, the convolution kernel group is selected by controlling the group selection signal kel, and the convolution kernel is selected by controlling the channel selection signal chl.

[0073] like Figure 11Figure 1 is a schematic diagram of an image recognition system according to an embodiment. The image recognition system includes an image sensor (e.g., an OV7670 image sensor), a convolutional neural network operation device, and a display (e.g., a liquid crystal (LCD) display). The convolutional neural network operation device includes an image acquisition module, an image preprocessing module, a storage module (e.g., FLASH), a convolutional neural network operation circuit, and an image display module. The convolutional neural network operation circuit includes the aforementioned image convolution circuit. The convolutional neural network can be based on a lightweight convolutional neural network (CNN), which is an improvement on the convolutional neural network of the LeNet-5 model.

[0074] The image sensor is used to capture images (e.g., images of handwritten letters); the image acquisition module is used to capture images transmitted by the image sensor to the convolutional neural network operation device and can also perform relevant configurations on the image sensor. The image acquisition module stores the captured source images in the storage module, which stores and schedules the source images. The image preprocessing module is used to preprocess the stored images. The preprocessed images can be stored in the storage module so that they can be sent to the convolutional neural network operation circuit for processing when needed. The preprocessed images can also be directly sent to the convolutional neural network operation circuit for processing. For example, the image can be grayscaled or fixed-point processed, that is, floating-point numbers representing pixel values ​​are converted to fixed-point numbers (e.g., represented by fixed 13-bit binary data). The convolutional neural network operation circuit performs operations such as convolution, pooling, and full connection on the preprocessed image to obtain calculation results for each category of the objects in the image. The numerical values ​​of the calculation results for each category are compared, and the category corresponding to the largest value is selected as the predicted category result of the object in the image. The predicted category result and the source image are output to a display through the image display module for display.

[0075] Establish as Figure 8 The structure of the convolutional neural network shown in FIG5 includes the following layers connected in sequence: input layer, first layer (convolutional layer C1), second layer (convolutional layer C2), third layer (first pooling layer S1), fourth layer (convolutional layer C3), fifth layer (convolutional layer C4), sixth layer (second pooling layer S2), seventh layer (convolutional layer C5), eighth layer (convolutional layer C6), ninth layer (maximum pooling layer GMP), tenth layer (fully connected layer Dense). Then, the convolutional neural network is trained. In some embodiments, the convolutional neural network is used to recognize handwritten letters. First, a handwritten letter dataset is established, and a large number of (for example, 213,962) handwritten letter pictures that conform to the writing habits of Chinese people are collected and screened. In order to facilitate the call of the convolutional neural network, the image data is converted into a format consistent with the MNIST dataset. Some pictures in the dataset are as follows: Figure 9 shown.

[0076] The input layer is used to input an image (which may be a pre-processed grayscale image) containing objects to be recognized, such as handwritten letters. In some embodiments, the input image is a grayscale image of handwritten letters with a size of 28×28.

[0077] The first layer is the convolution layer C1, which can contain multiple convolution kernels of different sizes and / or contained weights. The convolution kernels are implemented using the image convolution circuits of the various embodiments described above. In some embodiments, this layer contains four convolution kernels of size 3×3, and the weights of each convolution kernel are different (i.e., the weights stored in the weight registers at the center of the corresponding convolution kernel or the weight registers at the non-convolution kernel center are different). As mentioned above, each time a convolution is completed, the convolution kernel is translated by one pixel, that is, the step value of its movement is 1; in addition, the weight only includes the part multiplied by the pixel value of the corresponding pixel point, and does not include the bias term (in the prior art, for each pixel point, the bias term needs to be added to the corresponding product result). As mentioned above, there is no corresponding bias term register in the calculation unit of the image convolution circuit. Since the input of the convolution layer C1 is a grayscale image of size 28×28, the output feature map C1 of this layer is four images of size 28×28. The total number of training parameters of the convolutional layer C1 is: (3×3×1)×4=36, that is, 36 weights.

[0078] The second layer is the convolution layer C2, which can contain multiple convolution kernels of different sizes and / or contained weights. The convolution kernels are implemented using the image convolution circuits of the various embodiments described above. In some embodiments, this layer contains 4 convolution kernels of size 3×3×4 (i.e., a 3×3 convolution kernel with a depth of 4, which can be seen to be composed of 4 3×3 convolution kernels), and its moving step value is 1, without bias terms. The size of the 4 images input to this layer are all 28×28, so the image C2 generated by this layer is a 28×28 4-channel image. The total number of training parameters for this layer is: (3×3×4)×4=144, that is, 144 weights.

[0079] The third layer is the first pooling layer, S1. This layer contains a pooling window that can use a max pooling operation. The pooling window size can be 2×2. During pooling, the pooling window moves across the processed image, and the maximum pixel value within the 2×2 pooling window is used as the pixel value of the new pixel. The pooling window moves in increments of 2. After pooling, the resulting image size is reduced by half. This layer is fed with four 28×28 feature maps. Therefore, the image S1 generated by this layer is a 14×14 four-channel image. The pooling layer does not contain training parameters, that is, it does not contain the weights mentioned above.

[0080] The fourth layer is the convolution layer C3, which can contain multiple convolution kernels of different sizes and / or embedded weights. The convolution kernels are implemented using the image convolution circuits of the various embodiments described above. In some embodiments, this layer contains 12 convolution kernels of size 3×3×4, with a step value of 1, and no bias term. This layer is fed with 4 images of size 14×14, so C3 is 12 feature maps of size 14×14. The total number of training parameters for this layer is: (3×3×4)×12=432, that is, 432 weights.

[0081] The fifth layer is the convolution layer C4, which can contain multiple convolution kernels of different sizes and / or embedded weights. The convolution kernels are implemented using the image convolution circuits of the various embodiments described above. In some embodiments, this layer contains 12 convolution kernels of size 3×3×12, with a moving step value of 1 and no bias term. This layer is fed with 12 images of size 14×14, so the C4 layer is 12 feature maps of size 14×14. The total number of training parameters for the fifth layer is: (3×3×12)×12=1296, that is, 1296 weights.

[0082] The sixth layer is the second pooling layer, S2. This layer contains a pooling window that uses a max pooling operation. The pooling window size is 2×2, and the shift step is 2. This layer is fed with 12 feature maps of size 14×14. Therefore, the S2 layer is fed with 12 images of size 7×7. Pooling layers do not contain training parameters, that is, they do not contain the weights mentioned above.

[0083] The seventh layer is the convolution layer C5, which can contain multiple convolution kernels of different sizes and / or embedded weights. The convolution kernels are implemented using the image convolution circuits of the various embodiments described above. In some embodiments, this layer contains 16 convolution kernels of size 3×3×12, with a moving step value of 1, no bias term, and pixel expansion. This layer is fed with 12 images of size 7×7, so the C5 layer is 16 feature maps of size 7×7. The total number of training parameters for this layer is: (3×3×12)×16=1728, that is, 1728 weights.

[0084] The eighth layer is the convolution layer C6, which can contain multiple convolution kernels of different sizes and / or embedded weights. The convolution kernels are implemented using the image convolution circuits of the various embodiments described above. In some embodiments, this layer contains 16 convolution kernels of size 3×3×16, with a step value of 1, and no bias term. This layer is fed with 16 images of size 7×7, so the C6 layer is 16 feature maps of size 7×7. The total number of training parameters for the eighth layer is: (3×3×16)×16=2304, that is, 2304 weights.

[0085] The ninth layer is the global max pooling (GMP) layer. Global max pooling extracts the maximum value of each feature map input to this layer. Here, it can be considered a max pooling layer with a pooling window of 7×7. Global max pooling reduces images of various sizes to a 1×1 image. This layer is fed with 16 7×7 images. Therefore, the GMP layer generates 16 1×1 feature maps, which translates to 16 numerical values ​​to facilitate interfacing with the next fully connected layer. This layer does not contain any training parameters.

[0086] The final layer, the fully connected Dense layer, serves as the model's output layer and contains N nodes, where N represents the total number of categories to be classified. In some embodiments, the fully connected Dense layer has 27 nodes, representing 27 categories, including 26 uppercase letters and one non-letter category. The total number of training parameters for this layer is 16 × 27 = 432.

[0087] Adding up the number of training weights of each layer above, the total number of weights of this network structure is 6372. After completing the training of the convolutional neural network, the convolutional neural network can be used for image recognition. Figure 10 FIG. 1 is a flow chart of an image recognition method according to an embodiment, and the image recognition method includes the following steps.

[0088] At S100, the target image is sequentially convolved through the first image convolution circuit and the second image convolution circuit to obtain a first convolution image. As previously described, the target image is sequentially convolved through the first image convolution circuit of the first layer (convolution layer C1) and the second image convolution circuit of the second layer (convolution layer C2) to obtain the first convolution image, which can also be called the first feature image.

[0089] S200: Perform pooling processing on the first convolution image to obtain a first pooled image. As mentioned above, the first convolution image is pooled by the third layer (the first pooling layer S1) to obtain the first pooled image.

[0090] At S300 , the first pooled image is convolved sequentially with the third image convolution circuit and the fourth image convolution circuit to obtain a second convolved image. As previously described, the first pooled image is convolved with the third image convolution circuit of the fourth layer (convolution layer C3) and the fourth image convolution circuit of the fifth layer (convolution layer C4) to obtain the second convolved image.

[0091] S400: Perform pooling processing on the second convolution image to obtain a second pooled image. As mentioned above, the second convolution image is pooled by the sixth layer (the second pooling layer S2) to obtain the second pooled image.

[0092] At S500 , the second pooled image is sequentially convolved by the fifth image convolution circuit and the sixth image convolution circuit to obtain a third convolved image. As previously described, the second pooled image is sequentially convolved by the fifth image convolution circuit of the seventh layer (convolution layer C5) and the sixth image convolution circuit of the eighth layer (convolution layer C6) to obtain the third convolved image.

[0093] S600: Performing maximum pooling on the third convolution image to obtain a third pooled image. As described above, the ninth layer (maximum pooling layer GMP) performs global maximum pooling on the third convolution image to obtain a third pooled image.

[0094] S700, perform full connection processing on the third pooled image to obtain the category result of image recognition. As mentioned above, the 27 output nodes of the fully connected layer Dense output corresponding values. By judging which output node has the largest value, it can be judged that the object in the image belongs to the category corresponding to the output node. Specifically, it is necessary to take out the output results of all nodes of the fully connected layer in order from low to high addresses from the storage unit that stores the output results of each node of the fully connected layer. Afterwards, compare the output results in pairs, take out the larger value, and take out the category corresponding to the value for the next round of comparison. This cycle is repeated until the maximum value of all node results and its corresponding category are taken out.

[0095] Among them, the first image convolution circuit, the second image convolution circuit, the third image convolution circuit, the fourth image convolution circuit, the fifth image convolution circuit and the sixth image convolution circuit all adopt the image convolution circuit of the aforementioned embodiment.

[0096] The present invention also provides an image recognition chip, comprising the image convolution circuit.

[0097] The present invention also provides an embedded image recognition device, comprising the image recognition chip.

[0098] Compared with the traditional LeNet-5 convolutional neural network, the convolutional neural network of this embodiment has the following advantages:

[0099] 1) This convolutional neural network reduces the fully connected layer and only retains the output layer, which reduces the number of parameters and reduces the complexity.

[0100] 2) By increasing the network depth, the impact of missing image features caused by the reduction of fully connected layers is reduced, thereby improving the performance of the neural network. Specifically, the reduction of fully connected layers will lead to the loss of image features. By increasing the number of convolutional layers, the number of extracted image features can be increased, thereby compensating for the impact of missing image features caused by the reduction of fully connected layers. In addition, due to the weight sharing characteristics of the convolutional layer, adding convolutional layers will not significantly increase the number of training parameters, and the complexity will not increase much. Furthermore, in some embodiments, the size of the convolution window is changed from 5×5 to 3×3, further reducing the number of parameters.

[0101] 3) This convolutional neural network removes the bias term from the convolution kernel, making it a lightweight convolutional neural network. Removing the bias term does not significantly affect the recognition results of the neural network, but it can effectively reduce the number of parameters.

[0102] 4) Using the ReLU function as the activation function is computationally simple and avoids the aforementioned vanishing gradient problem in the forward direction, resulting in better training and recognition results. Specifically, the existing Sigmoid function and tanh function both suffer from the vanishing gradient problem and contain exponential terms, making the computational process more complex. Compared to these two functions, the ReLU function is computationally simple and avoids the aforementioned flaws in the forward direction, resulting in better training results.

[0103] 5) The pooling layer uses maximum pooling, which has position invariance, so that the relative position of image features in space remains unchanged, which helps to improve the recognition effect.

[0104] In some embodiments, the convolutional neural network achieves real-time recognition speed for handwritten letters. The number of processing cycles for a single letter image is 237,626, and at a working frequency of 50 MHz, the recognition rate is 4.75 ms / frame. The training dataset of more than 148,000 handwritten uppercase letter images enables the convolutional neural network to better adapt to the writing styles of different people. The recognition accuracy of the convolutional neural network in a test dataset containing more than 65,000 images is 94.41%. Compared with the existing solutions for using other neural networks to recognize handwritten letters, this solution reduces the number of parameters by nearly 90%, but the recognition rate only drops by about 2%. Compared with the LeNet-5 model, the network model parameters in this article are reduced by about 90%, and the test accuracy of the model is 94.41%. It can be seen that the convolutional neural network has obvious advantages.

[0105] In addition, in some embodiments, the entire system consumes less logic resources, accounting for 33%, and the total storage resource consumption accounts for 75%, and the static power consumption is about 147mW, which is suitable for implementation in portable mobile systems, and the network scale in the convolutional neural network can be further expanded.

[0106] It will be understood by those skilled in the art that, under the premise of no conflict, the above-mentioned preferred embodiments can be freely combined and superimposed. Among them, the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. The numbering of each step in this article is only for the convenience of description and reference, and is not used to limit the order of execution. The specific execution order is determined by the technology itself, and those skilled in the art can determine various allowable and reasonable orders based on the technology itself.

[0107] It should be noted that the use of step numbers in the present invention to refer to certain specific method steps is solely for the purpose of descriptive convenience and brevity, and is in no way intended to limit the order of these method steps by letters or numbers. Those skilled in the art will appreciate that the order of the relevant method steps is determined by the technology itself and should not be unduly limited by the presence of step numbers. Those skilled in the art can determine various permissible and reasonable step orders based on the technology itself.

[0108] Those skilled in the art will appreciate that, provided there is no conflict, the above preferred solutions can be freely combined and superimposed.

[0109] It should be understood that the above-mentioned embodiments are merely illustrative and non-restrictive. Without departing from the basic principles of the present invention, various obvious or equivalent modifications or substitutions that can be made by those skilled in the art to the above-mentioned details will be included in the scope of the claims of the present invention.

Claims

1. An image convolution circuit, comprising M cache units and a convolution operation circuit, characterized in that: The convolution operation circuit includes a convolution center operation unit and M 2 -1 non-convolution center operation unit, M cache units and the convolution center operation unit and M 2 -1 non-convolution center operation unit is connected in such a way that these operation units form an M*M operation unit matrix; the non-convolution center operation unit connected to the convolution center operation unit is the first non-convolution center operation unit; The convolution center operation unit includes a convolution center multiplier, a convolution center weight register and a center pixel register, wherein a first input end of the convolution center multiplier is connected to the convolution center weight register to input the weight of the convolution kernel center, and a second input end is connected to the center pixel register to input the pixel value of the convolution kernel center pixel point corresponding to the target image; The non-convolution center operation unit includes a non-convolution center multiplier, a non-convolution center weight register, a non-center pixel register, an adder and a gate. The first input end of the non-convolution center multiplier is connected to a non-convolution center weight register to input the weight of the corresponding non-convolution kernel center, and the second input end is connected to the corresponding non-center pixel register to input the pixel value of the non-convolution kernel center pixel point of the target image. The output end of the non-convolution center multiplier is connected to the first input end of the adder, and the output end of the adder is connected to the first input end of the gate. The gate control end of the gate inputs a gate control signal to control the first input end of the gate corresponding to the non-convolution kernel center that coincides with the pixel point in the target image to be gated, and controls the second input ends of the remaining gates to be gated; Each cache unit is used to cache pixel values ​​from a certain row of pixels in the target image, and provide the cached pixel values ​​to each center pixel register or non-center pixel register of the corresponding row of the operation unit matrix; In the first non-convolution center operation unit, the second input end of the adder is connected to the output end of the convolution center multiplier, and the second input end of the gate is connected to the output end of the convolution center multiplier; 2 -2 non-convolution center operation units, the output end of the gate is connected to the second input end of the adder in the next non-convolution center operation unit and the second input end of the gate; 2 The output end of the gate in the -1 non-convolution center operation unit is used to output the image convolution result; wherein M is an odd number greater than or equal to 3.

2. The image convolution circuit according to claim 1, wherein: The central pixel register, the first non-central pixel register to the M-1th non-central pixel register sequentially constitute a shift register group including M registers, The first register to the 1+(M-1) / 2th register sequentially store pixel values ​​from the center pixel point in the target image to the (M-1) / 2th pixel point to the right of the center pixel point, and the second+(M-1) / 2th register to the Mth register store pixel values ​​from the (M-1) / 2th pixel point to the left of the center pixel point in the target image; The corresponding cache unit inputs the pixel value of the pixel point in the row where the center pixel point is located from the 1+(M-1) / 2 register. Each time a pixel value is input: The pixel values ​​stored in the 2nd + (M-1) / 2th register to the M-1th register are respectively moved to the right to the next register, the pixel value in the 1st register is moved to the 2nd + (M-1) / 2th register, and the pixel values ​​stored in the 2nd register to the 1st + (M-1) / 2th register are respectively moved to the left to the previous register, or: The pixel values ​​stored in the 3rd + (M-1) / 2th register to the Mth register are respectively moved to the left to the previous register, the pixel value of the 1st register is moved to the Mth register, and the pixel values ​​stored in the 2nd register to the 1st + (M-1) / 2th register are respectively moved to the left to the previous register.

3. The image convolution circuit according to claim 1, wherein: Also includes M strobe switches, The input end of each strobe switch is connected to the output end of the target image memory, the output end of each strobe switch is connected to the input end of the corresponding cache unit, and the output end of each cache unit is connected to a non-center pixel register of a corresponding row of the operation unit matrix; Each strobe switch further includes a strobe terminal, which selectively controls the strobe switch of the corresponding row to be turned on and controls the strobe switches of the remaining rows to be turned off according to the value of the row where the current pixel point is located output by the target image memory.

4. The image convolution circuit according to claim 3, wherein: When the central pixel point is located at the upper edge row of the target image, (M-1) / 2 corresponding selection switches are controlled to remain off, and at least the second input ends of the selection switches of (M-1) / 2 corresponding rows in the control operation unit matrix are controlled to be selected.

5. The image convolution circuit according to claim 1, wherein: Also includes target image memory, Starting from the pixel point at the upper left corner of the target image, the pixels in the target image are numbered in increasing order from left to right and from top to bottom; Taking the address of the upper left corner pixel of the target image in the target image memory as the starting address, determining the offset address of the corresponding pixel in the target image according to the pixel sequence number, and sequentially storing the pixel values ​​of the pixel in the target image in the target image memory; The position of the pixel point in the target image is determined according to the offset address of the pixel point input into the central pixel register, and the gating control signal is determined according to the position.

6. The image convolution circuit according to claim 1, wherein: The image convolution circuit also includes an activation function circuit, which includes a comparator and an activation gate. The first input end of the comparator inputs the image convolution result output by the convolution operation circuit, the second input end inputs a low level, and the output end is connected to the selection control end of the activation gate; the first input end of the activation gate inputs the image convolution result output by the operation circuit, the second input end inputs a low level, and the output signal of the output end serves as the final image convolution result of the image convolution circuit.

7. An image convolution method, using the image convolution circuit according to any one of claims 1 to 6, comprising the following steps: Each cache unit caches pixel values ​​from a certain row of pixels in the target image, and provides the cached pixel values ​​to each center pixel register or non-center pixel register of the corresponding row of the operation unit matrix; The convolution center multiplier multiplies the weight of the convolution kernel center by the pixel value of the convolution kernel center pixel, and outputs the multiplication result to the second input end of the adder in the first non-convolution center operation unit and the second input end of the gate; The gating control signal of the gating control end of the gating device in each non-convolution center operation unit controls the gating of the first input end of the gating device corresponding to the non-convolution kernel center coinciding with the pixel point in the target image, and controls the gating of the second input ends of the remaining gating devices; 1st to Mth 2 - In two non-convolution central operation units, the gate outputs a gating result to the second input end of the adder in the next non-convolution central operation unit and the second input end of the gate according to the control of the gating control signal; No. M 2 -1 non-convolution center operation unit in the gate output image convolution result.

8. The convolution method according to claim 7, characterized in that The corresponding cache unit inputs the pixel value of the pixel point in the row where the center pixel point is located from the 1+(M-1) / 2 register. Each time a pixel value is input: The pixel values ​​stored in the 2nd + (M-1) / 2th register to the M-1th register are respectively moved to the right to the next register, the pixel value in the 1st register is moved to the 2nd + (M-1) / 2th register, and the pixel values ​​stored in the 2nd register to the 1st + (M-1) / 2th register are respectively moved to the left to the previous register, or: The pixel values ​​stored in the 3rd + (M-1) / 2th register to the Mth register are respectively moved to the left to the previous register, the pixel value of the 1st register is moved to the Mth register, and the pixel values ​​stored in the 2nd register to the 1st + (M-1) / 2th register are respectively moved to the left to the previous register.

9. The convolution method according to claim 7, characterized in that The strobe end of each strobe switch selects to control the strobe switch of the corresponding row to be turned on and controls the strobe switches corresponding to the remaining rows to be turned off according to the value of the row where the current pixel point is located output by the target image memory.

10. The convolution method according to claim 9, characterized in that When the center pixel point is located at the upper edge row of the target image, (M-1) / 2 corresponding selection switches are controlled to remain off, and the second input ends of the non-center pixel registers of (M-1) / 2 corresponding rows in the control operation unit matrix are controlled to be selected.

11. The convolution method according to claim 7, wherein: The position of the pixel point in the target image is determined according to the offset address of the pixel point input into the central pixel register, and the gating control signal is determined according to the position.

12. The convolution method according to claim 7, characterized in that When the image convolution result is greater than 0, the first input terminal of the activation gate is gated, and the final image convolution result is the same as the image convolution result; When the image convolution result is less than 0, the second input terminal of the activation gate is gated, and the final image convolution result is 0.

13. An image recognition method, characterized in that: The steps include: S100, sequentially subjecting a target image to convolution processing using a plurality of first image convolution circuits and a plurality of second image convolution circuits to obtain a first convolution image; S200, performing pooling processing on the first convolution image to obtain a first pooled image; S300, sequentially performing convolution processing on the first pooled image through a plurality of third image convolution circuits and a plurality of fourth image convolution circuits to obtain a second convolution image; S400, performing pooling processing on the second convolution image to obtain a second pooled image; S500, sequentially performing convolution processing on the second pooled image through a plurality of fifth image convolution circuits and a plurality of sixth image convolution circuits to obtain a third convolution image; S600, performing maximum pooling processing on the third convolution image to obtain a third pooled image; S700, performing full connection processing on the third pooled image to obtain a category result of image recognition; Among them, the first image convolution circuit, the second image convolution circuit, the third image convolution circuit, the fourth image convolution circuit, the fifth image convolution circuit and the sixth image convolution circuit all adopt the image convolution circuit as described in any one of claims 1-6.

14. The image recognition method according to claim 13, wherein: The pooling process in steps S200 and S400 is a maximum pooling process.

15. The image recognition method according to claim 13, wherein: The image recognition method is applied to handwritten letter recognition.

16. An image recognition device, characterized in that: include: a first convolution unit, configured to sequentially convolve a target image through a plurality of first image convolution circuits and a plurality of second image convolution circuits to obtain a first convolution image; A first pooling unit, configured to perform pooling processing on the first convolution image to obtain a first pooled image; a second convolution unit, configured to sequentially perform convolution processing on the first pooled image through a plurality of third image convolution circuits and a plurality of fourth image convolution circuits to obtain a second convolution image; a second pooling unit, configured to perform pooling processing on the second convolution image to obtain a second pooled image; a third convolution unit, configured to sequentially perform convolution processing on the second pooled image through a plurality of fifth image convolution circuits and a plurality of sixth image convolution circuits to obtain a third convolution image; a third pooling unit, configured to perform maximum pooling on the third convolution image to obtain a third pooled image; a fully connected unit, configured to perform fully connected processing on the third pooled image to obtain a category result of image recognition; Among them, the first image convolution circuit, the second image convolution circuit, the third image convolution circuit, the fourth image convolution circuit, the fifth image convolution circuit and the sixth image convolution circuit all adopt the image convolution circuit as described in any one of claims 1-6.

17. An image recognition chip, characterized in that: The method comprises the image convolution circuit as described in any one of claims 1 to 6.

18. An embedded image recognition device, characterized in that: Comprising the image recognition chip as described in claim 17.

Images