Image processing unit, image processing method and storage medium

Abstract

The embodiment of the present application provides an image processing unit, an image processing method and a storage medium. The image processing unit includes: a fixed-point circuit and a plurality of neural network blocks cascaded in sequence; the fixed-point circuit is electrically connected to each neural network block, and is used to receive a feature signal corresponding to an output feature image output by each neural network block, perform fixed-point processing on the feature signal to obtain fixed-point data that meets the design accuracy range, and input the fixed-point data as an input feature image to the next level neural network block. The embodiment of the present application can avoid overflow of fixed-point processing and improve calculation accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology. Specifically, the present application relates to an image processing unit, an image processing method and a storage medium. Background Art

[0002] At present, the CNN algorithm (convolutional neural network algorithm) is deployed in hardware. Before the algorithm is deployed, the algorithm model building, parameter training, and parameter optimization must be performed on the server. The data types calculated in the server-side CNN algorithm are all float types (floating point types). Then the trained weight parameters and input feature maps are processed in hardware. The chip for hardware deployment uses FPGA (Field Programmable Gate Array). FPGA is called field programmable logic array and is suitable for parallel computing of large amounts of data. When deploying the hardware, the floating-point data type needs to be converted into a fixed-point data type to meet the real-time working requirements of the hardware.

[0003] The steps of the traditional method for fixed-point processing are as follows: calculate the maximum and minimum values ​​of all layers of the convolutional neural network (mainly including multiple convolutional layers and pooling layers), and then perform fixed-point processing on them. This method is prone to overflow problems in fixed-point processing, which reduces the calculation accuracy. Summary of the invention

[0004] In view of the shortcomings of the existing methods, the present application proposes an image processing unit, an image processing method and a storage medium to solve the technical problems of fixed-point processing overflow or reduced calculation accuracy caused by the use of convolutional neural networks in the existing technology.

[0005] In a first aspect, an embodiment of the present application provides an image processing unit, comprising: a fixed-point circuit and a plurality of neural network blocks cascaded in sequence;

[0006] The fixed-point circuit is electrically connected to each neural network block, and is used to receive characteristic signals corresponding to the output characteristic images output by each neural network block, perform fixed-point processing on the characteristic signals to obtain fixed-point data that meets the design accuracy range, and input the fixed-point data as the input characteristic image corresponding to the next-level neural network block.

[0007] In a possible implementation, the first input terminal and the second input terminal of the fixed-point circuit are used to receive the convolution layer indication signal and the characteristic signal respectively;

[0008] The fixed-point circuit is used to determine the corresponding neural network block according to the convolution layer indication signal, determine the design fixed-point range of the corresponding neural network block according to the design fixed-point range of each neural network block, and perform fixed-point processing on the characteristic signal according to the design fixed-point range of the corresponding neural network block to obtain fixed-point data that meets the design accuracy range.

[0009] In a possible implementation, the fixed-point circuit includes a first trigger module and a first gating module;

[0010] The first input terminal of the first trigger module is used as the first input terminal of the fixed-point circuit;

[0011] The second input terminal of the first trigger module is used to receive the first clock signal;

[0012] The first input terminal of the first gating module is electrically connected to the output terminal of the first triggering module;

[0013] The second input terminal of the first gating module is used as the third input terminal of the fixed-point circuit;

[0014] The third input terminals of the first gating modules are all grounded;

[0015] The first selection module is used to determine the corresponding neural network block according to the received convolution layer indication signal, determine the design fixed-point range of the corresponding neural network block according to the pre-stored design fixed-point range of the neural network block, and perform fixed-point processing on the feature signal according to the design fixed-point range of the corresponding neural network block to obtain fixed-point data that meets the design accuracy range.

[0016] In a possible implementation, the fixed-point circuit further includes a second trigger module;

[0017] A first input terminal of the second trigger module is electrically connected to an output terminal of the first gating module;

[0018] The second input terminal of the second trigger module is electrically connected to the second input terminal of the first trigger module;

[0019] The output end of the second trigger module is used to output fixed-point data.

[0020] In one possible implementation, the neural network block includes: a convolution operation module, a pooling module, and an activation module electrically connected in sequence;

[0021] The convolution operation module is electrically connected to the fixed-point circuit, and is used to receive an input feature image corresponding to the fixed-point data output by the fixed-point circuit, and perform convolution processing on the input feature image to obtain a first image to be processed;

[0022] A pooling module, used for performing pooling processing on the first image to be processed to obtain a second image to be processed;

[0023] The activation module is used to perform activation processing on the second image to be processed to obtain an output feature image.

[0024] In one possible implementation, the pooling module includes multiple stages of comparison modules cascaded in sequence;

[0025] The number of sub-comparison modules of the upper-level comparison module is a design multiple of the number of sub-comparison modules of the lower-level comparison module;

[0026] The input end of the sub-comparison module of the first-level comparison module is used to receive the first image to be processed;

[0027] The input terminals of the sub-comparison modules of the next-level comparison module are electrically connected to the output terminals of the sub-comparison modules of the previous-level comparison module;

[0028] A multi-level comparison module is used to compare the characteristic matrices in the input first image to be processed in turn, select the maximum value result, and obtain the second image to be processed;

[0029] The output terminal of the last stage comparison module is electrically connected to the input terminal of the activation module.

[0030] In a possible implementation, the sub-comparison module includes a comparator, a second gate, and a third trigger;

[0031] The input terminal of the comparator and the first input terminal of the second selector are used together as the input terminal of the sub-comparison module;

[0032] The second input terminal of the second selector is electrically connected to the output terminal of the comparator, and is used to receive the comparison result of the comparator;

[0033] The output terminal of the second selector is electrically connected to the first input terminal of a third trigger;

[0034] The second input terminal of the third trigger is used to receive the second clock signal;

[0035] The output end of the third trigger is used as the output end of the sub-comparison module.

[0036] In one possible implementation, the activation module includes a fourth trigger;

[0037] The first input terminal and the reset terminal of the fourth trigger are both electrically connected to the output terminal of the last-stage comparison module, and are used to activate each value of the second image to be processed to obtain an output feature image;

[0038] The second input terminal of the fourth trigger is used to receive the second clock signal;

[0039] The output end of the fourth trigger is used to output the output feature image.

[0040] In a second aspect, the embodiment of the present application further provides an image processing method, including:

[0041] Receiving a convolution layer indication signal;

[0042] Determine the corresponding neural network block according to the convolution layer indication signal, and determine the design fixed-point range of the corresponding neural network block according to the pre-stored design fixed-point range of the neural network block;

[0043] According to the design fixed-point range of the corresponding neural network block, the characteristic signal is processed in a fixed-point manner to obtain fixed-point data that meets the design accuracy range.

[0044] In a possible implementation, the feature signal is processed in a fixed-point manner according to the design fixed-point range of the corresponding neural network block to obtain fixed-point data that meets the design accuracy range, including:

[0045] Determine whether the difference between the design fixed-point range of the current corresponding neural network block and the design fixed-point range corresponding to the previous level neural network block is greater than a predetermined difference threshold;

[0046] If it is not greater than the predetermined difference threshold, the design fixed-point range corresponding to the previous level neural network block is used as the design fixed-point range of the current corresponding neural network block.

[0047] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed on a computer, the image processing method of the second aspect is implemented.

[0048] The beneficial technical effects brought about by the technical solution provided by the embodiment of the present application include:

[0049] The fixed-point circuit of the embodiment of the present application is electrically connected to each neural network block, and can perform fixed-point processing on the output feature image output by the neural network block of each layer to obtain fixed-point data that meets the design accuracy range. Therefore, the fixed-point circuit of the embodiment of the present application processes the output feature image of the neural network block of each layer, and can determine the fixed-point range according to the range of the maximum and minimum values ​​of the output feature image of the neural network block of each layer, thereby avoiding overflow of the fixed-point processing and improving the calculation accuracy.

[0050] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0052] Figure 1 A schematic diagram of a framework of an image processing unit provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a framework of an electrification circuit provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of an electrification circuit provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of a framework of a neural network block provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of the electrical connection between a pooling module and an activation module provided in an embodiment of the present application;

[0057] Figure 6 A flowchart of an image processing method provided in an embodiment of the present application;

[0058] Figure 7 A flowchart of another image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The present application is described in detail below, and examples of embodiments of the present application are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar components or components with the same or similar functions. In addition, if the detailed description of the known technology is unnecessary for the features of the present application shown, it will be omitted. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0060] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0061] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0062] The inventor of the present application has conducted research and found that the steps of the traditional method for fixed-point processing are as follows: calculate the maximum and minimum values ​​of all layers of the convolutional neural network (mainly including multiple convolutional layers and pooling layers), and then uniformly perform fixed-point processing. The traditional method is prone to overflow problems in fixed-point processing, especially when the maximum or minimum value difference between layers is large. Assuming that the range of convolutional layer 1 is (-1000, 1000) and the range of convolutional layer 2 is (-1, 1), according to the unified fixed-point processing of the traditional method, in order to ensure that (-1000, 1000) does not overflow, the convolution layer (-1, 1) will have a partial loss of accuracy. Therefore, there is an urgent need for an image processing method that can avoid the overflow problem of fixed-point processing and improve the calculation accuracy.

[0063] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0064] The present application embodiment provides an image processing unit, see Figure 1 The image processing unit includes: a fixed-point circuit 100 and a plurality of neural network blocks 200 cascaded in sequence.

[0065] The fixed-point circuit 100 is electrically connected to each neural network block 200, and is used to receive a characteristic signal corresponding to an output characteristic image output by each neural network block 200, perform fixed-point processing on the characteristic signal to obtain fixed-point data that meets the design accuracy range, and input the fixed-point data as an input characteristic image corresponding to the next-level neural network block 200.

[0066] The inventor of the present application further considered that since the layers in the convolutional neural network are independent of each other, if each convolutional layer calculates the maximum and minimum values ​​respectively, and each value range is obtained and then fixed-point processing is performed respectively. When the maximum or minimum value difference between the layers in the convolutional neural network is large, the error caused by the fixed-point processing can be effectively reduced, thereby improving the calculation accuracy.

[0067] For example: the maximum or minimum value difference between the convolutional layer Convolutional layer1 and the convolutional layer Convolutional layer3 is large, and the range value of the convolutional layer Convolutional layer3 is much larger than that of Convolutional layer1. The traditional method uniformly performs fixed-point processing and uses the largest range of convolutional layer Convolutional layer3 as the standard for fixed-point processing. At this time, for the convolutional layer Convolutional layer1, the fixed-point standard is too large, resulting in reduced accuracy, and the details in the feature map may not be distinguished, thereby affecting the output result. The image processing unit of the embodiment of the present application is used to determine the fixed-point processing range of the convolutional layers Convolutional layer1 and Convolutional layer3 using the maximum and minimum values ​​of their respective layers. Even if the difference is large, higher accuracy can be obtained than the traditional method.

[0068] Based on the above analysis, the fixed-point circuit 100 of the embodiment of the present application is electrically connected to each neural network block 200, and can perform fixed-point processing on the output feature image output by the neural network block 200 of each layer to obtain fixed-point data that meets the design accuracy range. Therefore, the fixed-point circuit 100 of the embodiment of the present application processes the output feature image of the neural network block 200 of each layer, and can determine the fixed-point range according to the range of the maximum and minimum values ​​of the output feature image of the neural network block 200 of each layer, thereby avoiding overflow of the fixed-point processing and improving the calculation accuracy.

[0069] In some embodiments, the first input terminal and the second input terminal of the fixed-point circuit 100 are used to receive the convolution layer indication signal and the characteristic signal respectively;

[0070] The fixed-point circuit 100 is used to determine the corresponding neural network block 200 according to the convolution layer indication signal, determine the design fixed-point range of the corresponding neural network block 200 according to the design fixed-point range of each neural network block 200, and perform fixed-point processing on the characteristic signal according to the design fixed-point range of the corresponding neural network block 200 to obtain fixed-point data that meets the design accuracy range.

[0071] Optionally, the image processing unit of the embodiment of the present application may be applicable to fixed-point processing based on heart rate detection.

[0072] Optionally, the design fixed-point range of the neural network block 200 is determined according to the range of the maximum and minimum values ​​of the feature image output by the neural network block 200 of each layer.

[0073] Optionally, the fixed-point processing is to multiply the data by a design multiple according to a design fixed-point range, and make the data not exceed the design accuracy range, so as to prevent overflow of the fixed-point processing and improve the calculation accuracy.

[0074] In some embodiments, see Figure 2 and Figure 3 As shown, the fixed-point circuit 100 includes a first trigger module 110 and a first gating module 120 .

[0075] The first input terminal of the first trigger module 110 is used as the first input terminal of the fixed-point circuit 100;

[0076] The second input terminal of the first trigger module 110 is used to receive the first clock signal;

[0077] The first input terminal of the first gating module 120 is electrically connected to the output terminal of the first triggering module 110;

[0078] The second input terminal of the first gating module 120 is used as the second input terminal of the fixed-point circuit 100;

[0079] The third input terminals of the first gating module 120 are all grounded;

[0080] The first selection module 120 is used to determine the corresponding neural network block 200 according to the received convolutional layer indication signal, determine the design fixed-point range of the corresponding neural network block 200 according to the pre-stored design fixed-point range of the neural network block 200, and perform fixed-point processing on the characteristic signal according to the design fixed-point range of the corresponding neural network block 200 to obtain fixed-point data that meets the design accuracy range.

[0081] Alternatively, see Figure 2 and Figure 3 As shown, the first trigger module 110 includes a first trigger, and the first gating module 120 includes at least one first gate.

[0082] The first input terminal, the second input terminal, and the output terminal of the first trigger serve as the first input terminal, the second input terminal, and the output terminal of the first trigger module 110 , respectively.

[0083] The output end of each first trigger is electrically connected to the first input end of a first selector; the second input end of each first selector serves as the second input end of the fixed-point circuit; the third input end of each first selector is electrically connected and grounded; the output end of each first trigger is used to output the data after fixed-point processing.

[0084] Optionally, the first selection module 120 is also used to determine whether the difference between the design fixed-point range of the current corresponding neural network block 200 and the design fixed-point range corresponding to the previous level neural network block 200 is greater than a predetermined difference threshold; if it is not greater than the predetermined difference threshold, the design fixed-point range corresponding to the previous level neural network block 200 is used as the design fixed-point range of the current corresponding neural network block 200.

[0085] Optionally, the function of the first selection module 120 can be directly implemented through the first selection device, or the first selection module 120 can include a control circuit electrically connected to the first selection device, and the fixed-point processing of the first selection device is controlled by the control circuit. The control circuit is used to receive the convolution layer indication signal to determine the corresponding neural network block 200, determine the design fixed-point range of the corresponding neural network block 200 according to the pre-stored design fixed-point range of the neural network block 200, and output the design fixed-point range of the corresponding neural network block 200 to the first selection device, so that the first selection device performs fixed-point processing according to the design fixed-point range.

[0086] In some embodiments, see Figure 2 and Figure 3 As shown, the fixed-point circuit 100 further includes a second trigger module 130 .

[0087] The first input terminal of the second trigger module 130 is electrically connected to the output terminal of the first gating module 120;

[0088] The second input terminal of the second trigger module 130 is electrically connected to the second input terminal of the first trigger module 110;

[0089] The output terminal of the second trigger module 130 is used to output fixed-point data.

[0090] Alternatively, see Figure 2 and Figure 3 As shown, the second trigger module 130 includes at least one second trigger. The first input terminal of each second trigger is electrically connected to the output terminal of a first selector, and the second input terminal of each second trigger is electrically connected to the second input terminal of the first trigger; the output terminal of each second trigger is used to output fixed-point data.

[0091] Optionally, each trigger can be used for input buffering.

[0092] As an example, see Figure 3 As shown, the first trigger module 110 includes a trigger RTL_REG, which serves as a first trigger; the first selection module 120 includes four selectors RTL_MUX (corresponding to r_fm1_dout_temp_i, r_fm2_dout_temp_i, r_fm3_dout_temp_i, and r_fm4_dout_temp_i, respectively), and the four selectors RTL_MUX serve as four first selectors respectively; the second trigger module 130 includes four triggers RTL_REG, which serve as second triggers.

[0093] The four first selectors are equivalent to four channels respectively, corresponding to different characteristic signals of received data information.

[0094] The first clock signal input to the second input end of the first trigger is the system working main clock i_clk_100m, with a frequency of 100 MHz (megahertz); the i_conv_layer_num input to the first input end of the first trigger is the convolution layer indication signal, with a bit width of 6 bits (bits); the four first selectors i_fm1_dout_temp, i_fm2_dout_temp, i_fm3_dout_temp, and i_fm4_dout_temp represent the feature signals corresponding to the input feature images of the convolution layer. The number of input channels is 4, and the bit width of each channel is 38 bits.

[0095] Alternatively, see Figure 3 As shown, the characteristic signals of the four first selector input terminals are i_fm1_dout_temp[37:0], i_fm2_dout_temp[37:0], i_fm3_dout_temp[37:0], and i_fm4_dout_temp[37:0], respectively. The second input terminal of the first first selector has nine second sub-input terminals for receiving the characteristic signal i_fm1_dout_temp[37:0]; the second input terminals of the second to fourth first selectors have eight second sub-input terminals for receiving the characteristic signal Numbers are i_fm2_dout_temp[37:0], i_fm3_dout_temp[37:0], i_fm4_dout_temp[37:0]; the third input terminal of the first first selector has one, and the third input terminals of the second to fourth first selectors have two; the fixed-point data output by the four second triggers are r_fm1_dout_temp, r_fm2_dout_temp, r_fm3_dout_temp, r_fm4_dout_temp, respectively, with a bit width of 21 bits.

[0096] Optionally, based on Figure 3In the fixed-point circuit shown, the convolution layer indication signal i_conv_layer_num is connected to four first selectors respectively after passing through the input cache of the first trigger. The four first selectors select the fixed-point range according to the current neural network block 200 and output the processed data. The four second trigger output caches are responsible for caching the output fixed-point data for use by the lower-level circuits.

[0097] For example, the convolution layer indication signal indicates 3, which corresponds to the third neural network block 200. At this time, the fixed-point processing is performed according to the design fixed-point range of the third neural network block 200. For example, the feature signals i_fm1_dout_temp, i_fm2_dout_temp, i_fm3_dout_temp, and i_fm4_dout_temp corresponding to the input feature image are right-shifted by 10 bits, that is, multiplied by 2. 10 , by enlarging the accuracy, the corresponding fixed-point data that meets the design accuracy range can be obtained.

[0098] For example, the convolution layer indication signal indicates 7, which corresponds to the third neural network block 200. At this time, the fixed-point processing is performed according to the design fixed-point range of the seventh neural network block 200. For example, the feature signals i_fm1_dout_temp, i_fm2_dout_temp, i_fm3_dout_temp, and i_fm4_dout_temp corresponding to the input feature image are right-shifted by 12 bits, that is, multiplied by 2. 12 , by enlarging the accuracy, the corresponding fixed-point data that meets the design accuracy range can be obtained.

[0099] In some embodiments, see Figure 4 As shown, the neural network block 200 includes: a convolution operation module 210, a pooling module 220 and an activation module 230 which are electrically connected in sequence.

[0100] The convolution operation module 210 is electrically connected to the fixed-point circuit 100, and is used to receive an input feature image corresponding to the fixed-point data output by the fixed-point circuit 100, and perform convolution processing on the input feature image to obtain a first image to be processed.

[0101] The pooling module 220 is used to perform pooling processing on the first image to be processed to obtain a second image to be processed;

[0102] The activation module 230 is used to perform activation processing on the second image to be processed to obtain an output feature image.

[0103] The inventors of this application further discovered that fixed-point processing acts on the input and output feature maps of each computing layer, and deep learning algorithms require deeper and larger models to achieve higher precision and accuracy, which will lead to problems such as large number of parameters, high storage space occupancy, and complex calculations. For embedded devices, the hardware configuration cannot meet the large-scale computing requirements of deep learning.

[0104] Model compression refers to the reconstruction, simplification and acceleration technology of deep networks used in deep learning. After the network is trained, there is a lot of redundancy in the weight parameters. The redundant parameters are unimportant and can be deleted. Removing these parameters does not affect the accuracy of the network. At the same time, the parameters are reduced, the network model is simplified and the calculation speed is greatly improved, thereby improving the overall performance.

[0105] The relationship between the convolutional layer and the pooling layer of the traditional deep learning neural network model is that after the input feature image enters the convolutional layer, the convolution operation module 210 completes the extraction of feature parameters, and the activation function enters the pooling layer after completing the nonlinear mapping, and the feature map is output by the maximum pooling algorithm. Such a processing process will lead to the problem of hardware storage space occupancy.

[0106] The embodiment of the present application separates the activation module 230 from the convolution layer and places it at the output end of the pooling module 220. While achieving the same calculation accuracy as the traditional framework, it reduces the storage occupancy and calculation amount of the activation function, thereby achieving the effect of reducing storage space and increasing computing speed.

[0107] Optionally, the pooling module 220 of the embodiment of the present application may adopt a maximum pooling layer max pooling method, select a maximum pooling algorithm, and the pooling module 220 does not need fixed-point processing.

[0108] In some embodiments, see Figure 4 and Figure 5 As shown, the pooling module 220 includes multiple stages of comparison modules 221 that are cascaded in sequence.

[0109] The number of the sub-comparison modules 2211 of the upper-level comparison module 221 is a designed multiple of the number of the sub-comparison modules 2211 of the lower-level comparison module 221;

[0110] The input end of the sub-comparison module 2211 of the first-level comparison module 221 is used to receive the first image to be processed;

[0111] The input terminals of the sub-comparison modules 2211 of the next-level comparison module 221 are electrically connected to the output terminals of the sub-comparison modules 2211 of the previous-level comparison module 221;

[0112] A multi-level comparison module 221 is used to compare the characteristic matrices in the input first image to be processed in sequence, select the maximum value result, and obtain the second image to be processed;

[0113] The output terminal of the last-stage comparison module 221 is electrically connected to the input terminal of the activation module 230 .

[0114] In some embodiments, see Figure 4 and Figure 5 As shown, the sub-comparison module 2211 includes a comparator, a second gate and a third trigger;

[0115] The input terminal of the comparator and the first input terminal of the second selector are used together as the input terminal of the sub-comparison module 2211;

[0116] The second input terminal of the second selector is electrically connected to the output terminal of the comparator, and is used to receive the comparison result of the comparator;

[0117] The output terminal of the second selector is electrically connected to the first input terminal of a third trigger;

[0118] The second input terminal of the third trigger is used to receive the second clock signal;

[0119] The output end of the third trigger is used as the output end of the sub-comparison module 2211. The output end of the third trigger of the last-stage sub-comparison module is used as the output end of the last-stage comparison module.

[0120] Optionally, the main function of the pooling module 220 during model testing is to reduce parameters, dimensions and computational complexity. The convolution kernel size can be selected as 2*2, with a step of 2, to obtain each feature matrix in each first image to be processed, and select 1 maximum value from the pixel values ​​in each feature matrix. That is, select 1 maximum value from the 4 pixel values ​​of 2*2, and reduce the 4 pixels to 1 pixel. The storage occupancy rate of the activation module 230 is reduced to 1 / 4, and the computational complexity of the activation function is reduced to 1 / 4, while achieving the same computational accuracy as the traditional framework, thereby achieving the effect of reducing storage space and increasing computing speed.

[0121] In some embodiments, see Figure 4 and Figure 5 As shown, the activation module 230 includes a fourth trigger;

[0122] The first input terminal and the reset terminal of the fourth trigger are both electrically connected to the output terminal of the last-stage comparison module 221, and are used to activate each value of the second image to be processed to obtain an output feature image;

[0123] The second input terminal of the fourth trigger is used to receive the second clock signal;

[0124] The output end of the fourth trigger is used to output the output feature image.

[0125] As an example, see Figure 5 As shown, the pooling module 220 includes two cascaded comparison modules 221, and the number of sub-comparison modules 2211 of the upper comparison module 221 is twice the number of sub-comparison modules 2211 of the lower comparison module 221. The pooling module 220 and the activation module 230 include three comparators RTL_GEQ, three second selectors RTL_MUX, three third triggers and one trigger RTL_REG (e.g., D-flip-flop).

[0126] Specifically, the input end of the comparator includes two sub-input ends, the first input end of the second selector includes two sub-input ends, and the sub-input ends of the two comparators of the first-level sub-comparison module 2211 respectively input r_feature_map, r_feature_map0, r_feature_map1, and r_feature_map2, with a bit width of 8 bits. After the comparison processing of the two-level comparison module 221, the fourth trigger of the activation module 230 is activated and the output result o_dout is 8 bits.

[0127] Specifically, in the first-level comparison module 221, the comparator r_dout01_i compares the values ​​of r_feature_map and r_feature_map0, the second selector r_dout0_i outputs the maximum value of the two, and the third trigger stores and outputs the maximum value result r_dout0_reg. Similarly, the comparator r_dout11_i compares the values ​​of r_feature_map1 and r_feature_map2, the second selector r_dout1_i outputs the maximum value of the two, and the third trigger stores and outputs the maximum value result r_dout1_reg. In the second-level comparison module 221, the comparator r_dout1_i_0 completes the comparison of the maximum value, the second selector r_dout_i outputs the maximum value, that is, the result after pooling, and the third trigger stores and outputs r_dout_reg.

[0128] Optionally, the fourth trigger implements the activation function and performs activation processing. The fourth trigger can be a D trigger. The fourth trigger determines whether the current data is positive or negative according to r_dout_reg[7]. r_dout_reg[7] is connected to the reset terminal (RST pin) of the fourth trigger. When r_dout_reg[7]=1, it represents a negative number, and o_dout output is 0. When r_dout_reg[7]=0, it represents a positive number, and o_dout output is o_dout_reg.

[0129] Based on the above technical solution, i is the simulation time indication value, the second clock signal is the system working clock i_clk, the input is r_feature_map, r_feature_map0, r_feature_map1, r_feature_map2, and the output is o_dout. For example: when i=129, the convolution operation module 210 outputs hexadecimal data 0x7a, 0x7b, 0x7c, 0x7d, all of which are positive numbers, and the o_dout output is 0x7d; when i=132, the convolution operation module 210 outputs are all negative numbers, and the o_dout output is 0.

[0130] Optionally, based on the above technical solution, see Figure 6 As shown, a flowchart of the graphics processing method of the present application is provided.

[0131] Step 1: The fixed-point circuit 100 performs a first fixed-point processing on the original data according to the designed fixed-point range of the first neural network block 200 (corresponding to CNN block1) to obtain the first fixed-point data.

[0132] Step 2: Input the first fixed-point data in step 1 as the input feature image into the first neural network block 200. The first neural network block 200 processes the input feature image in sequence through the convolution operation module 210 (corresponding to Convolutional layer 1), the pooling module 220 (corresponding to Pooling layer 2) and the activation module 230 (corresponding to the activation function) to obtain the first output feature image. The feature signal corresponding to the first output feature image is input into the fixed-point circuit 100 again.

[0133] Step 3: The fixed-point circuit 100 performs a second fixed-point processing on the feature signal in step 2 according to the designed fixed-point range of the second neural network block 200 (corresponding to CNN block2) to obtain second fixed-point data.

[0134] Step 4: Input the second fixed-point data in step 3 as the input feature image into the second neural network block 200. The second neural network block 200 processes the input feature image in sequence through the convolution operation module 210 (corresponding to Convolutional layer 3), the pooling module 220 (corresponding to Pooling layer 4) and the activation module 230 (corresponding to the activation function) to obtain a second output feature image. The feature signal corresponding to the second output feature image continues to be input into the fixed-point circuit 100.

[0135] Step 5: The fixed-point circuit 100 performs a third fixed-point processing on the feature signal of step 4 according to the designed fixed-point range of the third neural network block 200 (corresponding to CNN block3) to obtain the third fixed-point data.

[0136] Step six: Input the third fixed-point data in step five as the input feature image into the third neural network block 200. The third neural network block 200 processes the input feature image in sequence through the convolution operation module 210 (corresponding to Convolutional layer 5), the pooling module 220 (corresponding to Pooling layer 6) and the activation module 230 (corresponding to the activation function) to obtain a third output feature image. The feature signal corresponding to the third output feature image continues to be input into the fixed-point circuit 100.

[0137] The process is repeated in this way until the last neural network block 200 outputs an output feature image.

[0138] Based on the same inventive concept, the present application also provides an image processing method, see Figure 7 As shown, the image processing method includes steps S701-S703.

[0139] S701. Receive a convolutional layer indication signal.

[0140] Optionally, the first gating module 120 receives a convolutional layer indication signal.

[0141] S702. Determine the corresponding neural network block 200 according to the convolution layer indication signal, and determine the design fixed-point range of the corresponding neural network block 200 according to the pre-stored design fixed-point range of the neural network block 200.

[0142] Optionally, the first selection module 120 determines the corresponding neural network block 200 according to the convolution layer indication signal, and determines the design fixed-point range of the corresponding neural network block 200 according to the pre-stored design fixed-point range of the neural network block 200.

[0143] S703. According to the design fixed-point range of the corresponding neural network block 200, the characteristic signal is processed in a fixed-point manner to obtain fixed-point data that meets the design accuracy range.

[0144] Optionally, the first gating module 120 performs fixed-point processing on the characteristic signal according to the design fixed-point range of the corresponding neural network block 200 to obtain fixed-point data that meets the design accuracy range.

[0145] In some embodiments, in step S703, the feature signal is processed in a fixed-point manner according to the design fixed-point range of the corresponding neural network block 200 to obtain fixed-point data that meets the design accuracy range, including:

[0146] Determine whether the difference between the design fixed-point range of the current corresponding neural network block 200 and the design fixed-point range corresponding to the previous neural network block 200 is greater than a predetermined difference threshold;

[0147] If it is not greater than the predetermined difference threshold, the design fixed-point range corresponding to the previous level neural network block 200 is used as the design fixed-point range of the current corresponding neural network block 200.

[0148] Optionally, for example, there are 9 neural network blocks 200 in the embodiment of the present application. According to the pre-stored design fixed-point range of the neural network block 200, if the difference between the design fixed-point ranges of the adjacent second, third and fourth neural network blocks 200 is not greater than the difference threshold, then the design fixed-point ranges of the second, third and fourth neural network blocks 200 are similar, and the same fixed-point range can be used for fixed-point processing; if the difference between the design fixed-point ranges of the adjacent seventh and eighth neural network blocks 200 is not greater than the difference threshold, then the design fixed-point ranges of the seventh and eighth neural network blocks 200 are similar, and the same fixed-point range can be used for fixed-point processing; the fixed-point ranges of other neural network blocks 200 are not similar and can be fixed-point processed individually.

[0149] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed on a computer, the image processing method of any embodiment of the present application is implemented.

[0150] The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM, RAM, EPROM (Erasable Programmable Read-Only Memory), EEPROM, flash memory, magnetic card or optical card. In other words, the readable medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0151] The embodiment of the present application provides a computer-readable storage medium applicable to any of the above-mentioned image processing methods, which will not be described in detail here.

[0152] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0153] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0154] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0155] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image processing unit, characterized in that: include: Fixed-point circuits and multiple neural network blocks cascaded in sequence; The fixed-point circuit is electrically connected to each of the neural network blocks, and is used to receive a characteristic signal corresponding to an output characteristic image output by each of the neural network blocks, perform fixed-point processing on the characteristic signal to obtain fixed-point data that meets the design accuracy range, and input the fixed-point data as an input characteristic image corresponding to the neural network block of the next level; The first input terminal and the second input terminal of the fixed-point circuit are used to receive the convolution layer indication signal and the characteristic signal respectively; The fixed-point circuit is used to determine the corresponding neural network block according to the convolution layer indication signal, determine the design fixed-point range of the corresponding neural network block according to the design fixed-point range of each neural network block, and perform fixed-point processing on the characteristic signal according to the design fixed-point range of the corresponding neural network block to obtain fixed-point data that meets the design accuracy range; The fixed-point circuit includes a first trigger module and a first gating module; The first input terminal of the first trigger module is used as the first input terminal of the fixed-point circuit; The second input terminal of the first trigger module is used to receive a first clock signal; The first input terminal of the first gating module is electrically connected to the output terminal of the first triggering module; The second input terminal of the first gating module is used as the second input terminal of the fixed-point circuit; The third input terminals of the first gating modules are all grounded; The first selection module is used to determine the corresponding neural network block according to the received convolution layer indication signal, determine the design fixed-point range of the corresponding neural network block according to the pre-stored design fixed-point range of the neural network block, and perform fixed-point processing on the characteristic signal according to the design fixed-point range of the corresponding neural network block to obtain fixed-point data that meets the design accuracy range.

2. The image processing unit according to claim 1, characterized in that The fixed-point circuit also includes a second trigger module; The first input terminal of the second trigger module is electrically connected to the output terminal of the first gating module; The second input terminal of the second trigger module is electrically connected to the second input terminal of the first trigger module; The output end of the second trigger module is used to output the fixed-point data.

3. The image processing unit according to claim 1, characterized in that The neural network block includes: a convolution operation module, a pooling module and an activation module electrically connected in sequence; The convolution operation module is electrically connected to the fixed-point circuit, and is used to receive an input feature image corresponding to the fixed-point data output by the fixed-point circuit, and perform convolution processing on the input feature image to obtain a first image to be processed; The pooling module is used to perform pooling processing on the first image to be processed to obtain a second image to be processed; The activation module is used to perform activation processing on the second image to be processed to obtain the output feature image.

4. The image processing unit according to claim 3, characterized in that: The pooling module includes multiple levels of comparison modules cascaded in sequence; The number of sub-comparison modules of the comparison module of the previous level is a design multiple of the number of sub-comparison modules of the comparison module of the next level; The input end of the sub-comparison module of the first-level comparison module is used to receive the first image to be processed; The input terminals of the sub-comparison modules of the next-level comparison module are electrically connected to the output terminals of the sub-comparison modules of the previous-level comparison module; The multi-level comparison module is used to compare the feature matrices in the first image to be processed in turn, select the maximum value result, and obtain the second image to be processed; The output terminal of the last stage comparison module is electrically connected to the input terminal of the activation module.

5. The image processing unit according to claim 4, characterized in that: The sub-comparison module includes a comparator, a second gate and a third trigger; The input end of the comparator and the first input end of the second selector are used together as the input end of the sub-comparison module; The second input terminal of the second selector is electrically connected to the output terminal of the comparator, and is used to receive the comparison result of the comparator; The output terminal of the second gate is electrically connected to the first input terminal of a third trigger; The second input terminal of the third trigger is used to receive a second clock signal; The output end of the third trigger serves as the output end of the sub-comparison module.

6. The image processing unit according to claim 5, characterized in that: The activation module includes a fourth trigger; The first input terminal and the reset terminal of the fourth trigger are both electrically connected to the output terminal of the last-stage comparison module, and are used to activate each value of the second image to be processed to obtain the output feature image; The second input terminal of the fourth trigger is used to receive the second clock signal; The output end of the fourth trigger is used to output the output feature image.

7. An image processing method, characterized in that: Applied to the image processing unit according to any one of claims 1 to 6, the method comprising: Receiving the convolutional layer indication signal; Determine the corresponding neural network block according to the convolution layer indication signal, and determine the design fixed-point range of the corresponding neural network block according to the pre-stored design fixed-point range of the neural network block; According to the design fixed-point range of the corresponding neural network block, the characteristic signal is processed in a fixed-point manner to obtain fixed-point data that meets the design accuracy range.

8. The image processing method according to claim 7, characterized in that: The method of performing fixed-point processing on the characteristic signal according to the design fixed-point range of the corresponding neural network block to obtain fixed-point data that meets the design accuracy range includes: Determine whether the difference between the design fixed-point range of the current corresponding neural network block and the design fixed-point range corresponding to the previous level neural network block is greater than a predetermined difference threshold; If it is not greater than the predetermined difference threshold, the design fixed-point range corresponding to the previous level neural network block is used as the design fixed-point range of the current corresponding neural network block.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer instructions. When the computer instructions are executed on a computer, the image processing method according to any one of claims 7 to 8 is implemented.

Images