Self-adaptive compression neural network calculation circuit based on energy attention mechanism

Abstract

The invention discloses a self-adaptive compression neural network computing circuit based on an energy attention mechanism, and relates to the technical field of hardware implementation of neural networks. According to the method, a compression mechanism based on signal energy and an attention compression mechanism based on a neural network are realized, the operand of an attention mechanism is simplified through a multi-stage compression processing mode, low-delay processing of compression operation is realized, and the processing strategy can effectively filter irrelevant signal frames in the time sequence signals, so that the accuracy of the time sequence signals is improved. Therefore, the input feature map size of the neural network operation core circuit is reduced, and the operation power consumption during neural network operation is reduced. Meanwhile, an adaptive compression neural network calculation circuit is realized aiming at compression of weight data of the neural network, and dynamic size adaptation aiming at a feature map and compression operation aiming at the weight of the neural network are realized. According to the invention, low-power-consumption and low-delay operation of the neural network is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of hardware implementation of neural networks, and in particular to an adaptive compression neural network calculation circuit. Background Art

[0002] In the signal processing process for time series, signal processing systems such as neural networks will process and calculate a continuous time series signal. However, the input continuous time series is usually mixed with meaningless time series signals, such as clutter signals in radar signals, sound signals, or background noise in physiological electrical signals. These meaningless signals mixed with valid information and sent to the signal processing system will bring additional processing delays and power consumption, and even reduce the accuracy of the calculation results, resulting in a decrease in the performance of the entire system. Under normal circumstances, the time series signal can be divided into independent events along the time dimension (for example, a certain time period will only contain irrelevant information such as noise, and other time periods will contain valid information). Existing technologies such as digital filters can effectively filter out irrelevant information mixed in valid signals, but it is difficult to directly ignore irrelevant information in the time dimension. Therefore, in real applications, a compression mechanism along the time dimension is needed to analyze the input time series in real time and compress irrelevant information along the time dimension to avoid additional processing delays and processing power consumption for the entire system. In addition, with the development of deep learning, attention mechanisms are increasingly being introduced into neural networks. The attention mechanism is a method that mimics the human visual and cognitive system. This mechanism uses neural networks to actively learn when processing input data, and selectively selects areas with important information from a large amount of irrelevant information, thereby processing data more efficiently. Therefore, introducing the attention mechanism into the screening of input time series signals can effectively extract valid information from a large amount of irrelevant information along the time dimension, and ignore a large amount of time series containing invalid information, avoiding additional processing delays and processing power consumption.

[0003] Introducing an energy-attention-based compression mechanism in neural network operations can effectively reduce the size of the input feature map, thereby reducing the amount of computation in neural network operations. However, this energy-attention-based compression mechanism will bring the following challenges: 1) The attention-based compression mechanism requires the neural network to calculate the attention information of the input signal in real time, but using the neural network to calculate attention will bring a large amount of computation, which is difficult to achieve real-time processing for continuous input time series signals; 2) The compression mechanism for the time dimension makes the input feature map size of the neural network dynamically variable, but the common edge neural network operation circuits on the market are designed based on application-specific integrated circuits (ASICs). This type of neural network operation circuit completes the operation through pre-compiled instructions, which means that this type of neural network operation circuit can only support fixed-size feature map inputs and cannot dynamically and adaptively adjust the operation mode according to the shape of the input features; if this dynamic size adaptation method needs to be supported, the instructions need to be modified in the processor core (such as ARM or RISC-V). This solution requires a lot of time to complete the recompilation of instructions and requires a large number of clock cycles of the processor core. Summary of the invention

[0004] The present invention provides an adaptive compression neural network computing circuit based on energy attention mechanism, which implements an adaptive compression neural network computing circuit based on a double compression strategy of energy attention and neural network attention to achieve low power consumption and low latency operation of the neural network computing circuit.

[0005] The technical solution adopted by the present invention is: an adaptive compression neural network computing circuit based on an energy attention mechanism, the circuit includes an interaction control unit and a storage and calculation control unit; wherein the interaction control unit includes a feature map interaction bus controller, an instruction and network parameter interaction bus controller, an interaction request and control circuit and an input data stream distribution circuit; the storage and calculation control unit includes an instruction decoding and calculation control circuit, an adaptive compression memory read and write control circuit and a neural network calculation circuit; The feature map interaction bus controller and the instruction and network parameter interaction bus controller mount the neural network computing circuit on the data bus on the chip and communicate with the processor core through the configured on-chip bus interaction protocol; The feature map interaction bus controller is used to interact the input feature map of the neural network to the inside of the circuit, and interact the neural network operation result to the outside of the neural network calculation circuit; wherein the input feature map of the neural network is the input feature map after performing energy attention compression and neural network attention compression; The instruction and network parameter interaction bus controller is used to interact the layer instructions, network weights, network bias values, and weight compression indexes required for neural network operations into the circuit; The interactive request and control circuit is used to generate an interactive request signal to the feature map interactive bus controller and the instruction and network parameter interactive bus controller. After receiving the interactive request signal, the two interactive bus controllers will start the interaction, interacting the relevant data required for the neural network operation to the inside of the neural network calculation circuit, or interacting the neural network operation results to the outside of the neural network calculation circuit; the interactive request and control circuit also generates an interactive control signal for controlling the working mode of the input data stream distribution circuit; The input data stream distribution circuit is used to decode the data stream interactively input by the feature map interactive bus controller and the instruction and network parameter interactive bus controller, and distribute the decoded input feature map data and network parameter data to the adaptive compression memory read-write control circuit, and distribute the decoded instruction data to the instruction decoding and operation control circuit; Among them, the interactive input data stream includes the input feature map data stream and the instruction and network parameter data stream; the input feature map data stream format fields include: feature map frame size, frame header of frame channel and feature map data stream carrier; the fields of the instruction and network parameter data stream include: instruction data, bias value data, compression index data and weight data.

[0006] The instruction decoding and operation control circuit is used to perform instruction correction on the decoded feature map size signal (i.e., feature map size synchronization signal) in the instruction data from the input data stream distribution circuit according to the current layer feature map size signal sent by the adaptive compression memory read-write control circuit when calculating the neural network layer by layer, and then generate the operation and read-write control signal and send it to the adaptive compression memory read-write control circuit and the neural network operation circuit to control the memory read-write state of the adaptive compression memory read-write control circuit and the operation state of the neural network operation circuit; The adaptive compression memory read-write control circuit includes: a feature map random access memory, a weight / bias value / compression index random access memory, a memory read-write control circuit, an MN selector, a detector (LOD) and a sequence 1 counter; The feature map random access memory is used to store the input and output feature map data of each network layer of the neural network, and is accompanied by a frame size register and a frame channel register. The two registers are used to save the frame size and frame channel number of the input feature map from the frame header of the input feature map data stream, generate the current feature map size signal and send it to the instruction decoding and operation control circuit to correct the decoded instruction data, and extract the operation output size signal from the operation and read-write control signal issued by the operation core data stream master controller and save it; The weight / bias value / compression index random access memory is used to store weight data, bias value data and compression index data of each network layer of the neural network; The read and write modes of the adaptive compression memory read and write control circuit include compressed reading and non-compressed reading and writing; In the non-compressed read / write mode, the memory read / write control circuit reads the required input feature map data from the feature map random access memory, reads the required weight data and bias value data from the weight / bias value / compression index random access memory, and sends the read data to the neural network operation circuit for operation, and the neural network operation circuit writes the operation result back to the feature map random access memory; In the compressed read-write mode, the memory read-write control circuit first reads a binary compressed index sequence of length M from the weight / bias value / compression index random access memory, and then sends the binary compressed index of length M to the sequence 1 counter and the detector of the leading 1 respectively. The sequence 1 counter is used to calculate the binary compressed index "1" data, and sends the calculation result as the compressed weight length N to the memory read-write control circuit. The detector is responsible for finding the location information of N "1" data from the compressed sequence of length M and sending it to the MN selector; After receiving the compressed weight length N, the memory read-write control circuit reads N compressed weight data and M feature map data, and sends the currently read M feature map data to the MN selector. The MN selector selects feature map data at N specified positions from the M feature map data based on the received N position information and sends them to the memory read-write control circuit; the memory read-write control circuit sends the read N feature map data and N weight data to the neural network operation circuit for calculation; after the current network layer operation is completed, the memory read-write control circuit writes the operation result output by the neural network operation circuit into the feature map random access memory as the input feature map of the next layer of the network.

[0007] Furthermore, energy attention compression is specifically as follows: the energy of an input signal of a fixed length in a certain time frame is calculated by accumulating the mean square value or the absolute value, and the energy result is compared with a pre-set energy threshold, and the input signal of the time frame that reaches the energy threshold is retained.

[0008] Furthermore, neural network attention compression is specifically as follows: The attention score and signal mask of the input signal that reaches the capability threshold are obtained through a preset attention neural network; wherein the length of the signal mask is consistent with the length of the input signal; The obtained attention score is compared with the preset attention threshold. For input signals that reach the attention threshold, the input signal is multiplied by its signal mask, and the result is stored in the specified buffer as the input feature map cache of the neural network.

[0009] Furthermore, when the attention score and signal mask of the input signal that reaches the capability threshold are obtained based on the attention neural network, it can be completed based on the adaptive compression neural network calculation circuit set by the present invention, using the input signal as the input data of the neural network, and completing the forward operation process of the attention neural network in the adaptive compression neural network calculation circuit based on the weight / bias value of the attention neural network. Of course, the attention score and signal mask of the current input signal can also be obtained based on a calculation circuit separately set for the operation of the attention mechanism module.

[0010] Further, the instruction decoding and operation control circuit includes an instruction decoding unit, an instruction correction unit, and an operation core data flow master controller; The instruction decoding unit decodes the instruction data into an uncorrected decoded instruction, and compares the decoded characteristic map size signal in the uncorrected decoded instruction with the current characteristic map size signal. If the two are the same, the instruction correction is not performed, and the uncorrected decoded instruction is directly used as the decoded instruction sent to the operation core data flow general controller; otherwise, the instruction correction is performed, and the corrected decoded instruction is used as the decoded instruction sent to the operation core data flow general controller; After receiving the decoding instruction, the operation core data flow master controller generates operation and read-write control signals to control the memory read-write state of the adaptive compression memory read-write control circuit and the operation state of the neural network operation circuit; Among them, the instruction correction is: calculate the difference in the shape of the input and output feature maps of the current layer according to the decoded feature map size signal, calculate the operational output feature map size of the current layer according to the feature map shape difference and the current feature map size signal in the instruction correction unit; correct the decoding instruction according to the three signals of shape difference, current feature map size and operational output feature map size of the current layer.

[0011] Furthermore, the working logic of the input data stream allocation circuit is as follows: when the feature map data stream is interactively input, the frame counter counts the data stream according to the data in the frame header, and the input interaction is completed when the specified number is reached; when the instruction and network parameter data stream are interactive, the frame decoding circuit identifies the frame header data, and sends the data stream of the field content of the specified number of fields set by the interactive instruction and network parameter data stream to the designated circuit.

[0012] Furthermore, when the input data stream allocation circuit interactively inputs the feature map data stream, it regards it as weight data for frame decoding and data allocation.

[0013] Further, the neural network operation circuit includes an input feature map buffer, a weight and bias value buffer, an operation unit input distribution circuit, a plurality of operation units, an activation function operation circuit, a pooling layer operation circuit and a quantization scaling / shifting circuit; The feature map buffer is used to store the feature map data input from the adaptive compression memory read-write control circuit; The weight and bias value buffer is used to store the weight and bias value input from the adaptive compression memory read and write control circuit; The computing unit input distribution circuit reads data from two buffers and dynamically distributes it to different computing units for calculation through multiple batches according to the network layer structure and feature map size; The operation unit then sends the operation results to the quantization scaling / shifting circuit, the activation function operation circuit, and the pooling layer operation circuit for calculation in sequence, and returns the final operation results to the adaptive compression memory read and write control circuit.

[0014] Furthermore, the neural network operation circuit also includes a softmax operation circuit that independently performs softmax operation, which is used to perform softmax operation on the operation result sent by the operation unit, and then send the operation result to the subsequent quantization scaling / shift circuit.

[0015] The technical solution provided by the present invention brings at least the following beneficial effects: 1) The present invention realizes multi-stage compression through a compression mechanism based on signal energy and an attention compression mechanism based on a neural network, thereby simplifying the computational complexity of the attention mechanism and realizing low-latency processing of the compression operation. This can effectively filter irrelevant signal frames in the time series signal to reduce the input feature map size of the neural network operation core circuit, thereby reducing the operating power consumption during the neural network operation; 2) The constructed adaptive compressed neural network computing circuit suitable for dual compression based on energy compression and attention realizes dynamic size adaptation for feature maps and compression operations for neural network weights, achieving low power consumption and low latency operations for neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 Schematic diagram of the multi-stage compression strategy based on the energy attention mechanism; Figure 2 Schematic diagram of the training process of a neural network for attention calculation; Figure 3 This is the overall structural block diagram of the adaptive compression neural network calculation circuit; Figure 4This is the structural block diagram of the instruction decoding and operation control circuit; Figure 5 It is a structural block diagram of the read-write control circuit of the adaptive compression memory; Figure 6 This is a block diagram of the neural network operation circuit structure. DETAILED DESCRIPTION

[0018] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0019] The feature compression method based on attention needs to use a neural network to calculate the attention of the input signal, and feed back the calculation result of the attention to the input signal to achieve the adjustment and compression of the input signal. However, the continuous input characteristics of the time series require that this compression strategy needs to have the ability to process in real time, but the neural network calculation of attention will bring a large amount of calculation and calculation delay. In view of this, in the implementation process of the adaptive compression neural network calculation circuit based on the energy attention mechanism provided in the embodiment of the present invention, a multi-stage compression strategy based on the energy attention mechanism is proposed. The strategy first pre-compresses the input signal through the first stage energy-based compression method, and then uses the pre-compressed input signal as the input data of the attention neural network for the second stage of attention compression. By combining the primary (first stage) energy-based compression method and the secondary (second stage) attention-based compression method, a low-latency and low-power feature compression strategy is achieved.

[0020] The implementation process of the multi-stage compression strategy based on the energy attention mechanism is as follows Figure 1 As shown. First, the input data is subjected to the first stage of energy compression method, which usually uses mean square value or absolute value accumulation to calculate the energy of a certain time frame of the input data, and compares the energy result with the pre-set energy threshold. If the energy threshold is not reached, it is considered that the current time frame does not contain a valid signal with sufficient energy, and the signal of the current time frame is ignored. If the energy threshold is exceeded, the input signal is passed back to perform the second-level attention calculation.

[0021] The signal after the first level of compression will be sent to the neural network calculation circuit to calculate the attention corresponding to the input signal. The attention neural network will output the current input attention score and signal mask (consistent with the dimension of the input data of the attention neural network). The attention score is used to compare with the attention threshold. If the threshold is not reached, the current input data will be discarded; if the threshold is reached, the input data will be multiplied by the signal mask and the result will be sent to the buffer as the input feature map cache of the neural network. As a possible implementation method, the training strategy of the attention neural network is as follows Figure 2 As shown, in the training phase, a data set with added noise will first be generated and used as a training set for the attention neural network. The output result of the attention neural network will simulate the compression method based on the attention mechanism to perform threshold comparison, data compression and mask calculation, and the input data of the compression and mask calculation will be sent to the signal processing system to obtain the calculation result. The accuracy error of the calculation result and the similarity error of the input data will be used as update indicators to guide the parameter update and training of the neural network. The attention operation neural network trained by this training method can efficiently match the target signal processing system.

[0022] The existing edge-end neural network operation circuit can only complete the calculation according to the pre-compiled instructions, which means that it can only statically support feature map data of fixed size and shape, and cannot dynamically adjust the operation state of the circuit according to feature maps of different shapes. Or it is necessary to recompile the instructions with the cooperation of the external processor core, but this method consumes a large number of clock cycles to complete the recompilation of the instructions. Therefore, in an embodiment of the present invention, a dynamic size adaptation method for feature maps is introduced in the neural network operation core. This method does not need to occupy a large number of clock cycles and can achieve dynamic size adaptation of feature maps. In addition, in neural network training, a weight pruning method is usually used to generate a large number of zero values ​​in the weights, reducing the amount of neural network calculation while ensuring that the accuracy remains unchanged. The embodiment of the present invention pre-compresses the zero values ​​of the pruned neural network weights off-chip, and designs a read, write and calculation method for the compressed weights in the neural network operation circuit, which is combined with the dynamic size adaptation method to further reduce the amount of neural network calculation and achieve low power consumption processing of the circuit. The embodiment of the present invention also designs a bus controller for data interaction in the circuit, which can mount the entire neural network operation circuit as an IP core on the data bus as a whole through on-chip interaction protocols such as APB / AHB / AXI, and support processor cores of different architectures such as ARM or RISC-V.

[0023] As a possible implementation method, the overall structure of the adaptive compression neural network computing circuit based on the energy attention mechanism proposed in the embodiment of the present invention is as follows: Figure 3As shown in the dotted box, the entire operation circuit can be divided into two parts: an interactive control unit and a storage and calculation control unit. The left half is the interactive control unit, which includes four parts: a feature map interactive bus controller 12, an instruction and network parameter interactive bus controller 11, an interactive request and control circuit 13, and an input data stream distribution circuit 14. The right half is the storage and calculation control unit, which includes three parts: an instruction decoding and operation control circuit 21, an adaptive compression memory read and write control circuit 22, and a neural network operation circuit 23.

[0024] Among them, the feature graph interaction bus controller 12 and the instruction and network parameter interaction bus controller 11 can mount the neural network computing circuit on the data bus in the chip and communicate with the processor core (such as ARM / RISC-V, etc.) through various on-chip bus interaction protocols such as APB / AHB / AXI. The feature graph interaction bus controller 12 is responsible for interacting the input feature graph to the inside of the circuit and interacting the neural network operation results to the outside of the neural network computing circuit; the instruction and network parameter interaction bus controller 11 is responsible for interacting the layer instructions, network weights, network bias values, and weight compression indexes required for the neural network operation to the inside of the circuit.

[0025] The interaction request and control circuit 13 is responsible for generating an interaction request signal to the two interaction bus controllers. After receiving the interaction request signal, the two interaction bus controllers will start the interaction, interacting the relevant data to the inside of the neural network computing circuit, or interacting the neural network output results to the outside of the neural network computing circuit; the interaction request and control circuit also generates an interaction control signal for controlling the working mode of the input data stream distribution circuit.

[0026] The input data stream distribution circuit 14 is responsible for decoding the data streams interactively input by the two interactive bus controllers, and distributing the input feature map data and network parameter data to the adaptive compression memory read-write control circuit 22, and distributing the instruction data to the instruction decoding and operation control circuit 21. The format of the input feature map data stream is: a frame header containing the feature map frame size and frame channel + a feature map data stream. The interactive content of each part of the instruction and network parameter data stream includes: instruction data + bias value data + compression index data + weight data, where the bias value, compression index, and weight are all regarded as network parameter data. If the current network layer is not compressed, the compression index data is not included. The data stream format of each part is a frame header containing the number of interactions + interactive data stream. The working logic of the input data stream distribution circuit is as follows: when the feature map data is interactively input, the frame counter will count the data stream according to the data in the frame header, and the input interaction will be completed when the specified number is reached. When the interactive instruction and network parameter data stream are in progress, the frame decoding circuit will identify the frame header data and send the specified number of data streams to the specified circuit according to the current interactive content. In addition, when the feature map data stream is interactively input, it can also be regarded as weight data for frame decoding and data allocation.

[0027] The structure of the instruction decoding and operation control circuit 21 is as follows: Figure 4 As shown in the figure, the dotted box contains the main circuit module, including the instruction decoding unit, the instruction correction unit, and the operation core data flow controller. Since the input feature map data of the neural network operation circuit is double compressed by energy attention and neural network attention, the input feature map size cannot match the feature map size encoded in the instruction. Therefore, when calculating the neural network layer by layer, it is necessary to dynamically correct the decoding instruction of the current layer according to the feature map size synchronization signal and the shape of the feature map of the current layer. When calculating a certain layer of the neural network, the instruction decoding unit decodes the instruction data into an uncorrected decoding instruction, and compares the decoded feature map size signal in the uncorrected decoding instruction with the current feature map size signal. If the two are the same, the instruction correction is not performed; otherwise, the instruction will be corrected. The instruction correction process is as follows: first, the difference in the input and output feature map shapes of the current layer is calculated according to the decoded feature map size signal. In the instruction correction unit, the operation output feature map size of the current layer is first calculated according to the feature map shape difference and the current feature map size signal. Then, the decoding instruction is corrected according to the shape interpolation, the current feature map size, and the operation output feature map size of the current layer. After receiving the decoding instruction, the operation core data flow controller generates operation and read-write control signals to control the memory read-write state of the adaptive compression memory read-write control circuit and the operation state of the neural network operation circuit.

[0028] The structure of the adaptive compression memory read-write control circuit 22 is as follows: Figure 5As shown, the dotted box contains the main circuit modules. The feature map random access memory is responsible for storing the input and output feature map data of each network layer of the neural network, and is accompanied by a frame size register and a frame channel register. The two registers are used to save the frame size and frame channel number of the input feature map from the frame header of the input feature map data stream, generate the current feature map size signal to correct the decoding instruction data, and extract the operation output size signal from the read-write control signal and save it. The feature map random access memory is responsible for storing the input and output feature map data of each network layer of the neural network, and the weight / bias value / compression index random access memory is responsible for storing the weight data, bias value data and compression index data of each network layer of the neural network. The adaptive compression memory read-write control circuit has two modes: compressed reading and non-compressed reading and writing. In the non-compressed reading and writing mode, the memory read-write control circuit reads the required input feature map data from the feature map random access memory, reads the required weight data and bias value data from the weight / bias value / compression index random access memory, and sends the data to the neural network operation circuit for operation, and the operation result is written back to the feature map random access memory. In the compressed read / write mode, the memory read / write control circuit first reads a binary compressed index sequence of length M from the weight / bias value / compressed index random access memory (can support reading of compressed index sequence of length 32 at most), and the binary compressed index of length M is sent to the sequence 1 counter and the leading 1 detector (LOD) respectively. Assuming that the binary compressed index of length M contains N "1" data (N is less than or equal to M), the sequence 1 counter is used to count the "1" data of the binary compressed index and send it to the memory read / write control circuit as the compressed weight length N. The LOD is responsible for calculating the position information of the N "1" data from the compressed sequence of length M and sending it to the MN selector. After receiving the compressed weight length N, the memory read / write control circuit will read the N compressed weight data and the M feature map data. The read M feature map data first pass through the MN selector. The MN selector selects the feature map data of the N specified positions from the M feature map data according to the received N position information. Then the memory read / write control circuit will send the read N feature map data and N weight data to the neural network operation circuit for calculation. After the current network layer operation is completed, the memory read and write control circuit will write the output results of the neural network operation circuit into the feature map random access memory as the input feature map data of the next layer of the network.

[0029] The structure of the neural network operation circuit 23 is as follows: Figure 6As shown, it includes an input feature map buffer, a weight and bias value buffer, an operation unit input distribution circuit, multiple operation units, an activation function operation circuit, a pooling layer operation circuit, a softmax operation circuit, and a quantization scaling / shifting circuit. In order to improve the operation speed of the neural network operation circuit, multiple operation units are usually configured for parallel operation, and each operation unit is numbered #1, #2, ..., #P. The value of P can be adjusted according to the parallel degree of the neural network operation, and the typical value is 4-32. When performing neural network operations, the feature map data input from the adaptive compression memory read and write control circuit is first written into the input feature map buffer, and the weight and bias values ​​are first written into the weight and bias value buffer. Then the data of the two buffers are sent to the operation unit input distribution circuit. The operation unit input distribution circuit is divided into multiple batches according to the network layer structure and the feature map size and dynamically distributed to different operation units for operation. In addition, the neural network operation circuit can also complete the softmax operation alone. After the calculation unit or softmax calculation circuit completes the calculation, the calculation result is sent to the quantization scaling / shifting circuit, the activation function calculation circuit, and the pooling layer calculation circuit for calculation in sequence, and the final calculation result is sent back to the adaptive compression memory read and write control circuit.

[0030] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0031] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. An adaptive compressed neural network computing circuit based on energy attention mechanism, characterized in that: It includes an interactive control unit and a storage and calculation control unit; wherein the interactive control unit includes a feature map interactive bus controller, an instruction and network parameter interactive bus controller, an interactive request and control circuit, and an input data stream distribution circuit; the storage and calculation control unit includes an instruction decoding and calculation control circuit, an adaptive compression memory read and write control circuit, and a neural network calculation circuit; The feature map interaction bus controller and the instruction and network parameter interaction bus controller mount the neural network computing circuit on the data bus on the chip and communicate with the processor core through the configured on-chip bus interaction protocol; The feature map interaction bus controller is used to interact the input feature map of the neural network to the inside of the circuit, and interact the neural network operation result to the outside of the neural network calculation circuit; wherein the input feature map of the neural network is the input feature map after performing energy attention compression and neural network attention compression; The instruction and network parameter interaction bus controller is used to interact the layer instructions, network weights, network bias values, and weight compression indexes required for neural network operations into the circuit; The interactive request and control circuit is used to generate an interactive request signal to the feature map interactive bus controller and the instruction and network parameter interactive bus controller. After receiving the interactive request signal, the two interactive bus controllers will start the interaction, interacting the relevant data required for the neural network operation to the inside of the neural network calculation circuit, or interacting the neural network operation results to the outside of the neural network calculation circuit; the interactive request and control circuit also generates an interactive control signal for controlling the working mode of the input data stream distribution circuit; The input data stream distribution circuit is used to decode the data stream interactively input by the feature map interactive bus controller and the instruction and network parameter interactive bus controller, and distribute the decoded input feature map data and network parameter data to the adaptive compression memory read-write control circuit, and distribute the decoded instruction data to the instruction decoding and operation control circuit; Among them, the interactive input data stream includes the input feature map data stream and the instruction and network parameter data stream; the input feature map data stream format fields include: feature map frame size, frame header of frame channel and feature map data stream carrier; the fields of the instruction and network parameter data stream include: instruction data, bias value data, compression index data and weight data; The instruction decoding and operation control circuit is used to perform instruction correction on the decoded feature map size signal in the instruction data from the input data stream distribution circuit according to the current layer feature map size signal sent by the adaptive compression memory read-write control circuit when calculating the neural network layer by layer, and then generate the operation and read-write control signal and send it to the adaptive compression memory read-write control circuit and the neural network operation circuit to control the memory read-write state of the adaptive compression memory read-write control circuit and the operation state of the neural network operation circuit; The adaptive compression memory read-write control circuit includes: a feature map random access memory, a weight / bias value / compression index random access memory, a memory read-write control circuit, an MN selector, a detector and a sequence 1 counter; The feature map random access memory is used to store the input and output feature map data of each network layer of the neural network, and is accompanied by a frame size register and a frame channel register. The two registers are used to save the frame size and frame channel number of the input feature map from the frame header of the input feature map data stream, generate the current feature map size signal and send it to the instruction decoding and operation control circuit to correct the decoded instruction data, and extract the operation output size signal from the operation and read-write control signal issued by the operation core data stream master controller and save it; The weight / bias value / compression index random access memory is used to store weight data, bias value data and compression index data of each network layer of the neural network; The read and write modes of the adaptive compression memory read and write control circuit include compressed reading and non-compressed reading and writing; In the non-compressed read / write mode, the memory read / write control circuit reads the required input feature map data from the feature map random access memory, reads the required weight data and bias value data from the weight / bias value / compression index random access memory, and sends the read data to the neural network operation circuit for operation, and the neural network operation circuit writes the operation result back to the feature map random access memory; In the compressed read-write mode, the memory read-write control circuit first reads a binary compressed index sequence of length M from the weight / bias value / compressed index random access memory, and then sends the binary compressed index of length M to the sequence 1 counter and the detector of the leading 1 respectively. The sequence 1 counter is used to calculate the binary compressed index "1" data, and sends the calculation result as the compressed weight length N to the memory read-write control circuit. The detector is responsible for finding the location information of N "1" data from the compressed sequence of length M and sending it to the MN selector; After receiving the compressed weight length N, the memory read-write control circuit reads N compressed weight data and M feature map data, and sends the currently read M feature map data to the MN selector. The MN selector selects feature map data at N specified positions from the M feature map data based on the received N position information and sends them to the memory read-write control circuit; the memory read-write control circuit sends the read N feature map data and N weight data to the neural network operation circuit for calculation; after the current network layer operation is completed, the memory read-write control circuit writes the operation result output by the neural network operation circuit into the feature map random access memory as the input feature map of the next layer of the network.

2. The adaptive compressed neural network computing circuit based on energy attention mechanism as claimed in claim 1, characterized in that: Energy attention compression is specifically: The energy of a fixed-length input signal in a certain time frame is calculated by cumulative mean square value or absolute value, and the energy result is compared with a preset energy threshold, and the input signal of the time frame that reaches the energy threshold is retained.

3. The adaptive compressed neural network computing circuit based on energy attention mechanism as claimed in claim 1, characterized in that: Neural network attention compression is specifically: The attention score and signal mask of the input signal that reaches the capability threshold are obtained through a preset attention neural network; wherein the length of the signal mask is consistent with the length of the input signal; The obtained attention score is compared with the preset attention threshold. For input signals that reach the attention threshold, the input signal is multiplied by its signal mask, and the result is stored in the specified buffer as the input feature map cache of the neural network.

4. The adaptive compressed neural network computing circuit based on energy attention mechanism as claimed in claim 1, characterized in that: The instruction decoding and operation control circuit includes an instruction decoding unit, an instruction correction unit, and an operation core data flow controller; The instruction decoding unit decodes the instruction data into an uncorrected decoded instruction, and compares the decoded characteristic map size signal in the uncorrected decoded instruction with the current characteristic map size signal. If the two are the same, the instruction correction is not performed, and the uncorrected decoded instruction is directly used as the decoded instruction sent to the operation core data flow general controller; otherwise, the instruction correction is performed, and the corrected decoded instruction is used as the decoded instruction sent to the operation core data flow general controller; After receiving the decoding instruction, the operation core data flow master controller generates operation and read-write control signals to control the memory read-write state of the adaptive compression memory read-write control circuit and the operation state of the neural network operation circuit; Among them, the instruction correction is: calculate the difference in the shape of the input and output feature maps of the current layer according to the decoded feature map size signal, calculate the operational output feature map size of the current layer according to the feature map shape difference and the current feature map size signal in the instruction correction unit; correct the decoding instruction according to the three signals of shape difference, current feature map size and operational output feature map size of the current layer.

5. The adaptive compressed neural network computing circuit based on energy attention mechanism as claimed in claim 1, characterized in that: The working logic of the input data stream allocation circuit is: when the feature map data stream is interactively input, the frame counter counts the data stream according to the data in the frame header, and the input interaction is completed when the specified number is reached; when the instruction and network parameter data stream are interactive, the frame decoding circuit identifies the frame header data, and sends the data stream of the field content of the specified number of fields set by the interactive instruction and network parameter data stream to the designated circuit.

6. The adaptive compressed neural network computing circuit based on energy attention mechanism as claimed in claim 1, characterized in that: When the input data stream allocation circuit interactively inputs the feature map data stream, it regards it as weight data for frame decoding and data allocation.

7. The adaptive compressed neural network computing circuit based on energy attention mechanism as claimed in claim 1, characterized in that: The neural network operation circuit includes an input feature map buffer, a weight and bias value buffer, an operation unit input distribution circuit, a plurality of operation units, an activation function operation circuit, a pooling layer operation circuit and a quantization scaling / shifting circuit; The feature map buffer is used to store the feature map data input from the adaptive compression memory read-write control circuit; The weight and bias value buffer is used to store the weight and bias value input from the adaptive compression memory read and write control circuit; The computing unit input distribution circuit reads data from two buffers and dynamically distributes it to different computing units for calculation through multiple batches according to the network layer structure and feature map size; The operation unit then sends the operation results to the quantization scaling / shifting circuit, the activation function operation circuit, and the pooling layer operation circuit for calculation in sequence, and returns the final operation results to the adaptive compression memory read and write control circuit.

8. The adaptive compressed neural network computing circuit based on energy attention mechanism as claimed in claim 7, characterized in that: The neural network operation circuit also includes a softmax operation circuit that performs softmax operation independently, which is used to perform softmax operation on the operation result sent by the operation unit, and then send the operation result to the subsequent quantization scaling / shifting circuit.

Images