Adaptive compressed neural network computing circuit based on energy attention mechanism

Abstract

The present invention discloses an adaptive compression neural network computing circuit based on an energy attention mechanism, and relates to the technical field of hardware implementation of neural networks. The present invention simplifies the computational complexity of the attention mechanism through a multi-stage compression processing method by implementing a compression mechanism based on signal energy and an attention compression mechanism based on a neural network, and realizes low-latency processing of compression operations. This processing strategy can effectively filter irrelevant signal frames in time series signals, thereby reducing the input feature map size of the neural network operation core circuit, and thus reducing the operating power consumption during neural network operation. At the same time, an adaptive compression neural network computing circuit is implemented for the compression of the weight data of the neural network, which realizes dynamic size adaptation for the feature map and compression operations for the neural network weights. The present invention realizes low-power and low-latency operations of neural networks.

Description

Technical Field

[0001] The present invention relates to the technical field of hardware implementation of neural networks, and in particular to an adaptive compression neural network calculation circuit. Background Art

[0002] In the signal processing of time series, signal processing systems such as neural networks process and compute on a continuous time series signal. However, the input continuous time series often contains meaningless signals, such as clutter in radar signals, acoustic signals, or background noise in physiological electrical signals. The mixing of these meaningless signals with valid information and their input into the signal processing system can lead to additional processing delays and power consumption, and can even reduce the accuracy of the computational results, resulting in a decrease in overall system performance. Typically, time series signals can be divided into independent events along the time dimension (for example, certain time segments contain only irrelevant information such as noise, while other time segments contain valid information). Existing technologies such as digital filters can effectively filter out irrelevant information mixed with valid signals, but it is difficult to directly ignore irrelevant information along the time dimension. Therefore, in real-world applications, a compression mechanism along the time dimension is needed that can analyze the input time series in real time and compress irrelevant information along the time dimension to avoid additional processing delays and power consumption for the entire system. Furthermore, with the development of deep learning, attention mechanisms are increasingly being incorporated into neural networks. This mechanism, a method that mimics the human visual and cognitive system, uses neural networks to actively learn when processing input data and selectively select areas containing important information from a large amount of irrelevant information, thereby processing data more efficiently. Therefore, introducing the attention mechanism into the screening of input time series signals can effectively extract valid information from a large amount of irrelevant information along the time dimension and ignore large amounts of time series containing invalid information, thus avoiding additional processing delays and power consumption.

[0003] Introducing an energy-attention-based compression mechanism into neural network operations can effectively reduce the size of input feature maps, thereby reducing the amount of computation required. However, this energy-attention-based compression mechanism presents the following challenges: 1) Attention-based compression requires the neural network to compute the attention information of the input signal in real time. However, using a neural network to calculate attention requires a large amount of computation, making it difficult to process continuously input time series signals in real time. 2) The compression mechanism targeting the time dimension makes the size of the neural network's input feature maps dynamically variable. However, common edge neural network circuits on the market are designed based on application-specific integrated circuits (ASICs). These circuits perform operations using pre-compiled instructions, meaning they can only support fixed-size feature map inputs and cannot dynamically adapt their operation mode to the shape of the input features. Supporting this dynamic size adaptation method requires modifying instructions in the processor core (such as ARM or RISC-V). This solution requires significant time to recompile instructions and consumes a large number of processor core clock cycles. Summary of the Invention

[0004] The present invention provides an adaptive compressed neural network computing circuit based on an energy attention mechanism, which implements an adaptive compressed neural network computing circuit based on a dual compression strategy of energy attention and neural network attention to achieve low power consumption and low latency operation of the neural network computing circuit.

[0005] The technical solution adopted by the present invention is: an adaptive compressed neural network computing circuit based on the energy attention mechanism, which includes an interaction control unit and a storage and calculation control unit; wherein the interaction control unit includes a feature map interaction bus controller, an instruction and network parameter interaction bus controller, an interaction request and control circuit, and an input data stream distribution circuit; the storage and calculation control unit includes an instruction decoding and calculation control circuit, an adaptive compressed memory read and write control circuit, and a neural network calculation circuit;

[0006] The feature map interaction bus controller and the instruction and network parameter interaction bus controller mount the neural network computing circuit on the on-chip data bus and communicate with the processor core through the configured on-chip bus interaction protocol;

[0007] The feature map interaction bus controller is used to interact the input feature map of the neural network to the inside of the circuit, and interact the neural network operation results to the outside of the neural network calculation circuit; wherein the input feature map of the neural network is the input feature map after performing energy attention compression and neural network attention compression;

[0008] The instruction and network parameter interaction bus controller is used to interact the layer instructions, network weights, network bias values, and weight compression indexes required for neural network operations into the circuit;

[0009] The interaction request and control circuit is used to generate an interaction request signal to the feature map interaction bus controller and the instruction and network parameter interaction bus controller. After receiving the interaction request signal, the two interaction bus controllers will start the interaction, interacting the relevant data required for the neural network operation to the inside of the neural network calculation circuit, or interacting the neural network operation results to the outside of the neural network calculation circuit; the interaction request and control circuit also generates an interaction control signal for controlling the working mode of the input data stream distribution circuit;

[0010] The input data stream distribution circuit is used to decode the data streams interactively input by the feature map interactive bus controller and the instruction and network parameter interactive bus controller, and distribute the decoded input feature map data and network parameter data to the adaptive compression memory read and write control circuit, and distribute the decoded instruction data to the instruction decoding and operation control circuit;

[0011] Among them, the interactive input data stream includes the input feature map data stream and the instruction and network parameter data stream; the input feature map data stream format fields include: feature map frame size, frame channel frame header and feature map data stream carrier; the fields of the instruction and network parameter data stream include: instruction data, bias value data, compressed index data and weight data.

[0012] The instruction decoding and operation control circuit is used to perform instruction correction on the decoded feature map size signal (i.e., the feature map size synchronization signal) in the instruction data from the input data stream distribution circuit according to the current layer feature map size signal sent by the adaptive compression memory read / write control circuit when calculating the neural network layer by layer, and then generate operation and read / write control signals and send them to the adaptive compression memory read / write control circuit and the neural network operation circuit to control the memory read / write state of the adaptive compression memory read / write control circuit and the operation state of the neural network operation circuit;

[0013] The adaptive compression memory read-write control circuit includes: a feature map random access memory, a weight / bias value / compression index random access memory, a memory read-write control circuit, an MN selector, a detector (LOD) and a sequence 1 counter;

[0014] The feature map random access memory is used to store the input and output feature map data of each network layer of the neural network, and is accompanied by a frame size register and a frame channel register. The two registers are used to save the frame size and number of frame channels of the input feature map from the frame header of the input feature map data stream, generate the current feature map size signal and send it to the instruction decoding and operation control circuit to correct the decoded instruction data, and extract the operation output size signal from the operation and read-write control signal issued by the operation core data stream controller and save it;

[0015] The weight / bias value / compression index random access memory is used to store weight data, bias value data and compression index data of each network layer of the neural network;

[0016] The read and write modes of the adaptive compression memory read and write control circuit include compressed reading and non-compressed reading and writing;

[0017] In the non-compressed read / write mode, the memory read / write control circuit reads the required input feature map data from the feature map random access memory, reads the required weight data and bias value data from the weight / bias value / compression index random access memory, and sends the read data to the neural network operation circuit for operation, and the neural network operation circuit then writes the operation result back to the feature map random access memory;

[0018] In the compressed read / write mode, the memory read / write control circuit first reads a binary compressed index sequence of length M from the weight / bias value / compression index random access memory, and then sends the binary compressed index of length M to the sequence 1 counter and the leading 1 detector respectively. The sequence 1 counter is used to calculate the binary compressed index "1" data and send the calculation result as the compressed weight length N to the memory read / write control circuit. The detector is responsible for finding the location information of N "1" data from the compressed sequence of length M and sending it to the MN selector;

[0019] After receiving the compressed weight length N, the memory read-write control circuit reads the N compressed weight data and M feature map data, and sends the currently read M feature map data to the MN selector. The MN selector selects the feature map data of N specified positions from the M feature map data based on the received N position information and sends them to the memory read-write control circuit; the memory read-write control circuit sends the read N feature map data and N weight data to the neural network operation circuit for calculation; after the current network layer operation is completed, the memory read-write control circuit writes the operation result output by the neural network operation circuit into the feature map random access memory as the input feature map of the next layer of network.

[0020] Furthermore, energy attention compression is specifically as follows: the energy of an input signal of fixed length in a certain time frame is calculated by accumulating the mean square value or absolute value, and the energy result is compared with a pre-set energy threshold, and the input signal of the time frame that reaches the energy threshold is retained.

[0021] Furthermore, the neural network attention compression is specifically as follows:

[0022] Obtaining the attention score and signal mask of the input signal that reaches the capacity threshold through a preset attention neural network; wherein the length of the signal mask is consistent with the length of the input signal;

[0023] The obtained attention score is compared with the preset attention threshold. For input signals that reach the attention threshold, the input signal is multiplied by its signal mask, and the result is stored in the specified buffer as the input feature map cache of the neural network.

[0024] Furthermore, when obtaining the attention score and signal mask of an input signal that reaches the capability threshold based on the attention neural network, this can be accomplished based on the adaptive compressed neural network computing circuit provided by the present invention. The input signal is used as the input data of the neural network, and the forward operation process of the attention neural network is completed in the adaptive compressed neural network computing circuit based on the weight / bias value of the attention neural network. Of course, the attention score and signal mask of the current input signal can also be obtained based on a separate computing circuit for the operation of the attention mechanism module.

[0025] Furthermore, the instruction decoding and operation control circuit includes an instruction decoding unit, an instruction correction unit, and an operation core data flow controller;

[0026] The instruction decoding unit decodes the instruction data into an uncorrected decoded instruction, and compares the decoded feature map size signal in the uncorrected decoded instruction with the current feature map size signal. If the two are the same, instruction correction is not performed, and the uncorrected decoded instruction is directly used as the decoded instruction sent to the operation core data flow controller; otherwise, instruction correction is performed, and the corrected decoded instruction is then used as the decoded instruction sent to the operation core data flow controller.

[0027] After receiving the decoding instruction, the operation core data flow controller generates operation and read / write control signals to control the memory read / write state of the adaptive compression memory read / write control circuit and the operation state of the neural network operation circuit;

[0028] Among them, the instruction correction is: calculate the difference in the shape of the input and output feature maps of the current layer according to the decoded feature map size signal, and calculate the operational output feature map size of the current layer according to the feature map shape difference and the current feature map size signal in the instruction correction unit; correct the decoding instruction according to the three signals of shape difference, current feature map size, and operational output feature map size of the current layer.

[0029] Furthermore, the working logic of the input data stream distribution circuit is: when the feature map data stream is interactively input, the frame counter counts the data stream according to the data in the frame header, and the input interaction is completed when the specified number is reached; when the instruction and network parameter data stream are interactive, the frame decoding circuit identifies the frame header data, and sends the data stream of the field content of the specified number of fields set by the interactive instruction and network parameter data stream to the designated circuit.

[0030] Furthermore, when the input data stream distribution circuit interactively inputs the feature map data stream, it regards it as weight data for frame decoding and data distribution.

[0031] Furthermore, the neural network operation circuit includes an input feature map buffer, a weight and bias value buffer, an operation unit input distribution circuit, a plurality of operation units, an activation function operation circuit, a pooling layer operation circuit and a quantization scaling / shifting circuit;

[0032] The feature map buffer is used to store the feature map data input from the adaptive compression memory read and write control circuit;

[0033] The weight and bias value buffer is used to store the weight and bias values ​​input from the adaptive compression memory read and write control circuit;

[0034] The arithmetic unit input distribution circuit reads data from two buffers and dynamically distributes it to different arithmetic units for calculation through multiple batches according to the network layer structure and feature map size;

[0035] The operation unit then sends the operation results to the quantization scaling / shift circuit, activation function operation circuit, and pooling layer operation circuit for calculation in sequence, and returns the final operation results to the adaptive compression memory read and write control circuit.

[0036] Furthermore, the neural network operation circuit also includes a softmax operation circuit that independently performs softmax operation, which is used to perform softmax operation on the operation result sent by the operation unit, and then send the operation result to the subsequent quantization scaling / shift circuit.

[0037] The technical solution provided by the present invention brings at least the following beneficial effects:

[0038] 1) This invention implements multi-stage compression through a signal energy-based compression mechanism and a neural network-based attention compression mechanism, simplifying the computational complexity of the attention mechanism and achieving low-latency compression processing. This effectively filters irrelevant signal frames in time series signals, reducing the size of the input feature map of the neural network's core circuitry and, in turn, lowering the power consumption of the neural network's operations.

[0039] 2) The constructed adaptive compressed neural network computing circuit suitable for dual compression based on energy compression and attention realizes dynamic size adaptation of feature maps and compression operations on neural network weights, realizing low power consumption and low latency operations of neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 Schematic diagram of the multi-stage compression strategy based on the energy attention mechanism;

[0042] Figure 2 Schematic diagram of the training process of a neural network for attention calculation;

[0043] Figure 3 This is the overall structural block diagram of the adaptive compression neural network calculation circuit;

[0044] Figure 4 This is the structural block diagram of the instruction decoding and operation control circuit;

[0045] Figure 5 This is a structural block diagram of the adaptive compression memory read and write control circuit;

[0046] Figure 6 This is a block diagram of the neural network operation circuit structure. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0048] Attention-based feature compression methods require the use of neural networks to calculate the attention of the input signal and feed the attention calculation results back to the input signal to achieve adjustment and compression of the input signal. However, the continuous input characteristics of time series require this compression strategy to have real-time processing capabilities, but neural network attention calculations will bring a large amount of computation and computational delay. In view of this, in the implementation process of the adaptive compression neural network calculation circuit based on the energy attention mechanism provided in the embodiment of the present invention, a multi-stage compression strategy based on the energy attention mechanism is proposed. This strategy first pre-compresses the input signal through a first-stage energy-based compression method, and then uses the pre-compressed input signal as the input data of the attention neural network for a second-stage attention compression. By combining the primary (first-stage) energy-based compression method and the secondary (second-stage) attention-based compression method, a low-latency and low-power feature compression strategy is achieved.

[0049] The implementation process of the multi-stage compression strategy based on the energy attention mechanism is as follows: Figure 1 As shown in the figure, the first stage of energy compression is performed on the input data. This method usually uses the mean square value or absolute value accumulation to calculate the energy of a certain time frame of the input data, and compares the energy result with a pre-set energy threshold. If the energy threshold is not reached, it is considered that the current time frame does not contain a valid signal with sufficient energy, and the signal of the current time frame is ignored. If the energy threshold is exceeded, the input signal is passed back to the second stage of attention calculation.

[0050] The signal after the first level of compression will be sent to the neural network calculation circuit to calculate the attention corresponding to the input signal. The attention neural network will output the attention score of the current input and the signal mask (consistent with the dimension of the input data of the attention neural network). The attention score is used to compare with the attention threshold. If the threshold is not reached, the current input data will be discarded; if the threshold is reached, the input data will be multiplied by the signal mask, and the result will be sent to the buffer as the input feature map cache of the neural network. As a possible implementation method, the training strategy of the attention neural network is as follows: Figure 2 As shown, in the training phase, a data set with added noise will first be generated and used as the training set of the attention neural network for training. The output result of the attention neural network will simulate the compression method based on the attention mechanism to perform threshold comparison, data compression and mask calculation, and the input data of the compression and mask calculation will be sent to the signal processing system to obtain the calculation result. The accuracy error of the calculation result and the similarity error of the input data will be used as update indicators to guide the parameter update and training of the neural network. The attention operation neural network trained by this training method can efficiently match the target signal processing system.

[0051] The existing edge-end neural network operation circuit can only complete calculations according to pre-compiled instructions, which means that it can only statically support feature map data of fixed size and shape, and cannot dynamically adjust the circuit's operation state according to feature maps of different shapes. Or it is necessary to recompile the instructions with the cooperation of an external processor core, but this method consumes a large number of clock cycles to complete the recompilation of instructions. Therefore, in an embodiment of the present invention, a dynamic size adaptation method for feature maps is introduced in the neural network operation core. This method does not require a large number of clock cycles and can achieve dynamic size adaptation of feature maps. In addition, in neural network training, a weight pruning method is usually used to generate a large number of zero values ​​in the weights, reducing the amount of neural network calculations while ensuring that the accuracy remains unchanged. The embodiment of the present invention pre-compresses the zero values ​​of the pruned neural network weights off-chip, and designs a reading, writing and calculation method for the compressed weights in the neural network operation circuit. Combined with the dynamic size adaptation method, it further reduces the amount of neural network calculations and achieves low-power processing of the circuit. The embodiment of the present invention also designs a bus controller for data interaction in the circuit, which can mount the entire neural network operation circuit as an IP core on the data bus as a whole through on-chip interaction protocols such as APB / AHB / AXI, and support processor cores of different architectures such as ARM or RISC-V.

[0052] As a possible implementation method, the overall structure of the adaptive compression neural network computing circuit based on the energy attention mechanism proposed in the embodiment of the present invention is as follows: Figure 3 As shown in the dashed box, the entire arithmetic circuit can be divided into two parts: the interaction control unit and the storage and computation control unit. The left half is the interaction control unit, which includes four parts: the feature map interaction bus controller 12, the instruction and network parameter interaction bus controller 11, the interaction request and control circuit 13, and the input data stream distribution circuit 14. The right half is the storage and computation control unit, which includes three parts: the instruction decoding and arithmetic control circuit 21, the adaptive compression memory read and write control circuit 22, and the neural network arithmetic circuit 23.

[0053] The feature graph interaction bus controller 12 and the instruction and network parameter interaction bus controller 11 can connect the neural network computing circuit to the on-chip data bus and communicate with the processor core (such as ARM / RISC-V) through various on-chip bus interaction protocols such as APB / AHB / AXI. The feature graph interaction bus controller 12 is responsible for communicating the input feature graph to the circuit and communicating the neural network calculation results to the outside of the neural network computing circuit; the instruction and network parameter interaction bus controller 11 is responsible for communicating the layer instructions, network weights, network bias values, and weight compression indexes required for neural network calculations to the circuit.

[0054] The interaction request and control circuit 13 is responsible for generating an interaction request signal to the two interaction bus controllers. After receiving the interaction request signal, the two interaction bus controllers will start the interaction, interacting the relevant data into the neural network calculation circuit, or interacting the neural network output results to the outside of the neural network calculation circuit; the interaction request and control circuit also generates an interaction control signal for controlling the working mode of the input data stream distribution circuit.

[0055] The input data stream distribution circuit 14 is responsible for decoding the data streams exchanged between the two interactive bus controllers, distributing the input feature map data and network parameter data to the adaptive compression memory read / write control circuit 22, and distributing the instruction data to the instruction decoding and operation control circuit 21. The input feature map data stream format is: a frame header containing the feature map frame size and frame channel + a feature map data stream. The interactive content of each component of the instruction and network parameter data stream includes: instruction data + bias value data + compression index data + weight data. The bias value, compression index, and weight are all considered network parameter data. If the current network layer is not compressed, the compression index data is not included. The data stream format of each component is a frame header containing the number of exchanges + an exchange data stream. The operating logic of the input data stream distribution circuit is as follows: When interactively inputting feature map data, the frame counter counts the data stream based on the frame header data. When a specified number is reached, the input exchange is completed. When exchanging instructions and network parameter data streams, the frame decoding circuit recognizes the frame header data and sends the specified number of data streams to the designated circuit based on the current exchange content. In addition, when interactively inputting feature map data streams, they can also be regarded as weight data for frame decoding and data distribution.

[0056] The structure of the instruction decoding and operation control circuit 21 is as follows: Figure 4As shown, the dotted box contains the main circuit modules, including the instruction decoding unit, the instruction correction unit, and the core data flow controller. Because the input feature map data of the neural network operation circuit is double-compressed by energy attention and neural network attention, the input feature map size cannot match the feature map size encoded in the instruction. Therefore, when calculating the neural network layer by layer, the decoded instruction of the current layer needs to be dynamically corrected based on the feature map size synchronization signal and the shape of the feature map of the current layer. When calculating a neural network layer, the instruction decoding unit decodes the instruction data into an uncorrected decoded instruction and compares the decoded feature map size signal in the uncorrected decoded instruction with the current feature map size signal. If the two are the same, the instruction correction is not performed; otherwise, the instruction is corrected. The instruction correction process is as follows: First, the difference in the input and output feature map shapes of the current layer is calculated based on the decoded feature map size signal. The instruction correction unit first calculates the output feature map size of the current layer based on the feature map shape difference and the current feature map size signal. Then, the decoded instruction is corrected based on the shape interpolation, the current feature map size, and the output feature map size of the current layer. After receiving the decoding instruction, the operation core data flow controller generates operation and read-write control signals to control the memory read and write status of the adaptive compression memory read and write control circuit and the operation status of the neural network operation circuit.

[0057] The structure of the adaptive compression memory read and write control circuit 22 is as follows: Figure 5As shown, the dashed box represents the main circuit modules. The feature map random access memory (FRAM) is responsible for storing the input and output feature map data for each neural network layer. It also includes a frame size register and a frame channel register. These registers are used to store the input feature map frame size and number of frame channels from the frame header of the input feature map data stream, generate the current feature map size signal to modify the decoding instruction data, and extract and store the computation output size signal from the read / write control signal. The FRAM is responsible for storing the input and output feature map data, weight / bias / compression index RAM, and weight data, bias data, and compression index data for each neural network layer. The adaptive compression memory read / write control circuit has two modes: compressed read and non-compression read / write. In non-compression read / write mode, the memory read / write control circuit reads the required input feature map data from the FRAM and the required weight and bias data from the weight / bias / compression index RAM. These data are then sent to the neural network computation circuit for computation, and the computation results are written back to the FRAM. In compressed read / write mode, the memory read / write control circuit first reads an M-length binary compressed index sequence (supporting compressed index sequences up to 32 in length) from the weight / bias / compressed index random access memory. This M-length binary compressed index is fed into a sequence 1 counter and a leading 1 detector (LOD), respectively. Assuming the M-length binary compressed index contains N "1" data (N is less than or equal to M), the sequence 1 counter counts the number of "1" data in the binary compressed index and feeds this into the memory read / write control circuit as the compressed weight length N. The LOD calculates the location information of the N "1" data from the M-length compressed sequence and feeds this information into the MN selector. After receiving the compressed weight length N, the memory read / write control circuit reads the N compressed weight data and M feature map data. The M feature map data first passes through the MN selector, which selects N feature map data at specified locations from the M feature map data based on the received N location information. The memory read / write control circuit then sends the N feature map data and N weight data to the neural network operation circuit for calculation. After the current network layer operation is completed, the memory read and write control circuit will write the output results of the neural network operation circuit into the feature map random access memory as the input feature map data of the next layer of the network.

[0058] The structure of the neural network operation circuit 23 is as follows Figure 6As shown, it includes an input feature map buffer, a weight and bias value buffer, an operation unit input distribution circuit, multiple operation units, an activation function operation circuit, a pooling layer operation circuit, a softmax operation circuit, and a quantization scaling / shifting circuit. In order to improve the operation speed of the neural network operation circuit, multiple operation units are usually configured for parallel operation. Each operation unit is numbered #1, #2, ..., #P. The value of P can be adjusted according to the parallelism of the neural network operation, and the typical value is 4-32. When performing neural network operations, the feature map data input from the adaptive compression memory read and write control circuit is first written into the input feature map buffer, and the weights and bias values ​​are first written into the weight and bias value buffer. Then the data from the two buffers is sent to the operation unit input distribution circuit. The operation unit input distribution circuit is divided into multiple batches and dynamically distributed to different operation units for operation according to the network layer structure and feature map size. In addition, the neural network operation circuit can also perform softmax operations independently. After the calculation is completed by the operation unit or softmax operation circuit, the operation results are sent to the quantization scaling / shift circuit, the activation function operation circuit, and the pooling layer operation circuit for calculation in sequence, and the final operation results are sent back to the adaptive compression memory read and write control circuit.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0060] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. An adaptive compressed neural network computing circuit based on energy attention mechanism, characterized in that: It includes an interaction control unit and a storage and calculation control unit; wherein the interaction control unit includes a feature map interaction bus controller, an instruction and network parameter interaction bus controller, an interaction request and control circuit, and an input data stream distribution circuit; the storage and calculation control unit includes an instruction decoding and calculation control circuit, an adaptive compression memory read and write control circuit, and a neural network calculation circuit; The feature map interaction bus controller and the instruction and network parameter interaction bus controller mount the neural network computing circuit on the on-chip data bus and communicate with the processor core through the configured on-chip bus interaction protocol; The feature map interaction bus controller is used to interact the input feature map of the neural network to the inside of the circuit, and interact the neural network operation results to the outside of the neural network calculation circuit; wherein the input feature map of the neural network is the input feature map after performing energy attention compression and neural network attention compression; The instruction and network parameter interaction bus controller is used to interact the layer instructions, network weights, network bias values, and weight compression indexes required for neural network operations into the circuit; The interaction request and control circuit is used to generate an interaction request signal to the feature map interaction bus controller and the instruction and network parameter interaction bus controller. After receiving the interaction request signal, the two interaction bus controllers will start the interaction, interacting the relevant data required for the neural network operation to the inside of the neural network calculation circuit, or interacting the neural network operation results to the outside of the neural network calculation circuit; the interaction request and control circuit also generates an interaction control signal for controlling the working mode of the input data stream distribution circuit; The input data stream distribution circuit is used to decode the data streams interactively input by the feature map interactive bus controller and the instruction and network parameter interactive bus controller, and distribute the decoded input feature map data and network parameter data to the adaptive compression memory read and write control circuit, and distribute the decoded instruction data to the instruction decoding and operation control circuit; The interactive input data stream includes the input feature map data stream and the instruction and network parameter data stream; the input feature map data stream format fields include: feature map frame size, frame channel frame header and feature map data stream carrier; the instruction and network parameter data stream fields include: instruction data, bias value data, compression index data and weight data; The instruction decoding and operation control circuit is used to perform instruction correction on the decoded feature map size signal in the instruction data from the input data stream distribution circuit according to the current layer feature map size signal sent by the adaptive compression memory read-write control circuit when calculating the neural network layer by layer, and then generate operation and read-write control signals and send them to the adaptive compression memory read-write control circuit and the neural network operation circuit to control the memory read-write state of the adaptive compression memory read-write control circuit and the operation state of the neural network operation circuit; The adaptive compression memory read-write control circuit includes: a feature map random access memory, a weight / bias value / compression index random access memory, a memory read-write control circuit, an MN selector, a detector, and a sequence 1 counter; The feature map random access memory is used to store the input and output feature map data of each network layer of the neural network, and is accompanied by a frame size register and a frame channel register. The two registers are used to save the frame size and number of frame channels of the input feature map from the frame header of the input feature map data stream, generate the current feature map size signal and send it to the instruction decoding and operation control circuit to correct the decoded instruction data, and extract the operation output size signal from the operation and read-write control signal issued by the operation core data stream controller and save it; The weight / bias value / compression index random access memory is used to store weight data, bias value data and compression index data of each network layer of the neural network; The read and write modes of the adaptive compression memory read and write control circuit include compressed reading and non-compressed reading and writing; In the non-compressed read / write mode, the memory read / write control circuit reads the required input feature map data from the feature map random access memory, reads the required weight data and bias value data from the weight / bias value / compression index random access memory, and sends the read data to the neural network operation circuit for operation, and the neural network operation circuit then writes the operation result back to the feature map random access memory; In the compressed read / write mode, the memory read / write control circuit first reads a binary compressed index sequence of length M from the weight / bias value / compression index random access memory, and then sends the binary compressed index of length M to the sequence 1 counter and the leading 1 detector respectively. The sequence 1 counter is used to calculate the binary compressed index "1" data and send the calculation result as the compressed weight length N to the memory read / write control circuit. The detector is responsible for finding the location information of N "1" data from the M-length compressed sequence and sending it to the MN selector; After receiving the compressed weight length N, the memory read-write control circuit reads the N compressed weight data and M feature map data, and sends the currently read M feature map data to the MN selector. The MN selector selects the feature map data of N specified positions from the M feature map data based on the received N position information and sends them to the memory read-write control circuit; the memory read-write control circuit sends the read N feature map data and N weight data to the neural network operation circuit for calculation; after the current network layer operation is completed, the memory read-write control circuit writes the operation result output by the neural network operation circuit into the feature map random access memory as the input feature map of the next layer of network.

2. The adaptive compressed neural network computing circuit based on the energy attention mechanism according to claim 1, characterized in that: Energy attention compression is specifically as follows: The energy of a fixed-length input signal in a certain time frame is calculated by accumulating the mean square value or absolute value, and the energy result is compared with a preset energy threshold. The input signal of the time frame that reaches the energy threshold is retained.

3. The adaptive compressed neural network computing circuit based on energy attention mechanism according to claim 1, characterized in that: Neural network attention compression is specifically: Obtaining the attention score and signal mask of the input signal that reaches the capacity threshold through a preset attention neural network; wherein the length of the signal mask is consistent with the length of the input signal; The obtained attention score is compared with the preset attention threshold. For input signals that reach the attention threshold, the input signal is multiplied by its signal mask, and the result is stored in the specified buffer as the input feature map cache of the neural network.

4. The adaptive compressed neural network computing circuit based on energy attention mechanism according to claim 1, characterized in that: The instruction decoding and operation control circuit includes an instruction decoding unit, an instruction correction unit, and an operation core data flow controller; The instruction decoding unit decodes the instruction data into an uncorrected decoded instruction, and compares the decoded feature map size signal in the uncorrected decoded instruction with the current feature map size signal. If the two are the same, instruction correction is not performed, and the uncorrected decoded instruction is directly used as the decoded instruction sent to the operation core data flow controller; otherwise, instruction correction is performed, and the corrected decoded instruction is then used as the decoded instruction sent to the operation core data flow controller. After receiving the decoding instruction, the operation core data flow controller generates operation and read / write control signals to control the memory read / write state of the adaptive compression memory read / write control circuit and the operation state of the neural network operation circuit; Among them, the instruction correction is: calculate the difference in the shape of the input and output feature maps of the current layer according to the decoded feature map size signal, and calculate the operational output feature map size of the current layer according to the feature map shape difference and the current feature map size signal in the instruction correction unit; correct the decoding instruction according to the three signals of shape difference, current feature map size, and operational output feature map size of the current layer.

5. The adaptive compressed neural network computing circuit based on energy attention mechanism according to claim 1, characterized in that: The working logic of the input data stream distribution circuit is as follows: when interactively inputting feature map data streams, the frame counter counts the data streams according to the data in the frame header, and the input interaction is completed when the specified number is reached; when interactive instructions and network parameter data streams are input, the frame decoding circuit identifies the frame header data and sends the data stream with the specified number of field contents according to the fields set by the interactive instructions and network parameter data streams to the designated circuit.

6. The adaptive compressed neural network computing circuit based on energy attention mechanism according to claim 1, characterized in that: When the input data stream distribution circuit interactively inputs the feature map data stream, it regards it as weight data for frame decoding and data distribution.

7. The adaptive compressed neural network computing circuit based on energy attention mechanism according to claim 1, characterized in that: The neural network operation circuit includes an input feature map buffer, a weight and bias value buffer, an operation unit input distribution circuit, a plurality of operation units, an activation function operation circuit, a pooling layer operation circuit and a quantization scaling / shifting circuit; The feature map buffer is used to store the feature map data input from the adaptive compression memory read and write control circuit; The weight and bias value buffer is used to store the weight and bias values ​​input from the adaptive compression memory read and write control circuit; The arithmetic unit input distribution circuit reads data from two buffers and dynamically distributes it to different arithmetic units for calculation through multiple batches according to the network layer structure and feature map size; The operation unit then sends the operation results to the quantization scaling / shift circuit, activation function operation circuit, and pooling layer operation circuit for calculation in sequence, and returns the final operation results to the adaptive compression memory read and write control circuit.

8. The adaptive compressed neural network computing circuit based on energy attention mechanism according to claim 7, characterized in that: The neural network operation circuit also includes a softmax operation circuit that performs softmax operation independently, which is used to perform softmax operation on the operation result sent by the operation unit, and then send the operation result to the subsequent quantization scaling / shift circuit.

Images