Hardware accelerator for full-dimensional attention mechanism of convolutional neural network

Abstract

The invention discloses a convolutional neural network-oriented hardware accelerator for a full-dimensional attention mechanism, and the hardware accelerator comprises a convolution operation unit which is used for carrying out the convolution operation of an input feature map; the full-dimensional attention unit is used for performing full-dimensional attention operation on the feature map output by the convolution operation unit; and the fusion unit is used for fusing the feature maps output by the full-dimensional attention unit to obtain a final output feature map. Channel and space horizontal and vertical dimension feature expressions are enhanced at the same time through a three-branch structure, the parameter quantity is reduced through grouping and channel shuffling, a 16-channel parallel architecture is designed based on an FPGA, DSP resource utilization is optimized, multiplication of two weight values and an activation value is achieved through a single DSP, and the method has the advantages of being simple in structure, convenient to operate and high in practicability. And a special attention module circuit is designed to be seamlessly embedded into the convolutional neural network, so that the method has advantages in the aspects of calculation performance, energy efficiency ratio, DSP efficiency and the like, and reference is provided for full-dimensional attention embedding application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a hardware accelerator for a full-dimensional attention mechanism of a convolutional neural network. BACKGROUND

[0002] In the development process of convolutional neural networks, attention mechanisms gradually become key technologies for improving model representation capabilities by simulating the information filtering characteristics of human vision. The evolution path presents the characteristics of optimization from a single dimension to multi-dimensional fusion, but there are still significant limitations in each stage of technology.

[0003] Channel attention mechanisms (represented by SENet) compress the spatial dimension of the feature map through global average pooling operations, learn the dependency between channels through fully connected layers, and realize feature recalibration in the channel dimension. For example, the "Squeeze-Excitation" operation of SENet can adaptively generate weights for each channel, and strengthen the feature channels with rich semantic information. However, this type of mechanism only focuses on "which features to pay attention to", and ignores the position information in the spatial dimension, resulting in a lack of sensitivity of the model to the spatial distribution of features. In tasks such as target detection and semantic segmentation that require positioning information, it is difficult to capture the spatial correlation of key regions of the image.

[0004] Spatial attention mechanisms (represented by CBAM) focus on "paying attention to the spatial position of features", and extract attention weights in the spatial dimension through large-size convolution kernels such as 7x7. For example, the spatial branch of CBAM generates a two-dimensional mask through convolution operations to emphasize key regions (such as object outlines) in the image. However, the introduction of large convolution kernels inevitably leads to an explosion in the number of parameters. In addition, relying only on local convolution operations in the spatial dimension makes it difficult to establish long-distance spatial dependency relationships, limiting the model's ability to integrate global features in complex scenes.

[0005] Hybrid attention mechanisms (represented by TA) attempt to combine the advantages of channel and spatial dimensions, and realize cross-dimensional interaction through a three-branch structure: two branches rotate the feature map and use 7x7 convolution to capture the correlation between the channel and the horizontal / vertical direction of the space, and the other branch directly extracts spatial attention. Although TA first attempts to build dynamic associations between channels and spaces, its core defect is the significant increase in computational complexity. In addition, this mechanism does not model the horizontal and vertical directions of the spatial dimension differently, resulting in insufficient detail in feature expression and difficulty in fully exploiting the feature interaction potential of three-dimensional space.

[0006] Through the entire development process, it can be found that the existing attention mechanisms have the following common defects:

[0007] Firstly, the one-sidedness of dimension optimization - only single or simple combination of channel and spatial dimensions, unable to simultaneously optimize the channel, spatial horizontal and vertical directions, and unable to fully utilize the interaction information of feature maps in the CxHxW three-dimensional space; secondly, the bottleneck of calculation efficiency - the fully connected layer of channel attention, the large convolution kernel of spatial attention, and the cross-dimensional operation of hybrid attention all lead to a significant increase in model parameter quantity and calculation quantity, making it difficult for the model to be deployed in lightweight devices; thirdly, the lack of hardware adaptability - without considering the parallel characteristics of hardware accelerators (such as input channel parallelism design of FPGA), the calculation mode of attention modules does not match the hardware architecture, resulting in a significant decrease in the efficiency of the embedded accelerator.

[0008] Chinese invention patent application with publication number CN117746123A discloses a high-resolution remote sensing scene recognition method based on splicing convolution and attention mechanism, which performs double-branch fusion based on coordinate attention and channel attention, but the spatial dimension of this technical solution is not independently separated and processed, limiting the precision of feature extraction. SUMMARY

[0009] The present application aims to solve the problems of one-sided dimension optimization, calculation efficiency bottleneck and lack of hardware adaptability in the prior art.

[0010] The technical solution adopted by the present application to solve its technical problems is to provide a hardware accelerator for full-dimensional attention mechanism of convolutional neural network, comprising:

[0011] a convolution operation unit for performing convolution operation on input feature maps;

[0012] a full-dimensional attention unit for grouping the feature maps output by the convolution operation unit into G groups of sub-feature maps along the channel dimension; each group of sub-feature maps respectively performs channel attention, spatial vertical attention and spatial horizontal attention to obtain three attention matrices;

[0013] a fusion unit for fusing the three attention matrices, then performing weighted calculation on the input sub-feature maps based on the fused attention matrices to obtain a fusion result, and combining the G groups of fusion results to obtain the final output feature map;

[0014] Wherein, each group of sub-feature maps respectively performs channel attention, spatial vertical attention and spatial horizontal attention to obtain three attention matrices, including the following steps:

[0015] The channel attention branch receives the sub-feature map, constructs attention in the channel dimension direction by using a channel attention module; the vertical spatial attention branch receives the sub-feature map, performs dimension transformation first, and then constructs attention in the vertical direction of the spatial dimension by using a first spatial attention module; and the horizontal spatial attention branch receives the sub-feature map, performs dimension transformation first, and then constructs attention in the horizontal direction of the spatial dimension by using a second spatial attention module.

[0016] Preferably, the channel attention branch receives the sub-feature map, constructs attention in the channel dimension direction by using a channel attention module, and is expressed as:

[0017] ;

[0018] ;

[0019] ;

[0020] ;

[0021] wherein, represents an input sub-feature map, , , respectively represent a global average pooling, a full connection and an activation operation, represents dimension adjustment, represents adjusting the dimension to be consistent with the bracket; , and are intermediate calculation results, represents a first attention matrix output by the channel attention branch.

[0022] Preferably, the vertical spatial attention branch receives the sub-feature map, performs dimension transformation first, and then constructs attention in the vertical direction of the spatial dimension by using a first spatial attention module, and is expressed as:

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] wherein, represents an input sub-feature map; the function is used to change the dimension order of a tensor; The function represents a mean operation; and represents a learnable weight and a learnable parameter attached to each point along the vertical direction of the spatial dimension; represents a Sigmoid activation function, The function can adjust the dimension to be the same as in the parentheses; , , and are intermediate calculation results, represents the second attention matrix output by the vertical spatial attention branch.

[0029] Preferably, the horizontal spatial attention branch receives the sub-feature map, first performs dimension transformation, and then uses the second spatial attention module to construct attention in the horizontal direction of the spatial dimension, represented as:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] wherein, represents the input sub-feature map, The function is used to change the dimension order of the tensor, The function represents a standard deviation operation, and represents a learnable weight and a learnable parameter attached to each point along the horizontal direction of the spatial dimension, represents a Sigmoid activation function, The function can adjust the dimension to be the same as in the parentheses; , , and are intermediate calculation parameters, represents the third attention matrix output by the horizontal spatial attention branch.

[0036] Preferably, the fusion unit fuses the three attention matrices, then performs weighted calculation on the input sub-feature map based on the fused attention matrices to obtain a fusion result, and combines a plurality of fusion results to obtain a final output feature map, including the following steps:

[0037] Feature fusion, feature fusion based on the fused attention matrix, represented as:

[0038] ;

[0039] ;

[0040] wherein OD represents a synthesis result corresponding to one attention matrix, represents a calculation of matrix multiplication at corresponding positions, is a fusion result;

[0041] channel reorganization, splitting the G groups of into a two-dimensional structure of “group number x channel number per group”;

[0042] dimension transposition, exchanging the dimension order of the group number and the channel number;

[0043] structure flattening, restoring the two-dimensional structure after transposition to one-dimensional representation.

[0044] Preferably, the hardware accelerator is designed based on an FPGA development board, and adopts input channel inter-parallel, output channel inter-parallel and convolution window inter-parallel operation;

[0045] The number of input channels and the number of output channels are both , n=0, 1, 2, 3, 4;

[0046] The convolution window size of the convolution operation unit is 3x3.

[0047] Preferably, the hardware accelerator uses 16 input channel inter-parallel and 16 output channel inter-parallel.

[0048] Preferably, the channel attention module comprises, in sequence, a global average pooling submodule, a first full connection submodule, a ReLU activation function, a second full connection submodule and a first Sigmoid activation function.

[0049] The first full connection submodule adopts 16 input channels and 4 output channels, and the second full connection submodule adopts 4 input channels and 16 output channels.

[0050] Preferably, the attention module in the vertical direction of the spatial dimension comprises, in sequence, an average value submodule, a multiplication-addition submodule and a second Sigmoid activation function.

[0051] Preferably, the attention module in the horizontal direction of the spatial dimension comprises, in sequence, a standard deviation submodule, a multiplication-addition submodule and a third Sigmoid activation function.

[0052] The application has the following beneficial effects: the application simultaneously enhances the channel, space horizontal and vertical dimension feature expression through the three-branch structure, reduces the parameter quantity by using grouping and channel mixing, designs a 16-channel parallel architecture based on FPGA, optimizes the DSP resource utilization, realizes the multiplication of two weight values and an activation value with a single DSP, and designs a special attention module circuit to seamlessly embed a convolutional neural network, and the hardware accelerator designed in the embodiment of the application has advantages in computing performance, energy efficiency ratio and DSP efficiency, thereby providing a reference for full-dimensional attention embedding applications.

[0053] The application will be further described in detail below with reference to the drawings and embodiments, but the application is not limited to the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A schematic diagram of the full-dimensional attention mechanism of the embodiment of the application;

[0055] Figure 2 A schematic diagram of the attention in the channel dimension direction of the embodiment of the application;

[0056] Figure 3 A schematic diagram of the attention in the vertical direction of the spatial dimension of the embodiment of the application;

[0057] Figure 4 A schematic diagram of the attention in the horizontal direction of the spatial dimension of the embodiment of the application;

[0058] Figure 5 A schematic diagram of the three-branch synthesis of the full-dimensional attention of the embodiment of the application;

[0059] Figure 6 A schematic diagram of the channel mixing of the embodiment of the application;

[0060] Figure 7 A schematic diagram of the overall architecture of the hardware accelerator of the embodiment of the application;

[0061] Figure 8 A schematic diagram of the inter-input channel parallelism of the embodiment of the application;

[0062] Figure 9 A schematic diagram of the inter-output channel parallelism of the embodiment of the application;

[0063] Figure 10 A schematic diagram of the intra-convolution window parallelism of the embodiment of the application;

[0064] Figure 11 A schematic diagram of the DSP multiplication of the embodiment of the application;

[0065] Figure 12 A structural schematic diagram of the attention module in the channel dimension direction of the embodiment of the application;

[0066] Figure 13 The structural schematic diagram of the global average pooling submodule of the embodiment of the application is shown in the figure.

[0067] Figure 14 The structural schematic diagram of the ReLU activation and ReLU6 activation function of the embodiment of the application is shown in the figure.

[0068] Figure 15 The structural schematic diagram of the fully connected submodule of the embodiment of the application is shown in the figure.

[0069] Figure 16 The structural schematic diagram of the Sigmoid function structure principle of the embodiment of the application is shown in the figure.

[0070] Figure 17 The structural schematic diagram of the attention module in the vertical direction of the spatial dimension of the embodiment of the application is shown in the figure.

[0071] Figure 18 The structural schematic diagram of the multiplication-addition module of the embodiment of the application is shown in the figure.

[0072] Figure 19 The structural schematic diagram of the attention in the vertical direction of the spatial dimension of the embodiment of the application is shown in the figure.

[0073] Figure 20 The structural schematic diagram of the mean module in the vertical direction of the space of the embodiment of the application is shown in the figure.

[0074] Figure 21 The structural schematic diagram of the attention module in the horizontal direction of the spatial dimension of the embodiment of the application is shown in the figure.

[0075] Figure 22 The structural schematic diagram of the standard deviation module of the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0076] Referring to Figure 1 The structural schematic diagram of the full-dimensional attention mechanism of the embodiment of the application is shown in the figure, and for a given feature map N, C, H, and W respectively represent the batch size, the number of channels, the height of the spatial dimension, and the width of the spatial dimension. The original feature map is divided into G groups along the channel, , Each group has 16 channels, and each sub-feature map has three branches. One branch does not perform dimension transformation, calculates the attention in the channel dimension direction, and the other two branches respectively perform dimension transformation, calculate the attention in the vertical direction and the horizontal direction of the spatial dimension, obtain the attention weight values of the three branches, and simultaneously weight the grouped sub-feature maps, and finally mix the channels of all the sub-feature maps.

[0077] Referring to Figure 2The first branch is shown to group the sub-feature maps and construct the attention in the channel dimension direction by using squeeze-and-excitation operation. As shown in Figure 2 , the original sub-feature maps are compressed into feature maps with dimension by using global average pooling operation. The global information is collected by using the squeeze operation of global average pooling, and the feature maps with size are squeezed into feature maps with size , which reduces the parameter amount in the process of calculating attention. Then, the full connection operation is performed, and a Sigmoid function is followed to capture the relationship between channels and output an attention vector with dimension . Finally, the dimension of the attention vector is expanded to to obtain the attention vector that can represent the channel dimension direction. The specific calculation process of the first branch is represented by equations (1) to (4) as follows:

[0078] (1)

[0079] (2)

[0080] (3)

[0081] (4)

[0082] wherein , , represent global average pooling, full connection and activation operation respectively, represents dimension adjustment, adjusts the dimension to be consistent with the bracket.

[0083] Referring to Figure 3 , the second branch is shown to perform dimension transformation on the grouped original sub-feature maps to become . The global average pooling operation is used on the dimension-transformed feature maps to compress them into feature maps with dimension . The average pooling along the vertical direction of the spatial dimension compresses the feature maps with size CxW in each vertical direction into one pixel point, and each pixel point contains the information of the horizontal direction (W) of the spatial dimension and the channel dimension direction (C). The values of different pixel points respectively represent the importance of different pixel points in the vertical direction (H) of the spatial dimension. A Sigmoid function with a gating mechanism is followed to obtain an attention vector with dimension The gating mechanism adds a learnable weight and a learnable bias to each point along the vertical direction of the spatial dimension, and the role of the gating mechanism is to learn the importance of the pixel points along the vertical direction of the spatial dimension. After the above operation, the dimension of the feature map is expanded to , and finally the feature map with the dimension of is dimensionally transformed back to the dimension of , to obtain the attention that can represent the vertical direction of the spatial dimension. The specific calculation process of the second branch is represented by formulas (5) to (9) as follows:

[0084] (5)

[0085] (6)

[0086] (7)

[0087] (8)

[0088] (9)

[0089] wherein, is a function for changing the dimension order of a tensor, is a function representing the mean operation, and represent the learnable weight and the learnable parameter added to each point along the vertical direction of the spatial dimension, represents a Sigmoid activation function, is a function that can adjust the dimension to the same as in the parentheses.

[0090] Referring to FIG. 5, the third branch dimensionally transforms the grouped original sub-feature map Figure 4 into . The standard deviation of the horizontal direction is calculated for the dimensionally transformed feature map , and is compressed into a feature map with the dimension of . For the input tensor (N is the batch size, C is the number of channels, H and W are the height and width), the mean formula is formula (10), and the standard deviation formula is formula (11).

[0091] (10)

[0092] (11)

[0093] wherein represents a pixel point, represents a mean value, represents a variance. ​

[0094] The standard deviation is calculated by averaging all data for each channel, adding global second-order information and enriching the resulting information. The standard deviation along the horizontal spatial dimension compresses each C×H feature map into a single pixel. Each pixel contains information from both the vertical (H) and channel dimensions of the feature map, with different pixel values ​​representing the importance of different pixels in the horizontal (W) spatial dimension. A sigmoid function with a gating mechanism is then applied after the standard deviation to obtain a dimension... The feature map is then used. In the gating mechanism, attaching learnable weights and learnable biases to each point along the horizontal direction of the spatial dimension helps learn the importance of pixels in that direction. Then, the feature map dimension is expanded to... Finally, the dimension is The feature map is then transformed back to its original dimension. Dimension, to obtain attention that can represent the horizontal dimension of space. The specific calculation process for the second branch is represented by formulas (12) to (16), as follows:

[0095] (12)

[0096] (13)

[0097] (14)

[0098] (15)

[0099] (16)

[0100] in, The function is used to change the dimensional order of a tensor. The function represents the operation of calculating the standard deviation. and This represents the learnable weights and learnable parameters assigned to each point along the horizontal direction of the spatial dimension. This represents the Sigmoid activation function. The function can adjust the dimensions to be the same as those within the parentheses.

[0101] See Figure 5 As shown, the calculation process of synthesizing full-dimensional attention is represented by formula (17).

[0102] (17)

[0103] in, , and respectively represent vectors of three-branch attention output.

[0104] Since the original feature map is divided into G groups of sub-feature maps, attention is calculated for each group of sub-feature maps, and finally the attention of each group is only calculated by the features of a part of the input channel sub-feature maps, which hinders the flow of information between different channels. Channel shuffling operation is used to realize the flow of cross-group information in the channel dimension.

[0105] Referring to Figure 6 As shown, the channel shuffling operation includes three processing steps:

[0106] Channel reorganization, which splits the input channel into a two-dimensional structure of (group number x channel number per group);

[0107] Dimension transposition, which exchanges the dimension order of the group number and the channel number;

[0108] Structure flattening, which restores the two-dimensional structure after transposition to one-dimensional representation.

[0109] Referring to Figure 7 As shown, the overall architecture of the hardware accelerator of the embodiment of the application is shown, and the running process of the accelerator system is as follows:

[0110] 1) The 8-bit unsigned image data is moved from the SD card to the DDR of the PS end, and then the image data in the DDR of the PS end is moved to the PL end DDR through the AXI protocol;

[0111] 2) The ARM hard core processor reads the 8-bit signed weight data and 8-bit signed attention weight data of the convolutional neural network after IAO quantization from the SD card to the DDR of the PS end;

[0112] 3) The ARM hard core processor writes the control instruction to the register stack of the PL end;

[0113] 4) The ARM hard core processor sends a data preparation completion signal to the PL end, and the PL end starts running;

[0114] 5) The PL end moves the image data in the DDR of the PL end of the current layer to the input buffer, moves the weight data in the DDR of the PS end to the weight buffer, and stores the image data obtained by the convolution operation in the attention input buffer in the layer where attention is needed. Then start the forward inference of the convolutional neural network, until the calculation of the current layer is completed, and the result is cached in the output buffer, and then the image data in the output buffer is moved to the DDR of the PL end;

[0115] 6) Repeat the steps in 5), until the calculation of all layers of the convolutional neural network is completed, the result is obtained and written to the DDR of the PL end, and the ARM reads the result in the DDR of the PL end.

[0116] The control unit is a "central scheduling core" of the entire FPGA hardware accelerator, responsible for coordinating the collaborative work of all sub-modules, and ensuring that the full-dimensional attention embedded convolutional neural network (CNN) can efficiently complete the forward inference according to the predetermined process, including:

[0117] State initialization and standby: after system reset, the control unit automatically enters the IDLE default state, waits for the ARM processor to send the "data preparation completion signal" (star_pre), at this time all sub-modules (convolution operation, attention calculation, cache) are in a ready state, avoiding invalid power consumption;

[0118] Data preparation phase trigger: after the ARM processor completes the image data (PL_DDR3) and weight data (PS_DDR3) transfer, the control unit receives the "star_pre" signal, switches to the PREPARE state, triggers the DDR controller and off-chip access module, and loads the required input feature map (stored in the input cache), convolution weight (stored in the weight cache), and attention parameter (stored in the attention input cache) into the on-chip.

[0119] Convolution and attention operation coordination: after data preparation is completed, the control unit switches to the CONV state, sends a "start signal" to the convolution operation module to start 16 input / output channel parallel convolution calculation; when the convolution operation is completed (conv_over signal is pulled high), the control unit reads the configuration bit (bit 127) in the register stack "whether to pass through the attention module": if attention calculation is required (configuration bit = 1), switch to the ATTENTION state, start the attention calculation module (first calculate C / H / W three-branch weight, then perform weighting), and after attention is completed (attention_over signal is pulled high) Enter the CONV_END state; if no attention is required (configuration bit = 0), directly enter the CONV_END state;

[0120] Multi-layer inference loop and termination: in the CONV_END state, the control unit determines whether the number of completed layers is equal to the total number of layers through the "network layer number counter" (register stack bits 121-126): if not completed, return to the CONV state and continue the calculation of the next layer; if completed, return to the IDLE state, send the "inference end signal" to the ARM processor, and wait for the next inference task.

[0121] Specifically, the accelerator of the embodiment of the present application adopts a parallel strategy. Through analysis of the algorithm, it can be found that there are four parallel operations in convolution operation: input channel parallel, output channel parallel, convolution window parallel, and convolution window parallel.

[0122] Referring toFigure 8 Figure 1 is a schematic diagram of inter-input-channel parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 2 is a schematic diagram of inter-output-channel parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 3 is a schematic diagram of intra-convolution window parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 4 is a schematic diagram of inter-input-channel parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 5 is a schematic diagram of inter-output-channel parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 6 is a schematic diagram of intra-convolution window parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map.

[0123] Figure 1 is a schematic diagram of inter-input-channel parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 9 Figure 5 is a schematic diagram of inter-output-channel parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 6 is a schematic diagram of intra-convolution window parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 5 is a schematic diagram of inter-output-channel parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map.

[0124] Figure 1 is a schematic diagram of inter-input-channel parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 10 Figure 6 is a schematic diagram of intra-convolution window parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 6 is a schematic diagram of intra-convolution window parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map. Figure 6 is a schematic diagram of intra-convolution window parallelism, one input feature map is convoluted with one convolution kernel to generate one output feature map.

[0125] The embodiment of the present application simultaneously adopts the three parallel operation modes of inter-input-channel parallelism, inter-output-channel parallelism and intra-convolution window parallelism.

[0126] The hardware DSP resource utilization analysis of the embodiment of the application: the DSP (Digital Signal Processor) on the FPGA platform is a hardware logic resource specially designed for intensive operations such as multiplication and addition. Using the DSP resource can avoid occupying a large number of LUT (Look-Up-Table) resources, and is more efficient than general logic to achieve the same function, and has less dynamic power consumption. The embodiment of the application uses the FPGA development board of the XC7Z100-2FFG900 model of Xilinx Company to design hardware acceleration, and the development board has 2020 DSP resources. The DSP of this development board is DSP48E1, which supports 25x18-bit multiplication operation.

[0127] The embodiment of the application uses an 8-bit quantization deployment strategy to quantize the activation value of the convolutional neural network model to 8-bit unsigned numbers and quantize the convolution kernel weight data to 8-bit signed numbers. The convolution operation is 8-bitx8-bit multiplication, and 16-bit signed data results are obtained. In order to fully utilize the DSP resources in the hardware, the weight value data can be adjusted to calculate the multiplication of one activation value and multiple weight values in one DSP resource.

[0128] As shown in Figure 11 , it is a schematic diagram for calculating the multiplication of one 8-bit activation value and two 8-bit weight values in one DSP resource. The first weight value data is arranged in the low 8 bits of the 25-bit space of the DSP, and the second weight value data is arranged in the 16th-23rd bits of the 25-bit space of the DSP, and the remaining positions are filled with 0. The activation value data is arranged in the low 8 bits of the 18-bit space of the DSP, and the remaining positions are filled with 0. The low 16 bits of the DSP output result are the multiplication result of the first weight value and the activation value, that is, , and the next low 16 bits of the DSP output result are the multiplication result of the second weight value and the activation value, that is, . Since the sign bit of the result of signed number multiplication may be 1, if the result obtained by is a negative number, the 1 of the sign bit of the low bit result will affect the result obtained by , so that obtained result has an error from the true result. For the case that is a negative number, an error correction operation is needed to correct the result of to . Using this method, the data arrangement of the weight value data can be performed to calculate the multiplication of one activation value and two weight values in one DSP. Compared with the default state that one DSP resource can only calculate the multiplication of one activation value and one weight value, half of the DSP resources are saved, that is, a larger parallel degree can be used under the condition of the same DSP resource, and the inference speed of the convolutional neural network model is accelerated.

[0129] The parallelism of the neural network hardware accelerator is mainly affected by two factors, the network model and the hardware resources. The goal of the present application is to maximize the use of hardware resources, minimize data transfer time, and calculate as much data as possible at the same time. Currently, the main convolution kernel size of the mainstream convolutional neural network is a 3x3 convolution kernel. Convolution kernels larger than 3x3 can be spliced by multiple 3x3 convolution kernels, such as the 7x7 convolution kernel in ResNet-18, which can be seen as 9 3x3 convolution kernels spliced. Convolution kernels smaller than 3x3 can also reuse 3x3 convolution kernels to achieve. Therefore, the embodiment of the present application uses a convolution window intra-parallel method with a parallelism of 9 for a 3x3 convolution kernel. It has been analyzed above that the inter-parallelism of the convolution window is 9, assuming that the inter-parallelism of the input channels is , and the inter-parallelism of the output channels is , then the number of multipliers required is From the previous subsection "Hardware DSP resource utilization analysis", it can be seen that using a method of data arrangement for weight value data can achieve the calculation of the multiplication of one activation value and two weight values in one DSP, i.e. one DSP is used as two multipliers. In summary, the DSP resources required are The FPGA development board used in the embodiment of the present application has 2020 DSP resources of the XC7Z100-2FFG900 model of Xilinx Corporation, so it can be derived that Currently, the input channel number and the output channel number of the convolutional neural network are generally , so there is the formula , If the input channel number and the output channel number are not consistent, it will cause the channel number of the output feature map result calculated by the current convolution layer to be inconsistent with the channel number of the input feature map data required to enter the next layer. This will require additional design of a data rearrangement module, which will increase the on-chip hardware resources, and data rearrangement will take time, affecting the performance of the accelerator. and , it can be derived that The maximum is 16. The hardware accelerator uses a strategy of 16 input channel inter-parallelism, 16 output channel inter-parallelism, and 3x3 convolution window intra-parallelism, using a total of 1152 DSP resources.

[0130] This invention proposes a "full-dimensional attention mechanism" and embeds it into classic models such as ResNet-18 and MobileNet-V2. Therefore, a dedicated module needs to be designed for full-dimensional attention. Since full-dimensional attention is not embedded in every layer of the convolutional neural network, the designed full-dimensional attention module needs to provide on / off functionality. This module is enabled in layers where attention is needed and disabled in layers where attention is not needed. For the convolutional output feature map of a layer, three branches are used to simultaneously calculate attention in the three dimensions of the output feature map, generating attention weights. These weights are then added to the output feature map to obtain the corrected output feature map. Considering the characteristics of full-dimensional attention, the hardware module is also designed with three branches, and finally, the three branches are combined to obtain the full-dimensional attention module.

[0131] See Figure 12 As shown, this is the attention module along the channel dimension. The input passes through a global average pooling submodule, then through a 16-input-channel, 4-output-channel fully connected submodule, followed by a ReLU activation function submodule to introduce non-linearity, then through a 4-input-channel, 16-output-channel fully connected submodule, and finally through a Sigmoid activation function to output the result. The global average pooling submodule, ReLU activation, ReLU6 activation functions, and fully connected submodule are described in [reference needed]. Figures 12 to 15 As shown, in the channel-dimensional attention module, the 16-channel 8-bit quantized sub-feature map output by the convolution operation module is first fed into the global average pooling sub-module (outputting a 16-channel 8-bit global mean). This output is then fed into the first fully connected sub-module with 16 input channels and 4 output channels (outputting a 4-channel 32-bit result). The output of the first fully connected sub-module is fed into the ReLU activation function sub-module (outputting a 4-channel 32-bit nonlinear result). The output of the ReLU sub-module is fed into the second fully connected sub-module with 4 input channels and 16 output channels (outputting a 16-channel 32-bit result). All sub-modules are scheduled by the synchronization control signals (such as enable and completion signals) of the control unit. Finally, the output of the second fully connected sub-module is fed into the Sigmoid sub-module to generate 16-channel 8-bit channel attention weights.

[0132] Specifically, the input end of the global average pooling submodule is connected to the output end of the convolution operation module. The input interface of the global average pooling submodule receives the 16-channel parallel 8-bit quantized feature map signal (after grouping, the sub-feature map is 16 channels / group, the quantization bit width is 8 bits, and the dimension is N x 16 x H x W, N is the batch size, H / W is the spatial height / width) from the “convolution operation module output cache”, corresponding to 16 independent 8-bit data signal lines (each corresponding to a channel of feature map pixel stream), and receives the “pooling enable signal (gap_en)” and “feature map size configuration signal (H / W value, from the register stack 0-7)” sent by the control unit. The output interface of the global average pooling submodule outputs a 16-channel parallel 8-bit global average signal (N x 16 x H x W is compressed to N x 16 x 1 x 1, only one global average is retained for each channel, and the quantization bit width is kept at 8 bits to match the subsequent fully connected input), which is directly connected to the 16 8-bit input interface of the first fully connected submodule, and outputs the “pooling completion signal (gap_done)” to the control unit to trigger the subsequent fully connected operation.

[0133] Specifically, the input interface of the first fully connected submodule (i.e. Figure 15 ConvModule11_16_4) in the global average pooling submodule is one-to-one corresponding to the 16 8-bit output signal lines of the global average pooling submodule, and receives 4 groups of 16-channel 8-bit quantized weight signals (fully connected multiplexing 1 x 1 convolution array, 16 input channels corresponding to 16 weights, 4 output channels corresponding to 4 independent weights, weights stored in the PS end DDR and loaded to the on-chip register through the weight cache module) from the “weight cache module”, and the “fully connected enable signal (fc1_en)” sent by the control unit. The output interface of the first fully connected submodule outputs a 4-channel parallel 32-bit operation result signal (fully connected operation is 8-bit x 8-bit multiplication + accumulation, and the intermediate result uses 32-bit width to avoid overflow), which is directly connected to the 4 32-bit input interface of the ReLU activation function submodule, and outputs the “first fully connected completion signal (fc1_done)” to the control unit.

[0134] Specifically, the input interface of the ReLU activation function submodule corresponds to the 4-way 32-bit output signal line of the first full connection submodule one by one, and only receives 32-bit signed data (because the full connection accumulation result may be negative), while receiving the "activation enable signal (relu_en)" sent by the control unit (no additional configuration signal, because ReLU does not require parameters, only enable). The output interface of the ReLU activation function submodule outputs a 4-channel parallel 32-bit nonlinear processing result signal (ReLU sets negative values to 0, and non-negative values remain unchanged, without changing the channel number and bit width), which is directly connected to the 4-way 32-bit input interface of the "second full connection submodule", and outputs the "ReLU completion signal (relu_done)" to the control unit.

[0135] Specifically, the input interface of the second full connection submodule (i.e. Figure 15 ConvModule11_4_16) corresponds to the 4-way 32-bit output signal line of the ReLU activation function submodule one by one, while receiving 16 groups of 4-channel 8-bit quantized weight signals from the "weight cache module" (4 input channels correspond to 4 weights, 16 output channels correspond to 16 independent weights, matching the grouped 16 channel number), and the "full connection enable signal (fc2_en)" sent by the control unit. The output interface of the second full connection submodule outputs a 16-channel parallel 32-bit operation result signal (4→16 channel dimensionality increasing operation, the intermediate result still uses 32-bit bit width), which is directly connected to the 16-way 32-bit input interface of the Sigmoid activation function submodule of the "full-dimensional attention module", and outputs the "second full connection completion signal (fc2_done)" to the control unit.

[0136] Specifically, the input interface of the Sigmoid activation function submodule corresponds to the 16-way 32-bit output signal line of the second full connection submodule one by one, while receiving the "Sigmoid enable signal (sigmoid_en)" sent by the control unit (no additional parameters, Sigmoid uses piecewise linear approximation method, and hardware implementation does not require external parameters). The output interface of the Sigmoid activation function submodule outputs a 16-channel parallel 8-bit channel attention weight signal (Sigmoid normalizes the 32-bit result to the range of 0~1, and quantizes it to 8-bit unsigned number, matching the subsequent element-wise multiplication with 8-bit feature map), which is finally connected to the channel weight input interface of the "three-branch weight fusion module", and outputs the "channel attention completion signal (ca_done)" to the control unit, triggering the fusion operation of the H / W branch weight.

[0137] Specifically, the Sigmoid function is a transcendental function, and is usually calculated by approximate methods, such as lookup table method, Taylor expansion, piecewise linear approximation method, and piecewise nonlinear approximation method. H. Amin proposed a linear approximation function to approximate the Sigmoid function, with a maximum error of 0.02, and a very small error when the input is close to 0. The expression is shown in equation (18).

[0138] (18)

[0139] The structural principle diagram of the Sigmoid function is shown in Figure 16 , the range of the output result is , and the output is expressed in the form of . The input data_in becomes an absolute value through a selector, corresponding to in the formula. When , is expressed as a constant 1 in the form of , so that the output can be expressed as The sigmoid function structure realizes hardware acceleration approximate calculation of the sigmoid function through cooperation of a plurality of selectors, an adder, a subtractor, a register and a condition judging module. Circuit connection relationships can be divided into the following core links: (1) an input and sign selection link. An input signal data_in (an activation value to be processed) and data_in[sign] (a sign bit of data_in, 0 for positive and 1 for negative) are inputs of a first-level selector: if data_in[sign]=0 (the input is positive), data_in is selected to be output; if data_in[sign]=1 (the input is negative), -data_in (the data after inversion) is selected to be output. (2) a value adjustment and addition operation link. The output of the first-level selector is connected to an adder, and a second-level selector selects one value (according to the design logic to adapt to different segments) from "27, 5, 2" and adds the output: the adder completes the operation of "selector output + multi-path selection value", and the result is sent to a register for temporary storage. (3) an m_out generation link (core weight of segmented linearity). The register value of the adder output is connected to the condition judging module ">5?": if the result is greater than 5, a third-level selector selects "1" to be output; if the result is less than or equal to 5, a fourth-level selector selects "one" (a value selected from "0, 4, 8, 32") to be output. The above result is connected to a subtractor and is subtracted from m_out_reg (a historical value stored in the register), and then combined with data_in[sign] to select the final m_out: if data_in[sign]=0 (the input is positive), the subtraction result is selected to be output; if data_in[sign]=1 (the input is negative), the inverted value is selected to be output. (4) an n_out generation link (offset of segmented linearity). A fifth-level selector selects one value from "0, 2, 3, 5" to be directly output as n_out, which is used for offset calculation of the sigmoid approximate formula. (5) an enable and output valid link. An input enable signal data_in_en is delayed by a register, and the output is data_out_avl (a data output valid signal), which is used to synchronize the operation timing of subsequent modules. The overall logic: the circuit realizes the hardware flow of "sign judgment -> value adjustment -> segmented selection -> linear operation"

[0140] Referring to Figure 17 As shown in the figure, for the attention module in the vertical direction of the spatial dimension, the input passes through the average value submodule in the vertical direction of the spatial dimension, then passes through the multiply-add submodule, and finally passes through the sigmoid function to output the result. Among them, the multiply-add submodule multiplies the input value by a learnable parameter and adds a learnable parameter to obtain the result. The two learnable parameters are constant values after training and exist in the Bram; the multiply-add submodule is as follows Figure 18As shown, through the cooperation of BlockRAM (Bram), multiplier, register, and adder, the combined operation of "multiplication + addition" is realized, and the circuit connection relationship is as follows: (1) Data input link, two-way output of Bram: the Bram module outputs two-way data signals at the same time, one way is directly connected to the adder, and the other way is connected to the multiplier; external input mult_add_in: external data to be operated is connected to the multiplier through the mult_add_in interface and multiplied with one-way data output by Bram. (2) Operation and storage link, multiplier operation: the multiplier receives "Bram one-way output + mult_add_in", and after completing the multiplication operation, the result is sent to the register for temporary storage (for timing synchronization or data buffering); adder operation: the multiplication result output by the register is added in the adder with the other way data output by Bram. (3) Result output link: the operation result of the adder is output through the mult_add_out interface as the final result of the multiplication-addition sub-module. Functional logic: this module realizes the process of "providing storage weight / intermediate data by Bram → completing primary operation by multiplier → guaranteeing timing by register → completing final accumulation by adder". As shown in Figure 19 As shown, the mean value in the vertical direction of the spatial dimension is the mean value of all image data in the blue box, and the global average pooling submodule calculates the mean value of all image data in the red box. Therefore, for the image mean value in the vertical direction of the spatial dimension, an additional mean value submodule needs to be designed.

[0141] The full-dimensional attention divides the input image into a group according to 16 channels, which matches the parallel hardware between 16 input channels. The mean value can be represented by equation (19) and equation (20), and the cumulative results of each group are added again, and finally divided by CxW.

[0142] (19)

[0143] (20)

[0144] wherein, represents a pixel point of the input image, represents a pixel point along the width direction of the spatial dimension, represents a pixel point along the channel direction, represents the number of channels, represents the width of the spatial dimension.

[0145] The mean value submodule is as shown in Figure 20As shown, the addition results of 16 channels of data are calculated through the adder tree, and then the accumulated sum of W adder tree results is calculated through the accumulator. After img_size, i.e. W accumulated sums are calculated, the results are stored in the register, and then the second row of data is accumulated, until the Hth row of data is accumulated, obtaining H accumulated results. Finally, the H results are divided by CxW to obtain H mean values, i.e. the mean values in the vertical direction of the spatial dimension. The circuit connection relationship is as follows: (1) AdderTree_16_1: multi-input parallel summation. Input: 16 parallel hat_data series signals (such as hat_data0-hat_data15), every 4 inputs a bottom-level adder (4 blue adders in total); operation: the bottom-level adder output is temporarily stored in the register and then input into the top-level adder to complete the final summation; output: the summation result is output through the add_out interface and sent to the subsequent accumulation module. (2) Accumulation and counting module: cumulative sum and sample number. Accumulator: the add_out and the historical accumulation value stored in the register group regs are added to realize the accumulation of multiple rounds of summation; counting logic: through the condition judgment of cnt==img_size? (when the count reaches the image size img_size), the start and stop of the accumulator are controlled, and the number of valid samples is recorded. (3) Division unit (Div): mean value calculation. Input: the final sum of the accumulator (temporarily stored in the register) and the "count reciprocal" (or the proportion coefficient) output by the multiplier; operation: the sum is scaled through the multiplier first, and then the division operation is completed through the divider (symbol ∧∧); output: the final mean value result is output through the hat_out interface. (4) Timing and control link. The registers are used to realize timing synchronization between modules to ensure the data consistency of multiple rounds of operation; the enable logic (cnt==img_size?) of the counting module ensures that the mean value calculation is only performed when the sample number meets the requirement, avoiding invalid operation. Function logic: this module is aimed at 16 parallel inputs, first completes parallel summation through the adder tree, then accumulates multiple rounds of sums and counts the sample number, and finally obtains the mean value through the division unit

[0146] Referring to Figure 21 As shown, the attention module in the horizontal direction of the spatial dimension is similar to the attention in the vertical direction of the spatial dimension, and the difference lies in that the standard deviation is used instead of the mean value. The input passes through the standard deviation submodule in the horizontal direction, then passes through the multiply-add submodule, and finally outputs the result through the Sigmoid function.

[0147] The calculation formula of the standard deviation is shown in formula (21) and formula (22).

[0148] (21)

[0149] (22)

[0150] wherein , , represents a pixel point, represents the mean of these pixel points, represents the standard deviation of these pixel points.

[0151] As shown in Figure 22 , the standard deviation submodule is composed of a mean submodule in the horizontal direction of the spatial dimension, a square value submodule, and a square root value submodule. The mean module in the horizontal direction of the spatial dimension is similar in design to the mean submodule in the vertical direction of the spatial dimension. After the W mean values are calculated by the mean submodule, the 16-channel input data is subtracted from the mean values and squared, and the results are added. Then, the W square sum results are obtained through the accumulation submodule, and the final W standard deviation values are obtained through the square root submodule. The W standard deviation values are merged into an output port for output. The circuit connection relationship is as follows: (1) One, mean calculation link (W_Avg module), input: 16 parallel wat_data series signals (such as wat_data0[7:0]-wat_data15[7:0]) and an enable signal wat_data_en; operation: the W_Avg module calculates the mean of the input data (the flow is the same as the addition tree + accumulation logic of the “mean submodule”); output: the mean result avg and the enable signal avg_en, which are sent to the subsequent subtraction module and control link, respectively. (2) Variance calculation link (subtraction + square + accumulation), subtraction operation: each wat_data is subtracted from the mean avg through the subtractor to obtain the difference value signal of “data-mean”; square operation: the difference value signal is sent to the Square module (multiple parallel square units) to obtain the square of the difference value; accumulation operation: all square results are accumulated through the adder, and the counting logic (cnt==img_size?) is used to judge whether the accumulation of all samples is completed, and the accumulation result is temporarily stored in the register. (3) Square root operation link (Sqrt module), input: variance result after accumulation is completed (synchronized through the register); operation: the Sqrt module performs square root operation on the input variance; output: the final standard deviation result std_out[447:0]. (4) Control and timing link, the enable signal wat_data_en triggers the W_Avg module to start mean calculation, and the avg_en synchronously triggers the subsequent variance operation; the counting logic (cnt==img_size?) controls the start and stop of the accumulator to ensure that only when the sample number meets the requirement, the complete variance accumulation is performed to ensure the accuracy of the result. Function logic: this module first calculates the mean of the input data through W_Avg, then calculates the square sum (variance) of “data-mean” element by element, and finally takes the square root of the variance to obtain the standard deviation.

[0152] It can be seen that the application proposes a lightweight full-dimensional attention mechanism, which can enhance the attention weights of a convolutional neural network (CNN) in the channel, spatial horizontal and spatial vertical directions at the same time, and significantly improve the image classification accuracy. Meanwhile, an FPGA hardware accelerator adapted to the mechanism is designed to solve the problems of large parameter quantity, low calculation efficiency and poor hardware adaptability of existing attention modules, and is suitable for edge computing device deployment. Specifically, the application adopts a three-branch independent parallel structure to finely model the channel, spatial height and spatial width dimensions respectively, and innovatively introduces grouped attention and channel shuffle operations, so that the attention module has the characteristics of lightweight and hardware friendliness while significantly improving the model accuracy. In addition, the scheme further reduces the parameter size through compression operation and gating activation mechanism, and supports 8bit quantization compression, providing convenience for edge deployment, and providing unique beneficial effects in realizing higher-dimensional feature capture, lower parameter quantity overhead and better hardware acceleration potential, and the hardware execution efficiency is significantly improved.

[0153] The above is only a preferred embodiment of the application and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A hardware accelerator for a full-dimensional attention mechanism for convolutional neural networks, characterized in that, The method comprises the following steps: The convolution operation unit performs convolution operation on the input feature map; The full-dimensional attention unit groups the feature map output by the convolution operation unit into G groups of sub-feature maps along the channel dimension; Each group of sub-feature maps respectively performs channel attention, spatial vertical attention and spatial horizontal attention to obtain three attention matrices; The fusion unit fuses the three attention matrices, and then performs weighted calculation on the input sub-feature map based on the fused attention matrix to obtain a fusion result, and combines G groups of fusion results to obtain a final output feature map; Wherein, each group of sub-feature maps respectively performs channel attention, spatial vertical attention and spatial horizontal attention to obtain three attention matrices, including the following steps: The channel attention branch receives the sub-feature map, and constructs the attention in the channel dimension direction by using the channel attention module; the vertical spatial attention branch receives the sub-feature map, first performs dimension transformation, and then constructs the attention in the vertical direction of the spatial dimension by using the first spatial attention module; the horizontal spatial attention branch receives the sub-feature map, first performs dimension transformation, and then constructs the attention in the horizontal direction of the spatial dimension by using the second spatial attention module.

2. The hardware accelerator for full-dimensional attention mechanism for convolutional neural network according to claim 1, wherein, The channel attention branch receives the sub-feature map, and constructs the attention in the channel dimension direction by using the channel attention module, which is represented as: ； ； ； ； wherein, represents the input sub-feature map, , , respectively represent global average pooling, fully connected and activation operations, represents dimension adjustment, represents adjusting the dimension to be consistent with that in the parentheses; , and are intermediate calculation results, represents the first attention matrix output by the channel attention branch.

3. The hardware accelerator for full-dimensional attention mechanism for convolutional neural networks according to claim 1, wherein, The vertical spatial attention branch receives the sub-feature map, first performs dimension transformation, and then constructs the attention in the vertical direction of the spatial dimension by using the first spatial attention module, which is represented as: ； ； ； ； ； wherein, represents an input sub-feature map; The function is used to change the dimension order of the tensor; The function represents the mean operation; and represents the learnable weight and the learnable parameter attached to each point along the vertical direction of the spatial dimension; represents the Sigmoid activation function, The function can adjust the dimension to the same as in the parentheses; , , and are intermediate calculation results, represents the second attention matrix output by the vertical spatial attention branch.

4. The hardware accelerator for full-dimensional attention mechanism for convolutional neural network according to claim 1, wherein, The horizontal spatial attention branch receives the sub-feature map, first performs dimension transformation, and then constructs the attention in the horizontal direction of the spatial dimension by using the second spatial attention module, which is represented as: ； ； ； ； ； wherein, represents an input sub-feature map, is used to change the dimension order of a tensor, represents a standard deviation operation, and represents a learnable weight and a learnable parameter attached to each point along the horizontal direction of the spatial dimension, represents a Sigmoid activation function, The function can adjust the dimension to the same as in the parentheses; , , and are intermediate calculation parameters, represents the third attention matrix output by the horizontal spatial attention branch.

5. The hardware accelerator for full-dimensional attention mechanism for convolutional neural networks according to claim 1, wherein, The fusion unit fuses the three attention matrices, and then performs weighted calculation on the input sub-feature map based on the fused attention matrix to obtain a fusion result, and combines G groups of fusion results to obtain a final output feature map, including the following steps: Feature fusion, feature fusion based on the fused attention matrix, represented as: ； ； wherein OD denotes the synthesis result corresponding to one attention matrix, denotes the calculation of the multiplication of the corresponding positions of the matrices, is the fusion result; Channel reorganization, G group will be Split into a two-dimensional structure of "group number x number of channels per group"; Dimension transposition, exchange the dimension order of the group number and the channel number; Structure flattening, restoring the transposed two-dimensional structure to one-dimensional representation.

6. The hardware accelerator for full-dimensional attention mechanism for convolutional neural networks according to claim 1, wherein, The hardware accelerator is designed based on an FPGA development board, and adopts input channel parallelism, output channel parallelism and convolution window parallelism; The number of input channels and the number of output channels are both n = 0, 1, 2, 3, 4; The convolution window size of the convolution operation unit is 3x3.

7. The hardware accelerator for full-dimensional attention mechanism for convolutional neural network according to claim 6, wherein, The hardware accelerator uses 16 input channel parallelism and 16 output channel parallelism.

8. The hardware accelerator for full-dimensional attention mechanism for convolutional neural network of claim 1, wherein, The channel attention module comprises a global average pooling submodule, a first full connection submodule, a ReLU activation function, a second full connection submodule and a first Sigmoid activation function connected in sequence; Wherein, the first full connection submodule adopts 16 input channels and 4 output channels, and the second full connection submodule adopts 4 input channels and 16 output channels.

9. The hardware accelerator for full-dimensional attention mechanism for convolutional neural network of claim 1, wherein, The attention module in the vertical direction of the spatial dimension comprises an average value submodule, a multiplication-addition submodule and a second Sigmoid activation function connected in sequence.

10. The hardware accelerator for full-dimensional attention mechanism for convolutional neural networks according to claim 1, wherein, The attention module in the horizontal direction of the spatial dimension comprises a standard deviation submodule, a multiplication-addition submodule and a third Sigmoid activation function connected in sequence.

Images