A multi-scale reconfigurable visual transformer accelerator and an acceleration method

Abstract

The application discloses a multiscale reconfigurable visual Transformer accelerator and an acceleration method, which comprise a top control module configured to receive configuration instructions from a CPU; a DMA module configured to carry input feature maps, weight parameters and normalization parameters in an off-chip memory to an on-chip cache module; the on-chip cache module configured to store the feature map data, the weight data and the parameter data obtained through the carrying; a reconfigurable computing array module connected to the on-chip cache module, configured to generate a query matrix Q, a key matrix K and a value matrix V; and a post-processing module connected to the reconfigurable computing array module, configured to sequentially perform normalization, an activation function, pooling, upsampling and quantization operations on the output feature maps. The application can improve the hardware resource utilization, reduce the chip area and power consumption, improve the inference throughput and energy efficiency of the visual Transformer model, support a dynamic mixed precision quantization strategy, and reduce the off-chip memory access frequency and data carrying cost.

Description

Technical Field

[0001] This invention relates to the field of deep learning processors and reconfigurable computing chips, specifically to a multi-scale reconfigurable visual Transformer accelerator and acceleration method. Background Technology

[0002] With the rapid development of deep learning technology, visual Transformers, with their powerful global feature modeling capabilities, have been widely used in computer vision tasks such as image classification, object detection, and semantic segmentation. However, these networks have a large number of parameters and computational demands, and the inference process involves a large number of matrix multiplications, convolutions, and attention calculations. General-purpose processors are unable to meet the requirements for real-time inference and low power consumption, and dedicated hardware accelerators are necessary for efficient deployment.

[0003] Lightweight visual Transformers, exemplified by EfficientViT, introduce multi-scale feature extraction structures. By extracting and fusing multi-scale features in parallel using convolutional kernels of different sizes, they improve model efficiency while also introducing diverse computational requirements, such as ordinary convolution, depthwise convolution, pointwise convolution, and linear attention computation. However, most existing accelerators are optimized for single computation types and cannot flexibly adapt to such multi-scale hybrid computation scenarios, resulting in low hardware resource utilization.

[0004] In terms of quantization acceleration, different network layers and operators in the visual Transformer exhibit significantly different sensitivities to quantization precision. To further improve computational efficiency, a hierarchical quantization strategy can be adopted: modules insensitive to quantization use 4-bit integer low-width quantization to significantly reduce storage and computational overhead; while precision-sensitive modules retain 8-bit integer high-width quantization to ensure that inference precision is not affected. However, existing accelerators generally adopt a fixed-precision design, which does not support dynamically reconfigurable 4-bit / 8-bit precision switching, making it difficult to deeply adapt to hierarchical mixed-precision quantization strategies and fully leverage the storage and computational advantages brought by quantization optimization.

[0005] Therefore, how to design a dedicated accelerator for visual Transformers that can flexibly adapt to multi-scale hybrid computing scenarios and support dynamic precision switching has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, one of the objectives of this invention is to provide a multi-scale reconfigurable visual Transformer accelerator that can improve hardware resource utilization, reduce chip area and power consumption, improve the inference throughput and energy efficiency of visual Transformer models, support dynamic mixed precision quantization strategies, and reduce the number of off-chip memory accesses and data transfer overhead.

[0007] The second objective of this invention is to provide a multi-scale reconfigurable visual Transformer acceleration method, which can improve hardware resource utilization, reduce chip area and power consumption, improve the inference throughput and energy efficiency of the visual Transformer model, support dynamic mixed precision quantization strategy, and reduce the number of off-chip memory accesses and data transfer overhead.

[0008] To achieve one of the objectives of this invention, the following solution is adopted: A multi-scale reconfigurable visual Transformer accelerator, comprising: The top-level control module is used to receive configuration instructions from the CPU, initialize the calculation mode, precision configuration, cache block size and post-processing mode according to the configuration instructions, and obtain initialization configuration parameters including the post-processing mode. The DMA module, connected to the off-chip memory, is used to transfer the input feature map, weight parameters and normalization parameters in the off-chip memory to the on-chip cache module based on the initialization configuration parameters under the control of the top-level control module. An on-chip cache module, connected to the DMA module, is used to store the transferred feature map data, weight data, and parameter data. The reconfigurable computing array module, connected to the on-chip cache module, is used to read corresponding data from the feature map data, weight data, and parameter data according to the initialized computing mode, perform multi-scale convolution and matrix multiplication operations, generate a query matrix Q, a key matrix K, and a value matrix V, and transpose the key matrix K to obtain the transposed key matrix K. T The reconfigurable computing array module is also used to read the query matrix Q and the transpose key matrix K. T Given a value matrix V, perform linear attention computation by sequentially combining the value matrix V with the transpose key matrix K. T Matrix multiplication is performed to obtain the attention score matrix score, and then the attention score matrix score is multiplied with the query matrix Q to obtain the attention feature; and the weight parameters of the subsequent convolutional layer are read from the on-chip cache module and convolved with the attention feature to obtain the output feature map; The post-processing module, connected to the reconfigurable computing array module, is used to perform normalization, activation function, pooling, upsampling and quantization operations on the output feature map in sequence according to the initialized post-processing mode to obtain the final inference result.

[0009] Furthermore, the reconfigurable computing array module internally includes an array controller, a configurable multiply-accumulate unit, a weight register group, and a partial cache; The array controller dynamically reconstructs the computing path of the reconfigurable computing array module according to the initialization computing mode configuration instructions issued by the CPU, supporting ordinary convolution computing mode, depthwise convolution computing mode, pointwise convolution computing mode, or linear projection computing mode.

[0010] Furthermore, the configurable multiply-accumulate supports a mixed precision packing mode of 8-bit integers and 4-bit integers; When configured for 8-bit integer precision, it operates in shared operand mode, where a single activation value is multiplied in parallel with two weights. When configured with 4-bit integer precision, it operates in multi-channel independent packaging mode, multiplying the two activation values ​​and the two weights in parallel, and outputting the four-channel multiplication and accumulation result.

[0011] Furthermore, in the ordinary convolutional computation mode, the reconfigurable computation array module performs cross-channel weighted multiplication and accumulation operations, and the output feature map part is obtained by summing the weights and the input feature map along the input channel dimension, and then accumulating them in the spatial dimension to generate the output feature map; In the deep convolutional computation mode, the reconfigurable computation array module performs channel-independent multiplication and accumulation operations. The output feature map is obtained directly by multiplying the weights and the input feature map, and then accumulated through spatial dimensions to generate the output feature map. In the pointwise convolutional computation mode, the reconfigurable computation array module performs cross-channel multiplication and accumulation operations without spatial dimension accumulation, and directly outputs the feature map; In the linear projection calculation mode, the reconfigurable calculation array module switches to a matrix-vector multiplication architecture, and the output activation value is obtained by weighted summation of the input features and projection weights along the dimension.

[0012] Furthermore, it also includes a configurable addition tree module, which is connected to the reconfigurable computing array module and the post-processing module respectively, and is used to dynamically adjust the accumulation path and parallelism according to the initialized computing mode, and to perform channel dimension accumulation and spatial dimension accumulation on the convolution part.

[0013] Furthermore, the on-chip cache module adopts a ping-pong cache structure, which supports parallel read and write operations. It reads historical data to participate in the calculation while writing new feature data, so as to hide the memory access latency. The process of generating the query matrix Q, key matrix K, and value matrix V adopts a block-based processing method. When the size of the input feature map exceeds the capacity of the on-chip cache module, the input feature map is divided into several sub-blocks and loaded and calculated block by block.

[0014] Furthermore, the post-processing module internally includes a post-processing controller, a normalization module, an activation function calculation module, a pooling module, an upsampling module, and a quantization calculation module; The post-processing controller selects the working mode according to the configuration instructions, the normalization module supports dual-mode switching between LayerNorm and BatchNorm, the activation function calculation module supports ReLU or HardSwish activation functions, and the quantization calculation module supports uniform quantization or logarithmic quantization.

[0015] Furthermore, in the layer normalization mode, the normalization module uses hardware to calculate the mean and variance online based on the current input features, and completes the normalization through parallel summation, sum of squares, fixed-point square root and division operations; In batch normalization mode, the normalization module uses pre-stored mean and variance parameters to complete the normalization. Both modes reuse the same set of subtraction, multiplication, square root, division, and addition operation pathways.

[0016] Furthermore, the reconfigurable computing array module also includes a data rearrangement module for transposing the key matrix K to obtain the transposed key matrix K. T ; When performing linear attention calculation, the reconfigurable computing array module uses a block caching method to cache the intermediate attention score matrix to the on-chip cache module or off-chip memory.

[0017] The specific details of the multi-scale reconfigurable visual Transformer accelerator of the present invention are as follows: The multi-scale reconfigurable visual Transformer accelerator of this invention includes: a top-level control module, a DMA module, an on-chip cache module, a reconfigurable computation array module, a configurable addition tree module, and a post-processing module; wherein: the top-level control module is used to receive configuration instructions from the CPU, control the on-chip and off-chip data interaction, data distribution and computation scheduling process, and report the execution completion status to the CPU; the on-chip cache module includes a feature map cache, a weight cache and a parameter cache, and is used to store the input feature map, weight parameters, bias parameters, normalization parameters and the Query matrix, Key matrix and Value matrix required for attention calculation of the current network layer; the reconfigurable computation array module is used to perform multiplication and accumulation operations of the input feature map and weights, and matrix multiplication operations of the QKV matrix according to the configuration instructions; the configurable addition tree module is used to accumulate the channel dimension and spatial dimension of the convolution part according to the currently configured computation mode; the post-processing module is used to collect the output results of the computation array and complete post-processing operations such as normalization, activation function, pooling, upsampling and quantization.

[0018] The reconfigurable computing array module includes: an array controller, a configurable multiply-accumulate unit, a weight register group, and a partial sum cache; wherein: the array controller configures the computing mode according to the instructions issued by the CPU, supporting ordinary convolution computing, depthwise convolution computing, pointwise convolution computing, and attention linear projection computing; the configurable multiply-accumulate unit performs corresponding multiply-accumulate operations according to the configuration mode and precision standard; the weight register group is used to cache the weight data required for the current computing cycle, improving data supply efficiency; the partial sum cache is used to temporarily store intermediate multiply-accumulate results, supporting multi-level pipelined convergence and iterative accumulation.

[0019] The post-processing module includes: a post-processing controller, a normalization module, an activation function calculation module, a pooling module, an upsampling module, and a quantization calculation module; wherein: the post-processing controller selects a working mode according to configuration instructions, including LayerNorm, BatchNorm, ReLU, HardSwish, uniform quantization, and logarithmic quantization; the normalization module is used to perform feature normalization operations; the activation function calculation module is used to output the corresponding activation values; the pooling module supports max pooling, average pooling, and min pooling; the upsampling module is used to upsample the feature map; and the quantization calculation module is used to output the quantization results according to the configuration mode.

[0020] The normalization module supports dual-mode switching: in batch normalization mode, it uses pre-stored mean and variance parameters to complete the calculation; in layer normalization mode, it performs online statistics on the current input features, calculates the mean and variance in real time, and completes the normalization; its operation path includes square root unit, multiplication unit, and addition and subtraction unit.

[0021] The feature map cache and weight cache adopt a ping-pong cache structure, which supports parallel read and write operations. It can read historical data to participate in the calculation while writing new feature data, thereby improving data throughput and hiding memory access latency.

[0022] The configurable multiply-accumulate supports mixed precision packing modes of INT8 and INT4: when configured for 8-bit integer precision, it operates in shared operand mode, with single-channel activation and parallel multiplication of two-channel weights, making full use of input data; when configured for 4-bit integer precision, it operates in multi-channel independent packing mode, with two-channel activation and parallel multiplication of two-channel weights, achieving a doubling of computing power density under the same DSP resources, significantly improving hardware utilization and computing energy efficiency.

[0023] To achieve the second objective of this invention, the following solution is adopted: A multi-scale reconfigurable visual Transformer acceleration method, applied to a multi-scale reconfigurable visual Transformer accelerator as described in one of the objectives of this invention, includes the following steps: Step S1: The top-level control module receives configuration instructions from the CPU and initializes the calculation mode, precision configuration, cache block size and post-processing mode according to the configuration instructions to obtain initialization configuration parameters including the post-processing mode. Step S2: Based on the initialization configuration parameters, the input feature map, weight parameters and normalization parameters in the off-chip memory are transferred to the on-chip cache module through the DMA module under the control of the top-level control module to obtain cached feature map data, weight data and parameter data; Step S3: The reconfigurable computing array module reads corresponding data from the feature map data, weight data, and parameter data according to the initialized computing mode, performs multi-scale convolution and matrix multiplication operations, generates a query matrix Q, a key matrix K, and a value matrix V, and transposes the key matrix K to obtain the transposed key matrix K. T ; Step S4: The reconfigurable computing array module reads the query matrix Q and the transpose key matrix K. T Given a value matrix V, perform linear attention computation by sequentially combining the value matrix V with the transpose key matrix K. T Matrix multiplication is performed to obtain the attention score matrix, and then the attention score matrix is ​​multiplied with the query matrix Q to obtain the attention features. Step S5: The reconfigurable computing array module reads the weight parameters of the subsequent convolutional layer from the on-chip cache module, performs convolution calculation with the attention features, and obtains the output feature map; Step S6: The post-processing module performs normalization, activation function, pooling, upsampling and quantization operations on the output feature map in sequence according to the initialized post-processing mode to obtain the final inference result.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention achieves efficient reuse of multiple operators through a unified computational pathway. The reconfigurable computational array module sequentially performs multi-scale convolution and matrix multiplication operations under the same hardware architecture to generate a query matrix Q, a key matrix K, and a value matrix V. The key matrix K is transposed, and linear attention computation is performed (by combining the value matrix V with the transposed key matrix K). T The attention score matrix (score) is obtained by multiplying the two matrices, and then the score is multiplied by the query matrix Q to obtain the attention features. The weight parameters and attention features of subsequent convolutional layers are also read from the on-chip cache module to perform convolution calculations. This design allows the same computing array to dynamically adapt to diverse computational needs such as multi-scale feature extraction, linear attention calculation, and subsequent convolutions, eliminating the need to deploy multiple independent hardware sets. This significantly improves hardware resource utilization and reduces chip area and power consumption.

[0025] 2. This invention supports end-to-end hardware acceleration. The top-level control module initializes the computation mode, precision configuration, cache block size, and post-processing mode; the DMA module moves the input feature map, weight parameters, and normalization parameters from off-chip memory to the on-chip cache module; the reconfigurable computing array module reads data from the on-chip cache module and performs core multiplication and matrix multiplication operations; the post-processing module sequentially performs normalization, activation function, pooling, upsampling, and quantization operations on the output feature map to obtain the final inference result. These modules work collaboratively to form a complete hardware pipeline from data transfer to inference result output, significantly reducing CPU intervention and on-chip / off-chip data interaction overhead, and improving the overall inference throughput and energy efficiency of the visual Transformer model.

[0026] 3. This invention provides an architectural foundation for dynamic mixed-precision quantization. The top-level control module can initialize the precision configuration, the reconfigurable computing array module performs multiply-accumulate operations of the corresponding precision according to the initialized precision configuration, and the post-processing module integrates quantization operations, enabling the entire accelerator to naturally support dynamic switching of different computing precisions. It can adapt to hierarchical mixed-precision quantization strategies and flexibly reduce storage and computing power overhead while ensuring inference accuracy.

[0027] 4. This invention optimizes storage access and reduces data transfer power consumption. The DMA module works in conjunction with the on-chip cache module, transferring the input feature map, weight parameters, and normalization parameters to the on-chip cache module in batches based on the initialization configuration parameters. The reconfigurable computing array module directly reads data from the on-chip cache module for reuse calculations, reducing repeated access to off-chip memory, effectively alleviating the memory access bandwidth bottleneck, and reducing the overall system power consumption. Attached Figure Description

[0028] Figure 1 This is a block diagram of the multi-scale reconfigurable visual Transformer accelerator in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the multi-scale reconfigurable visual Transformer accelerator in an embodiment of the present invention; Figure 3 This is a flowchart of the accelerated visual Transformer neural network computation in an embodiment of the present invention; Figure 4 This is a schematic diagram of the processing flow of the reconfigurable computing array module in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the calculation process of the reconfigurable computing array module with 8-bit and 4-bit integer calculation precision in an embodiment of the present invention; wherein, Figure 5 It contains subgraphs a and b, which correspond to 8-bit integer calculation precision flow and 4-bit integer calculation precision flow, respectively; Figure 6 This is a flowchart of the normalization calculation process in an embodiment of the present invention; Figure 7 This is a flowchart of a multi-scale reconfigurable visual Transformer acceleration method in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0030] Example 1 like Figure 1 As shown, this embodiment of the invention provides a multi-scale reconfigurable visual Transformer accelerator, comprising: The top-level control module is used to receive configuration instructions from the CPU, initialize the calculation mode, precision configuration, cache block size and post-processing mode according to the configuration instructions, and obtain initialization configuration parameters including the post-processing mode. The DMA module, connected to the off-chip memory, is used to transfer the input feature map, weight parameters and normalization parameters in the off-chip memory to the on-chip cache module based on the initialization configuration parameters under the control of the top-level control module. An on-chip cache module, connected to the DMA module, is used to store the transferred feature map data, weight data, and parameter data. The reconfigurable computing array module, connected to the on-chip cache module, is used to read corresponding data from the feature map data, weight data, and parameter data according to the initialized computing mode, perform multi-scale convolution and matrix multiplication operations, generate a query matrix Q, a key matrix K, and a value matrix V, and transpose the key matrix K to obtain the transposed key matrix K. T The reconfigurable computing array module is also used to read the query matrix Q and the transpose key matrix K. T Given a value matrix V, perform linear attention computation by sequentially combining the value matrix V with the transpose key matrix K. T Matrix multiplication is performed to obtain the attention score matrix score, and then the attention score matrix score is multiplied with the query matrix Q to obtain the attention feature; and the weight parameters of the subsequent convolutional layer are read from the on-chip cache module and convolved with the attention feature to obtain the output feature map; The post-processing module, connected to the reconfigurable computing array module, is used to perform normalization, activation function, pooling, upsampling and quantization operations on the output feature map in sequence according to the initialized post-processing mode to obtain the final inference result.

[0031] The multi-scale reconfigurable visual Transformer accelerator of this invention will be further described in detail below.

[0032] The multi-scale reconfigurable visual Transformer accelerator of this invention balances accuracy and performance through multiple computing modes and optimizes power consumption and latency through on-chip storage.

[0033] The multi-scale reconfigurable visual Transformer accelerator of this invention includes: a top-level control module for receiving CPU instructions, controlling data distribution, and feeding back execution results; a DMA module; an on-chip cache module, including a feature map cache, a weight cache, and a parameter cache, for storing input feature maps, weights, parameters, QKV matrices, and output feature maps, using ping-pong caching technology to achieve parallel read and write operations; a reconfigurable computation array module, including a controller, a multiply-accumulate unit, a weight register, and a partial sum cache, configurable with multiple computation modes to perform multiply-accumulate and matrix multiplication, and its multiply-accumulate unit supports a packing mode to improve DSP utilization; a configurable addition tree module for accumulating convolution results according to the configuration mode; and a post-processing module, including a controller, a normalization module, an activation function calculation module, a pooling module, an upsampling module, and a quantization module, for collecting multiply-accumulate results and performing normalization, activation, quantization, pooling, and upsampling operations.

[0034] like Figure 2 As shown, the multi-scale reconfigurable visual Transformer accelerator of this invention includes: a top-level control module, multiple feature map caches, multiple weight caches, multiple parameter caches, a reconfigurable computation array module, a configurable addition tree module, and a post-processing module.

[0035] like Figure 3 As shown, the accelerated visual Transformer neural network computation process in this embodiment mainly includes three parts: multi-scale feature input, linear attention computation, and convolutional output. The multi-scale feature input extracts multi-scale feature maps through convolutional layers of different scales, concatenates them, and then inputs them into the linear attention computation module to enhance the visual Transformer's ability to perceive and recognize multi-scale targets. In the linear attention computation stage, the query matrix Q and the key matrix K are first nonlinearly transformed using the ReLU activation function; then, the transformed K matrix is ​​transposed to obtain Ki. T Matrix, the value matrix V and K T The attention score matrix is ​​obtained by multiplying the matrices; then the score matrix is ​​multiplied by the Q matrix, and after normalization and feature rearrangement, it is input into the subsequent convolutional layer to complete the convolution output.

[0036] like Figure 2and Figure 3 As shown, when the accelerator is working, the CPU first sends configuration parameters to the top-level control module. The top-level control module completes the initialization configuration of the computation mode, precision configuration, cache block size, and post-processing mode. After configuration, the accelerator uses the DMA module to move the feature map, weight parameters, and normalization parameters from the off-chip memory to the on-chip cache module. The data distribution module parses, aligns, and routes the input data, distributing the feature map to the feature map cache, the weight parameters to the weight cache, and parameters such as mean, variance, and quantization scale to the parameter cache. All caches adopt a ping-pong cache structure, and the data distribution module can write to multiple sets of BRAM caches in parallel, ensuring that subsequent computing units can read data from multiple BRAMs simultaneously, improving data throughput. If the input feature map size exceeds the cache capacity, block processing can be configured in the software to divide the feature map into several sub-blocks that meet the cache's carrying capacity, and load and compute them block by block.

[0037] The reconfigurable computation array module reads data from the feature map cache, weight cache, and parameter cache according to the configuration mode, performs multi-scale convolution, QKV matrix generation, and other multiplication-accumulation and matrix multiplication operations, stores intermediate results in the partial cache, and completes channel accumulation and spatial accumulation through the configurable addition tree module to generate Q, K, and V matrices. The K matrix is ​​transposed to KV matrix through the data rearrangement module. T The matrix. After each block of calculation is completed, the result is written back to off-chip memory via DMA, and then the next block of data is loaded, until Q and K are calculated. T The V matrix has been fully generated.

[0038] Complete QK T After V is generated, the accelerator enters the linear attention computation phase. The reconfigurable computing array module reads Q and K from off-chip memory and on-chip cache modules. T The V matrix, connecting V and K T Matrix multiplication is performed to obtain the score matrix, which is then cached in blocks to the feature map cache and off-chip storage. The accelerator then performs multiplication of the score matrix and the Q matrix to obtain attention features, which are also cached. Finally, the weight parameters for subsequent convolutional layers are read from the weight cache, and the reconfigurable computation array module performs convolution calculations. The output features undergo normalization, activation, pooling, upsampling, and quantization operations in the post-processing module, ultimately completing the entire hardware acceleration process from multi-scale feature input, linear attention calculation, to convolution output.

[0039] like Figure 4As shown, the processing flow of the reconfigurable computing array module is illustrated. The computing array can be reconfigured into different computing modes according to the configuration. When the reconfigurable computing array module is working, the controller dynamically reconfigures the computing path according to the upper layer configuration, switching to ordinary convolution computing, depthwise convolution computing, pointwise convolution computing, or linear projection computing modes, and configuring it to 4-bit integer or 8-bit integer computing precision. The weight register is used to cache the weight data required for the current calculation, and the multiply-accumulate unit performs multiply-accumulate operations according to the operation rules of the corresponding computing mode. The partial sum cache is used to temporarily store intermediate results to support iterative accumulation.

[0040] In standard convolution mode, the array performs cross-channel weighted multiplication and accumulation. The output feature map sum of ofm_p(C_out) is obtained by summing the weights wgt(C_in,C_out) and the input feature map ifm(C_in,C_out) along the input channel C_in using a configurable adder tree. Based on the convolution size, the accumulator array performs the corresponding number of spatial accumulations to generate the output feature map ofm(C_out). In depthwise convolution mode, the array uses a channel-independent computation mechanism, performing single-channel multiplication and accumulation. The output feature map sum of ofm_p(C_out) is obtained by directly multiplying the weights and the input feature map, without needing to sum along the input channel dimension. It is directly fed to the accumulator array for spatial accumulation to obtain the output feature map. In pointwise convolution mode, the operation logic is the same as standard convolution, but the feature map sum does not require spatial accumulation and is directly used as the final output feature map, achieving efficient channel transformation and dimension mapping. In linear projection computation mode, the array is switched to a matrix-vector multiplication architecture. The output activation value act(m) is obtained by weighted accumulation of the input feature in_f(n) and the projection weight wgt(m,n) along the dimension n, which is adapted to the core operations such as Q / K / V projection and post-attention projection in the attention mechanism.

[0041] like Figure 5 As shown, the computation flow of the reconfigurable computing array module with 8-bit integer and 4-bit integer computation precision is illustrated. Figure 5 As shown in 'a': When the calculation precision is 8-bit integers, the multiplier operates in a two-way shared operand mode. Inputs include one activation channel A1 and two weight channels W1 and W2. These are assembled by the data assembly module into a 27-bit integer and an 18-bit integer, which are then input to ports A and B of the DSP. The DSP calculates and outputs a 48-bit integer. The unpacking module unpacks the data into two outputs: A1×W1 and A1×W2. When the calculation precision is 4-bit integers, as shown in 'a': Figure 5As shown in b, the multiplier operates in four-way parallel mode, taking two activation values ​​A1 and A2 and two weights W1 and W2 as inputs. These are assembled by the data assembly module into 27-bit and 18-bit integers adapted to the DSP ports and input to ports A and B of the DSP. The DSP calculates and outputs a 48-bit integer. The unpacking module unpacks the data into four outputs: A1×W1, A1×W2, A2×W1, and A2×W2.

[0042] like Figure 6 The diagram illustrates the normalization calculation process. In this embodiment, the post-processing module, upon receiving the quantized input, performs three parallel processing steps during layer normalization calculation: summation, sum of squares, and zero-point subtraction and shifting operations on the input features. The summation result is used to calculate the fusion parameter A with the quantized input after zero-point subtraction and shifting. The sum of squares, the square of the sum, and the aforementioned processed quantized input are used to calculate the fusion parameter B. After obtaining fusion parameter A, it is multiplied by the γ weight, then multiplied by fusion parameter B, and the β weight is added. Finally, the quantization result is output after scaling factor multiplication and zero-point shifting. The calculation of fusion parameter B includes square root and reciprocal operations. First, the relevant values ​​are converted to fixed-point number format. The square root is calculated using a fixed-point square root module, and the reciprocal is calculated using a fixed-point division module. These square root and division modules are scalable and support vector input / output methods to efficiently complete the parallel calculation of layer normalization.

[0043] The post-processing module includes a normalization module that can flexibly switch between batch normalization and layer normalization calculations based on the controller configuration. In layer normalization mode, the mean and variance are obtained online by hardware based on the current input features. In batch normalization mode, the mean and variance use preset calculation parameters that are fixed after training. Both modes reuse the same set of subtraction, multiplication, square root, division, and addition operations, achieving hardware resource sharing. The control module in the post-processing module performs the corresponding number of accumulation operations according to the parameter configuration and determines the number of vector calculations required for layer normalization based on the configuration and hardware resources. The calculation is executed in a pipelined manner, and intermediate results are cached in the feature map cache. After the calculation is completed, the module sends a completion signal to the top-level control module, which writes the cached results to external storage according to the configuration, completing the overall post-processing process.

[0044] The multi-scale reconfigurable visual Transformer accelerator of this invention has the following advantages: Addressing the complex inference requirements of visual Transformers in multi-scale feature extraction, hybrid convolution, and linear attention computation, this invention performs integrated optimization of the overall hardware architecture and data path, making it reconfigurable, configurable, and highly reusable, thereby improving hardware utilization and reducing resource and power consumption. For models such as EfficientViT, which involve diverse computations including ordinary convolution, depthwise convolution, pointwise convolution, and linear projection, this invention unifies the design of the core reconfigurable computation array module. This allows it to flexibly switch computation modes based on configuration instructions issued by the CPU, enabling efficient computation of multiple operators on the same hardware, maximizing the reuse of computation units, and avoiding resource redundancy caused by deploying independent hardware for different operators. Simultaneously, this invention designs a configurable addition tree module, which can dynamically adjust the accumulation path and parallelism according to the current computation mode, compatible with the channel accumulation and spatial accumulation requirements of various convolutions, significantly improving module versatility and resource utilization. At the storage level, this invention employs an on-chip distributed cache and ping-pong cache structure. Controllable read / write scheduling reduces on-chip and off-chip data interaction, lowers memory access power consumption, and increases data throughput. In the post-processing stage, this invention integrates normalization, activation, pooling, upsampling, and quantization operations into a unified configurable post-processing unit. The unit switches operating modes based on configuration instructions, supporting dual-mode layer normalization and batch normalization, multiple activation functions, and flexible selection of two quantization methods, further improving hardware reusability. The overall architecture, through a unified computation path, configurable execution units, and efficient storage scheduling, significantly reduces hardware area, resource consumption, and unnecessary computational power consumption while ensuring the inference accuracy of the visual Transformer, making end-to-end inference more efficient, lightweight, and easy to deploy.

[0045] Example 2 like Figure 7 As shown, this embodiment of the invention also provides a multi-scale reconfigurable visual Transformer acceleration method, applied to the multi-scale reconfigurable visual Transformer accelerator as described in Embodiment 1, comprising the following steps: Step S1: The top-level control module receives configuration instructions from the CPU and initializes the calculation mode, precision configuration, cache block size, and post-processing mode according to the configuration instructions, thereby obtaining initialization configuration parameters including the post-processing mode.

[0046] Step S2: Based on the initialization configuration parameters, the input feature map, weight parameters and normalization parameters in the off-chip memory are transferred to the on-chip cache module through the DMA module under the control of the top-level control module, so as to obtain cached feature map data, weight data and parameter data.

[0047] Step S3: The reconfigurable computing array module reads corresponding data from the feature map data, weight data, and parameter data according to the initialized computing mode, performs multi-scale convolution and matrix multiplication operations, generates a query matrix Q, a key matrix K, and a value matrix V, and transposes the key matrix K to obtain the transposed key matrix K. T .

[0048] Step S4: The reconfigurable computing array module reads the query matrix Q and the transpose key matrix K. T Given a value matrix V, perform linear attention computation by sequentially combining the value matrix V with the transpose key matrix K. T Matrix multiplication is performed to obtain the attention score matrix, and then the attention score matrix is ​​multiplied with the query matrix Q to obtain the attention features.

[0049] Step S5: The reconfigurable computing array module reads the weight parameters of the subsequent convolutional layer from the on-chip cache module, performs convolution calculation with the attention features, and obtains the output feature map.

[0050] Step S6: The post-processing module performs normalization, activation function, pooling, upsampling and quantization operations on the output feature map in sequence according to the initialized post-processing mode to obtain the final inference result.

[0051] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A multi-scale reconfigurable vision Transformer accelerator, comprising: include: The top-level control module is used to receive configuration instructions from the CPU, initialize the calculation mode, precision configuration, cache block size and post-processing mode according to the configuration instructions, and obtain initialization configuration parameters including the post-processing mode. The DMA module, connected to the off-chip memory, is used to transfer the input feature map, weight parameters and normalization parameters in the off-chip memory to the on-chip cache module based on the initialization configuration parameters under the control of the top-level control module. An on-chip cache module, connected to the DMA module, is used to store the transferred feature map data, weight data, and parameter data. The reconfigurable computing array module, connected to the on-chip cache module, is used to read corresponding data from the feature map data, weight data, and parameter data according to the initialized computing mode, perform multi-scale convolution and matrix multiplication operations, generate a query matrix Q, a key matrix K, and a value matrix V, and transpose the key matrix K to obtain the transposed key matrix K. T The reconfigurable computing array module is also used to read the query matrix Q and the transpose key matrix K. T Given a value matrix V, perform linear attention computation by sequentially combining the value matrix V with the transpose key matrix K. T Matrix multiplication is performed to obtain the attention score matrix, and then the attention score matrix is ​​multiplied with the query matrix Q to obtain the attention features. And to read the weight parameters of the subsequent convolutional layers from the on-chip cache module, perform convolution calculation with the attention features, and obtain the output feature map; The post-processing module, connected to the reconfigurable computing array module, is used to perform normalization, activation function, pooling, upsampling and quantization operations on the output feature map in sequence according to the initialized post-processing mode to obtain the final inference result.

2. The multi-scale reconfigurable vision Transformer accelerator of claim 1, wherein, The reconfigurable computing array module internally includes an array controller, a configurable multiply-accumulate unit, a weight register group, and a partial cache. The array controller dynamically reconstructs the computing path of the reconfigurable computing array module according to the initialization computing mode configuration instructions issued by the CPU, supporting ordinary convolution computing mode, depthwise convolution computing mode, pointwise convolution computing mode, or linear projection computing mode.

3. The multi-scale reconfigurable vision Transformer accelerator of claim 2, wherein, The configurable multiply-accumulate supports a mixed precision packing mode of 8-bit integers and 4-bit integers. When configured for 8-bit integer precision, it operates in shared operand mode, where a single activation value is multiplied in parallel with two weights. When configured with 4-bit integer precision, it operates in multi-channel independent packaging mode, multiplying the two activation values ​​and the two weights in parallel, and outputting the four-channel multiplication and accumulation result.

4. The multi-scale reconfigurable vision Transformer accelerator of claim 2, wherein, In the ordinary convolution calculation mode, the reconfigurable computing array module performs cross-channel weighted multiplication and accumulation operations. The output feature map is obtained by summing the weights and the input feature map along the input channel dimension, and then accumulating them in the spatial dimension to generate the output feature map. In the deep convolutional computation mode, the reconfigurable computation array module performs channel-independent multiplication and accumulation operations. The output feature map is obtained directly by multiplying the weights and the input feature map, and then accumulated through spatial dimensions to generate the output feature map. In the pointwise convolutional computation mode, the reconfigurable computation array module performs cross-channel multiplication and accumulation operations without spatial dimension accumulation, and directly outputs the feature map; In the linear projection calculation mode, the reconfigurable calculation array module switches to a matrix-vector multiplication architecture, and the output activation value is obtained by weighted summation of the input features and projection weights along the dimension.

5. The multi-scale reconfigurable vision Transformer accelerator of claim 1, wherein, It also includes a configurable addition tree module, which is connected to the reconfigurable computing array module and the post-processing module respectively, and is used to dynamically adjust the accumulation path and parallelism according to the initialized computing mode, and to perform channel dimension accumulation and spatial dimension accumulation on the convolution part.

6. The multi-scale reconfigurable vision Transformer accelerator of claim 1, wherein, The on-chip cache module adopts a ping-pong cache structure, which supports parallel read and write operations. It reads historical data to participate in the calculation while writing new feature data, so as to hide the memory access latency. The process of generating the query matrix Q, key matrix K, and value matrix V adopts a block-based processing method. When the size of the input feature map exceeds the capacity of the on-chip cache module, the input feature map is divided into several sub-blocks and loaded and calculated block by block.

7. The multi-scale reconfigurable vision Transformer accelerator of claim 1, wherein, The post-processing module includes a post-processing controller, a normalization module, an activation function calculation module, a pooling module, an upsampling module, and a quantization calculation module. The post-processing controller selects the working mode according to the configuration instructions, the normalization module supports dual-mode switching between LayerNorm and BatchNorm, the activation function calculation module supports ReLU or HardSwish activation functions, and the quantization calculation module supports uniform quantization or logarithmic quantization.

8. The multi-scale reconfigurable vision Transformer accelerator of claim 7, wherein, In the layer normalization mode, the normalization module uses hardware to calculate the mean and variance online based on the current input features, and completes the normalization through parallel summation, sum of squares, fixed-point square root and division operations. In batch normalization mode, the normalization module uses pre-stored mean and variance parameters to complete the normalization. Both modes reuse the same set of subtraction, multiplication, square root, division, and addition operation pathways.

9. The multi-scale reconfigurable vision Transformer accelerator of claim 1, wherein, The reconfigurable computing array module further comprises a data rearrangement module for transposing the key matrix K to obtain a transposed key matrix K T ; When performing linear attention calculation, the reconfigurable computing array module uses a block caching method to cache the intermediate attention score matrix to the on-chip cache module or off-chip memory.

10. A multi-scale reconfigurable vision Transformer acceleration method applied to the multi-scale reconfigurable vision Transformer accelerator of any one of claims 1-9, characterized in that, Includes the following steps: Step S1: The top-level control module receives configuration instructions from the CPU and initializes the calculation mode, precision configuration, cache block size, and post-processing mode according to the configuration instructions to obtain initialization configuration parameters including the post-processing mode. Step S2: Based on the initialization configuration parameters, the input feature map, weight parameters and normalization parameters in the off-chip memory are transferred to the on-chip cache module through the DMA module under the control of the top-level control module, so as to obtain cached feature map data, weight data and parameter data; Step S3: The reconfigurable computing array module reads corresponding data from the feature map data, weight data, and parameter data according to the initialized computing mode, performs multi-scale convolution and matrix multiplication operations, generates a query matrix Q, a key matrix K, and a value matrix V, and transposes the key matrix K to obtain the transposed key matrix K. T ; Step S4: The reconfigurable computing array module reads the query matrix Q and the transpose key matrix K. T Given a value matrix V, perform linear attention computation by sequentially combining the value matrix V with the transpose key matrix K. T Matrix multiplication is performed to obtain the attention score matrix, and then the attention score matrix is ​​multiplied with the query matrix Q to obtain the attention features. Step S5: The reconfigurable computing array module reads the weight parameters of the subsequent convolutional layer from the on-chip cache module, performs convolution calculation with the attention features, and obtains the output feature map; Step S6: The post-processing module performs normalization, activation function, pooling, upsampling and quantization operations on the output feature map in sequence according to the initialized post-processing mode to obtain the final inference result.

Images