Spatial channel level joint structured sparse method and accelerator

Abstract

The application discloses a space channel level joint structured sparse method and an accelerator, and comprises the following steps: obtaining a pre-trained convolutional neural network model, and performing redundancy analysis on each network layer to evaluate the sensitivity of different network layers to sparse processing; determining the retention ratio of the basic sparse unit according to the redundancy index of the current network layer, the target overall sparsity, the calculation amount and the parameter size; configuring all dense data blocks in the current network layer according to the retention ratio of the basic sparse unit to generate the corresponding structured sparse mode; fine-tuning training based on the structured sparse model, restoring the model accuracy by updating the reserved weight parameters, and performing quantization processing on the fine-tuned model to map the floating-point weight and activation data to low-bit-width integer representation, and finally generating a structured sparse quantization model for hardware deployment. The accelerator adopts a heterogeneous computing architecture in which PS and PL work cooperatively, the PS side is responsible for system control, task scheduling, data communication and software and hardware cooperative management, and the PL side is responsible for inference calculation of the structured sparse neural network.

Description

Technical Field

[0001] This invention relates to the field of edge artificial intelligence hardware acceleration technology, and in particular to a spatial channel hierarchical joint structured sparse method and accelerator. Background Technology

[0002] CNNs have achieved remarkable results in artificial intelligence tasks such as computer vision, object detection, and image recognition. With the rapid development of edge intelligence applications such as autonomous driving, intelligent monitoring, wearable devices, and the Industrial Internet of Things, more and more CNN models need to be deployed on resource-constrained devices such as mobile terminals, IoT devices, and embedded platforms. [1][2] Therefore, achieving efficient hardware acceleration of CNNs while maintaining model accuracy on resource-constrained edge devices has become a critical challenge in the field of edge intelligence. CNN models are characterized by high computational complexity and large parameter storage requirements. Utilizing the sparsity in neural networks for model compression has become an important technical approach to improve the efficiency of CNN accelerators. Sparsity can be divided into two main categories: unstructured sparsity and structured sparsity.

[0003] The distribution of non-zero weights in unstructured sparse structures is highly irregular, requiring the introduction of additional mechanisms such as index storage, sparse decoding, and dynamic scheduling. This incurs significant hardware overhead. [3] Existing methods for accelerating unstructured sparse data mainly fall into two categories: one is based on data matching, which dynamically matches input activations and sparse weights before the multiply-accumulate operation to identify valid non-zero computation pairs. However, due to the highly irregular distribution of non-zero elements, the matching process is complex, making it difficult to construct a regular pipelined execution structure. Furthermore, the matching relationship changes dynamically with the channel and spatial location, which is not conducive to efficient parallel hardware execution. The other is based on result collection, which uses an outer product computation paradigm to multiply non-zero weights with all non-zero activations in pairs, thus avoiding the complex operand reordering and matching process at the front end. However, the generated intermediate results need to be dynamically routed to different accumulation units for aggregation. The back-end result collection and accumulation process itself has high complexity, usually requiring complex interconnect structures based on cross switches and multi-level buffer structures, leading to communication and storage overhead. At the same time, the dynamic spatiotemporal distribution of intermediate results makes efficient pipelined accumulation difficult, further increasing the pressure on accumulation latency, accumulator access frequency, and on-chip memory access. These factors together make it difficult to achieve efficient acceleration under the limited resources of edge devices.

[0004] Structured sparsity, due to its regular data organization and hardware-friendly characteristics, has become an important research direction for mobile inference and edge deployment. Existing structured sparsity methods mainly include: channel-level structured sparsity, block-level structured sparsity, and hierarchical structured sparsity. Channel-level structured sparsity achieves model compression by removing redundant entire channels or convolutional kernels, offering hardware-friendly advantages. However, since different channels typically encode specific semantic information or low-level visual features (e.g., edges, textures, and local structures), coarse-grained pruning can easily reduce the network's representational and overall expressive power. To mitigate the accuracy loss caused by channel-level sparsity under high compression ratios, methods such as S2TA have emerged in recent years. [4] Block-level structured sparsity methods, such as S2TA, divide input channels into structured groups and perform block-level sparsity, achieving more flexible redundancy removal while maintaining computational regularity. However, S2TA's sparsity modeling still mainly relies on channel-level structure, failing to fully utilize structural redundancy in the convolutional kernel's spatial dimensions (i.e., height and width), thus limiting its ability to capture finer-grained sparse patterns. Furthermore, existing research shows that different layers in a neural network exhibit significantly different sensitivities to sparsity, with marked inter-layer redundancy heterogeneity. S2TA, employing a uniform sparsity configuration across layers, fails to model this difference, leading to some layers retaining excessive redundancy while others are over-pruned, resulting in performance degradation. To further enhance sparsity flexibility, Yannan Nellie Wu et al. proposed a hierarchical structured sparsity method based on S2TA. [5] While retaining the block-level sparsity strategy, this method introduces a more advanced pruning mechanism, combining multiple sparse blocks into larger structured pruning units. This allows for finer control of the overall sparsity rate through the product of sparsity rates at different levels. However, this method still relies on relatively coarse-grained structural rules. Under high sparsity rates, this coarse-grained structure often weakens the network's representational power, leading to a decrease in accuracy.

[0005] This limitation prevents the method from fully leveraging the advantages of high sparsity in edge deployment scenarios. Under resource-constrained conditions, there is still considerable computational and memory redundancy, which limits its actual efficiency.

[0006] References

[0007] [1]Y. Feng, P. Whatmough, and Y. Zhu, “Asv: Accelerated Stereo VisionSystem,” in Proceedings of IEEE / ACM International Symposium on Microarchitecture (MICRO), 2019, pp. 643–656.

[0008] [2]Z.-G. Liu, P. N. Whatmough, and M. Mattina, “Systolic TensorArray:An Efficient Structured-Sparse GEMM Accelerator for Mobile CNNInference,” IEEE Computer Architecture Letters, vol. 19, no. 1, pp. 34–37,202.

[0009] [3]M. Liu, C. Zhou, S. Qiu, Y. He, and H. Jiao, “CNN Accelerator atthe Edge With Adaptive Zero Skipping and Sparsity-Driven Data Flow,” IEEETransactions on Circuits and Systems for Video Technology,vol. 33, no. 12,pp. 7084–7095, 2023.

[0010] [4]Z.-G. Liu, P. N. Whatmough, Y. Zhu, and M. Mattina, “S2TA:Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Acceleration,”in Proceedings of IEEE International Symposium on High-Performance ComputerArchitecture (HPCA), 2022, pp. 573–586.

[0011] [5]YN Wu, P.-A. Tsai, S. Muralidharan, A. Parashar, V. Sze, and J.Emer, “HighLight: Efficient and Flexible DNN Acceleration with HierarchicalStructured Sparsity,” in Proceedings of IEEE / ACM International Sym-posium onMicroarchitecture (MICRO), 2023, pp. 1106–1120 Summary of the Invention

[0012] To address the challenges of large parameter size, high computational cost, high storage overhead, and limited inference efficiency in convolutional neural network models on resource-constrained edge devices, this invention proposes a spatial channel-level joint structured sparse method and accelerator. By combining structured sparse compression with hardware co-optimization, this method effectively reduces model storage requirements and computational overhead while maintaining model accuracy, thereby improving the deployment efficiency and inference performance of neural networks on edge devices. Details are described below:

[0013] A first aspect is a spatial channel hierarchical joint structured sparsity method, the method comprising:

[0014] Obtain a pre-trained convolutional neural network model and perform redundancy analysis on each network layer to assess the sensitivity of different network layers to sparsification.

[0015] The retention ratio of basic sparse units is determined based on the redundancy index of the current network layer, the overall sparsity of the target, the computational cost, and the parameter scale.

[0016] Based on the retention ratio of the basic sparse units, all dense data blocks in the current network layer are configured to generate the corresponding structured sparse pattern.

[0017] Fine-tuning training is performed based on a structured sparse model. The model accuracy is restored by updating the retained weight parameters. Quantization processing is then performed on the fine-tuned model to map floating-point weights and activation data into low-bit-width integer representations, ultimately generating a structured sparse quantization model for hardware deployment.

[0018] The pre-trained convolutional neural network model is as follows:

[0019] For convolutional layers, the input channel dimension is divided into multiple consecutive sub-input channel groups, and each sub-input channel group and the spatial dimension of the corresponding convolutional kernel together form a three-dimensional structured data block. The computational units that participate in sparsification processing inside the structured data block are defined as dense data blocks.

[0020] For fully connected layers, continuous data groups are divided along the input neuron dimension to form dense data blocks, achieving a unified data organization method and sparsification process for convolutional and fully connected layers.

[0021] The second aspect is a spatial channel-level joint structured accelerator, wherein the accelerator adopts a heterogeneous computing architecture in which PS and PL work together. The PS side is responsible for system control, task scheduling, data communication and hardware and software collaborative management, while the PL side is responsible for inference computing of structured sparse neural networks.

[0022] The input feature map data, compressed non-zero weight data, and corresponding index data are transmitted to the data storage module through the bus interface. The data storage module adopts a double buffering mechanism.

[0023] A distributed, independently managed storage architecture is built based on the input feature map. The input feature map is divided into multiple independent storage blocks. Each storage block is configured with an independent address management unit. Different storage blocks can independently control parallel output data to achieve continuous data supply in the pipeline.

[0024] The cascaded multiplexer extracts the corresponding input data from the input feature map based on the index information and sends it to the multiplier to complete the multiplication operation between the input data and the non-zero weights. The cascaded multiplexer adopts a two-level hierarchical structure to complete the data selection step by step; the calculation result is sent to the quantization module for quantization processing.

[0025] The distributed, independently managed storage architecture constructed based on the input feature map is as follows:

[0026] When feature map data comes from convolutional layers, the feature maps are grouped and stored according to the input channel dimension. For each group of input channel data, the feature map is divided according to the row and column dimensions, and then mapped and stored in the corresponding BRAM. The storage address in each BRAM corresponds one-to-one with the spatial location of the feature map.

[0027] When the feature map data comes from the fully connected layer, the data is divided into multiple storage groups according to the input channel dimension. The number of groups is consistent with the number of groups in the BRAM of the convolutional layer storage structure. Data access control can be completed by maintaining a single address variable.

[0028] The double buffering mechanism is as follows:

[0029] The dual-buffering mechanism consists of multiple BRAMs in single-read, single-write port mode, and two logical buffers are constructed by dividing the address space. The dual buffers are managed in a unified manner according to their read / write status and empty / full status, so as to realize the parallel execution of data writing and data reading.

[0030] The feature map data first combines the data within the height and width range corresponding to the convolution kernel and some input channel data to form a dense block, and then sends the dense block to the computing module to perform convolution operation;

[0031] Once the data for the current dense block has been read, the feature map is accessed continuously. The subsequent dense block data is obtained in a pipeline manner by sliding the convolution window along the spatial dimension of the feature map, and convolution calculations are continuously performed.

[0032] As the convolution calculation proceeds, the part and data corresponding to the current output space block are continuously written into the accumulator buffer. When the accumulator buffer is full, the system switches to the next set of continuous input channel data.

[0033] While keeping the current output space block unchanged, the corresponding dense block data is obtained again in the same way as above, and the calculation results are accumulated into the existing partial data until all input channels are calculated. When all input channels of an output space block have been accumulated, the final result corresponding to the output space block is quantized and the quantized result is output to the off-chip memory.

[0034] The beneficial effects of the technical solution provided by this invention are:

[0035] 1. This invention proposes a spatial channel-level joint structured sparsity method. By jointly partitioning the spatial dimension and channel dimension of the convolutional layer weights and combining it with a layer-level sparsity configuration strategy, heterogeneous sparsity allocation of different network layers is achieved, which effectively reduces the decrease in model accuracy while improving model compression rate.

[0036] 2. This invention constructs a unified structured data block representation for feature map data, enabling convolutional layers and fully connected layers to adopt a consistent sparsification process, reducing the complexity of sparse data storage, index management, and data access, and improving the adaptability between the model and hardware architecture.

[0037] 3. This invention designs a hardware acceleration architecture for structured sparse neural networks. Through distributed data storage, double buffering mechanism and sparse computing unit based on cascaded multiplexers, it achieves efficient coordination between data access and computing processes, thereby reducing hardware resource overhead while improving system throughput and computing efficiency.

[0038] 4. This invention is highly adaptable to edge computing and embedded application scenarios where computing power, power consumption, and cost are all constrained. It can widely cover the needs of autonomous driving real-time perception, intelligent security edge analysis, industrial quality inspection high-throughput processing, AR / VR low-latency interaction, and mobile terminal intelligent enhancement. At the same time, its low power consumption and high energy efficiency characteristics are also suitable for resource-constrained fields such as intelligent IoT, drone control, and remote sensing image processing, providing a hardware foundation that combines performance and energy efficiency for extending AI inference from the cloud to the edge. Attached Figure Description

[0039] Figure 1 A schematic diagram of a spatial channel hierarchical structured sparse algorithm;

[0040] Figure 2 Design diagram of the overall hardware architecture of a spatial channel hierarchical structured sparse neural network accelerator;

[0041] Figure 3 Architecture diagram for distributed feature map storage;

[0042] Figure 4 This is a schematic diagram of a double buffering mechanism;

[0043] Figure 5 This is a schematic diagram of the partial and cumulative mechanism. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0045] Example 1

[0046] This invention proposes a spatial channel-level joint structured sparse method. Its core idea is to construct a hardware-friendly structured sparse model under the joint constraints of spatial dimension, channel dimension and network layer dimension, thereby reducing model storage overhead and computational complexity while ensuring model accuracy, and improving the deployment efficiency of neural networks on resource-constrained edge devices.

[0047] 101: Obtain a pre-trained convolutional neural network model and perform redundancy analysis on each network layer to assess the sensitivity of different network layers to sparsification.

[0048] like Figure 1As shown, for convolutional layers, the input channel dimension is divided into multiple continuous sub-input channel groups, and each sub-input channel group and the spatial dimension of the corresponding convolutional kernel together form a three-dimensional structured data block. At the same time, the computational units participating in sparsification processing within the structured data block are defined as dense data blocks. For fully connected layers, since there is no convolutional kernel spatial structure, continuous data groups are divided along the input neuron dimension to form dense data blocks, thereby achieving a unified data organization method and sparsification processing flow for convolutional layers and fully connected layers.

[0049] 102: Determine the retention ratio of basic sparse units based on the redundancy index of the current network layer, the overall sparsity of the target, the computational cost, and the parameter scale;

[0050] Here, a data block with a sparsity of 75% is defined as the basic sparse unit. That is, each basic sparse unit retains only 25% of the non-zero weights in the corresponding dense data block for calculation, and the remaining 75% of the weights are removed.

[0051] 103: Based on the retention ratio of the basic sparse units determined by each network layer, configure all dense data blocks in the current network layer to generate the corresponding structured sparse pattern;

[0052] Specifically, for network layers with high parameter redundancy, fewer basic sparse units are retained to achieve a higher sparsity rate; for network layers that are more sensitive to sparsity, more basic sparse units are retained to reduce the sparsity rate, thus forming a heterogeneous sparse configuration among different network layers. After completing the basic sparse unit configuration of all network layers, the corresponding structured sparse pattern is generated, realizing structured pruning under the joint constraints of spatial dimension, channel dimension, and network layer dimension.

[0053] 104: Fine-tuning training is performed based on the generated structured sparse model. The model accuracy is restored by updating the retained weight parameters. Furthermore, quantization processing is performed on the fine-tuned model to map floating-point weights and activation data into low-bit-width integer representations, ultimately generating a structured sparse quantization model for hardware deployment.

[0054] This step simultaneously exports the non-zero weight data, index data, and hierarchical sparse configuration information corresponding to the model, providing support for subsequent hardware accelerators to perform structured sparse inference.

[0055] In summary, by using steps 101-104 above, while maintaining the structured sparse regularity, the redundancy characteristics of different network layers are fully utilized to achieve an effective balance between model compression rate and model accuracy, and to reduce the storage requirements and computational overhead during the deployment of neural networks.

[0056] Example 2

[0057] This invention proposes a hardware accelerator for executing a spatial channel-level joint structured sparse neural network compression method. It employs a heterogeneous computing architecture with PS and PL working collaboratively to meet the requirements of resource-constrained edge devices for low latency, high throughput, and low resource overhead in neural network inference. Specifically, the PS side is responsible for system control, task scheduling, data communication, and hardware / software collaborative management, while the PL side is responsible for the inference computation of the structured sparse neural network.

[0058] like Figure 2 As shown, firstly, the input feature map data, compressed non-zero weight data, and corresponding index data are transmitted to the data storage module via a bus interface. To improve data transmission efficiency, the data storage module adopts a double-buffering mechanism, dividing the storage space into two independent buffers. One buffer is responsible for reading data for the current computation task, while the other buffer is responsible for loading the next batch of data. After the current computation task is completed, the two buffers switch functions, thereby achieving overlapping execution of data transmission and computation. Simultaneously, a distributed, independently managed storage architecture is constructed for the input feature map, dividing it into multiple independent storage blocks. Each storage block is configured with an independent address management unit, and different storage blocks can independently control parallel data output, achieving continuous data supply in the pipeline. For convolutional layers, only the address information corresponding to the spatial dimension of the convolution kernel needs to be maintained to complete structured sparse data access; for fully connected layers, data access is completed through a unified address control mechanism, and the address change strategy is dynamically adjusted according to the structured sparse configuration of the current network layer, such as... Figure 3 As shown, the system dynamically adjusts the change interval of the corresponding address in the address controller according to the current sparsity and layer type, so as to realize the reuse of feature map data under the corresponding sparsity and realize data access control under different sparsity modes.

[0059] Subsequently, the data storage module sends the input data to the sparse computation module for structured sparse convolution operations according to the current network layer configuration. The sparse computation module consists of multiple parallel processing groups, each responsible for the computation task of one output channel. Each processing group contains multiple processing units (PEs), which perform multiplication operations in parallel on a structured sparse data block, and the computation results are accumulated through an addition tree. Each processing unit reads compressed non-zero weight data from the weight register and the corresponding index from the index register. The cascaded multiplexer extracts the corresponding input data from the input feature map based on the index information and sends it to the multiplier to complete the multiplication operation between the input data and the non-zero weights. To reduce the logical resource overhead of large-scale input selection networks, the cascaded multiplexer adopts a two-level hierarchical structure, performing data selection step by step. Simultaneously, a pipeline register is set between the cascaded multiplexer and the addition tree to reduce critical path latency and increase system operating frequency.

[0060] Finally, the calculated results are sent to the quantization module for quantization processing, and then enter the pooling module to perform pooling operations according to the network structure requirements. The output results after processing are written back to the external memory through the bus interface, thus completing the entire inference process of the structured sparse neural network.

[0061] The proposed hardware-software co-processing solution ensures hardware friendliness while maintaining model accuracy and significantly reducing storage and computational overhead. This solution is highly adaptable to edge computing and embedded applications where computing power, power consumption, and cost are constrained, and can broadly cover the needs of real-time perception in autonomous driving, edge analysis in intelligent security, high-throughput processing in industrial quality inspection, low-latency AR / VR interaction, and intelligent enhancement of mobile terminals. Simultaneously, its low power consumption and high energy efficiency characteristics are also suitable for resource-constrained fields such as intelligent IoT, drone control, and remote sensing image processing, providing a hardware foundation that combines performance and energy efficiency for extending AI inference from the cloud to the edge.

[0062] Example 3

[0063] This embodiment of the invention begins with the storage process after the feature map data is transmitted to the PL terminal, and then describes the specific implementation of this embodiment. For example... Figure 3 As shown, the data organization and storage method is first selected according to the network layer type to which the feature map data belongs. When the feature map data comes from a convolutional layer, the feature maps are grouped and stored according to the input channel dimension. For each group of input channel data, it is further divided according to the row and column dimensions of the feature map and mapped and stored in the corresponding BRAM (Block Random Access Memory). The storage address in each BRAM corresponds one-to-one with the spatial location of the feature map, thereby realizing the regularized storage and efficient access of the feature map data. Since the data in the same storage group is organized according to a fixed mapping relationship, only a small number of address variables corresponding to the convolution kernel window need to be maintained during the convolution calculation to complete the data access control, thereby effectively reducing the complexity of address generation and management logic. After all the input channel data in the current storage group has been stored, the data of the next group of consecutive input channels is stored according to the same data mapping rule until all feature map data is stored. When the feature map data comes from a fully connected layer, the data is divided into multiple storage groups according to the input channel dimension. The number of groups is consistent with the number of groups in the BRAM of the convolutional layer storage structure, thereby realizing a unified data organization and access method. Since fully connected layers do not involve convolutional window sliding operations, data access control can be completed by maintaining only a single address variable, further simplifying address management logic. Furthermore, as... Figure 4As shown, all types of data buffers employ a double-buffered structure for data storage and retrieval. The double-buffered structure consists of multiple BRAMs operating in single-read, single-write port mode, and is constructed into two logical buffers through address space partitioning. The system manages the two buffers uniformly based on their read / write status and full / empty state, enabling parallel execution of data writing and reading, thereby improving data transmission efficiency and reducing the waiting time of the computing units. Specifically, when the current write buffer is full, a buffer full signal is generated. The control module determines whether subsequent data should be written immediately based on the full / empty state of the other buffer. When the other buffer is idle, subsequent data will be written to that buffer immediately, achieving switching between the two buffers. Simultaneously, when the computing module issues a data read request, it first checks the status of the current buffer. When the buffer is full, all BRAMs within the buffer are accessed synchronously according to their addresses, and the corresponding data is output in parallel to the processing unit array for computation. As data continues to be read, when all data in the buffer has been read, a buffer empty signal is generated. The control module switches the buffer roles based on the current read / write status, causing the buffer to re-enter the write state, while the other buffer enters the read state, thus achieving continuous pipelined execution of data storage and computation. By using the aforementioned feature map storage organization method and dual buffer management mechanism, not only can the overhead of address generation and storage management be significantly reduced, but also the overlapping execution of data transmission and computation processes can be achieved, effectively improving the accelerator's resource utilization and overall computing throughput while ensuring data access bandwidth.

[0064] Example 4

[0065] This invention describes the specific implementation of the invention, starting with the reading of feature map data from the PL end. For example... Figure 5As shown, the feature map data is first read in the manner indicated by ① in the figure. The data within the height and width range corresponding to the convolution kernel, along with some input channel data, are combined to form a dense block, which is then sent to the computation module for convolution operation. Once the data of the current dense block is read, the feature map is accessed continuously in the direction indicated by ② in the figure. The subsequent dense block data is obtained in a pipeline manner by sliding the convolution window along the feature map spatial dimension, and convolution calculations are continuously performed. This process enables the reuse of convolution kernel weights, reducing the repeated loading of weights and feature map data, thereby reducing data transmission overhead and improving the utilization of the computation unit. As the convolution calculation progresses, the corresponding part and data of the current output spatial block are continuously written into the accumulation buffer. When the accumulation buffer is full, the convolution window moves to the position indicated by ③ in the figure. Subsequently, as indicated by ④ in the figure, the system switches to the next set of continuous input channel data. While keeping the current output spatial block unchanged, the corresponding dense block data is obtained again in the same manner as above, and the calculation results are accumulated into the existing part and data until all input channels have been calculated. After all input channels of an output space block have completed accumulation, the final result corresponding to that output space block is quantized, and the quantized result is output to off-chip memory. This data reading and calculation method effectively reduces the storage pressure on on-chip partial sums and data, avoids frequent write-back of partial sums and data to external memory, thereby reducing the bandwidth requirements of external memory, reducing data transmission latency, and improving the overall computational efficiency of the system. The output result of the addition tree is then sent to the accumulation module. The accumulation module generates corresponding control signals based on the counter determined by the current sparsity. If the input channels corresponding to the current output position have not yet completed calculation, the output result of the addition tree is accumulated with the partial sum data stored in the accumulation register, and the updated result is rewritten to the accumulation register; if all input channels corresponding to the current output position have completed calculation, the final result in the accumulation register is directly output and saved to the accumulation buffer.

[0066] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0067] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A spatial channel hierarchical joint structured sparse method, characterized in that, The method includes: Obtain a pre-trained convolutional neural network model and perform redundancy analysis on each network layer to assess the sensitivity of different network layers to sparsification processing. The retention ratio of basic sparse units is determined based on the redundancy index of the current network layer, the overall sparsity of the target, the computational cost, and the parameter scale. Based on the retention ratio of the basic sparse units, all dense data blocks in the current network layer are configured to generate the corresponding structured sparse pattern. Fine-tuning training is performed based on a structured sparse model. The model accuracy is restored by updating the retained weight parameters. Quantization processing is then performed on the fine-tuned model to map floating-point weights and activation data into low-bit-width integer representations, ultimately generating a structured sparse quantization model for hardware deployment.

2. The spatial channel hierarchical joint structured sparse method according to claim 1, characterized in that, The pre-trained convolutional neural network model is: For convolutional layers, the input channel dimension is divided into multiple consecutive sub-input channel groups, and each sub-input channel group and the spatial dimension of the corresponding convolutional kernel together form a three-dimensional structured data block. The computational units that participate in sparsification processing inside the structured data block are defined as dense data blocks. For fully connected layers, continuous data groups are divided along the input neuron dimension to form dense data blocks, achieving a unified data organization method and sparsification process for convolutional and fully connected layers.

3. A spatial channel-level joint structured accelerator, characterized in that, The accelerator adopts a heterogeneous computing architecture in which the PS and PL work together. The PS side is responsible for system control, task scheduling, data communication and hardware and software collaborative management, while the PL side is responsible for inference computing of structured sparse neural networks. The input feature map data, compressed non-zero weight data, and corresponding index data are transmitted to the data storage module through the bus interface. The data storage module adopts a double buffering mechanism. A distributed, independently managed storage architecture is built based on the input feature map. The input feature map is divided into multiple independent storage blocks. Each storage block is configured with an independent address management unit. Different storage blocks can independently control parallel output data to achieve continuous data supply in the pipeline. The cascaded multiplexer extracts the corresponding input data from the input feature map based on the index information and sends it to the multiplier to complete the multiplication operation between the input data and the non-zero weights. The cascaded multiplexer adopts a two-level hierarchical structure to complete the data selection step by step; the calculation result is sent to the quantization module for quantization processing.

4. A spatial channel hierarchical co-structured accelerator according to claim 3, characterized in that, The distributed, independently managed storage architecture based on the input feature map is as follows: When the feature map data comes from the convolutional layer, the feature maps are grouped and stored according to the input channel dimension. For each group of input channel data, the feature map is divided according to the row and column dimensions and then mapped and stored in the corresponding BRAM. Each BRAM memory address corresponds one-to-one with the spatial location of the feature map. When the feature map data comes from the fully connected layer, the data is divided into multiple storage groups according to the input channel dimension. The number of groups is consistent with the number of groups in the BRAM of the convolutional layer storage structure. Data access control can be completed by maintaining a single address variable.

5. A spatial channel hierarchical co-structured accelerator according to claim 3, characterized in that, The double buffering mechanism is as follows: The dual-buffering mechanism consists of multiple BRAMs in single-read, single-write port mode, and two logical buffers are constructed by dividing the address space. The dual buffers are managed in a unified manner according to their read / write status and empty / full status, so as to realize the parallel execution of data writing and data reading.

6. A spatial channel hierarchical co-structured accelerator according to claim 3, characterized in that, The feature map data first combines the data within the height and width range corresponding to the convolution kernel and some input channel data to form a dense block, and then sends the dense block to the computing module to perform convolution operation; Once the data for the current dense block has been read, the feature map is accessed continuously. The subsequent dense block data is obtained in a pipeline manner by sliding the convolution window along the spatial dimension of the feature map, and convolution calculations are continuously performed. As the convolution calculation proceeds, the part and data corresponding to the current output space block are continuously written into the accumulator buffer. When the accumulator buffer is full, the system switches to the next set of continuous input channel data. While keeping the current output space block unchanged, the corresponding dense block data is obtained again in the same way as above, and the calculation results are accumulated into the existing partial data until all input channels are calculated. When all input channels of an output space block have been accumulated, the final result corresponding to the output space block is quantized and the quantized result is output to the off-chip memory.

Images