A Spatial-Spectrum Dual-Axis Cross-Layer State Compression Method and System for Visual Transformers

Abstract

This invention discloses a spatial-spectral dual-axis cross-layer state compression method and system for visual Transformers. The method first extracts spatial summaries of the hidden states at each layer, retaining category tokens and pooling image patch tokens by fixed regions to generate a small number of spatial summary tokens. Then, along the network depth direction, the values ​​at the same summary position in each layer are treated as sequences, and the K pre-stored spectral buffers are updated using a Walsh-Hadamard transform in a streaming accumulation manner to achieve depth-dimensional compression. Next, a weighted retrieval of the spectral buffers is performed using a learnable query vector to obtain the cross-layer summary context of the current layer. Finally, the token space is filled back into the complete token space through difference calculation and static routing, and then adaptively fused using learnable gating to generate the enhanced state. This invention reduces the cross-layer state storage scale and significantly reduces cache overhead through dual compression of spatial and depth dimensions. It employs fixed region partitioning and symbol transformation, making it hardware-friendly and suitable for deployment on FPGAs, ASICs, and low-power NPUs.

Description

Technical Field

[0001] This invention belongs to the field of deep learning model compression and acceleration technology, and provides a spatial-spectral dual-axis cross-layer state compression method and system for visual Transformers. Background Technology

[0002] Visual Transformers have been widely applied to tasks such as image classification, object detection, semantic segmentation, and industrial vision inspection. These models typically consist of stacked Transformer blocks, each containing multi-head self-attention, a feedforward network, a normalization layer, and residual connections. To enhance information flow in deep networks, research has further introduced cross-layer residuals, cross-layer attention, or historical state retrieval mechanisms, enabling the current layer to access intermediate representations from earlier layers. Meanwhile, some visual Transformer compression methods reduce the number of tokens involved in attention computation in the current layer through token pruning, token merging, or dynamic token selection, thereby reducing computational cost and inference latency.

[0003] However, existing methods still have the following significant shortcomings on resource-constrained platforms: 1) If the cross-layer mechanism directly saves the complete token state of each layer, it is necessary to maintain a historical cache of size L×T×d, where L is the number of layers, T is the number of tokens, and d is the number of channels, resulting in high on-chip storage and external bandwidth overhead. 2) Existing token compression mainly operates on the spatial dimension, while cross-layer residuals mainly operate on the depth dimension, and there is a lack of a unified compression framework for the two; 3) Dynamic token routing depends on the input content, which is complex to control in hardware and not conducive to pipeline implementation; 4) Directly injecting cross-layer information may disrupt the original residual path of the pre-trained model, causing instability or decreased accuracy in the early stages of training.

[0004] Therefore, there is an urgent need for a method that can efficiently compress cross-layer states from both spatial and depth dimensions, and is hardware-friendly and stable in training. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a spatial-spectral dual-axis cross-layer state compression method and system for visual Transformers, which aims to solve the problems of large cross-layer state storage overhead, low hardware deployment efficiency, and unstable training in the prior art. It is especially suitable for visual inference tasks on hardware platforms such as resource-constrained edge computing, FPGA, ASIC and low-power NPU.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A spatial-spectral biaxial cross-layer state compression method for visual Transformers, which specifically includes the following steps: S1: Spatial summary extraction: For the hidden state X(l) at the l-th cross-layer position in the visual Transformer model, it is compressed into a spatial summary state Z(l) according to a fixed spatial partitioning strategy; wherein, the spatial summary state Z(l) contains a global semantic summary and multiple region summaries, and the region summaries are obtained by performing pooling operations on image block tokens within a fixed region; S2: Depth Spectrum Accumulation: The spatial summary state Z(l) generated at each cross-layer position is regarded as a sequence along the network depth direction. The K pre-generated spectrum buffers F(k) are updated in a streaming accumulation manner using Walsh-Hadamard transformation, so that the K spectrum buffers F(k) accumulate the spectrum components representing the cross-layer historical information; where K is a preset positive integer, representing the number of spectrum components retained. S3: Spectral Context Retrieval: At the l-th cross-layer position, the K spectral buffers F(k) are weighted and summed using the learnable query vector W(l) to calculate the cross-layer summary context Zc(l) of the current layer; S4: Cross-layer information backfilling and adaptive fusion: The cross-layer summary context Zc(l) is compared with the spatial summary state Z(l) of the current layer to obtain summary supplementary information. The summary supplementary information is backfilled into the complete token space according to the fixed space partitioning strategy to obtain the supplementary state D(l). The injection intensity of the supplementary state D(l) is controlled by the learnable channel adjustment vector G(l). The adjusted supplementary state is fused with the current hidden state X(l) to generate the enhanced state Xe(l).

[0007] Furthermore, in step S1, the fixed space partitioning strategy specifically involves: dividing the image block token into several non-overlapping fixed regions according to its two-dimensional image position; performing average pooling or weighted average on all image block tokens within each fixed region to obtain a region summary; and simultaneously retaining the category token directly as a global semantic summary.

[0008] Further, in step S2, the updating of the spectrum buffer F(k) using the Walsh-Hadamard transform in a streaming accumulation manner specifically involves: pre-generating a Hadamard symbol table S(k,l); when the spatial summary state Z(l) of the l-th cross-layer position is generated, performing an accumulation or subtraction operation on the k-th spectrum buffer F(k) according to the symbol of the Hadamard symbol table S(k,l); wherein, if S(k,l) is positive, Z(l) is accumulated into F(k), and if S(k,l) is negative, Z(l) is subtracted from F(k).

[0009] Furthermore, in step S3, the specific process of calculating the cross-layer summary context Zc(l) is as follows: First, each spectrum buffer F(k) is scored using the learnable query vector W(l); second, the scoring results are converted into normalized weights A(k,l); finally, the K spectrum buffers F(k) are weighted and summed according to the weights A(k,l) to obtain Zc(l) = Σ A(k,l)·F(k).

[0010] Furthermore, step S4 specifically includes: Calculate the difference between the cross-layer summary context Zc(l) and the spatial summary state Z(l) of the current layer, and use it as supplementary information for the summary; Based on the fixed space partitioning strategy, the summary supplementary information is backfilled into the corresponding category token and each image block token through the static routing table to obtain the supplementary state D(l) with the same size as the current hidden state X(l); The intensity of D(l) is adjusted channel by channel using a learnable channel adjustment vector G(l), and the adjusted result is superimposed on X(l) to obtain the enhanced state Xe(l).

[0011] Furthermore, the region division method in the fixed space division strategy is independent of the input image content and has the same division structure for different input images; preferably, when the image block token grid is 14×14, it is divided into 3×3 regions, and the segmentation method of each region in the row direction and column direction adopts 5, 5, 4 or similar non-uniform division.

[0012] Furthermore, the number K of the K spectral components is a configurable hyperparameter, and preferably, K is 3.

[0013] The present invention also provides a spatial-spectral dual-axis cross-layer state compression system for vision Transformers, which is used to perform the method described above.

[0014] The beneficial effects of this invention are as follows: This invention significantly reduces the storage and access overhead of cross-layer states in the visual Transformer through two independent compression paths: spatial summarization and depth spectrum. Traditional cross-layer mechanisms, if storing all historical token states, have a state size of L×T×d; this scheme reduces it to K×(g+1)×d. Taking a common setting of T=197, L=24, g=9, K=3 as an example, the state size is reduced from 24×197×d to 3×10×d, a theoretical compression ratio of approximately 157.6 times. Because the spatial summarization uses fixed region partitioning and rebroadcasting uses a static routing table, the inference process does not rely on dynamic token sorting or content-related routing, resulting in simple control logic suitable for FPGA, ASIC, and low-power NPU implementations. The depth direction uses Walsh-Hadamard spectrum accumulation, and the update process is mainly completed by addition, subtraction, and sign control, eliminating the need for complex multipliers. Simultaneously, zero-initialization gating ensures that the original visual Transformer behavior is not altered during the initial module insertion, differential injection reduces redundant information accumulation, and channel-by-channel gating improves fusion flexibility. Therefore, this solution maintains the ability to reuse information across layers while improving on-chip cache usage, external bandwidth pressure, hardware regularity, and training stability, making it particularly suitable for visual Transformer inference deployment on resource-constrained platforms.

[0015] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0016] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the structure of spatial summarization, deep spectrum, and cross-layer fusion. Detailed Implementation

[0017] The present invention will be further described below with reference to specific embodiments. The accompanying drawings are for illustrative purposes only, representing schematic diagrams rather than actual physical objects, and should not be construed as limiting the scope of this patent. To better illustrate the embodiments of the present invention, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0018] This invention provides a spatial spectrum biaxial cross-layer state compression method for visual Transformers. Figure 1 The diagram below illustrates the process of the method of this invention, which includes four steps: spatial summary extraction, deep spectrum accumulation, spectrum context retrieval, cross-layer information backfilling, and adaptive fusion. Figure 2 This is a schematic diagram of the structure of spatial summarization, deep spectrum, and cross-layer fusion.

[0019] 1. Spatial Summary Extraction During inference, the visual Transformer generates hidden states at multiple layer positions. For ease of explanation, this scheme denotes the hidden state at the l-th cross-layer position as X(l), whose size can be understood as T×d, where T represents the number of tokens and d represents the number of channels for each token. If each X(l) is directly saved as a complete cross-layer history state, then all token information needs to be saved at each cross-layer position. When the input image is 224×224 and the patch size is 16×16, a common visual Transformer will generate 196 patch tokens, plus 1 class token, so T is usually 197. If the model contains multiple Transformer blocks, and cross-layer positions are set before self-attention and before the feedforward network, the number of history states that need to be saved will increase rapidly.

[0020] This scheme first performs a spatial summary on X(l) to obtain Z(l). Z(l) is not a complete token sequence, but consists of a small number of summary tokens. Specifically, the class token is directly retained as a global semantic summary; the patch token is divided into several fixed regions according to its two-dimensional image position, and the patch tokens in each region are averaged to form a region summary. Taking a 14×14 patch grid as an example, it can be divided into 3×3 regions, which is 9 region summaries. Since 14 is not divisible by 3, a segmentation method of 5, 5, 4 can be used in each direction to ensure that all patch tokens are covered. In this way, only 1 class token summary and 9 region summaries need to be retained at each cross-layer position, for a total of 10 summary tokens.

[0021] After this step, the number of tokens required to enter the cross-layer cache at each layer is reduced from 197 to 10, and the spatial cache size is reduced to approximately 1 / 19.7 of its original size. This spatial digest method uses fixed region partitioning and does not depend on the input image content, thus different images have the same routing structure. Compared to dynamic token pruning or dynamic token merging methods, this scheme does not require sorting, selecting, or reorganizing tokens; the hardware implementation only requires fixed address reading, region accumulation, and a static routing table. It should be noted that this step only compresses the cross-layer auxiliary state and does not replace the complete token representation in the backbone network. That is, the self-attention and feedforward networks inside the Transformer block can still use the original token sequence; the spatial digest is only used to reduce the cost of cross-layer information storage and retrieval.

[0022] 2. Depth Spectrum Accumulation After spatial summarization is completed, a summary state Z(l) is generated for each cross-layer location. If all Z(l) are still saved layer by layer in the usual way, the cross-layer cache will still grow linearly with the number of layers. Therefore, this scheme further compresses the network depth. The core idea is that the change of the same summary location in different layers can be regarded as a signal that unfolds along the depth direction, and this signal usually has strong inter-layer correlation and does not need to be completely saved layer by layer.

[0023] This scheme uses the Walsh-Hadamard transform to represent the spectral density of the depth-direction summary state. Since the coefficients of the Walsh-Hadamard transform only have positive and negative signs, it is well-suited for hardware implementation. The system pre-generates a Hadamard symbol table and retains only the first K spectral components. Here, K is a configurable parameter used to control the amount of information retained in the depth direction. A larger K retains more cross-layer information but also increases caching and computational overhead; a smaller K results in higher compression efficiency but may lose some cross-layer details. In a preferred configuration, K of 3 is sufficient to represent the main cross-layer trends.

[0024] This scheme does not require waiting for all layers to complete the calculation before performing the full transformation; instead, it adopts a streaming accumulation method. The system maintains K spectrum buffers, denoted as F(0), F(1), ..., F(K-1). When the l-th digest state Z(l) is generated, the system reads the corresponding symbol S(k,l) from the Hadamard symbol table. If S(k,l) is positive, Z(l) is added to the k-th spectrum buffer F(k); if S(k,l) is negative, Z(l) is subtracted from F(k). Therefore, for each new cross-layer digest state, the system only needs to perform a symbol-controlled addition or subtraction update on the K spectrum buffers once.

[0025] In this way, the system does not need to store all historical summary states, but only needs to maintain a small spectral buffer. Taking 24 cross-layer locations and K equal to 3 as an example, the number of states that need to be stored in the depth direction is reduced from 24 sets to 3 sets. Combined with spatial summarization, the scale of cross-layer states is reduced from 24 layers and 197 tokens per layer to 3 spectral components and 10 summary tokens per component. This process does not rely on complex multiplication, but only involves addition, subtraction and fixed scaling, making it suitable for implementation as a regular pipeline structure in FPGAs, ASICs and low-power NPUs.

[0026] 3. Spectrum Context Retrieval The spectrum buffer stores compressed historical cross-layer information, but different layers have different needs for historical information. For example, shallow layers may need more stable global trends, while deeper layers may need more supplementary information containing inter-layer differences; classification tasks may rely more on the spectrum information corresponding to class tokens, while detection or segmentation tasks may rely more on the spectrum information corresponding to region summaries. Therefore, this scheme sets a learnable query vector, denoted as W(l), for each cross-layer location to select the most needed cross-layer context from the spectrum buffer.

[0027] Specifically, at the l-th cross-layer position, the system first scores the K spectral buffers according to W(l). Each spectral buffer F(k) receives a score representing the current layer's demand for that spectral component. Subsequently, the system converts these scores into weights, denoted as A(k,l). The sum of all A(k,l) is 1, representing the current layer's allocation ratio among different spectral components. Finally, the system performs a weighted summation of F(0), F(1), ..., F(K-1) according to these weights to obtain the cross-layer summary context of the current layer, denoted as Zc(l).

[0028] This retrieval method differs from traditional cross-layer attention. Traditional cross-layer attention typically requires searching across all historical layer states, meaning it needs to access L historical states; this scheme only searches across K spectrum buffers. Since K is usually much smaller than L, the number of search objects is significantly reduced. Furthermore, because each set of spectrum states contains only a small number of digest tokens instead of the full 197 tokens, the overall memory access and computational cost are further reduced.

[0029] The advantage of this step is that it preserves the adaptive selection capability of cross-layer information while avoiding the direct storage and access of large-scale historical token states. The query vector W(l) can be automatically learned during training, enabling different combinations of cross-layer information for different layers, channels, and summary positions. Thus, this scheme does not simply use a fixed historical layer, nor does it roughly average all historical states, but rather performs lightweight contextual retrieval in the compressed spectral space.

[0030] 4. Cross-layer information backfilling and adaptive fusion After spectral context retrieval, the system obtains the cross-layer summary context Zc(l) required for the current layer. Since Zc(l) is still in the summary space and cannot be directly sent as a complete token sequence to the subsequent Transformer block, this step backfills it into the original token space and fuses it into the current hidden state X(l) in a controlled manner.

[0031] First, the system calculates the difference between Zc(l) and the current layer summary Z(l) to obtain supplementary summary information. Z(l) comes from the current hidden state X(l) and already contains information about the current layer itself. Therefore, using the difference form avoids redundant superposition and highlights the additional information provided by the historical spectrum state for the current layer. Subsequently, the system fills the complete token space with the supplementary summary information according to a fixed routing table: class tokens receive class token supplementary summary information, and each patch token receives supplementary summary information for the corresponding region based on its spatial region. This process strictly corresponds to the aforementioned spatial summary extraction steps and does not require dynamic sorting, dynamic matching, or content-related selection.

[0032] After backfilling, the system obtains a supplementary state D(l) with the same size as X(l). To ensure fusion stability, this scheme sets a learnable channel adjustment vector G(l) for each cross-layer position to control the injection intensity of D(l) on different channels. After adjustment, the system adds the supplementary state to the current hidden state to obtain the enhanced state Xe(l). Preferably, G(l) is initialized to 0 at the beginning of training, so that Xe(l) is consistent with X(l) in the initial state, and the model behavior is equivalent to the original visual Transformer. As training or fine-tuning proceeds, G(l) gradually learns to open, allowing the model to absorb cross-layer spectral information as needed for the task.

[0033] Through the above process, the spectral summary context is fed back into the complete token space and participates in subsequent computations in an adaptive manner. The enhanced Xe(l) continues to be input into the self-attention or feedforward network of the current Transformer block until all layers have been computed.

[0034] This solution addresses the issues of high overhead in cross-layer state storage and low hardware deployment efficiency in visual Transformers, and has broad application prospects, including but not limited to the following areas: (1) Edge vision reasoning: for drones, mobile robots, smart cameras and portable detection devices, deploying visual Transformer models with limited cache and power budget.

[0035] (2) Industrial vision inspection: used for defect detection, production line monitoring, part identification and equipment status analysis, reducing the inference latency and memory access pressure of embedded vision models.

[0036] (3) Remote sensing image processing: used for remote sensing image classification, ground object recognition and change detection, making deep vision models more suitable for edge or low power accelerator card deployment.

[0037] (4) Intelligent traffic perception: used for road target recognition, traffic condition analysis, vehicle vision and roadside perception equipment to improve real-time perception efficiency.

[0038] (5) Medical image analysis: used for medical image classification, lesion detection and image segmentation, reducing model caching requirements on a dedicated acceleration platform.

[0039] (6) FPGA, ASIC and low power NPU: As a cross-layer state compression module, it is combined with matrix multiplication array, attention accelerator and on-chip cache system to build an efficient visual Transformer inference system.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A spatial-spectral biaxial cross-layer state compression method for visual Transformers, characterized in that, The method specifically includes the following steps: S1: Spatial summary extraction: For the hidden state X(l) at the l-th cross-layer position in the visual Transformer model, it is compressed into a spatial summary state Z(l) according to a fixed spatial partitioning strategy; wherein, the spatial summary state Z(l) contains a global semantic summary and multiple region summaries, and the region summaries are obtained by performing pooling operations on image block tokens within a fixed region; S2: Depth Spectrum Accumulation: The spatial summary state Z(l) generated at each cross-layer position is regarded as a sequence along the network depth direction. The K pre-generated spectrum buffers F(k) are updated in a streaming accumulation manner using Walsh-Hadamard transformation, so that the K spectrum buffers F(k) accumulate the spectrum components representing the cross-layer historical information; where K is a preset positive integer, representing the number of spectrum components retained. S3: Spectral Context Retrieval: At the l-th cross-layer position, the K spectral buffers F(k) are weighted and summed using the learnable query vector W(l) to calculate the cross-layer summary context Zc(l) of the current layer; S4: Cross-layer information backfilling and adaptive fusion: The cross-layer summary context Zc(l) is compared with the spatial summary state Z(l) of the current layer to obtain summary supplementary information. The summary supplementary information is backfilled into the complete token space according to the fixed space partitioning strategy to obtain the supplementary state D(l). The injection intensity of the supplementary state D(l) is controlled by the learnable channel adjustment vector G(l). The adjusted supplementary state is fused with the current hidden state X(l) to generate the enhanced state Xe(l).

2. The spatial-spectral dual-axis cross-layer state compression method for visual Transformers according to claim 1, characterized in that, In step S1, the fixed space partitioning strategy is as follows: the image block token is divided into several non-overlapping fixed regions according to its two-dimensional image position, and average pooling or weighted average is performed on all image block tokens in each fixed region to obtain a region summary; at the same time, the category token is directly retained as the global semantic summary.

3. The spatial-spectral dual-axis cross-layer state compression method for visual Transformers according to claim 2, characterized in that, In step S2, the method of updating the spectrum buffer F(k) using Walsh-Hadamard transform in a streaming accumulation manner specifically involves: pre-generating a Hadamard symbol table S(k,l); when the spatial summary state Z(l) of the l-th cross-layer position is generated, performing an accumulation or subtraction operation on the k-th spectrum buffer F(k) according to the symbol of the Hadamard symbol table S(k,l); wherein, if S(k,l) is positive, Z(l) is accumulated into F(k), and if S(k,l) is negative, Z(l) is subtracted from F(k).

4. The spatial-spectral biaxial cross-layer state compression method for visual Transformers according to claim 3, characterized in that, In step S3, the specific process of calculating the cross-layer summary context Zc(l) is as follows: First, each spectrum buffer F(k) is scored using the learnable query vector W(l); second, the scoring results are converted into normalized weights A(k,l); finally, the K spectrum buffers F(k) are weighted and summed according to the weights A(k,l) to obtain Zc(l) = Σ A(k,l)·F(k).

5. The spatial-spectral biaxial cross-layer state compression method for visual Transformers according to claim 4, characterized in that, Step S4 specifically includes: Calculate the difference between the cross-layer summary context Zc(l) and the spatial summary state Z(l) of the current layer, and use it as supplementary information for the summary; Based on the fixed space partitioning strategy, the summary supplementary information is backfilled into the corresponding category token and each image block token through the static routing table to obtain the supplementary state D(l) with the same size as the current hidden state X(l); The intensity of D(l) is adjusted channel by channel using a learnable channel adjustment vector G(l), and the adjusted result is superimposed on X(l) to obtain the enhanced state Xe(l).

6. The spatial-spectral biaxial cross-layer state compression method for visual Transformers according to claim 2, characterized in that, The region division method in the fixed space partitioning strategy is independent of the input image content and has the same partitioning structure for different input images. Preferably, when the image block token grid is 14×14, it is divided into 3×3 regions, and the segmentation method of each region in the row direction and column direction adopts 5, 5, 4 or similar non-uniform partitioning.

7. The spatial-spectral dual-axis cross-layer state compression method for visual Transformers according to claim 1, characterized in that, The number K of the K spectral components is a configurable hyperparameter, preferably K is 3.

8. A spatial-spectral dual-axis cross-layer state compression system for vision Transformers, characterized in that, The system is used to perform the method as described in any one of claims 1 to 7.

Images