A large model super-long context processing method

Abstract

The application discloses a large model super-long context processing method, belongs to the field of artificial intelligence and natural language processing technology, and realizes large model super-long context processing based on visual text compression and sparse attention cooperation. The implementation of the method comprises the following steps: S1, visual representation learning of text; S2, adaptive multi-granularity visual text compression; S3, dynamic sparse attention of compressed sensing; S4, hierarchical feature fusion and text reconstruction; and S5, end-to-end optimization and adaptive strategy. The application realizes the two-dimensional visualization of text and collaborative design with a sparse attention mechanism, solves the problems of high computational complexity, insufficient semantic modeling and insufficient structural information utilization of traditional methods in processing super-long sequences, significantly reduces the computational and memory overhead, realizes accurate modeling of deep semantic and global structure of long documents, and provides efficient core technical support for long text intelligent analysis, multi-round dialogue systems and other applications.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, specifically a method for processing ultra-long contexts of large models. Background Technology

[0002] The ability of large language models to efficiently process extremely long text sequences is key to their value in complex document understanding, long-range dialogue, and knowledge-intensive tasks, and is of core demand in scenarios such as intelligent document analysis, multi-turn dialogue systems, and code understanding. However, existing large model architectures and methods generally face severe computational bottlenecks, memory limitations, and semantic modeling challenges when processing text far exceeding their training context window.

[0003] The first type of method directly extends the context window of the standard Transformer architecture. This type of method captures dependencies over longer distances by directly increasing the scope of attention computation. Its main limitation is: (1) The computational complexity and memory overhead increase dramatically: The computational complexity of the standard attention mechanism is quadratic (O(n²)) with the sequence length, which leads to an exponential increase in the demand for computing resources when processing sequences of tens of thousands or even hundreds of thousands of tokens, resulting in high actual deployment costs.

[0004] (2) Information processing uniformity and “intermediate layer decay”: The model has difficulty paying attention to the information at all positions equally in a very long context, which easily leads to insufficient attention to the information at both ends of the input, while the information in the middle part is diluted or forgotten, which is the so-called “intermediate layer decay” phenomenon, which seriously affects the grasp of the overall structure of long documents and the key information at the beginning and end.

[0005] (3) Unable to effectively model the hierarchical and structured characteristics of text: Purely serialized modeling ignores the inherent multi-level structure of natural language, such as paragraphs and chapters, making it difficult to use this structural information to efficiently organize and retrieve relevant content.

[0006] To alleviate the aforementioned computational pressure, the second type of method employs various sparsification or approximate attention mechanisms. These methods reduce computational complexity by limiting the scope of context each lexical unit can focus on (e.g., sliding window, block sparsity, random attention). While significantly improving efficiency, they still suffer from the following inherent drawbacks: (1) Global semantic coherence is impaired: In order to ensure local efficiency, most sparsification methods forcibly cut off a large number of long-distance potential dependency connections, which may damage the ability to model key information such as the global logic of the document and cross-chapter reference relationships.

[0007] (2) Separation of compression and modeling processes: Existing methods usually treat sequence compression (such as pooling and block division) as an independent preprocessing step, which is disconnected from the core attention mechanism design of the model. The compression process is often heuristic or information-lossy, and the compressed representation fails to effectively coordinate with attention computation for optimization, making it difficult to guarantee modeling accuracy while improving efficiency.

[0008] (3) Lack of adaptive perception of semantics at different granularities: The model lacks a unified mechanism to adaptively and collaboratively process information at different levels of abstraction (from lexical units and phrases to paragraphs and chapters). For ultra-long text tasks that require grasping both details and main ideas at the same time, a single, fixed-pattern attention mechanism is difficult to achieve the optimal balance between performance and efficiency.

[0009] Solving the above-mentioned technical problems is a matter that urgently needs to be addressed by those skilled in the art. Summary of the Invention

[0010] The technical task of this invention is to address the shortcomings of existing large models in processing ultra-long text sequences, such as high computational complexity, insufficient semantic modeling, and the separation of compression and attention mechanisms. This invention provides a method for processing ultra-long contexts in large models that can achieve both efficiency and accuracy and can synergistically utilize text structure and semantic information.

[0011] The technical solution adopted by this invention to solve its technical problem is: A method for processing extremely long contexts in large models is proposed, which achieves this by combining visual text compression and sparse attention. The implementation of this method includes the following steps: S1. Learning the visual representation of text: Treat long text sequences as two-dimensional structured documents, and use a learnable text visualization encoder to convert the text into a visual representation with spatial awareness. This representation simultaneously encodes the content information of words, two-dimensional spatial location information, and local structural features of the text, forming a two-dimensional feature grid that can be processed by visual algorithms. S2, Adaptive Multi-Granularity Visual Text Compression: Based on the visual representation generated by S1, multi-layer deformable convolution and adaptive pooling operations are applied to downsample the text features at multiple scales while preserving key semantic information, generating compressed representations with different granularity levels, forming a visual feature pyramid from fine-grained word level to coarse-grained semantic level, with explicit spatial correspondence between each level. S3. Dynamic Sparse Attention in Compressed Sensing: Based on the spatial geometric relationships, semantic similarity, and feature importance information retained during compression at different levels in the compressed feature pyramid, multi-level sparse attention maps are dynamically generated. Sparse attention computation is performed collaboratively at multiple abstract levels of the original sequence to achieve an optimized balance between computational complexity and modeling capability. S4. Hierarchical Feature Fusion and Text Reconstruction: Through cross-level feature aggregation and upsampling operations, a top-down semantic information propagation path and a bottom-up detail supplementation path are established to accurately propagate high-level semantic information to low-level specific word representations, thereby achieving long-distance dependency modeling and final text representation reconstruction while maintaining the sequential structure of the original sequence. S5. End-to-end optimization and adaptive strategy: Integrate the modules described in S1 to S4 into an end-to-end differentiable framework, train by jointly optimizing the language modeling loss function and the compressed sensing loss function, and dynamically adjust the compression strategy and attention sparsity according to the length of the input sequence during the inference stage.

[0012] This method provides a comprehensive technical solution that integrates text visualization representation, adaptive multi-granularity compression, sparse attention collaboration with compressed sensing, and hierarchical feature fusion and reconstruction. It can support large language models to efficiently and accurately process ultra-long text inputs with tens of thousands or even hundreds of thousands of tokens.

[0013] Furthermore, step S1 further includes: S101, Remapping of sequence to 2D grid: For a length of L input word sequence ,in For the first i Embedding vectors of 1 word element d For the embedding dimension, it is first divided into N A series of consecutive and partially overlapping text blocks Each text block contains K One word element, adjacent text blocks overlap O Each word is represented by a lexical unit, and each text block is then treated as a two-dimensional "visual block," arranged according to their order in the original sequence to form a visual unit. H OK W Two-dimensional grid of columns ,in H×W=N ; S102, Two-dimensional learnable position encoding: For each position in a two-dimensional grid G ​​( h , w Assign a learnable two-dimensional position code This positional encoding is achieved by learning row encoding separately. Column encoding It was assembled later, that is ; S103, Visual Feature Extraction: Extracting position-encoded grids G+P The input is fed into a shallow visual encoder, which is... It consists of several deformable convolutional layers, and each deformable convolutional layer can be represented as: ; in, p To output the spatial location on the feature map, For the first k The weight of each sampling point For a predefined sampling offset, The learnable offset is predicted from the input features through a single convolutional layer, ultimately outputting a spatially aware visual representation. ,in Dimensions of visual representation.

[0014] Furthermore, step S2 further includes: S201, Deformable Convolutional Compression: Apply a set of deformable convolutional layers to the visual representation V, each with a learnable offset field. The sampling position of the convolution kernel is adaptively adjusted according to the input content, achieving adaptive compression of irregular text structures; S202, Adaptive Multi-Scale Pooling: Applying adaptive pooling operations across multiple compression levels to generate compressed representations at different scales; let the... l The compression ratio of the layers is The size of the output feature map is then... This forms a multi-scale compressed feature set. ,in To compress the feature dimension; S203, Pyramid Feature Organization: Organizing features at different compression levels into a feature pyramid. ,in Corresponding to the k The pyramid uses a hierarchical compression feature structure, with fine-grained features at the bottom and coarse-grained semantic features at the top. Features are associated between adjacent levels through spatial and channel attention.

[0015] Furthermore, step S3 further includes: S301, Dynamic Sparse Attention Map Generation: For each level of features in the feature pyramid... First, flatten it into a sequence form. ,in Then generate a sparse attention adjacency matrix. : S302, Cross-Level Attention Propagation: A cross-level attention propagation mechanism is designed to allow information exchange between feature levels of different granularities. Higher-level features guide the attention focus of lower-level features, while lower-level features provide detailed supplementation for higher-level features. The cross-level attention weights are calculated as follows: ; in, For the first k Hierarchical position i The query vector, For the first k' Hierarchical position j The key vector, It is a mask matrix based on the spatial correspondence between layers; S303. Highly efficient sparse attention computation: The block sparse attention mechanism is adopted to divide the feature map into multiple overlapping local windows. Dense attention computation is performed in each window, and information is exchanged between windows through a small number of global connection points, achieving linear complexity attention computation.

[0016] Furthermore, a sparse attention adjacency matrix is ​​generated through the following steps. : Spatial proximity calculation: Calculating location i and j Euclidean distance in a two-dimensional grid ,in For position i Two-dimensional coordinates; Semantic similarity calculation: Calculate the cosine similarity between feature vectors. ,in , Positions i and j eigenvectors; Composite Importance Score: Calculate the composite score ,in For distance scaling parameters, For connection indications based on compressed feature importance prediction, , and These are learnable weight parameters; Sparsification: For each position i ,reserve highest score K Connect 1 element and set the rest to 0 to obtain a sparse adjacency matrix. .

[0017] Furthermore, step S4 further includes: S401, Feature Pyramid Fusion: This method fuses high-level semantic features with low-level detail features through a top-down path and lateral connections. A gating mechanism is used during the fusion process to adaptively control the contribution weights of features at each level. The fusion process can be represented as follows: ; in, For the first kThe output features of the layer feature pyramid after cross-layer fusion operation. For element-wise multiplication, For upsampling, The learnable gated tensor is computed as follows: ; in, It is the Sigmoid activation function. For learnable weight matrix, For splicing operations; S402, Iterative Upsampling Reconstruction: From the coarsest-grained compressed features Initially, spatial resolution is gradually restored through iterative upsampling operations. Each upsampling step incorporates skip connections to introduce the pre-compressed features of the corresponding layer. Ultimately, an enhanced representation aligned with the original word sequence is reconstructed. ; S403, Residual Connectivity and Layer Normalization: The reconstructed enhanced representation is combined with the original lexical embeddings through residual connectivity, and then layer normalization is applied to output the final context-aware lexical representation. Specifically: ; in, The output projection matrix is ​​used to adjust the dimensional alignment between the reconstructed features and the original features. LayerNorm(x) = For layer normalization; For learnable scaling parameters, Given the mean of all elements in x, Given the variance of all elements of input x, To minimize the constant that prevents the denominator from being zero, This is a learnable bias parameter used to adjust the mean after normalization.

[0018] Furthermore, step S5 further includes: S501, End-to-End Joint Optimization Framework: Integrates the visual encoder, compression module, sparse attention module, and reconstruction module into an end-to-end differentiable framework, minimizing the language modeling loss function through joint optimization. and compressed sensing loss function : + ; in, To balance hyperparameters, the language modeling loss function Defined as: ; in, L For sequence length, For the first in the sequence i The true value of each word element For the first i The final lexical representation of each word, and Learnable parameters for modeling the head of language. V For vocabulary size; Compressed sensing loss function Defined as: ; Where MSE represents the mean squared error. This represents a proxy task that randomly samples from the original features. This refers to the hierarchical weight coefficient; S502, Progressive Compression Training: In the initial stage of training, a small compression ratio and dense attention patterns are used. As training progresses, the compression levels and sparsity are gradually increased, allowing the model to gradually learn text understanding capabilities from fine-grained to coarse-grained. The change in compression ratio follows the following scheduling strategy: ; in, Indicates the first t The training step is the first l The compression ratio of the layers, and These are the minimum and maximum compression ratios, respectively. This refers to the number of preheating steps; S503, Multi-task Collaborative Training: Simultaneously train the model's performance on various long text tasks, including language modeling, text classification, question answering, and summary generation. Improve the model's generalization ability through multi-task learning. The total loss function is expanded to: ; in, M For the number of tasks, For the first m The loss function for each task. This refers to the task weighting coefficient; S504, Dynamic computing resource allocation: Dynamically adjust the compression strategy and attention sparsity based on the length of the input sequence.

[0019] Furthermore, the dynamic computing resource allocation, for the input sequence length... L Length threshold and ( The computing resource allocation strategy is as follows: when When using single-layer compression, the sparsity factor... ; when At that time, two layers of compression were used, and the sparsity factor was adjusted. (Enable partial compression, using only the top K features from feature pyramid F) 浅 (Cooperative attention computation is performed on layers with shallower compression) when When, use all K Layer compression, sparsity factor (Enable full compression and use all K layers of the feature pyramid F for collaborative attention computation) in, and These are the minimum and maximum sparsity factors, respectively, which control the number of connections at each position.

[0020] The present invention also claims a large-model ultra-long context processing device, comprising: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the above method.

[0021] The present invention also claims a computer-readable medium storing computer instructions that, when executed by a processor, enable the implementation of the above-described method.

[0022] Compared with existing technologies, the large-model ultra-long context processing method of the present invention has the following advantages: This invention creatively introduces the structured processing ideas of the visual domain into the long context modeling of pure text by constructing a complete technical link from text visualization, multi-granularity compression, collaborative sparse attention to hierarchical reconstruction. It effectively solves the problems of traditional methods, such as the inability to handle ultra-long sequences due to limited computing resources, shallow semantic modeling due to insufficient utilization of structural information, and difficulty in achieving both efficiency and accuracy due to the separation of compression and attention. It realizes deep semantic understanding of ultra-long text under controllable computing overhead.

[0023] By introducing a core architecture that collaboratively designs "visual text compression" and "sparse attention," this method achieves unified modeling and efficient utilization of the two-dimensional spatial structure and multi-granular semantic information of text. Theoretical analysis and experiments show that the proposed method can significantly reduce the computational complexity of the core attention function for processing sequences of length L from O(L²) of the traditional Transformer to near O(L log L), while maintaining performance comparable to or even better than the benchmark model for processing segmented text in tasks such as long document QA and summarization.

[0024] By designing an end-to-end differentiable joint optimization framework and a dynamic adaptive strategy, the method can intelligently adjust the compression granularity and attention sparsity mode according to task requirements and resource constraints, achieving an optimal balance between performance, efficiency, and generalization. Simulation experiments show that when processing academic papers with more than 32K tokens, compared with mainstream efficient attention mechanisms such as standard sliding window attention and block sparse attention, the method of this invention improves the inference speed by an average of 2-3 times and reduces memory usage by 40%-60% while maintaining a key information recall rate of no less than 95%. This provides key technical support for building practical large-model ultra-long context processing systems. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall process of a large-model ultra-long context processing method based on visual text compression and sparse attention collaboration provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the process of text visualization representation learning and adaptive multi-granularity compression in an embodiment of the present invention; Figure 3 This is a schematic diagram of the dynamic sparse attention collaborative computing mechanism of compressed sensing in an embodiment of the present invention. Detailed Implementation

[0026] The present invention will be further described below with reference to specific embodiments.

[0027] This invention provides a method for processing extremely long context sequences in large models. Based on the collaborative processing of visual text compression and sparse attention, this method achieves efficient and accurate semantic understanding and modeling of extremely long text sequences (e.g., tens of thousands to hundreds of thousands of words). It is applicable to applications requiring efficient and accurate semantic understanding and modeling of extremely long text sequences, such as intelligent document analysis, long text summarization, multi-turn dialogue systems, codebase understanding, and academic literature processing. The aim is to overcome the memory and computational bottlenecks caused by the quadratic increase in computational complexity of self-attention in traditional Transformer models due to the sequence length. The implementation of this method includes the following steps: S1. Learning the visual representation of text: Treat long text sequences as two-dimensional structured documents, and use a learnable text visualization encoder to convert the text into a visual representation with spatial awareness. This representation simultaneously encodes the content information of words, two-dimensional spatial location information, and local structural features of the text, forming a two-dimensional feature grid that can be processed by visual algorithms. S2, Adaptive Multi-Granularity Visual Text Compression: Based on the visual representation generated by S1, multi-layer deformable convolution and adaptive pooling operations are applied to downsample the text features at multiple scales while preserving key semantic information, generating compressed representations with different granularity levels, forming a visual feature pyramid from fine-grained word level to coarse-grained semantic level, with explicit spatial correspondence between each level. S3. Dynamic Sparse Attention in Compressed Sensing: Based on the spatial geometric relationships, semantic similarity, and feature importance information retained during compression at different levels in the compressed feature pyramid, multi-level sparse attention maps are dynamically generated. Sparse attention computation is performed collaboratively at multiple abstract levels of the original sequence to achieve an optimized balance between computational complexity and modeling capability. S4. Hierarchical Feature Fusion and Text Reconstruction: Through cross-level feature aggregation and upsampling operations, a top-down semantic information propagation path and a bottom-up detail supplementation path are established to accurately propagate high-level semantic information to low-level specific word representations, thereby achieving long-distance dependency modeling and final text representation reconstruction while maintaining the sequential structure of the original sequence. S5. End-to-end optimization and adaptive strategy: Integrate modules S1 to S4 into an end-to-end differentiable framework, and jointly optimize language modeling loss and compressed perception loss for training; during training, a progressive compression strategy is adopted to gradually transition the model from fine-grained learning to coarse-grained understanding, while combining multi-task collaborative learning to improve generalization ability; during inference, the compression level and attention sparsity are dynamically adjusted according to the length of the input sequence to achieve an adaptive balance between efficiency and accuracy.

[0028] Step S1 further includes: S101, the sequence to two-dimensional grid remapping sub-step, is used to segment the input word sequence into partially overlapping text blocks and arrange them in order into a two-dimensional grid; S102, Two-dimensional learnable location coding sub-step, used to assign a learnable location code that integrates row and column information to each location in the grid; S103, Visual Feature Extraction Sub-step, is used to process the position-encoded grid through deformable convolutional layers and output a spatially aware visual representation.

[0029] Step S2 further includes: S201, Deformable convolutional compression sub-step, used to compress visual representations through content-adaptive sampling offset; S202, Adaptive multi-scale pooling sub-step, used to generate multi-scale compressed features according to different compression ratios; S203, Pyramid Feature Organization Sub-step, is used to organize multi-scale features into a feature pyramid with clear hierarchical relationships.

[0030] Step S3 further includes: S301, Dynamic Sparse Attention Graph Generation Sub-step, which integrates spatial proximity, semantic similarity and feature importance to dynamically generate a sparse attention adjacency matrix for each level of the feature pyramid. S302, Cross-level attention propagation sub-step, is used to achieve bidirectional information interaction and guidance between feature levels of different granularities; S303, a memory-efficient sparse attention computation sub-step, is used to achieve near-linear complexity attention computation by employing a strategy that combines block partitioning and global connectivity.

[0031] Step S4 further includes: S401, Feature Pyramid Fusion Sub-step, is used to adaptively fuse high-level semantic features and low-level detail features through a gating mechanism; S402, Iterative upsampling reconstruction sub-step, is used to gradually reconstruct an enhanced representation aligned with the original sequence from coarse-grained features through upsampling and skip connections; S403, the residual connection and layer normalization sub-step, is used to fuse and normalize the reconstructed features with the original features, and output the final context-aware lexical representation.

[0032] Step S5 further includes: S501, an end-to-end joint optimization framework, is used to collaboratively train all modules by jointly optimizing the language modeling loss and compressed perception loss; S502, a progressive compression training strategy, is used to gradually increase the compression ratio and sparsity during training. S503, a multi-task collaborative training strategy, is used to improve the model's generalization ability through multi-task learning; S504, a dynamic computing resource allocation strategy, is used to adaptively adjust the compression level and attention sparsity based on the length of the input sequence during inference.

[0033] The specific implementation is as follows: Step S1 further includes: S101, Remapping of sequence to 2D grid: For a length of L input word sequence ,in For the first i Embedding vectors of 1 word element d For the embedding dimension, it is first divided into N A series of consecutive and partially overlapping text blocks Each text block contains K One word element, adjacent text blocks overlap OEach word is represented by a lexical unit, and each text block is then treated as a two-dimensional "visual block," arranged according to their order in the original sequence to form a visual unit. H OK W Two-dimensional grid of columns ,in H×W=N ; S102, Two-dimensional learnable position encoding: For each position in a two-dimensional grid G ​​( h , w Assign a learnable two-dimensional position code This positional encoding is achieved by learning row encoding separately. Column encoding It was assembled later, that is ; S103, Visual Feature Extraction: Extracting position-encoded grids G+P The input is fed into a shallow visual encoder, which is... It consists of several deformable convolutional layers, and each deformable convolutional layer can be represented as: ; in, p To output the spatial location on the feature map, For the first k The weight of each sampling point For a predefined sampling offset, The learnable offset is predicted from the input features through a single convolutional layer, ultimately outputting a spatially aware visual representation. ,in Dimensions of visual representation.

[0034] Step S2 further includes: S201, Deformable Convolutional Compression: Apply a set of deformable convolutional layers to the visual representation V, each with a learnable offset field. The sampling position of the convolution kernel is adaptively adjusted according to the input content, achieving adaptive compression of irregular text structures; S202, Adaptive Multi-Scale Pooling: Applying adaptive pooling operations across multiple compression levels to generate compressed representations at different scales; let the... l The compression ratio of the layers is The size of the output feature map is then... This forms a multi-scale compressed feature set. ,in To compress the feature dimension; S203, Pyramid Feature Organization: Organizing features at different compression levels into a feature pyramid. ,in Corresponding to the kThe pyramid uses a hierarchical compression feature structure, with fine-grained features at the bottom and coarse-grained semantic features at the top. Features are associated between adjacent levels through spatial and channel attention.

[0035] Step S3 further includes: S301, Dynamic Sparse Attention Map Generation: For each level of features in the feature pyramid... First, flatten it into a sequence form. ,in Then, the sparse attention adjacency matrix is ​​generated through the following steps. : Spatial proximity calculation: Calculating location i and j Euclidean distance in a two-dimensional grid ,in For position i Two-dimensional coordinates; Semantic similarity calculation: Calculate the cosine similarity between feature vectors. ,in , Positions i and j eigenvectors; Composite Importance Score: Calculate the composite score ,in For distance scaling parameters, For connection indications based on compressed feature importance prediction, , and These are learnable weight parameters; Sparsification: For each position i ,reserve highest score K Connect 1 element and set the rest to 0 to obtain a sparse adjacency matrix. ; S302, Cross-Level Attention Propagation: A cross-level attention propagation mechanism is designed to allow information exchange between feature levels of different granularities. Higher-level features guide the attention focus of lower-level features, while lower-level features provide detailed supplementation for higher-level features. The cross-level attention weights are calculated as follows: ; in, For the first k Hierarchical position i The query vector, For the first k' Hierarchical position j The key vector, It is a mask matrix based on the spatial correspondence between layers; S303. Highly efficient sparse attention computation: The block sparse attention mechanism is adopted to divide the feature map into multiple overlapping local windows. Dense attention computation is performed in each window, and information is exchanged between windows through a small number of global connection points, achieving linear complexity attention computation.

[0036] Step S4 further includes: S401, Feature Pyramid Fusion: This method fuses high-level semantic features with low-level detail features through a top-down path and lateral connections. A gating mechanism is used during the fusion process to adaptively control the contribution weights of features at each level. The fusion process can be represented as follows: ; in, For the first k The output features of the layer feature pyramid after cross-layer fusion operation. For element-wise multiplication, For upsampling, The learnable gated tensor is computed as follows: ; in, It is the Sigmoid activation function. For learnable weight matrix, For splicing operations; S402, Iterative Upsampling Reconstruction: From the coarsest-grained compressed features Initially, spatial resolution is gradually restored through iterative upsampling operations. Each upsampling step incorporates skip connections to introduce the pre-compressed features of the corresponding layer. Ultimately, an enhanced representation aligned with the original word sequence is reconstructed. ; S403, Residual Connectivity and Layer Normalization: The reconstructed enhanced representation is combined with the original lexical embeddings through residual connectivity, and then layer normalization is applied to output the final context-aware lexical representation. Specifically: ; in, The output projection matrix is ​​used to adjust the dimensional alignment between the reconstructed features and the original features. LayerNorm(x) = For layer normalization; For learnable scaling parameters, Given the mean of all elements in x, Given the variance of all elements of input x, To minimize the constant that prevents the denominator from being zero, This is a learnable bias parameter used to adjust the mean after normalization.

[0037] Step S5 further includes: S501, End-to-End Joint Optimization Framework: Integrates the visual encoder, compression module, sparse attention module, and reconstruction module into an end-to-end differentiable framework, minimizing the language modeling loss function through joint optimization. and compressed sensing loss function : + ; in, To balance hyperparameters, the language modeling loss function Defined as: ; in, L For sequence length, For the first in the sequence i The true value of each word element For the first i The final lexical representation of each word, and Learnable parameters for modeling the head of language. V For vocabulary size; Compressed sensing loss function Defined as: ; Where MSE represents the mean squared error. This represents a proxy task that randomly samples from the original features. This refers to the hierarchical weight coefficient; S502, Progressive Compression Training: In the initial stage of training, a small compression ratio and dense attention patterns are used. As training progresses, the compression levels and sparsity are gradually increased, allowing the model to gradually learn text understanding capabilities from fine-grained to coarse-grained. The change in compression ratio follows the following scheduling strategy: ; in, Indicates the first t The training step is the first l The compression ratio of the layers, and These are the minimum and maximum compression ratios, respectively. This refers to the number of preheating steps; S503, Multi-task Collaborative Training: Simultaneously train the model's performance on various long text tasks, including language modeling, text classification, question answering, and summary generation. Improve the model's generalization ability through multi-task learning. The total loss function is expanded to: ; in,M For the number of tasks, For the first m The loss function for each task. This refers to the task weighting coefficient; S504, Dynamic computing resource allocation: Dynamically adjust the compression strategy and attention sparsity based on the length of the input sequence.

[0038] Furthermore, the dynamic computing resource allocation, for the input sequence length... L Length threshold and ( The computing resource allocation strategy is as follows: when When using single-layer compression, the sparsity factor... ; when At that time, two layers of compression were used, and the sparsity factor was adjusted. (Enable partial compression, using only the top K features from feature pyramid F) 浅 (Cooperative attention computation is performed on layers with shallower compression) when When, use all K Layer compression, sparsity factor (Enable full compression and use all K layers of the feature pyramid F for collaborative attention computation) in, and These are the minimum and maximum sparsity factors, respectively, which control the number of connections at each position.

[0039] As attached Figure 1-3 As shown, this embodiment provides a method for processing ultra-long contexts in large models based on visual text compression and sparse attention collaboration, which includes the following steps: S1. Learning the visual representation of text Receive a paper of length L The abstract text of an academic paper containing 32,768 tokens is used as input for visual representation learning. The specific steps are as follows: S101, Sequence to 2D Grid Remapping: Remapping the input word sequence T (Embedding Dimension) d =768) divided into lengths of K =64, overlap O =16-word text block, a total of N =1024 blocks. Arrange these blocks in order into a... H =32 lines W =32 columns of 2D grid .

[0040] S102, Two-dimensional learnable position coding: Assign a learnable two-dimensional position coding to each position in grid G. By learning the 384-dimensional row encoding separately. And 384-dimensional column encoding These are then concatenated to form the final 768-dimensional positional code for each location. (In practice, this can be adjusted according to the model dimensions), and then added to the grid G ​​to obtain G+P.

[0041] S103, Visual Feature Extraction: Input G+P into a... =A shallow encoder consisting of 2 layers of deformable convolutions. Each convolution kernel is 3×3 in size, and the learnable offset is... It is predicted by a single 3×3 convolutional layer. The final output is a visual representation. (Visual Representation Dimension) ).

[0042] S2, Adaptive Multi-Granularity Visual Text Compression Multi-scale compression is performed on the visual representation V to construct a feature pyramid: S201, Deformable Convolution Compression: Apply two layers of deformable convolution with a stride of 2 to V, followed by a ReLU activation function after each layer to achieve preliminary downsampling and feature transformation.

[0043] S202, Adaptive Multi-Scale Pooling: Three compression levels are set, with compression ratios of [missing information]. , , Adaptive average pooling is applied to V and the feature map after deformable convolution to adjust the size to 16×16 (resulting in...). ), 8×8 (to obtain) ), 4×4 (to obtain) ), the feature dimensions are uniformly maintained as .

[0044] S203, Pyramid Feature Organization: Combining the original visual representation V (considered as layer 0) with three compressed features The joint organization is a four-level feature pyramid. ,in .

[0045] S3, Dynamic Sparse Attention in Compressed Perception Perform collaborative sparse attention computation on the constructed feature pyramid: S301, Dynamic Sparse Attention Map Generation: Using Layer 2 Compressed Features (After flattening) For example, (=64). Regarding the position within it... i(Corresponding to two-dimensional coordinates (1,1)): Calculate its relationship with all other locations in the grid. j Euclidean distance ; Calculate its 1024-dimensional eigenvector. With all cosine similarity ; A lightweight network is used to predict the feature importance score for each location, and a connection indication is generated based on this score. ; Combining the above three factors, according to the formula Calculate the composite score. In this example, the initial weights are set to... =0.4, =0.4, =0.2, Distance scaling parameter =2.0.

[0046] Position i Only retain highest score K =16 connections, forming a sparse adjacency matrix The i OK.

[0047] S302, Cross-level attention propagation: Let the second level ( k =2) a position i Not only focusing on the same floor K One neighbor, and also through cross-level attention, pay attention to its position on the 3rd floor ( k' =3, a coarser-grained approach, encompasses all positions within the corresponding parent region (e.g., a 2×2 region), enabling high-level semantics to guide low-level features. During attention weight calculation, a mask matrix generated based on spatial coordinate correspondence is used. To constrain valid connections.

[0048] S303, Sparse attention computing with high memory efficiency: The 8×8 grid is divided into four overlapping 4×4 local windows. Standard dense attention computation is performed within each window. Simultaneously, each window has two pre-defined "global connectivity points," which interact with corresponding points in all other windows to ensure the flow of remote information. This strategy reduces computational complexity from... Significantly reduced.

[0049] S4. Hierarchical Feature Fusion and Text Reconstruction The multi-level features enhanced by sparse attention are fused and reconstructed into the final lexical representation: S401, Feature Pyramid Fusion: From the Top Level Begin by upsampling (e.g., bilinear interpolation) and then connecting it to the next layer. Fusion is achieved through gating mechanisms. Gating tensor. The concatenated features are calculated using a lightweight convolutional layer, controlling the proportion of information derived from higher-level semantic information. This process is repeated layer by layer until the features are fused together. (i.e., V).

[0050] S402, Iterative Upsampling Reconstruction: The fused... The features are progressively upsampled back to the original 32×32 spatial resolution through a decoder consisting of deconvolutional layers and skip connections (connected to the intermediate layer features in S103), and finally mapped back to their corresponding original word sequence positions through a projection layer to generate an enhanced representation. .

[0051] S403, Residual Connectivity and Layer Normalization: [This section appears to be incomplete and requires further context.] Through a projection matrix Then, it is added to the original input lexical embedding T, and layer normalization is performed to output the final context-aware lexical representation. This is used by downstream language modeling heads.

[0052] S5, End-to-End Optimization and Adaptive Strategies Applying optimization strategies in training and inference: S501, End-to-End Joint Optimization Framework: During overall model training, the total loss is the language modeling loss. (Cross-entropy loss) and compressed sensing loss and. Requires features for each compression layer It can recover some key information (such as the representation of the central word) of the corresponding uncompressed region through a small reconstruction network, and the hierarchical weights are also important. Set to [0.1, 0.2, 0.3] to balance the hyperparameters. =0.05.

[0053] S502, Progressive Compression Training: In a total of 100,000 steps of training, the first 10,000 steps ( This is the preheating stage. During this period, the first... l Layer compression ratio From initial value =1.0 (i.e., no compression) linearly increase to the target value (i.e., [0.5, 0.25, 0.125]), while the number of attention connections at each position. K Also from the maximum value =64 linearly reduced to K =16.

[0054] S503, Multi-task Collaborative Training: In addition to the language modeling task, the model is simultaneously fine-tuned on long document question answering (e.g., the Qasper dataset) and summarization generation (e.g., the arXiv dataset). The total loss is the weighted sum of the losses from each task. (weight) (Set according to task importance).

[0055] S504, Dynamic Computing Resource Allocation: During inference, a length threshold is set. , When the input length L =32768> At that time, activate all =3-layer compressed feature pyramid, and using lower =8 connections for the sparsest attention computation. When L =8000 (at the level of) When the interval is specified, only the first part is used. =2 layers of compressed features, number of connections K Based on the input length =8 and =32 linear interpolation, achieving a balance between efficiency and accuracy.

[0056] Based on the above method, this invention also provides a large-model ultra-long context processing device based on visual text compression and sparse attention collaboration, including: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the large model ultra-long context processing method described in the above embodiments.

[0057] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0058] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0059] This invention also provides a computer-readable medium storing computer instructions that, when executed by a processor, implement the large model ultra-long context processing method described in the above embodiments. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the above embodiments can be provided, enabling the computer (or CPU or MPU) of the system or apparatus to read and execute the program code stored in the storage medium.

[0060] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0061] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0062] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0063] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing extremely long contexts in large models, characterized in that, This method achieves ultra-long context processing for large models based on the collaborative approach of visual text compression and sparse attention. The implementation includes the following steps: S1. Learning the visual representation of text: Treat long text sequences as two-dimensional structured documents, and use a learnable text visualization encoder to convert the text into a visual representation with spatial awareness. This representation simultaneously encodes the content information of words, two-dimensional spatial location information, and local structural features of the text, forming a two-dimensional feature grid that can be processed by visual algorithms. S2, Adaptive Multi-Granularity Visual Text Compression: Based on the visual representation generated by S1, multi-layer deformable convolution and adaptive pooling operations are applied to downsample the text features at multiple scales while preserving key semantic information, generating compressed representations with different granularity levels, forming a visual feature pyramid from fine-grained word level to coarse-grained semantic level, with explicit spatial correspondence between each level. S3. Dynamic Sparse Attention in Compressed Sensing: Based on the spatial geometric relationships, semantic similarity, and feature importance information retained during compression at different levels of the compressed feature pyramid, multi-level sparse attention maps are dynamically generated, and sparse attention computation is performed collaboratively at multiple abstract levels of the original sequence. S4. Hierarchical Feature Fusion and Text Reconstruction: Through cross-level feature aggregation and upsampling operations, a top-down semantic information propagation path and a bottom-up detail supplementation path are established to accurately propagate high-level semantic information to low-level specific word representations, thereby achieving long-distance dependency modeling and final text representation reconstruction while maintaining the sequential structure of the original sequence. S5. End-to-end optimization and adaptive strategy: Integrate the modules described in S1 to S4 into an end-to-end differentiable framework. The language modeling loss function and the compressed sensing loss function are jointly optimized for training, and the compression strategy and attention sparsity are dynamically adjusted according to the length of the input sequence during the inference stage. Step S1 further includes: S101, Remapping of sequence to 2D grid: For a length of L input word sequence ,in For the first i Embedding vectors of 1 word element d For the embedding dimension, it is first divided into N A series of consecutive and partially overlapping text blocks Each text block contains K One word element, adjacent text blocks overlap O Each word is represented by a lexical unit, and each text block is then treated as a two-dimensional "visual block," arranged according to their order in the original sequence to form a visual unit. H OK W Two-dimensional grid of columns ,in H×W=N ; S102, Two-dimensional learnable position encoding: For each position in a two-dimensional grid G ​​( h , w Assign a learnable two-dimensional position code This positional encoding is achieved by learning row encoding separately. Column encoding It was assembled later, that is ; S103, Visual Feature Extraction: Extracting position-encoded grids G+P The input is fed into a shallow visual encoder, which is... It consists of several deformable convolutional layers, and each deformable convolutional layer can be represented as: ； in, p To output the spatial location on the feature map, For the first k The weight of each sampling point For a predefined sampling offset, The learnable offset is predicted from the input features through a single convolutional layer, ultimately outputting a spatially aware visual representation. ,in Dimensions of visual representation; Step S3 further includes: S301, Dynamic Sparse Attention Map Generation: For each level of features in the feature pyramid... First, flatten it into a sequence form. ,in Then generate a sparse attention adjacency matrix. : S302, Cross-Level Attention Propagation: A cross-level attention propagation mechanism is designed to allow information exchange between feature levels of different granularities. Higher-level features guide the attention focus of lower-level features, while lower-level features provide detailed supplementation for higher-level features. The cross-level attention weights are calculated as follows: ； in, For the first k Hierarchical position i The query vector, For the first k' Hierarchical position j The key vector, It is a mask matrix based on the spatial correspondence between layers; S303, Memory-efficient sparse attention computation: The block sparse attention mechanism is adopted to divide the feature map into multiple overlapping local windows. Dense attention computation is performed in each window, and information is exchanged between windows through a small number of global connection points to achieve linear complexity attention computation. The sparse attention adjacency matrix is ​​generated through the following steps. : Spatial proximity calculation: Calculating location i and j Euclidean distance in a two-dimensional grid ,in For position i Two-dimensional coordinates; Semantic similarity calculation: Calculate the cosine similarity between feature vectors. ,in , Positions i and j eigenvectors; Composite Importance Score: Calculate the composite score ,in For distance scaling parameters, For connection indications based on compressed feature importance prediction, , and These are learnable weight parameters; Sparsification: For each position i ,reserve highest score K Connect 1 element and set the rest to 0 to obtain a sparse adjacency matrix. ; In step S5, the length of the input sequence is... L Length threshold and ( The computing resource allocation strategy is as follows: when When using single-layer compression, the sparsity factor... ; when At that time, two layers of compression were used, and the sparsity factor was adjusted. ; when When, use all K Layer compression, sparsity factor ; in, and These are the minimum and maximum sparsity factors, respectively, which control the number of connections at each position.

2. The method for processing extremely long contexts in large models according to claim 1, characterized in that, Step S2 further includes: S201, Deformable Convolutional Compression: Apply a set of deformable convolutional layers to the visual representation V, each with a learnable offset field. The sampling position of the convolution kernel is adaptively adjusted according to the input content, achieving adaptive compression of irregular text structures; S202, Adaptive Multi-Scale Pooling: Applying adaptive pooling operations across multiple compression levels to generate compressed representations at different scales; let the... l The compression ratio of the layers is The size of the output feature map is then... This forms a multi-scale compressed feature set. ,in To compress the feature dimension; S203, Pyramid Feature Organization: Organizing features at different compression levels into a feature pyramid. ,in Corresponding to the k The pyramid uses a hierarchical compression feature structure, with fine-grained features at the bottom and coarse-grained semantic features at the top. Features are associated between adjacent levels through spatial and channel attention.

3. The method for processing extremely long contexts in large models according to claim 1, characterized in that, Step S4 further includes: S401, Feature Pyramid Fusion: This method fuses high-level semantic features with low-level detail features through a top-down path and lateral connections. A gating mechanism is used during the fusion process to adaptively control the contribution weights of features at each level. The fusion process can be represented as follows: ； in, For the first k The output features of the layer feature pyramid after cross-layer fusion operation. For element-wise multiplication, For upsampling, The learnable gated tensor is computed as follows: ； in, It is the Sigmoid activation function. For learnable weight matrix, For splicing operations; S402, Iterative Upsampling Reconstruction: From the coarsest-grained compressed features Initially, spatial resolution is gradually restored through iterative upsampling operations. Each upsampling step incorporates skip connections to introduce the pre-compressed features of the corresponding layer. Ultimately, an enhanced representation aligned with the original word sequence is reconstructed. ; S403, Residual Connectivity and Layer Normalization: The reconstructed enhanced representation is combined with the original lexical embeddings through residual connectivity, and then layer normalization is applied to output the final context-aware lexical representation. Specifically: ； in, The output projection matrix is ​​used to adjust the dimensional alignment between the reconstructed features and the original features. LayerNorm(x) = For layer normalization; For learnable scaling parameters, Given the mean of all elements in x, Given the variance of all elements of input x, To minimize the constant that prevents the denominator from being zero, This is a learnable bias parameter used to adjust the mean after normalization.

4. The method for processing extremely long contexts in large models according to claim 3, characterized in that, Step S5 further includes: S501, End-to-End Joint Optimization Framework: Integrates the visual encoder, compression module, sparse attention module, and reconstruction module into an end-to-end differentiable framework, minimizing the language modeling loss function through joint optimization. and compressed sensing loss function : + ； in, To balance hyperparameters, the language modeling loss function Defined as: ； in, L For sequence length, For the first in the sequence i The true value of each word element For the first i The final lexical representation of each word, and Learnable parameters for modeling the head of language. V For vocabulary size; Compressed sensing loss function Defined as: ； Where MSE represents the mean squared error. This represents a proxy task that randomly samples from the original features. This refers to the hierarchical weight coefficient; S502, Progressive Compression Training: In the initial stage of training, a small compression ratio and dense attention patterns are used. As training progresses, the compression levels and sparsity are gradually increased, allowing the model to gradually learn text understanding capabilities from fine-grained to coarse-grained. The change in compression ratio follows the following scheduling strategy: ； in, Indicates the first t The training step is the first l The compression ratio of the layers, and These are the minimum and maximum compression ratios, respectively. This refers to the number of preheating steps; S503, Multi-task Collaborative Training: Simultaneously train the model's performance on various long text tasks, including language modeling, text classification, question answering, and summary generation. Improve the model's generalization ability through multi-task learning. The total loss function is expanded to: ； in, M For the number of tasks, For the first m The loss function for each task. This refers to the task weighting coefficient; S504, Dynamic computing resource allocation: Dynamically adjust the compression strategy and attention sparsity based on the length of the input sequence.

5. A large-model ultra-long context processing device, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to implement the method according to any one of claims 1 to 4.

6. A computer-readable medium, characterized in that, The computer-readable medium stores computer instructions that, when executed by a processor, enable the implementation of the method described in any one of claims 1 to 4.

Images