Context compression method and system based on visual modality and medium

Abstract

The invention provides a context compression method and system based on a visual modality and a medium, and the method comprises the steps: obtaining long text context data, and rendering the long text context data into a corresponding document image; performing context optical coding on the document image, and performing segmentation processing on the document image by adopting a visual segmentation model to obtain a plurality of initial visual tokens; performing down-sampling compression on the initial visual token based on a convolution compression algorithm to obtain a compressed token; capturing a long-distance dependency relationship among different compression tokens based on a global attention mechanism, extracting high-level semantic knowledge, and outputting a final visual token sequence; executing context optical decoding on the final visual token sequence, and fusing context optical decoding data with a text prompt input by a user to generate a target text; through visual modal compression, computing resources and memory occupation required for processing a long text are reduced, and relatively high text decoding precision can still be maintained while a high compression ratio is maintained.

Description

Technical Field

[0001] This application relates to the field of text compression technology, and more specifically, to a context compression method, system, and medium based on visual modalities. Background Technology

[0002] In recent years, Large Language Models (LLMs) based on the Transformer architecture have achieved revolutionary success in the field of natural language processing, demonstrating outstanding capabilities in tasks such as human-computer dialogue, content creation, and code generation. However, when applied to long text dialogue scenarios, the self-attention mechanism, the core component of LLMs, exposes a fundamental computational bottleneck.

[0003] Self-attention mechanisms capture global dependencies by interactively computing all word pairs in the input sequence. The computational complexity and memory consumption of this mechanism increase quadratically (O(n²)) relative to the length n of the input sequence. This means that as the dialogue history (i.e., the context sequence) lengthens, the computational resources and memory required by the model increase dramatically. For example, processing an 8K sequence requires 16 times the computation of processing a 2K sequence. This quadratic growth characteristic places enormous hardware pressure and high inference costs on the model during deployment, severely limiting its application in scenarios requiring long-term memory, such as long-text reading and dialogue systems. Summary of the Invention

[0004] The purpose of this application is to provide a context compression method, system and medium based on visual modality. By using visual modality compression, the computational resources and memory usage required for processing long texts are significantly reduced, while maintaining a high compression ratio and high text decoding accuracy.

[0005] This application also provides a context compression method based on visual modality, including: Obtain long text context data and render the long text context data into a corresponding document image; The document image is subjected to contextual optical encoding, and a visual segmentation model is used to segment the document image to obtain multiple image blocks, which are then converted into multiple initial visual tokens. The initial visual token is downsampled and compressed using a convolutional compression algorithm to obtain a compressed token. Based on the global attention mechanism, long-distance dependencies between different compressed tokens are captured, high-level semantic knowledge is extracted, and the final visual token sequence is output. Contextual optical decoding is performed on the final visual token sequence, and the contextual optical decoding data is fused with the text prompts entered by the user to generate the target text.

[0006] Optionally, in the context compression method based on visual modality described in this application embodiment, obtaining long text context data and rendering the long text context data into a corresponding document image specifically includes: Obtain long text context data, convert the long text context data into a format, and obtain long text context data in a standard format. The long text context data includes at least one of dialogue history, paper document, and long document content. The standard format long text context data is processed according to preset rules to obtain the processing result. The preset rules include font uniformity, paragraph alignment or page margin standardization. Based on the sorting results, analyze whether the text is cluttered; If confusion occurs, adjust the preset rules and reorganize. If no confusion occurs, the long text context data will be rendered into a single, continuous high-resolution document image based on the sorting results.

[0007] Optionally, in the context compression method based on visual modality described in this application embodiment, the document image is subjected to contextual optical encoding, and the document image is segmented using a visual segmentation model to obtain multiple image blocks and convert them into multiple initial visual tokens, specifically including: Input high-resolution document images into the visual segmentation model, set a sliding window, and traverse all high-resolution document images based on the sliding window; The continuous document image is segmented to obtain multiple independent image blocks; Extract the visual features of each image patch, and then combine the visual features of each image patch. The visual features of each image patch are quantized and encoded to obtain encoded data; Each image patch is converted into a fixed-dimensional initial visual token based on the encoded data, resulting in multiple initial visual tokens.

[0008] Optionally, in the context compression method based on visual modality described in the embodiments of this application, the initial visual token is downsampled and compressed based on a convolutional compression algorithm to obtain a compressed token, specifically including: Obtain the initial visual token and parse the data from the initial visual token; The parsed data is then processed into a standardized format to obtain standard data; The parameters are configured based on the convolutional compression algorithm, and the standard data is compressed. The compressed data is then enhanced in terms of features, and a compressed token is obtained based on the enhanced data.

[0009] Optionally, in the context compression method based on visual modality described in the embodiments of this application, the long-distance dependency relationship between different compressed tokens is captured based on a global attention mechanism, high-level semantic knowledge is extracted, and the final visual token sequence is output, specifically including: Receive the compressed token, preprocess and adapt the format of the compressed token to obtain an optimized compressed token; Configure the core parameters for the global attention mechanism; Based on the global attention mechanism after successful configuration, high-level semantic features of compressed tokens are captured to obtain the logical connections and key information of long texts; The logical connections and key information of long texts are evaluated to obtain evaluation results; The final visual token sequence is generated based on the evaluation results.

[0010] Optionally, in the context compression method based on visual modality described in this application embodiment, context optical decoding is performed on the final visual token sequence, and the context optical decoding data is fused with the text prompts input by the user to generate the target text, specifically including: Obtain the visual token sequence, filter and optimize the visual token sequence to obtain the final visual token sequence; Perform context optical decoding on the final visual token sequence. The context optical decoding includes combining the spatial positional relationship, semantic association features and context dependency information of the visual tokens to parse the context optical decoding data containing the semantic information of the visual content. Obtain text prompts input by the user, wherein the text prompts include at least one of the subject, style, and constraints of the target generated text; A fusion model for contextual optical decoding data and text prompts is established. The fusion model is used to perform semantic alignment, feature fusion, and weight allocation on the contextual optical decoding data and text prompts to generate target text.

[0011] Secondly, embodiments of this application provide a context compression system based on visual modality. The system includes a memory and a processor. The memory includes a program for a context compression method based on visual modality. When the program for the context compression method based on visual modality is executed by the processor, it implements the following steps: Obtain long text context data and render the long text context data into a corresponding document image; The document image is subjected to contextual optical encoding, and a visual segmentation model is used to segment the document image to obtain multiple image blocks, which are then converted into multiple initial visual tokens. The initial visual token is downsampled and compressed using a convolutional compression algorithm to obtain a compressed token. Based on the global attention mechanism, long-distance dependencies between different compressed tokens are captured, high-level semantic knowledge is extracted, and the final visual token sequence is output. Contextual optical decoding is performed on the final visual token sequence, and the contextual optical decoding data is fused with the text prompts entered by the user to generate the target text.

[0012] Optionally, in the visual modality-based context compression system described in this application embodiment, acquiring long text context data and rendering the long text context data into a corresponding document image specifically includes: Obtain long text context data, convert the long text context data into a format, and obtain long text context data in a standard format. The long text context data includes at least one of dialogue history, paper document, and long document content. The standard format long text context data is processed according to preset rules to obtain the processing result. The preset rules include font uniformity, paragraph alignment or page margin standardization. Based on the sorting results, analyze whether the text is cluttered; If confusion occurs, adjust the preset rules and reorganize. If no confusion occurs, the long text context data will be rendered into a single, continuous high-resolution document image based on the sorting results.

[0013] Optionally, in the context compression system based on visual modality described in this application embodiment, the document image is subjected to contextual optical encoding, and a visual segmentation model is used to segment the document image to obtain multiple image blocks and convert them into multiple initial visual tokens, specifically including: Input high-resolution document images into the visual segmentation model, set a sliding window, and traverse all high-resolution document images based on the sliding window; The continuous document image is segmented to obtain multiple independent image blocks; Extract the visual features of each image patch, and then combine the visual features of each image patch. The visual features of each image patch are quantized and encoded to obtain encoded data; Each image patch is converted into a fixed-dimensional initial visual token based on the encoded data, resulting in multiple initial visual tokens.

[0014] Thirdly, embodiments of this application also provide a computer-readable storage medium, which includes a visual modality-based context compression method program. When the visual modality-based context compression method program is executed by a processor, it implements the steps of the visual modality-based context compression method as described in any of the preceding claims.

[0015] As can be seen from the above, the context compression method, system, and medium based on visual modality provided in this application obtain long text context data and render it into a corresponding document image; perform contextual optical encoding on the document image, segment the document image using a visual segmentation model to obtain multiple image blocks and convert them into multiple initial visual tokens; perform downsampling compression on the initial visual tokens based on a convolutional compression algorithm to obtain compressed tokens; capture the long-distance dependencies between different compressed tokens based on a global attention mechanism, extract high-level semantic knowledge, and output the final visual token sequence; perform contextual optical decoding on the final visual token sequence, and fuse the contextual optical decoding data with the text prompts input by the user to generate the target text; through visual modality compression, the computational resources and memory usage required for processing long text are significantly reduced, and high text decoding accuracy is maintained while maintaining a high compression ratio. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating the context compression method based on visual modality provided in this application embodiment; Figure 2 A flowchart illustrating the document image acquisition process of the context compression method based on visual modality provided in this application embodiment; Figure 3 A flowchart illustrating the initial visual token acquisition process of the context compression method based on visual modality provided in this application embodiment. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0020] Please refer to Figure 1 , Figure 1 This is a flowchart of a visual modality-based context compression method according to some embodiments of this application. This visual modality-based context compression method is used in a terminal device and includes the following steps: S101, Obtain long text context data and render the long text context data into the corresponding document image; S102, perform contextual optical encoding on the document image, use a visual segmentation model to segment the document image, obtain multiple image blocks and convert them into multiple initial visual tokens; S103, the initial visual token is downsampled and compressed based on the convolutional compression algorithm to obtain a compressed token; S104 captures long-distance dependencies between different compressed tokens based on a global attention mechanism, extracts high-level semantic knowledge, and outputs the final visual token sequence. S105, perform contextual optical decoding on the final visual token sequence, fuse the contextual optical decoding data with the text prompts entered by the user, and generate the target text.

[0021] It should be noted that the long text content is rendered into an image, and then an efficient Visual Language Model (VLM) is used to read (i.e. decode) the original text from the image. Since the image can be represented with far fewer visual tokens than the original text, an extremely high information compression ratio is achieved. This invention preserves more long-range dependencies through visual modalities, reducing problems such as context inconsistencies, factual omissions, or logical breaks.

[0022] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a document image acquisition process using a visual modality-based context compression method according to some embodiments of this application. According to embodiments of the present invention, acquiring long text context data and rendering the long text context data into a corresponding document image specifically includes: S201, Obtain long text context data, convert the long text context data into a format, and obtain long text context data in a standard format. The long text context data includes at least one of dialogue history, paper document, and long document content. S202, standard format long text context data is processed according to preset rules to obtain the processing result. The preset rules include font uniformity, paragraph alignment or page margin standardization. S203, Analyze whether the text is disordered based on the sorting results; S204. If confusion occurs, adjust the preset rules and reorganize. S205, if no confusion occurs, the long text context data is rendered into a single continuous high-definition document image based on the sorting results.

[0023] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the initial visual token acquisition process of a context compression method based on visual modality in some embodiments of this application. According to embodiments of the present invention, contextual optical encoding is performed on the document image, and a visual segmentation model is used to segment the document image, obtaining multiple image blocks and converting them into multiple initial visual tokens. Specifically, this includes: S301: Input the high-definition document image into the visual segmentation model, set a sliding window, and traverse all high-definition document images based on the sliding window; S302, segment the continuous document image to obtain multiple independent image blocks; S303, extract the visual features of each image patch, and store the visual features of each image patch; S304, quantize and encode the visual features of each image block to obtain encoded data; S305 converts each image block into a fixed-dimensional initial visual token based on the encoded data, resulting in multiple initial visual tokens.

[0024] According to an embodiment of the present invention, a compressed token is obtained by downsampling and compressing an initial visual token based on a convolutional compression algorithm, specifically including: Obtain the initial visual token and parse the data from the initial visual token; The parsed data is then processed into a standardized format to obtain standard data; The parameters are configured based on the convolutional compression algorithm, and the standard data is compressed. The compressed data is then enhanced in terms of features, and a compressed token is obtained based on the enhanced data.

[0025] It should be noted that the initial visual tokens received from the Visual Segmentation Model (SAM) are in a two-dimensional matrix of token number × feature dimension, typically 1372 × 1024 (1372 tokens, each with 1024 features). This data needs to be converted to float32 using PyTorch's Tensor format conversion to avoid loss of data precision.

[0026] Data standardization: Channel-level standardization is performed on the token matrix. The mean and standard deviation of the 1024-dimensional feature channels are calculated. The feature values ​​of each token are normalized according to the formula (original value - mean) / standard deviation, and the feature distribution is constrained within the interval [-1,1] to eliminate the interference of the magnitude difference of different feature channels on the convolution calculation.

[0027] According to an embodiment of the present invention, a global attention mechanism is used to capture long-distance dependencies between different compressed tokens, extract high-level semantic knowledge, and output a final visual token sequence, specifically including: Receive the compressed token, preprocess and adapt the format of the compressed token to obtain an optimized compressed token; Configure the core parameters for the global attention mechanism; Based on the global attention mechanism after successful configuration, high-level semantic features of compressed tokens are captured to obtain the logical connections and key information of long texts; The logical connections and key information of long texts are evaluated to obtain evaluation results; The final visual token sequence is generated based on the evaluation results.

[0028] It should be noted that the global attention mechanism is executed in the order of layer-by-layer attention calculation → feature fusion → semantic enhancement to realize the transformation from compressed tokens to high-level semantic features, and to capture the logical connections and key information of long texts.

[0029] The underlying attention computation (layers 1-6): The input Tensor [1, 173, 768] is fed into the first layer attention module. The attention weight of each token with all 173 tokens is computed in parallel by 12 attention heads, with a focus on adjacent tokens (such as the word association within a sentence in the paper). Weight filtering: Randomly discard 10% of attention weights through the dropout mechanism to avoid over-reliance on local redundant features; Feature output: After residual connection and layer normalization, the output is a feature matrix that fuses local semantics, with dimensions maintained at [1, 173, 768].

[0030] High-level attention calculation (layers 7-12): Global Dependency Capture: The 7th layer and above modules strengthen the association between tokens across long distances (such as the semantic correspondence between the paper abstract and the conclusion) through attention weight calculation, and assign higher weights to key tokens (such as tokens corresponding to core words such as research methods and experimental results); Semantic Aggregation: Through the non-linear transformation of the FeedForward layer, the semantic information scattered in different tokens is aggregated into unified features, such as fusing the features of related tokens such as data samples, experimental equipment, and result indicators to form a complete representation of experimental information; Feature Refinement: The feature matrix output from the 12th layer is normalized to complete the final aggregation of global semantics.

[0031] According to an embodiment of the present invention, contextual optical decoding is performed on the final visual token sequence, and the contextual optical decoding data is fused with the text prompts input by the user to generate target text, specifically including: Obtain the visual token sequence, filter and optimize the visual token sequence to obtain the final visual token sequence; Contextual optical decoding is performed on the final visual token sequence. Contextual optical decoding includes combining the spatial positional relationship, semantic association features and contextual dependency information of the visual tokens to parse the contextual optical decoding data containing the semantic information of the visual content. Obtain text prompts input by the user, wherein the text prompts include at least one of the following: the topic, style, and constraints of the target generated text; A fusion model for contextual optical decoding data and text prompts is established. The fusion model performs semantic alignment, feature fusion, and weight allocation on the contextual optical decoding data and text prompts to generate the target text.

[0032] Secondly, embodiments of this application provide a context compression system based on visual modality. The system includes a memory and a processor. The memory includes a program for a context compression method based on visual modality. When the program for the context compression method based on visual modality is executed by the processor, it implements the following steps: Obtain the long text context data and render it into the corresponding document image; The document image is subjected to contextual optical encoding, and a visual segmentation model is used to segment the document image to obtain multiple image blocks, which are then converted into multiple initial visual tokens. The initial visual token is downsampled and compressed using a convolutional compression algorithm to obtain a compressed token. Based on the global attention mechanism, long-distance dependencies between different compressed tokens are captured, high-level semantic knowledge is extracted, and the final visual token sequence is output. Perform contextual optical decoding on the final visual token sequence, and fuse the contextual optical decoding data with the text prompts entered by the user to generate the target text.

[0033] According to an embodiment of the present invention, obtaining long text context data and rendering the long text context data into a corresponding document image specifically includes: Obtain long text context data, convert the long text context data into a format, and obtain long text context data in a standard format. The long text context data includes at least one of dialogue history, paper documents, and long document content. The standard format long text context data is processed according to preset rules to obtain the processing result. The preset rules include font uniformity, paragraph alignment or page margin standardization. Based on the sorting results, analyze whether the text is cluttered; If confusion occurs, adjust the preset rules and reorganize. If no confusion occurs, the long text context data will be rendered into a single, continuous high-resolution document image based on the sorting results.

[0034] According to embodiments of the present invention, contextual optical encoding is performed on a document image, and a visual segmentation model is used to segment the document image to obtain multiple image blocks, which are then converted into multiple initial visual tokens. Specifically, this includes: Input high-resolution document images into the visual segmentation model, set a sliding window, and traverse all high-resolution document images based on the sliding window; The continuous document image is segmented to obtain multiple independent image blocks; Extract the visual features of each image patch, and then combine the visual features of each image patch. The visual features of each image patch are quantized and encoded to obtain encoded data; Each image patch is converted into a fixed-dimensional initial visual token based on the encoded data, resulting in multiple initial visual tokens.

[0035] According to an embodiment of the present invention, a compressed token is obtained by downsampling and compressing an initial visual token based on a convolutional compression algorithm, specifically including: Obtain the initial visual token and parse the data from the initial visual token; The parsed data is then processed into a standardized format to obtain standard data; The parameters are configured based on the convolutional compression algorithm, and the standard data is compressed. The compressed data is then enhanced in terms of features, and a compressed token is obtained based on the enhanced data.

[0036] It should be noted that the initial visual tokens received from the Visual Segmentation Model (SAM) are in a two-dimensional matrix of token number × feature dimension, typically 1372 × 1024 (1372 tokens, each with 1024 features). This data needs to be converted to float32 using PyTorch's Tensor format conversion to avoid loss of data precision.

[0037] Data standardization: Channel-level standardization is performed on the token matrix. The mean and standard deviation of the 1024-dimensional feature channels are calculated. The feature values ​​of each token are normalized according to the formula (original value - mean) / standard deviation, and the feature distribution is constrained within the interval [-1,1] to eliminate the interference of the magnitude difference of different feature channels on the convolution calculation.

[0038] According to an embodiment of the present invention, a global attention mechanism is used to capture long-distance dependencies between different compressed tokens, extract high-level semantic knowledge, and output a final visual token sequence, specifically including: Receive the compressed token, preprocess and adapt the format of the compressed token to obtain an optimized compressed token; Configure the core parameters for the global attention mechanism; Based on the global attention mechanism after successful configuration, high-level semantic features of compressed tokens are captured to obtain the logical connections and key information of long texts; The logical connections and key information of long texts are evaluated to obtain evaluation results; The final visual token sequence is generated based on the evaluation results.

[0039] It should be noted that the global attention mechanism is executed in the order of "layer-by-layer attention calculation → feature fusion → semantic enhancement" to realize the transformation from compressed tokens to high-level semantic features, and to capture the logical connections and key information of long texts.

[0040] Low-level attention calculation (layers 1-6): Input the Tensor [1, 173, 768] into the first layer attention module, and compute the attention weight of each token with all 173 tokens in parallel through 12 attention heads, focusing on adjacent tokens (such as the word association within a sentence in the paper). Weight filtering: Randomly discard 10% of attention weights through the dropout mechanism to avoid over-reliance on local redundant features; Feature output: After residual connection and layer normalization, the output is a feature matrix that fuses local semantics, with dimensions maintained at [1, 173, 768].

[0041] High-level attention calculation (layers 7-12): Global Dependency Capture: The 7th layer and above modules strengthen the association of tokens across long distances (such as the semantic correspondence between the paper abstract and the conclusion) through attention weight calculation, and assign higher weights to key tokens (such as tokens corresponding to core words such as research methods and experimental results). Semantic aggregation: Through the non-linear transformation of the FeedForward layer, semantic information scattered in different tokens is aggregated into unified features, such as fusing features of related tokens such as data samples, experimental equipment, and result indicators to form a complete representation of experimental information; Feature Refinement: The feature matrix output from the 12th layer is normalized to complete the final aggregation of global semantics.

[0042] According to an embodiment of the present invention, contextual optical decoding is performed on the final visual token sequence, and the contextual optical decoding data is fused with the text prompts input by the user to generate target text, specifically including: Obtain the visual token sequence, filter and optimize the visual token sequence to obtain the final visual token sequence; Contextual optical decoding is performed on the final visual token sequence. Contextual optical decoding includes combining the spatial positional relationship, semantic association features and contextual dependency information of the visual tokens to parse the contextual optical decoded data containing the semantic information of the visual content. Obtain text prompts input by the user, wherein the text prompts include at least one of the following: the topic, style, and constraints of the target generated text; A fusion model for contextual optical decoding data and text prompts is established. The fusion model performs semantic alignment, feature fusion, and weight allocation on the contextual optical decoding data and text prompts to generate the target text.

[0043] A third aspect of the present invention provides a computer-readable storage medium including a visual modality-based context compression method program, wherein when the visual modality-based context compression method program is executed by a processor, it implements the steps of the visual modality-based context compression method as described in any of the preceding claims.

[0044] This invention discloses a visual modality-based context compression method, system, and medium. It acquires long text context data and renders it into a corresponding document image. The document image undergoes contextual optical encoding, and a visual segmentation model is used to segment the image, resulting in multiple image blocks that are converted into multiple initial visual tokens. A convolutional compression algorithm is used to downsample and compress the initial visual tokens to obtain compressed tokens. A global attention mechanism is used to capture long-distance dependencies between different compressed tokens, extracting high-level semantic knowledge and outputting a final visual token sequence. Contextual optical decoding is performed on the final visual token sequence, and the decoded data is fused with user-inputted text prompts to generate the target text. Through visual modality compression, the computational resources and memory required for processing long text are significantly reduced, maintaining a high compression ratio while still achieving high text decoding accuracy.

[0045] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0046] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0047] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0048] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0049] Alternatively, if the integrated units of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A context compression method based on visual modality, characterized in that, include: Obtain long text context data and render the long text context data into a corresponding document image; The document image is subjected to contextual optical encoding, and a visual segmentation model is used to segment the document image to obtain multiple image blocks, which are then converted into multiple initial visual tokens. The initial visual token is downsampled and compressed using a convolutional compression algorithm to obtain a compressed token. Based on the global attention mechanism, long-distance dependencies between different compressed tokens are captured, high-level semantic knowledge is extracted, and the final visual token sequence is output. Contextual optical decoding is performed on the final visual token sequence, and the contextual optical decoding data is fused with the text prompts entered by the user to generate the target text.

2. The context compression method based on visual modality according to claim 1, characterized in that, Obtaining long text context data and rendering the long text context data into a corresponding document image specifically includes: Obtain long text context data, convert the long text context data into a format, and obtain long text context data in a standard format. The long text context data includes at least one of dialogue history, paper document, and long document content. The standard format long text context data is processed according to preset rules to obtain the processing result. The preset rules include font uniformity, paragraph alignment or page margin standardization. Based on the sorting results, analyze whether the text is cluttered; If confusion occurs, adjust the preset rules and reorganize. If no confusion occurs, the long text context data will be rendered into a single, continuous high-resolution document image based on the sorting results.

3. The context compression method based on visual modality according to claim 2, characterized in that, The document image is subjected to contextual optical encoding, and a visual segmentation model is used to segment the document image to obtain multiple image patches, which are then converted into multiple initial visual tokens. Specifically, this includes: Input high-resolution document images into the visual segmentation model, set a sliding window, and traverse all high-resolution document images based on the sliding window; The continuous document image is segmented to obtain multiple independent image blocks; Extract the visual features of each image patch, and then combine the visual features of each image patch. The visual features of each image patch are quantized and encoded to obtain encoded data; Each image patch is converted into a fixed-dimensional initial visual token based on the encoded data, resulting in multiple initial visual tokens.

4. The context compression method based on visual modality according to claim 3, characterized in that, The initial visual token is downsampled and compressed using a convolutional compression algorithm to obtain a compressed token, specifically including: Obtain the initial visual token and parse the data from the initial visual token; The parsed data is then processed into a standardized format to obtain standard data; The parameters are configured based on the convolutional compression algorithm, and the standard data is compressed. The compressed data is then enhanced in terms of features, and a compressed token is obtained based on the enhanced data.

5. The context compression method based on visual modality according to claim 4, characterized in that, Based on a global attention mechanism, long-distance dependencies between different compressed tokens are captured, high-level semantic knowledge is extracted, and the final visual token sequence is output, including: Receive the compressed token, preprocess and adapt the format of the compressed token to obtain an optimized compressed token; Configure the core parameters for the global attention mechanism; Based on the global attention mechanism after successful configuration, high-level semantic features of compressed tokens are captured to obtain the logical connections and key information of long texts; The logical connections and key information of long texts are evaluated to obtain evaluation results; The final visual token sequence is generated based on the evaluation results.

6. The context compression method based on visual modality according to claim 5, characterized in that, Perform contextual optical decoding on the final visual token sequence, fuse the contextual optical decoding data with the text prompts input by the user, and generate the target text, specifically including: Obtain the visual token sequence, filter and optimize the visual token sequence to obtain the final visual token sequence; Perform context optical decoding on the final visual token sequence. The context optical decoding includes combining the spatial positional relationship, semantic association features and context dependency information of the visual tokens to parse the context optical decoding data containing the semantic information of the visual content. Obtain text prompts input by the user, wherein the text prompts include at least one of the subject, style, and constraints of the target generated text; A fusion model for contextual optical decoding data and text prompts is established. The fusion model is used to perform semantic alignment, feature fusion, and weight allocation on the contextual optical decoding data and text prompts to generate target text.

7. A context compression system based on visual modality, characterized in that, The system includes: a memory and a processor, wherein the memory includes a program for a visual modality-based context compression method, and when the program for the visual modality-based context compression method is executed by the processor, it performs the following steps: Obtain long text context data and render the long text context data into a corresponding document image; The document image is subjected to contextual optical encoding, and a visual segmentation model is used to segment the document image to obtain multiple image blocks, which are then converted into multiple initial visual tokens. The initial visual token is downsampled and compressed using a convolutional compression algorithm to obtain a compressed token. Based on the global attention mechanism, long-distance dependencies between different compressed tokens are captured, high-level semantic knowledge is extracted, and the final visual token sequence is output. Contextual optical decoding is performed on the final visual token sequence, and the contextual optical decoding data is fused with the text prompts entered by the user to generate the target text.

8. The context compression system based on visual modality according to claim 7, characterized in that, Obtaining long text context data and rendering the long text context data into a corresponding document image specifically includes: Obtain long text context data, convert the long text context data into a format, and obtain long text context data in a standard format. The long text context data includes at least one of dialogue history, paper document, and long document content. The standard format long text context data is processed according to preset rules to obtain the processing result. The preset rules include font uniformity, paragraph alignment or page margin standardization. Based on the sorting results, analyze whether the text is cluttered; If confusion occurs, adjust the preset rules and reorganize. If no confusion occurs, the long text context data will be rendered into a single, continuous high-resolution document image based on the sorting results.

9. The context compression system based on visual modality according to claim 8, characterized in that, The document image is subjected to contextual optical encoding, and a visual segmentation model is used to segment the document image to obtain multiple image patches, which are then converted into multiple initial visual tokens. Specifically, this includes: Input high-resolution document images into the visual segmentation model, set a sliding window, and traverse all high-resolution document images based on the sliding window; The continuous document image is segmented to obtain multiple independent image blocks; Extract the visual features of each image patch, and then combine the visual features of each image patch. The visual features of each image patch are quantized and encoded to obtain encoded data; Each image patch is converted into a fixed-dimensional initial visual token based on the encoded data, resulting in multiple initial visual tokens.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a visual modality-based context compression method program, which, when executed by a processor, implements the steps of the visual modality-based context compression method as described in any one of claims 1 to 6.

Images