A long text processing method, device, equipment and medium

Abstract

This application relates to the field of artificial intelligence technology and discloses a method, apparatus, device, and medium for processing long text. The method includes: processing a long natural language text to be parsed to determine an ultra-long visual image; performing multi-level encoding on the ultra-long visual image to generate standardized visual modal encoding; encoding input text prompts to obtain a text hidden state; extracting general semantic features of the text hidden state; and selecting modal visual features adapted to the text hidden state from the visual modal encoding; fusing the general semantic features, the modal visual features, and the text hidden state to obtain a target fusion feature, which is used to generate a text sequence based on the target fusion feature. The technical solution provided by this application can improve the ability of large language models to perceive global semantic and visual information when processing long text.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a long text processing method, apparatus, device and medium. Background Technology

[0002] Current natural language processing (NLP) technologies face numerous challenges in understanding and generating long texts, particularly in the legal and political fields. Large language models, limited by sequence length, struggle to fully understand global semantics when processing extremely long texts, resulting in a lack of coherence and completeness in the generated content. Furthermore, existing methods lack dynamic adaptability in utilizing cross-modal information, failing to flexibly integrate visual information with text, thus affecting the accuracy of the generated results. Finally, insufficient controllability of the generation methods prevents models from selecting the most relevant external information based on context, thereby reducing the relevance and practicality of the generated content, which is especially critical in highly specialized fields such as law.

[0003] Therefore, how to improve the ability of large language models to perceive global semantics and visual information when processing long texts is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a long text processing method, apparatus, device, and medium, which achieves the technical effect of improving the ability of large language models to perceive global semantic and visual information when processing long texts.

[0005] To achieve the above objectives, the main technical solutions adopted in this application include: In a first aspect, embodiments of this application provide a long text processing method, the method comprising: The ultra-long visual image is determined by processing the long natural language text to be parsed. The ultra-long visual image is encoded in multiple layers to generate a standardized visual modality code; The input text prompt words are encoded to obtain the text hidden state, the general semantic features of the text hidden state are extracted, and the modal visual features that are suitable for the text hidden state are selected from the visual modal encoding. The general semantic features of the text, the modal visual features, and the hidden state of the text are fused to obtain target fusion features, which are then used to generate text sequences based on the target fusion features.

[0006] In one implementation, the method for determining the ultra-long visual image includes: Based on the language type of the natural language long text, determine the layout configuration file adapted to the natural language long text; The natural language long text is formatted according to the formatting configuration file to obtain a formatted multi-page document; The multi-page document is converted into multiple bitmap images; The multiple bitmap images are stitched together according to the page order of the multi-page document to obtain the ultra-long visual image.

[0007] In one implementation, the step of performing multi-level encoding on the ultra-long visual image to generate standardized visual modality codes includes: Based on the relationship between the image size of the ultra-long visual image and the preset image block size, an image block that matches the preset image block size is determined; Fine-grained modeling is performed on the image patch to obtain local detail features, and global semantic modeling is performed on the image patch to obtain global semantic features; The local detail features and the global semantic features are concatenated to form a fused feature vector, and the fused feature vector is transformed into a feature sequence adapted to the dimensions of a large language model. The feature sequence is added with structural markers and positional information and then normalized to obtain the visual modality code.

[0008] In one implementation, determining the image block that matches the preset image block size based on the relationship between the image size of the ultra-long visual image and the preset image block size includes: If the image height in the image size is greater than the preset height in the preset image block size, then the ultra-long visual image is divided into multiple image blocks with a height equal to the preset height along the height direction of the ultra-long visual image; the height direction represents the longitudinal spatial extension direction of the ultra-long visual image; If the image height in the image size is less than the preset height in the preset image block size, then size padding is performed at the bottom of the ultra-long visual image to obtain an image block that matches the preset image block size; the padding height is determined according to the preset height in the preset image block size, and the height of the image block is divisible by the preset height.

[0009] In one implementation, extracting the general semantic features of the hidden text state includes: The text hiding state is normalized to obtain the normalized text hiding state; Extract the semantic gating signal and text feature representation corresponding to the normalized text hidden state; Semantic filtering is performed based on the semantic gating signal and the text feature representation to generate general semantic features of the text that match the dimension of the hidden text state.

[0010] In one implementation, the step of filtering modal visual features adapted to the text hiding state from the visual modal encoding includes: The text hiding state is normalized to obtain the normalized text hiding state; Using the normalized text hidden state as the query vector and the visual modality encoding as the key vector, the attention score between the query vector and the key vector is determined, and the visual modality encoding is aggregated based on the attention score to obtain the aggregated modality features. Extract the text modality gating signal corresponding to the normalized text hidden state, and extract the modality feature representation corresponding to the aggregated modality feature; Semantic filtering is performed based on the text modality gating signal and the modality feature representation to generate modality visual features that match the dimension of the text hidden state.

[0011] In one implementation, fusing the general semantic features of the text, the modal visual features, and the text hidden state to obtain the target fused features includes: The textual general semantic features and the modal visual features are added element-wise to obtain the dual-branch fusion features; The hidden state of the text is determined as the original text residual stream; The dual-branch fusion feature is superimposed on the original text residual stream to obtain the target fusion feature.

[0012] In one implementation, the method is achieved through a trained cross-modal model, which is trained in a phased manner, comprising a first phase and a second phase; wherein... The first stage includes: Obtain the long training natural language text and training text prompts; Determine the training visual modality encoding corresponding to the training natural language long text, and determine the training text tag sequence corresponding to the training text prompt words; The trained visual modality encoding and the trained text tag sequence are concatenated and expanded before being input into Transformer encoding. Cross-modal attention constraints are used to achieve coarse-grained spatial alignment between the visual and text modalities. The second phase includes: Based on the training text tag sequence, a dual-branch SwiGLU gating process is used to fuse the general semantic features of the training text with the adapted training visual modality encoding, thereby completing the cross-modal deep interaction fusion guided by text semantics.

[0013] Secondly, embodiments of this application provide a long text processing apparatus, the apparatus comprising: The long text processing unit is used to process long natural language text to be parsed and determine ultra-long visual images; A visual modality coding unit is used to perform multi-level coding on the ultra-long visual image to generate standardized visual modality codes. The dual feature extraction and filtering unit is used to encode the input text prompt words to obtain the text hidden state, extract the general semantic features of the text hidden state, and filter the modal visual features that are suitable for the text hidden state from the visual modal encoding. A cross-modal fusion unit is used to fuse the general semantic features of the text, the modal visual features, and the hidden state of the text to obtain target fusion features, which are then used to generate a text sequence based on the target fusion features.

[0014] Thirdly, embodiments of this application provide a computer device, including: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes these computer instructions to perform the long text processing method described above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the long text processing method described above.

[0016] The technical solutions provided in one or more real-time methods of this application transform long natural language text into ultra-long visual images. Then, multi-level encoding is performed on the ultra-long visual images to generate standardized visual modality codes, enabling the effective extraction of key information. This process ensures that the large language model can combine relevant visual information when parsing long text, improving its ability to understand the context. Simultaneously, the input text prompts are encoded to obtain corresponding text hidden states. By extracting the general semantic features of the text hidden states and selecting suitable visual features, effective matching between text and visual information is achieved. Finally, these features are fused to generate target fusion features, providing the large language model with a multi-dimensional information perspective. Ultimately, the required text sequence can be generated based on these target fusion features, significantly enhancing its perception ability when processing complex text content involving visual elements. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a long text processing method provided in this application embodiment; Figure 2 A flowchart illustrating the method for determining ultra-long visual images provided in embodiments of this application; Figure 3 A flowchart of step S3 provided in the embodiments of this application; Figure 4 A flowchart for extracting general semantic features of hidden text states provided in an embodiment of this application; Figure 5 A flowchart for filtering modal visual features adapted to the hidden state of text from visual modal coding, provided for embodiments of this application; Figure 6 A flowchart of step S7 provided in an embodiment of this application; Figure 7 A model architecture diagram of the first-stage coarse-grained alignment provided for embodiments of this application; Figure 8 A model architecture diagram of the second-stage deep semantic fusion provided in this application embodiment; Figure 9 A block diagram of a long text processing device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Current natural language processing technologies face several challenges in understanding and generating long texts, particularly in fields such as law and government. First, existing large language models, due to sequence length limitations, often focus only on local contexts when processing extremely long texts, demonstrating insufficient understanding of global semantics. This limitation makes it difficult for models to capture key information and complex logical relationships within long documents, potentially resulting in generated content lacking coherence and comprehensiveness.

[0021] Secondly, regarding the utilization of cross-modal information, traditional methods often employ static or fixed approaches to integrate visual information into language generation. This approach lacks a dynamic adaptation mechanism and cannot flexibly adjust the use of visual information according to the different needs of the text content. For example, in the drafting of legal documents, relevant charts or data may be crucial for text comprehension, but if the model cannot effectively combine this visual information with the textual context, it will affect the accuracy and usability of the final generated results.

[0022] Finally, existing generation methods also suffer from significant shortcomings in terms of controllability. Many large language models are unable to flexibly select the most relevant external information to assist in the generation process based on the current context, resulting in reduced relevance and practicality of the generated content. In highly specialized fields such as law and government affairs, generated content requires a high degree of accuracy and relevance; therefore, if the model cannot dynamically select and inject key information, it will severely hinder its application effectiveness.

[0023] Therefore, how to improve the ability of large language models to perceive global semantics and visual information when processing long texts is a technical problem that urgently needs to be solved.

[0024] To address the aforementioned technical problems, an embodiment of a long text processing method is provided according to an embodiment of this application. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] This embodiment provides a method for processing long text. Figure 1 A flowchart of a long text processing method provided in this application embodiment is shown below. Figure 1 As shown, the process includes the following steps: Step S1: Process the long natural language text to be parsed to determine the ultra-long visual image.

[0026] Specifically, with the core principles of preserving the semantic integrity of long texts and adapting to the characteristics of subsequent model processing, the processing is completed in two parallel links: On the one hand, for the visualization transformation link of long natural language texts, the semantic relationships of long natural language texts (such as paragraph hierarchy and page cross-page logic) are first transformed into visual layout relationships (line spacing, paragraph indentation, and page correspondence) that conform to human reading cognition through structured typesetting. Then, lossless conversion tools such as pdf2image are used to convert the structured document into multiple bitmap images. Finally, the images of multiple pages are vertically and seamlessly stitched together in the reading order to generate an ultra-long visual image with a fixed width and adaptive length. This image fully preserves the character details, typesetting structure, and semantic flow continuity of the text, indirectly encoding the semantics of the text into visual layout features, and building an adaptation bridge between natural language and visual encoder.

[0027] Step S3: Perform multi-level encoding on the ultra-long visual image to generate standardized visual modality codes.

[0028] Specifically, for ultra-long visual images with fixed width and variable height, dynamic slicing (including pixel-level overlapping areas + tail backing adjustment) or bottom padding strategies are used to transform them into standardized image blocks that conform to the input specifications of the visual encoder, avoiding size mismatch and semantic fragmentation at the source. Secondly, a local-global dual-path parallel feature extraction structure is adopted. The local branch captures fine-grained details such as character shape and text edges through multi-scale adaptive modeling, while the global branch mines macroscopic layout and cross-regional semantic associations through CLS token aggregation and adaptive positional encoding, achieving full-level coverage of visual features. Subsequently, the dual-path features are concatenated in the channel dimension to form a fused feature vector, which is transformed into a feature sequence matching the hidden layer dimension of the large language model through the MLP projection module. Then, line break / separator structure markers and dual positional encoding (hierarchical embedding + global vertical embedding) are injected, and RMSnorm normalization is used to standardize the feature distribution, enhance the structural awareness and positional accuracy of the features, and generate standardized visual modality codes. The entire encoding process not only ensures the semantic continuity of ultra-long texts, but also breaks down the fusion barriers between visual and textual modalities through dimensional adaptation and standardization, providing a core visual input that is structurally stable, semantically complete, and dimensionally compatible for subsequent cross-modal attention retrieval and feature fusion.

[0029] Step S5: Encode the input text prompt words to obtain the text hidden state, extract the general semantic features of the text hidden state, and select the modal visual features that are suitable for the text hidden state from the visual modal encoding.

[0030] Specifically, for user-inputted text prompts (such as whether the company's legal status and continued existence meet the basic requirements for listing? Whether the company's internal decision-making procedures regarding this IPO are complete?), the prompts are first broken down into several tokens, encoded into high-dimensional token vectors through an embedding layer. This vector forms the initial basis for generating the text hidden state. Then, through several layers of Transformer Blocks, the prompts are computed layer by layer, progressively extracting morphemes, syntax, semantics, and other features at different levels and updating the vectors. During the update process, the prompts are fused with visual modal features in each layer of the feedforward network (FFN). It should be noted that, apart from the initial token vectors output by the embedding layer and the final output of the last layer of the model, the token feature vectors generated by each intermediate layer within the model are collectively referred to as the text hidden state. This state, as the core foundation of text-dominant semantics, fully preserves the original semantic logic of long natural language texts, providing a semantic benchmark for text-dominant and visually adapted cross-modal fusion.

[0031] Based on the normalized text hidden state as a unified semantic benchmark, feature processing is carried out simultaneously in two independent branches: the first branch is pure text feature processing, which takes only the normalized text hidden state as input, and uses the SwiGLU gating structure to generate semantic gating signals through gating linear mapping, upsampling mapping to obtain text feature representations, and then achieves text semantic filtering and enhancement through element-wise multiplication gating modulation. Finally, it restores the original text hidden state dimension through downsampling mapping, extracts general semantic features of the text that remove redundant information and enhance the core semantics, and introduces no external modal information throughout the process, only achieving nonlinear self-enhancement of text semantics. The second branch is cross-modal feature selection. It uses the normalized text hidden state as the query benchmark and the visual modality encoding as the key-value vector. It completes cross-modal attention retrieval through an improved Softmax-one normalization strategy, aggregates visual features that are highly related to the text semantics, and then generates a dedicated text modality gating signal based on the text hidden state based on the SwiGLU gating structure of the same architecture. It performs a second precise selection of the aggregated visual features guided by text semantics. After dimensional adaptation and downsampling restoration, it selects modal visual features that are completely adapted to the semantics of the current text hidden state and have no redundant interference. The whole process takes text semantics as the core control logic to achieve cross-modal precise adaptation of visual features.

[0032] Step S7: The general semantic features, modal visual features and hidden states of the text are fused to obtain the target fusion features, which are used to generate text sequences based on the target fusion features.

[0033] Specifically, element-wise addition is performed on the general semantic features and modal visual features of the text to achieve initial integration of the two-branch features, allowing the non-linear enhanced semantics of the text itself and the precise visual supplementary semantics guided by the text semantics to form a synergistic effect. Subsequently, the original text hidden state is used as the original text residual stream, and additive superposition is performed again with the two-branch fusion features. The purest core semantic base of the text is preserved through residual connection, fundamentally preventing excessive interference from cross-modal information. The entire fusion process has no dimensional expansion or complex feature transformation, and only two pure linear additive operations are used to integrate the three types of features. The final target fusion feature not only preserves the original semantic logic and contextual association of the long text with the original text hidden state as the core, but also incorporates the deep enhanced semantics of the general semantic features of the text, and supplements the missing information of the text modality such as the layout structure and local details of the modal visual features, achieving triple information integration. At the same time, the target fusion feature is fully compatible with the input dimension of the large language model and can be directly used by the large language model for parsing, realizing a seamless connection from cross-modal fusion processing to long text parsing of the large language model, and keeping the injection of cross-modal information in a precise and controllable state.

[0034] This embodiment provides a long text processing method, the core of which is to modify the "input form" and "feature fusion" of large text generation models (this modification method is applicable to all large text generation models and has universality). Specifically, it involves converting long natural language text into ultra-long visual images. Next, the ultra-long visual images are encoded at multiple levels to generate standardized visual modal codes, enabling the effective extraction of key information. This process ensures that the large language model can combine relevant visual information when parsing long text, improving its ability to understand the context. Simultaneously, the input text prompts are encoded to obtain the corresponding text hidden states. By extracting the general semantic features of the text hidden states and selecting suitable visual features, effective matching between text and visual information is achieved. Finally, these features are fused to generate target fusion features, providing the large language model with a multi-dimensional information perspective. Ultimately, the required text sequence can be generated based on these target fusion features, significantly enhancing its perception ability when processing complex text content involving visual elements. The core role of the large language model in this process is to encode the background knowledge of ultra-long text in image form and the information of input text prompts, fuse the features of the two modalities, and generate the final text. Specifically, the model extracts the vector representation of text words and visual modal features, and performs deep feature fusion in the feedforward network layer (FFN) of each transformer. Through layer-by-layer forward computation and feature injection, it fully integrates ultra-long background knowledge and generates accurate text sequences. This method achieves efficient text generation by utilizing ultra-long text background knowledge and is applicable to various practical task scenarios, including but not limited to intelligent question answering, text summarization, intelligent writing, and conversational interaction.

[0035] Figure 2 A flowchart illustrating the method for determining ultra-long visual images provided in embodiments of this application is provided. This process may include the following steps: Step S11: Determine the appropriate typesetting configuration file for the long natural language text based on its language type.

[0036] Specifically, based on the language type of the long natural language text to be parsed (covering Chinese and English, and expandable to other languages), the system automatically matches and loads the corresponding typesetting configuration file. For example, the Chinese configuration file adapts to the block structure and stroke density characteristics of Chinese characters, while the English configuration file adapts to the linear arrangement of letters and the need for case sensitivity. This avoids typesetting errors caused by a single configuration from the source (such as using Chinese line spacing in English causing character crowding, or using English fonts in Chinese causing display abnormalities). The typesetting configuration file includes at least parameters such as page size, DPI precision, font path, line height, first-line indentation, and text alignment, ensuring that the visualization results of texts in different languages ​​have consistent readability and structural standardization, and avoiding the loss or distortion of semantic information during subsequent feature extraction due to typesetting differences.

[0037] Step S13: Format the long natural language text according to the formatting configuration file to obtain a formatted multi-page document.

[0038] Specifically, unstructured original long text (without fixed format, disordered paragraphs, or simply a sequence of characters) is transformed into a structured, uniformly formatted multi-page intermediate document. This process fully preserves the text's hierarchical relationships (headings / body text), paragraph structure, list logic, and other semantic connections, providing a foundation for subsequent image conversion and stitching. Specifically, a typesetting rendering engine (such as ReportLab) is used to unify fonts and sizes according to the typesetting configuration file (e.g., Chinese body text size 4, English body text size 12pt), eliminating formatting differences in the original text. The line width and height of each page are fixed to ensure even text distribution within the page, avoiding character crowding or excessive white space due to line width fluctuations. When the content of a single page reaches the page size limit, automatic pagination is performed. The natural paragraph indentation, heading hierarchy (first-level headings / second-level headings, etc.), and list structure (ordered lists / unordered lists) of the original text are fully replicated, ensuring that the formatted document conforms to human reading habits and that the semantic structure is not lost. After typesetting, output a multi-page document (preferably a PDF document). Each page has a fixed visual size and standardized layout, which ensures the readability of the text within a single page and provides a unified processing object for subsequent batch image conversion.

[0039] Step S15: Convert the multi-page document into images to obtain multiple bitmap images.

[0040] Specifically, an image conversion tool such as pdf2image is used to output RGB bitmap images in PNG format. PNG format is chosen because it supports lossless compression, preserving the text's color information (such as font weight and highlighting) and edge details, avoiding feature blurring caused by compression. The RGB color mode is compatible with the input requirements of mainstream visual encoders, ensuring compatibility for feature extraction. The structured multi-page document is converted into pixel-level bitmap images that can be directly processed by the visual encoder, achieving a mapping between text semantics and visual pixels, while preserving all layout details with high fidelity, providing raw data support for subsequent visual feature extraction. The number of bitmap images is the same as the number of pages in the multi-page document.

[0041] It should be noted that during the conversion process, it is necessary to ensure that the layout structure remains unchanged: line spacing, paragraph indentation, and page break positions are completely consistent with multi-page documents; character features remain unchanged: symbol shape, font weight, and font size are not distorted, ensuring that subsequent local feature extraction can accurately capture character details.

[0042] Step S17: Stitch together multiple bitmap images according to the page order of the multi-page document to obtain an ultra-long visual image.

[0043] Specifically, to break down the physical separation of multiple pages of images and restore them to a continuous visual carrier consistent with the reading order of the original long text, avoiding semantic fragmentation caused by pagination, it is necessary to stitch multiple pages of documents together. Specific stitching rules include: using a vertical stitching method to perfectly match the reading order from top to bottom and from left to right, ensuring that the semantic flow of the stitched image is consistent with the semantic flow of the original text; maintaining a fixed width for the stitched image (consistent with the width of a single-page image), accommodating multiple pages of content only through continuous expansion in the height direction, avoiding visual encoder input adaptation problems caused by width changes; not introducing any additional blank areas or separators during the stitching process, allowing text on adjacent pages to seamlessly connect in visual space (e.g., the last paragraph of the previous page is directly adjacent to the first paragraph of the next page), restoring the semantic continuity of cross-page text; and having no fixed upper limit on the stitching length, dynamically expanding according to the number of pages in the long text to meet the visualization needs of ultra-long texts (e.g., documents with thousands of pages).

[0044] This embodiment improves text readability by automatically selecting a suitable typesetting configuration file for long natural language texts, ensuring effective processing of different language characteristics. Based on the typesetting configuration file, the original unstructured long natural language text is converted into structured multi-page documents, preserving the hierarchical relationships and important information, enabling large language models to understand the content more clearly. These documents are then converted into high-quality bitmap images, ensuring the integrity of text details, which provides a solid foundation for visual analysis. Multiple bitmap images are sequentially stitched together to form an ultra-long visual image, eliminating the semantic fragmentation caused by pagination and promoting coherent understanding of the context.

[0045] Figure 3 The flowchart for step S3 provided in the embodiments of this application may include the following steps: Step S31: Based on the relationship between the image size of the ultra-long visual image and the preset image block size, determine the image block that matches the preset image block size.

[0046] Specifically, in order to achieve dynamic standardization of ultra-long visual image inputs, regardless of whether the original image size is much larger or slightly smaller than the preset image block size, it can be transformed into image blocks of uniform size that meet the encoder requirements, adapting to the fixed input specifications of the visual encoder. This avoids semantic loss or computational anomalies caused by size mismatch or boundary truncation from the source, laying a unified input foundation for subsequent feature extraction.

[0047] Specifically, it includes two scenarios: long image slicing and short image completion strategy. In the long image slicing scenario, if the image height in the image size is greater than the preset height in the preset image block size, then the ultra-long visual image is divided into multiple image blocks with a height equal to the preset height along the height direction of the ultra-long visual image; the height direction represents the vertical spatial extension direction of the ultra-long visual image.

[0048] It should be noted that, for long image slicing scenarios, pixel-level overlap areas are set between adjacent image blocks to avoid the hard truncation of structural information such as text, paragraphs, and lists at the slice boundaries, thus ensuring semantic continuity across image blocks. If the height of the last image block after segmentation is less than the preset height, the starting position of the tail image block is adjusted by backscaling forward by a preset length, so that the height of the adjusted tail image block strictly matches the preset height. This ensures the uniformity of the size of all image blocks and avoids feature extraction deviations caused by abnormal tail slice sizes.

[0049] The short image padding strategy works as follows: if the image height in the overall image size is less than the preset height in the preset image block size, then size padding is performed at the bottom of the extra-long visual image to obtain an image block that matches the preset image block size. The padding height is determined based on the preset height in the preset image block size, and the height of the image block is divisible by the preset height. The padding area is generated only at the bottom of the image and does not cover the original text content, ensuring that the visual features and semantic information of the original text are not tampered with or lost.

[0050] Step S33: Perform fine-grained modeling on the image patch to obtain local detail features, and perform global semantic modeling on the image patch to obtain global semantic features.

[0051] The core objective of local detail feature extraction is to capture fine-grained visual information in image blocks, including high-frequency details such as character shape, character edges, line spacing, paragraph indentation, and local list structure. This information is the core basis for restoring the basic semantics of text.

[0052] Specifically, a local feature extraction module (such as the SAM encoder based on the Vision Transformer architecture) is adopted to adapt to the fine-grained modeling requirements of image patches. Based on the dynamic height and width of the input image patch, bicubic interpolation resampling is performed on the preset absolute position embedding to ensure that the position encoding accurately matches the actual spatial layout characteristics of ultra-long visual images, avoiding position information deviation caused by the mismatch between fixed position encoding and dynamic image size. Windowing and global attention modeling are performed through multi-layer cascaded Transformer blocks to achieve refined feature mining of local regions. After the initial dimension mapping is completed by the convolutional network, feature extraction is performed through cascaded multi-layer downsampling convolutional layers. A dynamic scale selection strategy is adopted, and convolution operations with a stride of 2 are recursively performed according to the height of the input image patch, so that the number of feature channels gradually expands with the sampling depth (e.g., 256→512→1024), generating local feature representations of different granularities such as 64×64 and 128×128. If the image height is insufficient to support a certain scale, the largest available scale is automatically selected for modeling to ensure the adaptability and effectiveness of feature extraction.

[0053] The core objective of global semantic feature extraction is to capture macroscopic semantic and layout information in image blocks, including the overall text layout structure, heading hierarchy, cross-paragraph semantic relationships, style distribution, and other macroscopic information, to provide contextual support for the overall understanding of long texts.

[0054] Specifically, a global semantic encoding module (such as the CLIP-ViT-L encoder) is adopted to meet the global semantic aggregation requirements of image patches. First, the local detail features output by the local feature extraction module are mapped into feature sequences, and a category token (CLS Token) is introduced to achieve efficient aggregation of global semantic information. The position encoding is transformed by spatial scale through an interpolation mechanism, separating the position encoding into fixed token bits and variable spatial bits. Only the variable spatial bits are resampled to accurately match the actual resolution of the image patch, ensuring accurate alignment of spatial position information. The feature sequence is deeply modeled through a Transformer network structure, and a multi-head self-attention mechanism is used to capture the macroscopic layout structure and style distribution within the image patch. With the help of a nonlinear activation function (such as Quick GELU), the extraction and enhancement of global semantic features are completed, achieving cross-region semantic alignment.

[0055] Step S35: Concatenate the local detail features and global semantic features to form a fused feature vector, and transform the fused feature vector into a feature sequence adapted to the dimensions of a large language model.

[0056] Specifically, the local detail features output by the local detail feature extraction module and the global semantic features output by the global semantic encoding module are concatenated along the channel dimension to generate a fused feature vector. The fused feature vector is then processed using the feature mapping module (MLPProjector). Through a combination of linear projection and non-linear activation, the fused feature vector is transformed into a feature sequence whose dimension perfectly matches the hidden layer dimension of the large language model. For feature layers of different scales generated during the local detail feature extraction process, a parallel processing strategy is employed to ensure that the local detail features at each level are deeply aligned with the global semantic features before dimensionality transformation, thus guaranteeing the integrity of multi-scale features.

[0057] Step S37: Add structural markers and positional information to the feature sequence and normalize it to obtain the visual modality code.

[0058] Specifically, dedicated structural tokens are introduced into the feature sequence to explicitly represent the inherent structure of ultra-long visual images. This enables the model to accurately identify structural boundaries in ultra-long visual images and avoid contextual confusion during sequence modeling. The dedicated structural tokens include: a newline token, inserted at the horizontal splicing position of each image block to explicitly represent the line ending structure in the image, restoring the line arrangement logic of the text; and a separator token, inserted at the end of the sequence of each image block to explicitly distinguish different image blocks, avoiding semantic confusion between feature sequences of different slices during modeling and ensuring structural independence and semantic continuity across image blocks.

[0059] Furthermore, this embodiment introduces a dual positional encoding mechanism to inject precise positional information into the feature sequence, enabling the model to identify the positional attributes of features in both the scale dimension and the global spatial dimension, achieving multi-dimensional positional awareness. Specifically, this includes level embedding: injecting dedicated level embeddings into features at different downsampling levels, allowing the model to identify the scale level to which the feature belongs (e.g., 64×64, 128×128), thus possessing scale awareness. Global y-embedding: injecting global y-embedding into the feature sequence based on the vertical index of each image patch in the ultra-long visual image, enabling the model to accurately identify the physical location of the current image patch in the ultra-long visual image, thus possessing global spatial awareness.

[0060] Finally, the feature sequences with added structural markers and location information are subjected to root mean square layer normalization (RMSNorm) to standardize the distribution of the feature sequences and obtain highly standardized visual modality codes.

[0061] This embodiment ensures the effectiveness of information extraction by precisely matching image patch sizes. Fine-grained and global modeling is then performed, enabling the model to simultaneously capture the microscopic details and macroscopic semantics of the image, thereby enhancing a comprehensive understanding of the content. Subsequently, local detail features and global semantic features are concatenated to form a unified fused feature vector, further improving information integration capabilities. Furthermore, structural markers and positional information are added, and normalization processing is performed to obtain standardized visual modality encoding, ensuring consistency in feature representation and promoting the model's grasp of text structure and context.

[0062] Figure 4 The flowchart for extracting general semantic features of hidden text states provided in this application embodiment may include the following steps: Step S511: Normalize the text hiding state to obtain the normalized text hiding state.

[0063] Specifically, to eliminate the distribution differences of features across different dimensions in the original text hidden state, and to unify the feature mean and variance, thus avoiding numerical oscillations in subsequent linear mapping and gating calculations due to fluctuations in feature distribution and ensuring computational stability, root mean square normalization (RMSNorm) is used to standardize the text hidden state H, resulting in the normalized text hidden state H. norm .

[0064] Step S513: Extract the semantic gating signal and text feature representation corresponding to the normalized text hidden state.

[0065] Specifically, the normalized text hidden state H normInput gated linear mapping layer (gate) proj The signal undergoes linear feature transformation, followed by nonlinear mapping using the SiLU activation function, ultimately generating a semantically gated signal G1. The calculation formula is: G1 = SiLU(H) norm W gate1 ) (W gate1 (These are the weight parameters specific to the first branch-gated linear mapping layer). Among them, the semantic gating signal G1 and the normalized text hidden state H... norm The dimensions are consistent, and the numerical distribution changes dynamically with the semantics of the text, exhibiting text semantic-guided selectivity—positions with high values ​​correspond to the core semantic regions of the text, while positions with low values ​​correspond to redundant semantic regions of the text, providing a basis for subsequent accurate selection.

[0066] Hidden state H of the normalized text norm Input upsampling linear mapping layer (up) proj A linear feature transformation is performed to directly generate the upsampled text feature representation U1, calculated as follows: U1 = H norm W up1 (W) up1 (These are the weight parameters specific to the upsampling linear mapping layer of this first branch). The function of the upsampling linear mapping layer is to expand the dimension of the text features, transforming the original normalized text hidden state H... norm By mapping to a higher-dimensional feature space, we can uncover the hidden deep semantic information in the text, providing a richer feature base for subsequent refined semantic filtering.

[0067] Step S515: Semantic filtering is performed based on semantic gating signals and text feature representations to generate general semantic features of the text that match the dimensions of the hidden text state.

[0068] Specifically, the semantic gate signal G1 is multiplied element-wise with the text feature representation U1. The semantic selectivity of the semantic gate signal G1 is used to filter the high-dimensional semantic information in the text feature representation U1, retaining the core semantic features with high semantic gate signal G1 values ​​and filtering out redundant semantic information with low semantic gate signal G1 values. This achieves accurate filtering based on the semantics of the text itself, and the filtered features are G1⊙U1.

[0069] The filtered features G1⊙U1 are input into the downsampling linear mapping layer (down). proj A linear feature transformation is performed to map the high-dimensional filtered features back to a dimension completely consistent with the original text hidden state H, ultimately obtaining the general semantic feature O of the text output by the first branch. mlp The calculation formula is: O mlp =(G1⊙U1)W down1 =(SiLU(H norm W gate1)⊙(H norm W up1 ))W down1 (W) down1 (These are the weight parameters specific to the downsampling linear mapping layer of the first branch).

[0070] This embodiment significantly improves the computational stability and feature consistency of large language models when processing long texts by normalizing the text hidden state. This process eliminates the distribution differences between features of different dimensions, making subsequent linear mapping and gating calculations smoother and avoiding information loss due to numerical oscillations, thus laying a solid foundation for global semantic perception. Furthermore, by extracting the semantic gating signal and text feature representation corresponding to the normalized text hidden state, the model can effectively identify important features in the text, focusing on core semantic regions, thereby improving its ability to understand complex structures. Finally, semantic filtering based on the semantic gating signal and text feature representation generates general semantic features that match the dimensions of the original text, not only optimizing feature expression but also enhancing the model's ability to grasp deeper meanings.

[0071] Figure 5 A flowchart for filtering modal visual features adapted to text hiding states from visual modal encoding, provided for embodiments of this application, may include the following steps: Step S531: Normalize the text hiding state to obtain the normalized text hiding state.

[0072] Specifically, the processing procedure for this step is the same as that for step S511, and will not be repeated here.

[0073] Step S533: Using the normalized text hidden state as the query vector and the visual modality encoding as the key vector, determine the attention score between the query vector and the key vector, and aggregate the visual modality encoding based on the attention score to obtain the aggregated modality features.

[0074] Specifically, using the normalized text hidden state H norm Using the visual modality code E as the key vector, the similarity between the two is calculated to obtain the original cross-modal attention logits (similarity score). The calculation formula is as follows: Among them, Q H For normalized text hidden state H norm The text query vector generated after linear feature transformation; K E The visual key vector is generated after transforming the visual modality code E; d is the feature dimension of Q or K; T is the transpose.

[0075] Furthermore, because traditional Softmax forces the sum of all weights to 1 through exponential normalization, even features completely unrelated to text semantics will be assigned a certain weight, leading to the forced aggregation of redundant features and interfering with the dominant semantics of the text. This embodiment introduces a smoothing term based on maximum similarity into the normalization denominator, breaking the constraint that the weight sum is forced to 1. This allows the sum of attention weights to be less than 1 overall, assigning effective weights only to features that highly match the text semantics, and assigning zero weights to features with low or no match. Specifically, the cross-modal attention weights processed by Softmax-one are used to apply the attention weights to the visual value vector V. E Weighted summation is performed to aggregate features highly relevant to the current text semantics, filtering out unmatched features, ultimately yielding the aggregated modality feature P. The calculation formula is as follows: Among them, W q W k W v For weights.

[0076] Step S535: Extract the text modality gating signal corresponding to the normalized text hidden state, and extract the modality feature representation corresponding to the aggregated modality features.

[0077] Specifically, the normalized text hidden state H norm Input gated linear mapping layer (gate) proj The signal undergoes linear feature transformation, followed by nonlinear mapping using the SiLU activation function, ultimately generating the text modality-gated signal G2. The calculation formula is: G2 = SiLU(H) norm W gate2 ) (W gate2 (These are the weight parameters specific to the second branch gated linear mapping layer). Among them, the text modality gating signal G2 and the normalized text hidden state H... norm The dimensions are consistent, and the numerical distribution changes dynamically with the semantics of the text, exhibiting text semantic-guided selectivity—positions with high values ​​correspond to the core semantic regions of the text, while positions with low values ​​correspond to redundant semantic regions of the text, providing a basis for subsequent accurate selection.

[0078] The aggregated modal features P are input into the upsampling linear mapping layer (up proj A linear feature transformation is performed to directly generate a modal feature representation U2 with the same dimension as the text modal gating signal G2. The calculation formula is: U2 = PW up2 (W) up2 (These are the specific weight parameters for the upsampling linear mapping layer in this second branch).

[0079] Step S537: Semantic filtering is performed based on text modal gating signals and modal feature representations to generate modal visual features that match the dimensions of the text hidden state.

[0080] Specifically, the text modality gating signal G2 is multiplied element-wise with the modality feature representation U2 (⊙). The text semantic selectivity of the text modality gating signal G2 is used to precisely filter the visual features in the modality feature representation U2 point-to-point. At positions where the text modality gating signal G2 has a high value, the visual features in the modality feature representation U2 are fully preserved, achieving visual feature enhancement of the core text semantics. At positions where the text modality gating signal G2 has a low value, the visual features in the modality feature representation U2 are significantly suppressed / reduced to zero, achieving visual feature filtering of secondary text semantics.

[0081] The filtered visual features G2⊙U2 are input into the downsampling linear mapping layer (down). proj A linear feature transformation is performed to map the high-dimensional filtered features back to a dimension completely consistent with the original text hidden state H, ultimately obtaining the modal visual features O output by the second branch. patch The calculation formula is: O patch =(G2⊙U2)W down2 =(SiLU(H norm W gate2 )⊙(PW up2 ))W down2 (W) down2 (These are the weight parameters specific to the downsampling linear mapping layer of this second branch).

[0082] This embodiment normalizes the text hidden state to eliminate scale differences between inputs, thereby enhancing feature consistency and enabling the model to better capture global semantic information when processing long texts. Next, an attention mechanism is used to treat the normalized text hidden state as a query vector and the visual modality encoding as a key-value vector, calculating the attention score between them to effectively extract visual information relevant to the text semantics. Furthermore, by extracting text modality gating signals and modal feature representations, the model can dynamically select the most relevant features, adapting to changes in complex long text content. Finally, through semantic filtering based on gating signals, modal visual features matching the dimensions of the text hidden state are generated, ensuring accurate extraction of visual information. In summary, this embodiment effectively improves the ability of large language models to perceive global semantic and visual information when processing long texts.

[0083] Figure 6 The flowchart for step S7 provided in the embodiments of this application may include the following steps: Step S71: Perform element-wise addition of the general semantic features and modal visual features of the text to obtain the dual-branch fusion features.

[0084] Specifically, the general semantic features of the text and the modal visual features are fused by adding elements one by one to generate a dual-branch fusion feature O. mlp +O patch This achieves a synergistic effect between textual semantic enhancement and cross-modal visual semantic supplementation.

[0085] Step S73: Determine the hidden state of the text as the original text residual stream.

[0086] Specifically, in order to preserve the core semantics of the original text to the greatest extent and prevent semantic shift, the hidden state of the text is defined as the residual stream H of the original text. residual This prevents information shifts and loss after subsequent deep processing of large models.

[0087] Step S75: The dual-branch fusion features are superimposed with the original text residual stream to obtain the target fusion features for large language models to parse long natural language texts.

[0088] Specifically, the original text residual stream is fused with the dual-branch fusion features to generate the target fusion feature Output, which can be directly parsed by large language models for long natural language texts. The calculation formula is: Output = H residual +O mlp +O patch .

[0089] This embodiment generates a dual-branch fusion feature by adding general semantic features and modal visual features element-by-element, effectively combining linguistic and visual information to enhance the model's understanding of multimodal data. This not only improves the semantic representation of the text but also introduces supplementary visual information, helping large language models to more comprehensively grasp the global semantics of long texts. To preserve the core semantics of the text to the maximum extent, the hidden state of the text is defined as the residual stream of the original text, preventing semantic shifts and information loss during subsequent processing. Finally, the dual-branch fusion feature is superimposed with the original text residual stream to generate the target fusion feature. This target fusion feature integrates information from both textual and visual modalities, ensuring that the model can more efficiently understand and process long texts, significantly improving its ability to perceive global semantics and visual information.

[0090] It should be noted that the long text processing method provided in this application is optimized based on a cross-modal model through a phased training method, which includes a first phase and a second phase; wherein, The first stage includes: acquiring training natural language long text and training text prompt words; determining the training visual modality encoding corresponding to the training natural language long text, and determining the training text label sequence corresponding to the training text prompt words; concatenating and expanding the training visual modality encoding and the training text label sequence and inputting them into Transformer encoding, and performing coarse-grained spatial alignment between visual and text modalities through cross-modal attention constraints. The second stage includes: using dual-branch SwiGLU gating processing based on the training text label sequence, fusing the general semantic features of the training text with the adapted training visual modality encoding, and completing the cross-modal deep interaction fusion guided by text semantics.

[0091] The specific process is as follows: (1) Multimodal data preprocessing methods In the multi-stage cross-modal model training process, a unified multimodal data preprocessing method is used to construct training samples. First, the text data is segmented and encoded to convert the original text into a text tag sequence, while preserving its semantic information under the condition of satisfying the maximum length constraint. Then, the text tag sequence is divided into an input tag sequence and a supervision tag sequence according to a preset ratio, with the unsupervised part filled with an ignore marker to adapt to the autoregressive training objective.

[0092] For the image data corresponding to the samples, a unified format loading and size parsing are performed, and the number of corresponding image tags is calculated according to the spatial resolution of the images to generate visual modality codes that can be jointly modeled with text tags. Based on this, a joint attention mask for cross-modal modeling is constructed, and multiple samples are batch processed and uniformly filled to form multimodal training batch data with consistent structure that can be directly input into the model.

[0093] (2) Training phase description The first stage, coarse-grained alignment, uses multimodal data preprocessing methods to construct basic alignment training samples. A sequence-expansion-based fusion strategy is employed, utilizing an image encoder to map visual information to the text hidden layer space. (See also...) Figure 7The model architecture diagram for the first-stage coarse-grained alignment provided in this application embodiment is as follows: The input layer linearly concatenates the visual modality encoding (blue block) and the text tag sequence (green block) generated by the text embedding layer in the sequence length dimension, directly expanding the length of the input sequence, and simultaneously supplementing the label mask to adapt to the long sequence processing logic of large language models. The expanded input sequence enters a stacked Transformer Block (×L), each Block containing a classic structure of Root Mean Square Normalization (RMSNorm) + Attention + Residual Connection and Root Mean Square Normalization (RMSNorm) + Feed-Forward Network + Residual Connection. In the detail module of the attention layer, a cross-modal attention mask is introduced, and similarity calculation is performed ( Attention weights are generated using Softmax and then mapped to the output through the output projection layer. This process allows the large language model to initially perceive visual features using the logic of processing long texts, completing the cross-modal alignment between visual and textual modalities, and building a basic feature bridge for the deep interaction in the second stage.

[0094] It should be noted that, in order to achieve dominant modeling of text semantics during the decoding phase and to prevent cross-modal information from interfering with the historical text context, this embodiment introduces a cross-modal attention constraint mechanism in the attention layer of each Transformer layer to uniformly control the visibility relationships between tokens of different modalities. This attention constraint mechanism includes at least the following rules: Visibility constraints within visual modalities: In the visual modal encoding sequence, each visual token is visible to the others during attention computation to support holistic modeling and cross-regional association of ultra-long visual representations.

[0095] Text modal causality constraint: Autoregressive causal constraints are maintained between text tokens, ensuring that the current text position can only focus on its historical text state, thus avoiding future information leakage.

[0096] Full visibility constraint of text on vision: During the text generation process, the text token can access all visual tokens in the visual modality encoding sequence, thereby enabling semantic generation based on the global visual context.

[0097] Visual isolation constraint from text: Visual tokens must not follow back to text tokens, in order to avoid visual features being interfered with by the historical state of the text during the modeling process, thereby maintaining the stability of modal representation.

[0098] Through the above cross-modal attention constraint design, precise control over the direction and scope of cross-modal information flow is achieved, providing structural assurance for subsequent cross-modal feature fusion based on SwiGLU.

[0099] The second stage is deep semantic fusion: This stage further utilizes multimodal data preprocessing methods for cross-modal fusion training. An integration strategy based on internal interaction is adopted, directly injecting visual features as independent encoding tensors into the model, and introducing a dedicated attention-filled mask for feature constraints. In this stage, instead of simply relying on sequence concatenation, deep interaction between visual features and textual representations is achieved through retrieval in the feedforward network layer and modal feature fusion. This allows the model to dynamically select and utilize external modal information under the guidance of textual semantics, thereby enhancing the model's ability to understand complex visual semantics.

[0100] For details, please refer to Figure 8 The model architecture diagram for the second-stage deep semantic fusion provided in this application embodiment is as follows: The input is only text information, which is then processed by the text embedding layer to generate a text tag sequence before entering the stacked Transformer Blocks (×L). In the feedforward network layer (FFN) of the Block, a dual-branch parallel processing structure is introduced, including a first branch (pure text processing) to normalize the text hidden state H. norm As input, a SwiGLU gating structure is used (gated linear mapping layer + SiLU activation to generate semantic gating signals, upsampling linear mapping layer to generate text feature representation, element-wise multiplication (⊙) to achieve semantic filtering, and downsampling linear mapping layer to restore dimensionality) to generate general textual semantic features O that eliminate redundancy and enhance core semantics. mlp The second branch (cross-modal interaction) uses normalized text hidden state H. norm The query vector is visually modally encoded as a key-value vector, and similarity is calculated ( The improved Softmax-one normalized aggregation of text-related visual features is then processed through a SwiGLU gating structure with the same architecture (gated linear mapping layer + SiLU to generate text modal gating signals, upsampling linear mapping layer to adapt to text space, element-wise multiplication to achieve visual feature selection, and downsampling linear mapping layer to restore dimensionality), generating modal visual features O that are accurately adapted to the text semantics. patch Ultimately, the dual-branch output is combined with the original hidden text state H. residual Multi-path fusion is achieved through element-wise addition (⊕) to generate the target fusion feature. This process no longer relies on sequence concatenation, but instead uses the model's internal dual-branch gating and residual mechanism to dynamically guide the selection and injection of visual information through textual semantics, achieving deep semantic fusion and significantly enhancing the model's ability to understand complex visual semantics.

[0101] In a preferred embodiment, for the first and second training stages, in order to address the batch processing failure caused by inconsistent visual modality encoding lengths of different image blocks within the same batch, the number of tokens for all visual modality encodings within the current batch is counted to determine the maximum number of tokens T within the batch. max This serves as the unified length benchmark for this batch processing. Targeted processing is applied to each visual modality code within the batch: if its token count T... i Less than T max Zero padding is performed at the end of the sequence to make the number of tokens equal to T after padding. max If its token quantity T i equal to T max The original sequence remains unchanged. After the above processing, all visual modality codes within a batch are transformed into feature sequences of uniform length, i.e., visual modality codes of uniform length, with tensor dimensions of [B, max_seq_length, hidden_dim], where B is the batch size, i.e., the number of image blocks processed in the current batch; max_seq_length is the maximum number of tokens in the current batch, i.e., the uniform sequence length after length alignment; and hidden_dim is the feature dimension, which perfectly matches the hidden layer dimension of the large language model.

[0102] Accordingly, please refer to Figure 9 A block diagram of a long text processing apparatus provided in this application embodiment, the apparatus comprising: The long text processing unit 101 is used to process the long natural language text to be parsed to determine the ultra-long visual image. The visual modality coding unit 103 is used to perform multi-level coding on ultra-long visual images to generate standardized visual modality codes. The dual feature extraction and filtering unit 105 is used to encode the input text prompt words to obtain the text hidden state, extract the general semantic features of the text hidden state, and filter the modal visual features that are suitable for the text hidden state from the visual modal encoding. The cross-modal fusion unit 107 is used to fuse general semantic features, modal visual features and hidden states of text to obtain target fusion features, which are then used to generate a text sequence based on the target fusion features.

[0103] In some alternative implementations, the long text processing unit 101 includes: Based on the language type of the long natural language text, determine the appropriate typesetting configuration file for the long natural language text; The natural language long text is formatted according to the formatting configuration file to obtain a formatted multi-page document; Convert a multi-page document into multiple bitmap images; Multiple bitmap images are stitched together according to the page order of a multi-page document to obtain an ultra-long visual image.

[0104] In some alternative implementations, the visual modality coding unit 103 includes: Based on the relationship between the image size of ultra-long visual images and the preset image block size, determine the image blocks that match the preset image block size; Fine-grained modeling of image patches is performed to obtain local detail features, and global semantic modeling of image patches is performed to obtain global semantic features; The local detail features and global semantic features are concatenated to form a fused feature vector, and the fused feature vector is transformed into a feature sequence adapted to the dimensions of a large language model. By adding structural markers and positional information to the feature sequences and then normalizing them, visual modality coding is obtained.

[0105] In some alternative implementations, the visual modality coding unit 103 includes: If the image height in the image size is greater than the preset height in the preset image block size, then the ultra-long visual image is divided into multiple image blocks with a height equal to the preset height along the height direction of the ultra-long visual image; the height direction represents the vertical spatial extension direction of the ultra-long visual image. If the image height in the image size is less than the preset height in the preset image block size, then size padding is performed at the bottom of the ultra-long visual image to obtain an image block that matches the preset image block size; the padding height is determined according to the preset height in the preset image block size, and the height of the image block can be divided by the preset height.

[0106] In some optional implementations, the dual feature extraction and filtering unit 105 includes: The text hiding state is normalized to obtain the normalized text hiding state. Extract the semantic gating signals and text feature representations corresponding to the normalized text hidden states; Semantic filtering is performed based on semantic gating signals and text feature representations to generate general semantic features of text that match the dimensions of the hidden text state.

[0107] In some optional implementations, the dual feature extraction and filtering unit 105 includes: The text hiding state is normalized to obtain the normalized text hiding state. Using the normalized text hidden state as the query vector and the visual modality encoding as the key vector, the attention score between the query vector and the key vector is determined, and the visual modality encoding is aggregated based on the attention score to obtain the aggregated modality features. Extract the text modality gating signal corresponding to the normalized text hidden state, and extract the modality feature representation corresponding to the aggregated modality features; Semantic filtering is performed based on text modal gating signals and modal feature representations to generate modal visual features that match the dimensions of the text's hidden state.

[0108] In some alternative implementations, the cross-modal fusion unit 107 includes: By performing element-wise addition of the general semantic features and modal visual features of the text, a dual-branch fusion feature is obtained; The hidden state of the text is determined as the residual stream of the original text; The target fusion feature is obtained by superimposing the dual-branch fusion feature with the original text residual stream.

[0109] In some alternative implementations, the apparatus is used to implement a trained cross-modal model, which is trained in a phased manner, the phased manner including a first phase and a second phase; wherein, The first stage includes: Obtain the long training natural language text and training text prompts; Determine the training visual modality encoding corresponding to the training natural language long text, and determine the training text tag sequence corresponding to the training text prompt words; The trained visual modality encoding and the trained text tag sequence are concatenated and expanded before being input into Transformer encoding. Cross-modal attention constraints are used to achieve coarse-grained spatial alignment between the visual and text modalities. The second phase includes: Based on the training text tag sequence, a dual-branch SwiGLU gating process is used to fuse the general semantic features of the training text with the adapted training visual modality encoding, thereby completing the cross-modal deep interaction fusion guided by text semantics.

[0110] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0111] In this embodiment, a long text processing device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0112] Please see Figure 10 , Figure 10 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 10 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 10 Take a processor 10 as an example.

[0113] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0114] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0115] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0116] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0117] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0118] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0119] The apparatus, module, or unit described in the above embodiments can be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0120] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0121] Those skilled in the art will understand that the embodiments of this application can be provided as methods or apparatus. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and devices according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0126] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0127] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0128] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A long text processing method, characterized by, The method includes: The ultra-long visual image is determined by processing the long natural language text to be parsed. Based on the relationship between the image size of the ultra-long visual image and the preset image block size, an image block that matches the preset image block size is determined; Fine-grained modeling is performed on the image patch to obtain local detail features, and global semantic modeling is performed on the image patch to obtain global semantic features; The local detail features and the global semantic features are concatenated to form a fused feature vector, and the fused feature vector is transformed into a feature sequence adapted to the dimensions of a large language model. The feature sequence is added with structural markers and positional information and then normalized to obtain the visual modality code. The text prompt words are encoded to obtain a text hidden state. General semantic features of the text hidden state are extracted, and visual modal features adapted to the text hidden state are selected from the visual modal encoding. The extraction of general semantic features of the text hidden state includes: normalizing the text hidden state to obtain a normalized text hidden state; extracting the semantic gating signal and text feature representation corresponding to the normalized text hidden state; performing semantic filtering based on the semantic gating signal and the text feature representation to generate general semantic features matching the dimension of the text hidden state; and selecting suitable visual modal features from the visual modal encoding. The modal visual features of this hidden state include: normalizing the text hidden state to obtain a normalized text hidden state; using the normalized text hidden state as a query vector and the visual modality encoding as a key vector, determining the attention score between the query vector and the key vector, and aggregating the visual modality encoding based on the attention score to obtain aggregated modal features; extracting the text modality gating signal corresponding to the normalized text hidden state, and extracting the modal feature representation corresponding to the aggregated modal features; performing semantic filtering based on the text modality gating signal and the modal feature representation to generate modal visual features that match the dimension of the text hidden state; The general semantic features of the text, the modal visual features, and the text hidden state are filtered and fused in each Transformer Block of the trained cross-modal model to obtain target fusion features, which are then used to generate text sequences based on the target fusion features. Obtaining the target fusion features includes: performing element-wise addition on the general semantic features of the text and the modal visual features to obtain a two-branch fusion feature; determining the text hidden state as the original text residual stream; and superimposing the two-branch fusion feature with the original text residual stream to obtain the target fusion feature. In this process, forward computation is performed layer by layer in several Transformer Blocks of the cross-modal model. In each Transformer Block, the text hidden state of the current layer and the visual modality encoding are filtered, feature fused and vector updated in each feedforward network layer to preserve the original semantic logic of the text prompt words. The cross-modal model is trained in a phased manner, which includes a first phase and a second phase; wherein... The first stage includes: Obtain the long training natural language text and training text prompts; Determine the training visual modality encoding corresponding to the training natural language long text, and determine the training text tag sequence corresponding to the training text prompt words; The trained visual modality encoding and the trained text tag sequence are concatenated and expanded before being input into Transformer encoding. Cross-modal attention constraints are used to achieve coarse-grained spatial alignment between the visual and text modalities. The second phase includes: Based on the training text tag sequence, a dual-branch SwiGLU gating process is used to fuse the general semantic features of the training text with the adapted training visual modality encoding, thereby completing the cross-modal deep interaction fusion guided by text semantics.

2. The method of claim 1, wherein, The methods for determining the ultra-long visual image include: Based on the language type of the natural language long text, determine the layout configuration file adapted to the natural language long text; The natural language long text is formatted according to the formatting configuration file to obtain a formatted multi-page document; The multi-page document is converted into multiple bitmap images; The multiple bitmap images are stitched together according to the page order of the multi-page document to obtain the ultra-long visual image.

3. A long text processing apparatus, characterized by comprising: The device includes: The long text processing unit is used to process long natural language text to be parsed and determine ultra-long visual images; A visual modality coding unit is used to determine an image block that matches the preset image block size based on the relationship between the image size of the ultra-long visual image and the preset image block size; to perform fine-grained modeling on the image block to obtain local detail features, and to perform global semantic modeling on the image block to obtain global semantic features; to concatenate the local detail features and the global semantic features to form a fused feature vector, and to transform the fused feature vector into a feature sequence adapted to the large language model dimension; to add structural markers and positional information to the feature sequence and to perform normalization processing to obtain the visual modality code; A dual-feature extraction and filtering unit is used to encode the input text prompt words to obtain the text hidden state, extract the general semantic features of the text hidden state, and filter the modal visual features adapted to the text hidden state from the visual modal encoding. The extraction of the general semantic features of the text hidden state includes: normalizing the text hidden state to obtain a normalized text hidden state; extracting the semantic gating signal and text feature representation corresponding to the normalized text hidden state; performing semantic filtering based on the semantic gating signal and the text feature representation to generate general semantic features matching the dimension of the text hidden state; and filtering from the visual modal encoding. Selecting modal visual features adapted to the text hiding state includes: normalizing the text hiding state to obtain a normalized text hiding state; using the normalized text hiding state as a query vector and the visual modality encoding as a key vector, determining the attention score between the query vector and the key vector, and aggregating the visual modality encoding based on the attention score to obtain aggregated modal features; extracting the text modality gating signal corresponding to the normalized text hiding state, and extracting the modal feature representation corresponding to the aggregated modal features; performing semantic filtering based on the text modality gating signal and the modal feature representation to generate modal visual features that match the dimension of the text hiding state. A cross-modal fusion unit is used to filter and fuse the general semantic features of the text, the modal visual features, and the text hidden state in each Transformer Block of the trained cross-modal model to obtain target fusion features, which are then used to generate text sequences based on the target fusion features. Obtaining the target fusion features includes: performing element-wise addition on the general semantic features and the modal visual features to obtain a two-branch fusion feature; determining the text hidden state as the original text residual stream; and superimposing the two-branch fusion feature with the original text residual stream to obtain the target fusion feature. In this process, forward computation is performed layer by layer in several Transformer Blocks of the cross-modal model. In each Transformer Block, the text hidden state of the current layer and the visual modality encoding are filtered, feature fused and vector updated in each feedforward network layer to preserve the original semantic logic of the text prompt words. The cross-modal model is trained in a phased manner, which includes a first phase and a second phase; wherein... The first stage includes: Obtain the long training natural language text and training text prompts; Determine the training visual modality encoding corresponding to the training natural language long text, and determine the training text tag sequence corresponding to the training text prompt words; The trained visual modality encoding and the trained text tag sequence are concatenated and expanded before being input into Transformer encoding. Cross-modal attention constraints are used to achieve coarse-grained spatial alignment between the visual and text modalities. The second phase includes: Based on the training text tag sequence, a dual-branch SwiGLU gating process is adopted to fuse the general semantic features of the training text with the adapted training visual modality encoding, thereby completing the cross-modal deep interaction fusion guided by text semantics. The batch processing of visual modality encoding for different image patches in the same batch during the first and second training phases includes: counting the number of tokens in all training visual modality encodings in the current batch to determine the maximum number of tokens in the batch; if the number of tokens in the current batch training visual modality encodings is less than the maximum number of tokens, zero padding is performed at the end of the sequence to make the number of tokens in the training visual modality encodings equal to the maximum number of tokens; if it is equal to the maximum number of tokens, it remains unchanged to obtain a standardized length training visual modality encoding.

4. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the long text processing method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the long text processing method according to any one of claims 1 to 2.

Images