A visual compression and retrieval method, device and equipment of a document and a storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for visual compression and retrieval of documents, relating to the field of computer technology. The method includes: acquiring an image of a document page to be processed and segmenting it into multiple image blocks; determining the structural category and structural importance score of each image block; aggregating several structural regions based on structural category and spatial location; assigning a preset number of compression tokens to each region based on structural category weights and importance scores; generating structural anchor tokens corresponding to each region from the compression tokens, forming a set; receiving a query request and converting it into a query vector; retrieving the target structural region from the anchor token set; and performing local decoding and reconstruction based on the compression tokens of the target region, outputting a region image or structural mask. This method significantly improves the efficiency of long document processing through structure-guided adaptive compression and fine-grained retrieval, balancing compression effectiveness and retrieval accuracy.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for visual compression and retrieval of documents. Background Technology

[0002] With the deepening of digital transformation, in key industries such as engineering, law, scientific research, and manufacturing, a large amount of core knowledge and standards exist in the form of multi-page, complex documents, such as technical reports, engineering specifications, research papers, and compliance documents. These documents typically contain multiple modalities of information, including text, tables, formulas, charts, and flowcharts, and have strict cross-page referencing relationships and page layout logic. Efficiently and accurately extracting and understanding specific structured information from massive amounts of long documents has become a key requirement for improving knowledge management, intelligent retrieval, and automated analysis capabilities.

[0003] Currently, the mainstream solutions for processing such multimodal long documents mainly rely on large language models based on plain text and visual language models. Large language models typically process long texts by expanding the context window or using retrieval-enhanced generation techniques, while visual language models, although performing well in image description and question-answering tasks, are usually designed for single-page or small-scale image inputs, lacking the ability to model the global structure of multi-page documents, and are insufficient in accurately locating and understanding fine-grained structures (such as table cells and formula ranges).

[0004] However, existing technical solutions suffer from low efficiency when processing long documents. Summary of the Invention

[0005] This application provides a visual compression and retrieval method for documents, which can improve efficiency when processing long documents.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, this application provides a method for visual compression and retrieval of documents, including:

[0008] Obtain the page image of the document to be processed, and divide the page image into multiple image blocks;

[0009] Based on multiple image patches, the structure category and structure importance score of each image patch are obtained;

[0010] Based on the structural category and spatial location of each image block, several structural regions are obtained;

[0011] Based on the weights of structural categories and structural importance scores, a target number of compressed tokens is assigned to each structural region; based on the target number, the image data of each structural region is encoded using a visual compression encoder to obtain the corresponding number of compressed tokens.

[0012] Based on the compressed token, the structural anchor tokens corresponding to each structural region are obtained, forming a set of structural anchor tokens;

[0013] Receive a query request and obtain a query vector based on the query request;

[0014] Based on the query vector, a search is performed in the set of structural anchor tokens to obtain the target structural region;

[0015] Based on the compressed token corresponding to the target structural region, local decoding and reconstruction are performed to obtain the reconstructed regional image or structural mask.

[0016] Optionally, the target number of compressed lexical tokens allocated to each structural region based on the structural category weights and structural importance scores includes:

[0017] Calculate the unnormalized importance of each structural region based on the weight of the structural category and the structural importance score;

[0018] The unnormalized importance is normalized to obtain the token allocation ratio for each structural region;

[0019] Based on the token allocation ratio and the preset total page token budget, a target number of compressed tokens are allocated to each structural region.

[0020] Optionally, obtaining the structural anchor token corresponding to each structural region based on the compressed token includes:

[0021] Pool all compressed tokens within each structural region to obtain the semantic representation vector of the structural region;

[0022] The semantic representation vector is fused with the index encoding vector to obtain the structural anchor token.

[0023] Optionally, obtaining the query vector based on the query request includes:

[0024] If the query request contains text content, the text content is encoded using a text encoder to obtain a text query vector;

[0025] If the query request contains a structure type condition, the structure type is mapped to a structure query vector through the structure type embedding table;

[0026] The text query vector and the structure query vector are fused using feature fusion to obtain the query vector.

[0027] Optionally, the step of retrieving the target structural region from the set of structural anchor tokens based on the query vector includes:

[0028] Calculate the similarity between the query vector and each structural anchor token in the set of structural anchor tokens;

[0029] Select the structural regions corresponding to the K structural anchor tokens with high similarity as the target structural regions.

[0030] Optionally, the step of performing local decoding and reconstruction based on the compressed token corresponding to the target structural region to obtain the reconstructed region image or structural mask includes:

[0031] Map the compressed tokens of the target structural region to the initial feature map;

[0032] The initial feature map is upsampled to obtain the first feature map;

[0033] The first feature map is input into the convolutional layer, and the reconstructed region image or structure mask is output.

[0034] Optionally, based on the structural category and spatial location of each image block, several structural regions are obtained, including:

[0035] Based on the structural category and spatial location of each image block, image blocks with the same or similar structural categories and adjacent spatial locations are merged to obtain several structural regions.

[0036] Secondly, this application provides a visual compression and retrieval device for documents, comprising:

[0037] The acquisition module is used to acquire page images of the document to be processed and to segment the page images into multiple image blocks;

[0038] The data processing module is used to obtain the structural category and structural importance score of each image patch from multiple image patches; to obtain several structural regions based on the structural category and spatial location of each image patch; to allocate a target number of compressed tokens to each structural region based on the weight of the structural category and the structural importance score; to encode the image data of each structural region using a visual compression encoder based on the target number to obtain a corresponding number of compressed tokens; and to obtain the structural anchor tokens corresponding to each structural region based on the compressed tokens, thus forming a set of structural anchor tokens.

[0039] The retrieval module is used to receive query requests, obtain query vectors based on the query requests, search the set of structural anchor tokens based on the query vectors to obtain the target structural region, and perform local decoding and reconstruction based on the compressed tokens corresponding to the target structural region to obtain the reconstructed region image or structural mask.

[0040] Thirdly, this application provides a computing device, including a memory and a processor;

[0041] The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of the first aspects.

[0042] Fourthly, this application provides a computer-readable storage medium for storing a computer program for performing the method as described in any one of the first aspects.

[0043] As can be seen from the above technical solution, this application has at least the following beneficial effects:

[0044] In this application, firstly, by recognizing the structural category and importance score of image blocks and combining spatial location information to form structured regions, the distribution and boundaries of modalities such as tables, formulas, and charts in documents are accurately captured, solving the problem of insufficient understanding of the global structure and fine-grained semantic regions of long documents by existing models, laying the foundation for efficient processing in the future.

[0045] Secondly, the adaptive token allocation mechanism based on the weight of structural categories and the importance of regions tilts resources towards high-value structural regions under the preset total token budget. This not only significantly reduces the redundancy of long document representation, but also avoids the degradation of key information during compression, thus balancing compression efficiency and information fidelity.

[0046] Furthermore, by constructing a global retrieval index through structural anchor tokens, combined with accurate matching of query vectors and local reversible decoding, there is no need to perform global decoding on the entire page of the document. Instead, the index is reconstructed only for the target area as needed, which reduces the computational cost of retrieval and parsing and effectively improves the extraction efficiency of specific structured information in massive long documents.

[0047] Finally, it supports combined queries of text and structural types, accurately responding to retrieval needs for specific structural types and semantic content, greatly improving the practicality of long document knowledge management, intelligent retrieval, and automated analysis, and providing efficient support for the digital transformation of industries such as engineering, law, and scientific research.

[0048] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0049] Figure 1 A flowchart illustrating a document visual compression and retrieval method provided in an embodiment of this application;

[0050] Figure 2 A schematic diagram of a document visual compression and retrieval device provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0052] The terms "first," "second," and "third," etc., used in this application specification and accompanying drawings are used to distinguish different objects, not to limit a specific order.

[0053] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0054] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0055] Multimodal long documents refer to documents spanning tens to hundreds of pages, such as engineering specifications, scientific research papers, and compliance documents. They integrate various information formats, including text, tables, formulas, charts, and flowcharts, and have strict cross-page referencing relationships and page layout logic.

[0056] In the process of digital transformation in industries such as engineering, law, and scientific research, the efficient processing of multimodal long documents faces major technical challenges: existing solutions struggle to accurately understand the global structure and fine-grained semantic regions of long documents, cannot effectively capture the distribution boundaries and cross-page relationships of key modalities such as tables, formulas, and charts, and it is difficult to balance processing efficiency and information fidelity. Global decoding leads to excessively high computational costs, while uniform compression causes the degradation of key information and cannot efficiently respond to combined query needs for specific structural types and semantic content, making it difficult to quickly locate target structured information from massive amounts of long documents.

[0057] The problems arise because existing visual language models are mostly designed for single pages or a small number of images, lacking the ability to model the global layout logic of multi-page documents, and failing to design dedicated recognition mechanisms adapted to document structural characteristics; traditional compression schemes adopt a uniform resource allocation strategy, failing to consider the differences in information density in different structural regions, resulting in the loss of information in high-value regions or redundancy in low-value regions; existing retrieval schemes rely on pure text indexes or dense visual feature retrieval, which cannot cover non-textual modalities and have low retrieval efficiency, while lacking a closed-loop design of structure recognition, resource allocation, accurate retrieval, and on-demand reconstruction, resulting in poor overall processing performance due to the disconnect between each link.

[0058] In view of this, embodiments of this application provide a method for visual compression and retrieval of documents, which can be executed by a processing device. The processing device can be a terminal or a server. Terminals include, but are not limited to, smartphones, tablets, laptops, personal digital assistants, or smart wearable devices. The server can be a cloud server, such as a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Alternatively, the server can be a server in a local data center. A local data center refers to a data center directly controlled by the user.

[0059] To address the main problems in multimodal long document processing, such as insufficient global structural modeling, lack of fine-grained region understanding, difficulty in balancing processing efficiency and information fidelity, and low retrieval accuracy, this application firstly identifies the structural categories and importance scores of image blocks and aggregates them based on spatial location to form structured regions, thereby accurately capturing key modalities of the document. Secondly, it allocates compressed token resources based on structural value differences, reducing redundancy while ensuring that key information is not degraded. Next, it constructs a global retrieval index through structural anchor tokens to achieve composite query matching of structural types and semantic content. Finally, it performs local decoding and reconstruction only on the target region, significantly reducing computational costs, ultimately achieving the goal of efficient and accurate structured information extraction and understanding of multimodal long documents.

[0060] To make the technical solution of this application clearer and easier to understand, the following describes a document visual compression and retrieval method provided by an embodiment of this application, in conjunction with the accompanying drawings. Figure 1 As shown, this figure is a flowchart of a document visual compression and retrieval method provided in an embodiment of this application. The method includes:

[0061] S201. The processing device acquires the page image of the document to be processed and divides the page image into multiple image blocks.

[0062] The document to be processed refers to a long, multi-page document containing multimodal information such as text, tables, formulas, and charts, such as engineering specifications, scientific research papers, and compliance documents. It is characterized by strong structure and cross-page references, and requires the extraction of structured information through specific technologies.

[0063] Page images are visual image data obtained by rasterizing each page of the document to be processed (such as PDF pages or scanned pages). They are the basic data format for subsequent image segmentation and structure recognition, and have a fixed resolution, such as 1024×1024 pixels.

[0064] An image block is the smallest processing unit obtained by sliding and dividing a page image into segments (with or without overlap) according to a preset fixed window size. Its size is uniform, and it is the basic carrier for structural category recognition and importance scoring.

[0065] First, the processing device reads the content of each page of the document to be processed and converts it into a page image with a fixed resolution, completing the document-to-image format conversion. Then, according to a preset fixed window size, such as 32×32 pixels, the page image is evenly divided into multiple image blocks of the same size, ensuring that each image block can participate as an independent unit in the subsequent structural category judgment and importance assessment, avoiding computational redundancy or fragmentation of structural information caused by excessively high page image dimensions.

[0066] The expression for the page image is:

[0067]

[0068] in, This indicates the first document in the document to be processed. Page image, Indicates the image height. Indicates the image width. This represents the RGB three-channel (color image). The dimension of image data is A real matrix.

[0069] The expression for image block segmentation is:

[0070]

[0071] in, This indicates a preset fixed window side length for the image block. Indicates the first The total number of image blocks obtained after segmenting the page image. Indicates the first Page 1 of 1 Image patches, with dimensions of The number of channels should be consistent with the number of channels in the page image.

[0072] S202. The processing device obtains the structural category and structural importance score of each image block based on multiple image blocks.

[0073] Structure category is a classification definition of the information form of image blocks in a document. It is an attribute that distinguishes the information type to which an image block belongs, including text, tables, formulas, charts, titles, backgrounds, etc., reflecting the information carrier type of the image block.

[0074] The structural importance score is a quantitative evaluation value assigned to each image patch based on the density of information corresponding to the image patch and its value to downstream tasks (such as retrieval and parsing). The value range is [0,1]. The higher the score, the richer the key information contained in the image patch, and the more it needs to be retained.

[0075] After receiving multiple image blocks obtained from the previous steps, the processing device first extracts features from each image block through a preset structure prior estimation module (including a visual encoder, a context modeling network, and a dual-branch prediction head), then integrates the context information of the entire page, and finally outputs the probability distribution of the image block belonging to various structures through the classification branch (determining the structure category), and outputs the structure importance score of the image block through the regression branch, thus achieving an accurate judgment on what structure the image block is and how important it is, breaking the limitation of treating all image blocks equally in traditional processing.

[0076] Image patch feature extraction and context coding results:

[0077]

[0078]

[0079] in, Indicates the first After processing by the Transformer layer, the first The intermediate feature vector of an image patch, when hour, , This represents the initial features obtained from the image patch by the visual encoder. Indicates the first The lightweight Transformer layer is used to perform contextual modeling of image patch features, capturing the positional and semantic relationships between all image patches across the entire page. Indicates the first After processing by the Transformer layer, the first The intermediate feature vector of an image patch, Indicates the first Two-dimensional position encoding of image patches This indicates the total number of layers in the Transformer, controlling the depth of context modeling. Indicates the first Global context features of an image patch express Feature dimensions, Indicates the first After processing by the Transformer layer, the first Feature vectors of image patches.

[0080] The expression for calculating the structure category is:

[0081]

[0082] in, Indicates the first Page number The probability distribution of structural categories of each image patch This represents the activation function. Represents the classification weight matrix. Indicates the category bias term. This indicates the total number of structure categories.

[0083]

[0084] in, Indicates the first Page number The structural categories of image patches The function that retrieves the index corresponding to the maximum value. Indicates the first Page number The image patch belongs to the first The probability of class structure, This indicates the total number of structure categories.

[0085] The formula for calculating the structural importance score is:

[0086]

[0087] in, Indicates the first Page 1 of 1 Structural importance score of each image patch The importance prediction can be learned vector. This indicates the importance prediction bias term. This represents the Sigmoid activation function, used to constrain the score within the range [0,1].

[0088] The training loss of the structure prior estimation module (optimization objective):

[0089]

[0090]

[0091]

[0092] in, This represents the total training loss of the structural prior estimation module. The weight hyperparameters representing the structural classification loss are... Represents the structural classification loss. The weight hyperparameters representing the importance prediction loss, Indicate importance to predict losses. Indicates the first The total number of image blocks obtained after segmenting the page image. Indicates the first Page 1 The model predicts that a given image patch belongs to a true structure category. The probability, Indicates the first Page 1 The true structural importance score of each image patch.

[0093] S203. The processing device obtains several structural regions based on the structural category and spatial location of each image block.

[0094] Specifically, the processing device merges image blocks with the same or similar structural categories and adjacent spatial locations based on the structural category and spatial location of each image block to obtain several structural regions.

[0095] Spatial location refers to the specific coordinate information of an image block within the page image, such as the top-left pixel coordinates and center coordinates. It reflects the relative position of the image block within the page and is the basis for determining whether image blocks are adjacent.

[0096] A structural region is a logical unit formed by aggregating multiple image blocks that meet the conditions of consistent structural attributes and adjacent spatial positions. It is a basic component of document structure, such as a complete table region, a continuous formula region, or a whole paragraph of headings. It can correspond to information blocks with independent semantics or functions in the document.

[0097] The processing device first obtains the structural category (such as tables, formulas, etc.) and spatial location information of each image block, and then filters and merges them through region aggregation rules. On the one hand, it filters image blocks with the same structural category or similar functions (ensuring that the information type in the region is consistent), and on the other hand, it filters image blocks with adjacent spatial locations (ensuring that the regions are continuously distributed on the page). Finally, the image block groups that meet the dual conditions are integrated into several independent structural regions, so that the originally scattered image blocks are transformed into structured carriers corresponding to the actual information units of the document, which facilitates the subsequent allocation of resources and construction of retrieval indexes for complete structural units.

[0098] The expression for the structural region is:

[0099]

[0100] in, Indicates the first Page 1 of 1 Each structural region Indicates the first The total number of image blocks obtained after segmenting the page image. Indicates the first The total number of structural regions obtained after page aggregation, as shown by the formula, indicates that each structural region is the first... A subset of the set of page image patches (i.e., composed of partial image patches).

[0101] S204. The processing device assigns a target number of compressed tokens to each structural region based on the weight of the structural category and the structural importance score; based on the target number, it uses a visual compression encoder to encode the image data of each structural region to obtain the corresponding number of compressed tokens.

[0102] The weight of the structure category is a quantitative weight value preset based on the information density and task value of different structure categories (such as tables, formulas, and body text). It is used to strengthen the priority of resource allocation for high-value structure areas. For example, tables and formulas have a higher weight than body text and background.

[0103] The target number of compressed tokens is the number of tokens allocated to each structural region to carry the compressed information of that region, and is determined by the importance of the region.

[0104] Visual compression encoders are models used to compress image data (pixels / features) of structural regions into low-dimensional discrete tokens, such as convolutional networks with attention mechanisms and learnable clustering modules.

[0105] A compressed token is a low-dimensional discrete representation unit obtained by encoding structural region image data. It is a condensed carrier of regional information and can significantly reduce storage and computational overhead.

[0106] Specifically, the processing device calculates the unnormalized importance of each structural region based on the weight of the structural category and the structural importance score; normalizes the unnormalized importance to obtain the token allocation ratio for each structural region; and allocates a target number of compressed tokens to each structural region based on the token allocation ratio and the preset total page token budget.

[0107] The specific process is as follows: First, the processing device combines the preset structural category weights (e.g., tables and formulas have higher weights than the main text) with the structural importance scores of all image blocks within each structural region, and calculates the unnormalized importance of each structural region through weighted averaging, quantifying the relative value of the region; Second, the unnormalized importance of all structural regions is normalized to eliminate differences in numerical ranges, obtaining the allocation ratio of each region in the global token resource (the sum is 1); Third, with the preset total page token budget as the upper limit, the initial token quantity of each region is calculated according to the allocation ratio, and then converted into integers through discretization, ensuring that each region receives at least the minimum number of tokens, while the total allocation does not exceed the budget constraint, ultimately obtaining the compressed token target quantity that satisfies both the budget constraint and adapts to the region importance.

[0108] Calculation of unnormalized importance of structural regions (quantifying region value):

[0109]

[0110] in, Indicates the first Page 1 The unnormalized importance of each structural region Indicates the first Page 1 The number of image blocks contained in each structural region Indicates the first Each image patch corresponds to a structural category The preset weights.

[0111] Structural region importance normalization (determining allocation ratio):

[0112]

[0113] in, Indicates the first Page 1 The allocation ratio of each structural region in the global token resource. Indicates the first The total number of structural regions obtained after page aggregation Indicates the first Page 1 Normalization importance of each structural region This represents the minimum constant.

[0114] Initial allocation of floating-point tokens:

[0115]

[0116] in, Indicates the first Page 1 The initial number of floating-point tokens for each structural region Indicates the preset first Total token budget per page.

[0117] The final integer token allocation (satisfying the budget and minimum token constraints), i.e., the target number of compressed tokens:

[0118]

[0119]

[0120] in, Indicates the first Page 1 The target number of compressed tokens for each structural region. This represents the minimum number of tokens required for each structural region. This represents the initial number of floating-point tokens. Round to the nearest integer. Constraints require that the... The total number of tokens across all structural regions of the page will not exceed the preset total token budget; if it does, tokens will be reduced from low to high according to the region's normalized importance (ensuring it is not less than...). ).

[0121] Subsequently, the processing device performs variable-rate compression encoding on the structural region based on the target number of compressed tokens to generate compressed tokens. Traditional image compression typically uses a fixed-ratio downsampling, which is difficult to adapt to the highly uneven information density characteristics in documents (such as blank backgrounds and dense tables). This application proposes an attention-based dynamic resampling mechanism. Specifically, for the first... Page 1 A structural region, assuming it contains Each image patch feature is denoted as... .

[0122] The number of compressed tokens that should be allocated to this area has been calculated in the previous steps. The processing device first initializes. There are learnable latent query vectors, denoted as... These query vectors can be generated using sinusoidal positional encoding or learnable parameter matrices, with the aim of retrieving the most critical information from the raw features.

[0123] Subsequently, using the cross-attention mechanism, For querying, based on region image patch features Perform feature aggregation for keys and values:

[0124]

[0125] in, This represents the attention mechanism function. The query vector representing the attention mechanism. The key vector representing the attention mechanism. This represents the value vector of the attention mechanism. represents the scaling factor, d is the feature dimension, and Softmax(·) represents the activation function.

[0126] The attention output is fed into the feedforward network to obtain the final compressed token:

[0127]

[0128] in, Indicates the first Page 1 The final compressed token generated from each structural region, FeedForward(·) represents the feedforward network. This represents a learnable hidden query vector. Image block features representing structural regions.

[0129] Through the above steps, this mechanism... Controlling compression granularity: More tokens are allocated to important areas (such as tables) to retain details, while fewer tokens are allocated to background areas to retain only macro information, achieving adaptive and efficient compression.

[0130] S205. The processing device obtains the structural anchor token corresponding to each structural region based on the compressed token, forming a set of structural anchor tokens.

[0131] The structural anchor token is a feature carrier that integrates the semantic information and index information of the structural region. It is the index unit for subsequent global retrieval, containing both the semantic content of the region (such as whether it is a table region) and the identification information such as the region's location and type.

[0132] The structure anchor token set is a global index library formed by aggregating the structure anchor tokens corresponding to all structure regions, used to support subsequent fast retrieval and matching.

[0133] Specifically, the processing device performs a pooling operation on all compressed tokens within each structural region to obtain the semantic representation vector of the structural region; the semantic representation vector is then fused with the index encoding vector to obtain the structural anchor token.

[0134] The specific process is as follows: First, the processing device aggregates the scattered token features into a unified vector for multiple compressed tokens within each structural region through pooling operations (such as mean pooling). This vector represents the semantic information of the region. For example, this is a table region containing experimental data, i.e., the semantic representation vector of the structural region. Second, the semantic representation vector is fused with the index encoding vector (containing the region's page number, spatial coordinates, structural category, and other identifying information) to generate a structural anchor token that simultaneously carries the region's content and location / type, ensuring that subsequent retrieval can both match semantics and locate the region.

[0135] Calculation of semantic representation vectors for structural regions (pooling operation):

[0136]

[0137] in, Indicates the first Page 1 Semantic representation vectors of structural regions Indicates the first Page 1 The final integer number of tokens for each structural region. Indicates the first Page 1 The first structural region A compressed token, The learnable weight matrix representing the pooling operation. This represents the learnable bias term for pooling operations.

[0138] Region coordinate encoding vector:

[0139]

[0140] in, Indicates the first Page 1 Each structural region is encoded by a coordinate vector, recording the spatial location information of that region. The learnable weight matrix represents the coordinate encoding used to map normalized coordinates to vectors. Represents a normalized spatial information vector of the region. Indicates the first Page 1 Normalized center x-coordinate of each structural region Indicates the first Page 1 Normalized center ordinate of each structural region Indicates the first Page 1 Normalized width of each structural region Indicates the first Page 1 Normalized height of each structural region This represents the learnable bias term for coordinate encoding.

[0141] Comprehensive index encoding vector:

[0142]

[0143] in, Indicates the first Page 1 A comprehensive index encoding vector for each structural region. Indicates the first Page number embedding vector, Indicates the first Each structural region corresponds to a structural category. Embedded vector, Indicates the first Page 1 The coordinate encoding vector of each structural region Indicates the dimension of the index-encoded vector.

[0144] Structural anchor token (semantic and index fusion):

[0145]

[0146] in, Indicates the first Page 1 The structural anchor token of each structural region This represents the learnable linear mapping matrix of the index encoding.

[0147] Finally, the structural anchor tokens of all structural regions are aggregated to form a global structural anchor token set. This allows subsequent searches to quickly locate the target region by simply matching the anchors in this set, without having to traverse all compressed tokens.

[0148] S206. The processing device receives the query request and obtains the query vector based on the query request.

[0149] A query request is a search need initiated by a user, containing semantic content (such as an experimental data table) and / or structural type conditions (such as searching only a table range), and is an instruction to locate target information from a long document.

[0150] A query vector is a high-dimensional vector obtained by encoding a query request. It is used for similarity matching with structural anchor tokens and serves as a carrier for achieving accurate semantic and structural retrieval.

[0151] After receiving a query request submitted by a user, if the query request contains text content, the text content is encoded by a text encoder to obtain a text query vector; if the query request contains structure type conditions, the structure type is mapped to a structure query vector through a structure type embedding table; the text query vector and the structure query vector are linearly combined to obtain a query vector.

[0152] Specifically, when a query request contains text content, the hidden state of each word in the text is first obtained through a text encoder (such as a Transformer encoder), and then these states are averaged and pooled to obtain a text query vector that can represent the overall semantics of the text.

[0153] When a query request contains structure type conditions (such as a table range), the original embedding vector of the type is first obtained through the structure type embedding table, and then the structure query vector is obtained through linear transformation to reflect the structure constraints.

[0154] In one embodiment of this application, the text query vector and the structure query vector are finally fused by linear weighting to obtain a final query vector that simultaneously carries semantic requirements and structural type constraints, providing a unified vector carrier for subsequent retrieval and matching.

[0155] Text query vector encoding:

[0156]

[0157] in, This represents the global text query vector obtained through mean pooling. This represents the first output of a text encoder (such as a Transformer encoder). The token-level hidden state of each word; This indicates the number of words (tokens) contained in the text query sequence. Indicates the dimension of the text query vector.

[0158] Structure type query vector encoding:

[0159]

[0160] in, This represents the structure query vector obtained after mapping. The learnable linear transformation matrix represents the embedding of the structure type. Represents the original embedding vector (from a separate structure type embedding table) corresponding to the structure type (such as a table or formula).

[0161] Final query vector fusion:

[0162]

[0163] in, This represents the final query vector after fusion; This indicates the fusion weight.

[0164] In another preferred embodiment of this application, to achieve more adaptive feature selection, feature fusion employs a gated fusion mechanism. Specifically, the processing device automatically calculates the contribution (fusion coefficient) of text and structural features through a learned gated network:

[0165]

[0166]

[0167] in, Indicates the gating coefficient. This represents a vector concatenation operation. This represents the learnable weight matrix of the gated network. This represents the learnable bias term of the gating network. It is the Sigmoid activation function. This is element-wise multiplication.

[0168] This mechanism can dynamically suppress redundant features and strengthen key features based on the specific distribution of the query content.

[0169] In another embodiment of this application, feature fusion can also be achieved through a cross-attention layer. By using text query vectors as queries and structure query vectors as keys and values, information from the two modalities can deeply interact and align in the latent space. This diverse fusion implementation ensures that this application can flexibly handle complex situations where user queries have different emphases on semantic content and structural types, thus improving the robustness of cross-modal retrieval.

[0170] S207. The processing device retrieves the target structural region from the set of structural anchor tokens based on the query vector.

[0171] The target structural region is the structural region in the set of structural anchor tokens that has the highest similarity to the query vector (or meets the preset similarity threshold). It is the information region that matches the user's query requirements.

[0172] Specifically, the processing device first calculates the similarity between the query vector and each structural anchor token in the set of structural anchor tokens; then selects the structural regions corresponding to the K structural anchor tokens with the highest similarity as the target structural regions.

[0173] First, the matching degree between the query vector and each structural anchor token is quantified by similarity calculation. Then, the most relevant regions are selected based on the matching degree. The specific process is as follows: The processing device first calculates the similarity between the query vector and each structural anchor token in the set of structural anchor tokens (the higher the value, the stronger the matching degree). Then, it selects the top K similarity results (K is a preset number, such as the top 5) and determines the structural regions corresponding to these highly similar anchor tokens as the target structural regions, thereby quickly locating the information region that best matches the query requirements.

[0174] Similarity calculation between query vector and structural anchor token:

[0175]

[0176] in, Indicates the first The similarity value between the structural anchor token of each structural region and the query vector. This represents the final query vector after fusion. Indicates the first Page 1 The structural anchor token of each structural region This represents the L2 norm.

[0177] Selection of anchor tokens with high similarity structures:

[0178]

[0179] in, Indicates all After sorting in descending order, select the top K corresponding anchor token indices. , This indicates the index corresponding to the top-ranked structural anchor token. This represents the index corresponding to the Kth ranked structural anchor token.

[0180] Determining the set of target structural regions:

[0181]

[0182] in, This represents the set of target structural regions corresponding to the selected K highly similar structural anchor tokens. This indicates that it is an index. The corresponding structural region, Indicates index The corresponding structural region.

[0183] S208. The processing device performs local decoding and reconstruction based on the compressed token corresponding to the target structural region to obtain the reconstructed regional image or structural mask.

[0184] The compressed token corresponding to the target structural region refers to the low-dimensional discrete representation unit obtained by compressing the target structural region in the preprocessing stage, and is a condensed carrier of information in that region.

[0185] Local decoding and reconstruction is the process of restoring a compressed token from a low-dimensional compressed state to its original information form using a decoding model (such as the Transformer decoder) for a target structural region.

[0186] The reconstructed region image is the original visual content (such as an image of a table or formula) corresponding to the target structural region obtained by decoding the compressed token.

[0187] A structure mask is a contour / range identifier of the target structure region obtained after decoding (used to mark the position and shape of the region in the document).

[0188] Specifically, the processing device maps the compressed token of the target structural region to an initial feature map; upsamples the initial feature map to obtain a first feature map; inputs the first feature map into a convolutional layer and outputs the reconstructed region image or structural mask.

[0189] Upsampling is an operation that increases the resolution of a feature map / image. Simply put, it enlarges a low-sized feature map to a higher size, making it closer to the size of the original region.

[0190] First, the compressed tokens of the target structural region are converted into a low-resolution initial feature map (converting discrete tokens into feature matrix form); then, the initial feature map is enlarged (resolution is increased) to obtain a first feature map that is closer to the original size; finally, the first feature map is input into a convolutional layer to generate a region image that can be directly viewed (such as the visual content of a table / formula), or a structural mask that marks the range of the region.

[0191] Compressed tokens are mapped to the initial feature map:

[0192]

[0193] in, Represents the initial feature map; This indicates a reshaping operation, which converts the result of a linear transformation into a feature map form. This represents a compressed token stacking matrix representing the target structural region. The weight matrix represents the linear transformation. The bias term represents the linear transformation; Indicates the height of the initial feature map. This represents the width of the initial feature map. This indicates the number of channels in the feature map.

[0194] Multi-layer upsampling magnification resolution:

[0195]

[0196] in, Indicates the first Feature map after layer upsampling; Indicates the first Layer upsampling module (including transposed convolution, local attention, non-linear layers, etc.) Indicates the first Feature map of the layer This indicates the total number of upsampling layers, which progressively increase the spatial resolution of the feature map through multiple layers.

[0197] Mapped to region image / structure mask:

[0198]

[0199] in, This represents the reconstructed target region image or structure mask. Indicates the output convolutional layer. Indicates the process The final feature map after layer upsampling.

[0200] To balance compression quality and retrieval accuracy, the model adopts a joint optimization strategy, designing a joint loss function with multiple loss terms to achieve end-to-end collaborative training.

[0201] The first part is the retrieval comparison loss:

[0202] To optimize the semantic alignment between the structural anchor token and the query vector, matching accuracy is enhanced by constructing positive and negative sample pairs:

[0203] The sample is constructed as: text query vector As an anchor point, the corresponding structural anchor token As positive samples, structural anchor tokens from other regions in the same batch As a negative sample.

[0204] The expression for the retrieval contrastive loss function is:

[0205]

[0206] in, This indicates the loss of comparison in retrieval. Indicates the temperature coefficient. Represents cosine similarity. Represents a text query vector. This represents all structural anchor tokens (including positive and negative samples) in the same batch. This represents the index of the anchor token within the set. This represents the anchor token for the positive sample structure.

[0207] This loss function forces the model to learn to bring semantically matched text and image regions closer together in the feature space.

[0208] The second part describes the losses from local reconstruction:

[0209] To ensure that the compressed token can restore the details of the original image, the difference between the decoded reconstructed image and the original region image is calculated. The expression for the local reconstruction loss function is as follows:

[0210]

[0211] in, Indicates partial reconstruction losses. Represents the original region image. This represents the reconstructed image after decoding. Represented as L1 norm, This represents a pre-trained feature extraction network. The weighting coefficients represent the perceived loss. It is represented as the L2 norm.

[0212] The expression for the overall optimization objective is:

[0213] By jointly analyzing the retrieval comparison loss, local reconstruction loss, and the loss from the structure prior estimation module (SPE), multi-task collaborative optimization is achieved.

[0214]

[0215] in, This represents the overall optimization objective. This represents the loss of the structural prior estimation module. This represents the first loss coefficient. This represents the second loss coefficient. This represents the third loss coefficient.

[0216] Through end-to-end joint training, the model in this application not only learned to describe images (retrieval), but also learned to memorize on demand (compression and reconstruction), achieving a significant performance leap in long document comprehension tasks.

[0217] Based on the above description, this application has the following beneficial effects:

[0218] In this application, firstly, by recognizing the structural category and importance score of image blocks and combining spatial location information to form structured regions, the distribution and boundaries of modalities such as tables, formulas, and charts in documents are accurately captured, solving the problem of insufficient understanding of the global structure and fine-grained semantic regions of long documents by existing models, laying the foundation for efficient processing in the future.

[0219] Secondly, the adaptive token allocation mechanism based on the weight of structural categories and the importance of regions tilts resources towards high-value structural regions under the preset total token budget. This not only significantly reduces the redundancy of long document representation, but also avoids the degradation of key information during compression, thus balancing compression efficiency and information fidelity.

[0220] Furthermore, by constructing a global retrieval index through structural anchor tokens, combined with accurate matching of query vectors and local reversible decoding, there is no need to perform global decoding on the entire page of the document. Instead, the index is reconstructed only for the target area as needed, which reduces the computational cost of retrieval and parsing and effectively improves the extraction efficiency of specific structured information in massive long documents.

[0221] Finally, it supports combined queries of text and structural types, accurately responding to retrieval needs for specific structural types and semantic content, greatly improving the practicality of long document knowledge management, intelligent retrieval, and automated analysis, and providing efficient support for the digital transformation of industries such as engineering, law, and scientific research.

[0222] The above text combined Figure 1 The document visual compression and retrieval method provided in the embodiments of this application has been described in detail. The apparatus and device provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0223] like Figure 2 As shown, this figure is a schematic diagram of a document visual compression and retrieval device provided in an embodiment of this application. The device includes:

[0224] The acquisition module 301 is used to acquire the page image of the document to be processed and to divide the page image into multiple image blocks;

[0225] The data processing module 302 is used to obtain the structural category and structural importance score of each image patch based on multiple image patches; obtain several structural regions based on the structural category and spatial location of each image patch; allocate a target number of compressed tokens to each structural region based on the weight of the structural category and the structural importance score; encode the image data of each structural region using a visual compression encoder based on the target number to obtain a corresponding number of compressed tokens; and obtain the structural anchor tokens corresponding to each structural region based on the compressed tokens, thus forming a set of structural anchor tokens.

[0226] The retrieval module 303 is used to receive a query request, obtain a query vector based on the query request, search in the set of structural anchor tokens based on the query vector to obtain a target structural region, and perform local decoding and reconstruction based on the compressed token corresponding to the target structural region to obtain a reconstructed regional image or structural mask.

[0227] Optionally, the data processing module 302 is specifically used to calculate the unnormalized importance of each structural region based on the weight of the structural category and the structural importance score; to normalize the unnormalized importance to obtain the token allocation ratio of each structural region; and to allocate a target number of compressed word tokens to each structural region based on the token allocation ratio and the preset total page token budget.

[0228] Optionally, the data processing module 302 is specifically used to perform pooling operations on all compressed tokens within each structural region to obtain a semantic representation vector of the structural region; and to fuse the semantic representation vector with the index encoding vector to obtain a structural anchor token.

[0229] Optionally, the data processing module 302 is specifically used to encode the text content using a text encoder to obtain a text query vector if the query request contains text content; to map the structure type to a structure query vector using a structure type embedding table if the query request contains structure type conditions; and to perform feature fusion between the text query vector and the structure query vector to obtain a query vector.

[0230] Optionally, the data processing module 302 is specifically used to calculate the similarity between the query vector and each structural anchor token in the set of structural anchor tokens; and select the structural regions corresponding to the K structural anchor tokens with high similarity as the target structural regions.

[0231] Optionally, the data processing module 302 is specifically used to map the compressed token of the target structural region to an initial feature map; perform upsampling processing on the initial feature map to obtain a first feature map; input the first feature map into a convolutional layer, and output the reconstructed region image or structural mask.

[0232] Optionally, the data processing module 302 is specifically used to merge image blocks with the same or similar structural categories and adjacent spatial locations based on the structural category and spatial location of each image block, to obtain several structural regions.

[0233] The document visual compression and retrieval apparatus according to the embodiments of this application can correspondingly execute the method described in the embodiments of this application, and the other operations and / or functions of each module / unit of the document visual compression and retrieval apparatus are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0234] This application also provides a computing device. For example... Figure 3 As shown in the figure, this is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 700 includes a bus 701, a processor 702, a communication interface 703, and a memory 704. The processor 702, the memory 704, and the communication interface 703 communicate with each other via the bus 701.

[0235] The 701 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0236] The processor 702 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0237] The communication interface 703 is used for external communication.

[0238] Memory 704 may include volatile memory, such as random access memory (RAM). Memory 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0239] The memory 704 stores executable code, which the processor 702 executes to perform the aforementioned visual compression and retrieval method for documents.

[0240] Specifically, in achieving Figure 2 In the case of the illustrated embodiment, and Figure 2 When the modules or units of the document visual compression and retrieval device described in the embodiments are implemented by software, the following steps are performed: Figure 2 The software or program code required for the functions of each module / unit can be partially or wholly stored in the memory 704. The processor 702 executes the program code corresponding to each unit stored in the memory 704 to execute the aforementioned visual compression and retrieval method for documents.

[0241] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute the aforementioned visual compression and retrieval method for documents.

[0242] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0243] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0244] When the computer program product is executed by a computer, the computer executes any of the aforementioned methods of visual compression and retrieval of documents. The computer program product can be a software installation package; when any of the aforementioned methods of visual compression and retrieval of documents is required, the computer program product can be downloaded and executed on the computer.

[0245] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0246] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A method for visual compression and retrieval of documents, characterized in that, The method includes: Obtain the page image of the document to be processed, and divide the page image into multiple image blocks; Based on multiple image patches, the structure category and structure importance score of each image patch are obtained; Based on the structural category and spatial location of each image block, several structural regions are obtained; Based on the weights of structural categories and structural importance scores, a target number of compressed tokens is assigned to each structural region; based on the target number, the image data of each structural region is encoded using a visual compression encoder to obtain the corresponding number of compressed tokens. Based on the compressed token, the structural anchor tokens corresponding to each structural region are obtained, forming a set of structural anchor tokens; Receive a query request and obtain a query vector based on the query request; Based on the query vector, a search is performed in the set of structural anchor tokens to obtain the target structural region; Based on the compressed token corresponding to the target structural region, local decoding and reconstruction are performed to obtain the reconstructed regional image or structural mask.

2. The method according to claim 1, characterized in that, The target number of compressed lexical tokens allocated to each structural region based on the weights of structural categories and structural importance scores includes: Calculate the unnormalized importance of each structural region based on the weight of the structural category and the structural importance score; The unnormalized importance is normalized to obtain the token allocation ratio for each structural region; Based on the token allocation ratio and the preset total page token budget, a target number of compressed tokens are allocated to each structural region.

3. The method according to claim 1, characterized in that, The step of obtaining the structural anchor token corresponding to each structural region based on the compressed token includes: Pool all compressed tokens within each structural region to obtain the semantic representation vector of the structural region; The semantic representation vector is fused with the index encoding vector to obtain the structural anchor token.

4. The method according to claim 1, characterized in that, The step of obtaining the query vector based on the query request includes: If the query request contains text content, the text content is encoded using a text encoder to obtain a text query vector; If the query request contains a structure type condition, the structure type is mapped to a structure query vector through the structure type embedding table; The text query vector and the structure query vector are fused using feature fusion to obtain the query vector.

5. The method according to claim 1, characterized in that, The step of retrieving the target structural region from the set of structural anchor tokens based on the query vector includes: Calculate the similarity between the query vector and each structural anchor token in the set of structural anchor tokens; Select the structural regions corresponding to the K structural anchor tokens with high similarity as the target structural regions.

6. The method according to claim 1, characterized in that, The step of performing local decoding and reconstruction based on the compressed token corresponding to the target structural region to obtain the reconstructed region image or structural mask includes: Map the compressed tokens of the target structural region to the initial feature map; The initial feature map is upsampled to obtain the first feature map; The first feature map is input into the convolutional layer, and the reconstructed region image or structure mask is output.

7. The method according to claim 1, characterized in that, Based on the structural category and spatial location of each image block, several structural regions are obtained, including: Based on the structural category and spatial location of each image block, image blocks with the same or similar structural categories and adjacent spatial locations are merged to obtain several structural regions.

8. A visual compression and retrieval device for documents, characterized in that, The device includes: The acquisition module is used to acquire page images of the document to be processed and to segment the page images into multiple image blocks; The data processing module is used to obtain the structural category and structural importance score of each image patch from multiple image patches; to obtain several structural regions based on the structural category and spatial location of each image patch; to allocate a target number of compressed tokens to each structural region based on the weight of the structural category and the structural importance score; to encode the image data of each structural region using a visual compression encoder based on the target number to obtain a corresponding number of compressed tokens; and to obtain the structural anchor tokens corresponding to each structural region based on the compressed tokens, thus forming a set of structural anchor tokens. The retrieval module is used to receive query requests, obtain query vectors based on the query requests, search the set of structural anchor tokens based on the query vectors to obtain the target structural region, and perform local decoding and reconstruction based on the compressed tokens corresponding to the target structural region to obtain the reconstructed region image or structural mask.

9. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 7.

Images