A general document image segmentation method based on query decoupling and prototype classification

Abstract

The application relates to the field of image segmentation, and provides a general document image segmentation method based on query decoupling and prototype classification, which comprises the following steps: extracting multi-scale visual features from an input document image by using a visual skeleton network, and optimizing the multi-scale visual features to obtain optimized multi-scale visual features; encoding the text description of each type of region to be segmented into a semantic query vector by using a text encoder, and randomly initializing a preset number of instance query vectors; inputting the optimized multi-scale visual features, the semantic query vector and the instance query vectors into a mixed query decoder to obtain updated semantic query vectors and instance query vectors; and performing semantic segmentation and instance segmentation by using the updated semantic query vectors and instance query vectors respectively to obtain semantic segmentation results and instance segmentation results. The application can not only process various segmentation tasks on various types of document images, but also can input any document image and segmentation task prompt to output corresponding segmentation results.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a general document image segmentation method based on query decoupling and prototype classification. Background Technology

[0002] Document image segmentation (DIS) is a fundamental task in the field of document analysis and recognition (DAR). It aims to break down complex document image content into smaller, more meaningful components, such as text regions, images, tables, and other elements, thereby serving downstream applications such as text recognition, information extraction (IE), and document visual question answering (DocVQA). Accurately segmenting text regions, images, tables, and other elements improves the accuracy and efficiency of these techniques, enabling machines to better understand and parse information within documents, which is crucial for automating document processing workflows. Despite its important role in document processing, document image segmentation still faces numerous challenges due to the diversity of document types, the complexity of page layouts, the heterogeneity of content annotation, and the complexity of document structure.

[0003] Existing document image segmentation methods can be broadly categorized into two types: rule-based methods and machine learning-based methods. Rule-based methods segment document images using predefined rules and templates. These methods perform well with simple and regular documents, but perform poorly with complex and diverse documents. Machine learning-based methods, especially deep learning-based methods, have made significant progress in recent years. These methods automatically extract features and patterns by learning from large amounts of labeled data, thus achieving more accurate segmentation. Specifically, these methods can be roughly divided into the following categories: object detection-based methods, instance segmentation-based methods, graph model-based methods, and Transformer-based methods, etc. Among these methods, object detection-based approaches transform document image segmentation into an object detection problem, utilizing techniques such as Faster R-CNN, Mask R-CNN, and YOLO to detect different regions in a document and generate corresponding bounding boxes, achieving segmentation of text, tables, images, and other content. Instance segmentation-based approaches transform document image segmentation into an image segmentation problem, using fully convolutional neural networks (FCNs) and U-Net for pixel-level classification of documents, achieving segmentation of different regions in document images. Graph model-based methods treat documents as hierarchical graph structures, employing graph neural networks (GNNs) to model the structural relationships within the document, enabling tasks such as layout analysis, table structure recognition, and text detection. Transformer-based methods use Transformer-based object detection or instance segmentation techniques to detect or segment regions in document images. Furthermore, some multimodal methods combine visual, textual, and layout features, such as LayoutLMv3. These methods improve model performance and generalization ability by pre-training on large-scale unsupervised documents, aligning features from different modalities.

[0004] The field of document image segmentation has accumulated a large number of document image segmentation datasets. These datasets exhibit great diversity and heterogeneity in terms of data sources, document types, writing languages, processing tasks, annotation categories, and annotation formats. For example, PubLayNet contains digitally generated English PDF documents with region-level border annotations, used for page segmentation tasks of modern documents; SCUT-CAB and MTHv2 contain scanned historical Chinese documents with region, line, and character-level border annotations, used for region segmentation, text line extraction, and character detection tasks of ancient books; SCUT-HCCDoc contains handwritten documents with line-level border annotations, used for handwritten text line segmentation tasks; M6Doc contains documents of various types and languages ​​with region-level border annotations, used for page segmentation tasks of various document types; CTW1500 and Total-Text contain natural scene images with text border annotations of arbitrary shapes, used for text detection and segmentation tasks in natural scenes.

[0005] However, existing deep learning-based document image segmentation methods typically focus on a specific segmentation subtask and dataset, transforming various document image segmentation tasks into general object detection or image segmentation problems. They then modify general object detection and image segmentation methods based on the characteristics of the current task and data to better suit specific tasks and datasets. While this specialized and customized solution performs well on specific tasks, it falls short when dealing with comprehensive document segmentation tasks, often lacking versatility and flexibility. For example, a table detection model trained on PDF document images cannot perform text line segmentation on handwritten documents. Furthermore, this customization not only increases system complexity but also leads to redundant computational and storage overhead. Users need to design, develop, and maintain a corresponding algorithm and model for each task and dataset, making these methods highly inefficient for large-scale deployment in real-world scenarios. Summary of the Invention

[0006] This invention provides a general document image segmentation method based on query decoupling and prototype classification to address the limitations of existing document image segmentation methods, particularly their insufficient adaptability and generalization ability when faced with diverse segmentation tasks and labeled datasets. This method can handle various segmentation tasks on multiple types of document images and can output corresponding segmentation results from any document image and segmentation task prompts as input, improving the model's versatility and flexibility. The technical solution proposed in this invention is as follows:

[0007] In a first aspect, the present invention provides a general document image segmentation method based on query decoupling and prototype classification, comprising:

[0008] Obtain the input document image and a text description of each category of the region to be segmented;

[0009] Multi-scale visual features are extracted from the input document image using a pre-built visual skeleton network, and the extracted multi-scale visual features are optimized using a pre-built deformable attention network to obtain optimized multi-scale visual features.

[0010] A text encoder is used to encode the text description of each category of region to be segmented into a semantic query vector, and a preset number of instance query vectors are randomly initialized.

[0011] The optimized multi-scale visual features, the semantic query vector, and the instance query vector are input into a pre-built hybrid query decoder. The optimized multi-scale visual features are decoded in multiple rounds using the semantic query vector and the instance query vector through a cross-attention mechanism to obtain the updated semantic query vector and the updated instance query vector.

[0012] Semantic segmentation and instance segmentation are performed using the updated semantic query vector and the updated instance query vector, respectively, to obtain semantic segmentation results and instance segmentation results. The segmented instances are then classified using the updated semantic query vector as a category prototype to obtain the category classification result of the instance.

[0013] Optionally, the step of extracting multi-scale visual features from the input document image using a pre-built visual skeleton network, and optimizing the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features, includes:

[0014] The input document image is normalized in terms of scale and pixel values ​​to obtain a normalized document image;

[0015] Multi-scale visual features are extracted from the normalized document image using a pre-built visual skeleton network.

[0016] The multi-scale visual features are optimized using a pre-constructed deformable attention network to obtain optimized multi-scale visual features.

[0017] Optionally, the text encoder includes a word segmenter, an embedding layer, and an encoding layer; the step of encoding the text description of each category of regions to be segmented into a semantic query vector using the text encoder, and randomly initializing a preset number of instance query vectors, includes:

[0018] The word segmenter is used to segment the text description of each category of regions to be segmented, resulting in a token sequence;

[0019] The embedding layer is used to map the token sequence into a vector sequence;

[0020] The vector sequence is encoded using the encoding layer to obtain the encoded token feature sequence;

[0021] The global average of the encoded token feature sequence along the sequence dimension is calculated to obtain the semantic query vector for each category of the region to be segmented.

[0022] A preset number of instance query vectors are randomly initialized, and the dimension of each instance query vector is the same as the dimension of the semantic query vector.

[0023] Optionally, the optimized multi-scale visual features, the semantic query vector, and the instance query vector are input into a pre-constructed hybrid query decoder. A cross-attention mechanism is used to decode the optimized multi-scale visual features multiple times using the semantic query vector and the instance query vector, resulting in updated semantic query vectors and updated instance query vectors, including:

[0024] The semantic query vector and the instance query vector are subjected to a first information exchange to obtain the semantic query vector and the instance query vector after the exchange.

[0025] By using a multi-head cross-attention mechanism and a feedforward neural network, the semantic query vector after the interaction is used to decode the optimized multi-scale visual features to obtain a semantic query vector that integrates multi-scale features.

[0026] The optimized multi-scale visual features are decoded using the interactive instance query vector through a multi-head cross-attention mechanism and a feedforward neural network to obtain an instance query vector that integrates the multi-scale features.

[0027] The semantic query vector that incorporates multi-scale features and the instance query vector that incorporates multi-scale features are subjected to a second information exchange to obtain the updated semantic query vector and the updated instance query vector.

[0028] Optionally, the method further includes: fusing the optimized multi-scale visual features using a feature pyramid network to obtain hybrid multi-scale visual features.

[0029] Optionally, the step of performing semantic segmentation and instance segmentation using the updated semantic query vector and the updated instance query vector respectively to obtain semantic segmentation results and instance segmentation results, and using the updated semantic query vector as a category prototype to classify the segmented instances to obtain the category classification result of the instance, includes:

[0030] The updated semantic query vector is used to perform semantic segmentation on the hybrid multi-scale visual features to obtain the semantic segmentation result;

[0031] The updated instance query vector is used to perform instance segmentation on the hybrid multi-scale visual features to obtain the instance segmentation result.

[0032] Calculate the inner product of each updated instance query vector and each updated semantic query vector, and normalize it to obtain a similarity score. Select the semantic category corresponding to the semantic query vector with the highest similarity score for each updated instance query vector as the category classification result of the instance.

[0033] Secondly, the present invention also provides a general document image segmentation device based on query decoupling and prototype classification, comprising the following modules:

[0034] The acquisition module is used to acquire the input document image and the text description of each category of the region to be segmented;

[0035] The extraction module is used to extract multi-scale visual features from the input document image using a pre-built visual skeleton network, and to optimize the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features.

[0036] The encoding module is used to encode the text description of each category of the region to be segmented into a semantic query vector using a text encoder, and randomly initialize a preset number of instance query vectors.

[0037] The decoding module is used to input the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-constructed hybrid query decoder, and use the semantic query vector and the instance query vector to perform multiple rounds of decoding on the optimized multi-scale visual features using a cross-attention mechanism to obtain updated semantic query vectors and updated instance query vectors.

[0038] The segmentation module is used to perform semantic segmentation and instance segmentation using the updated semantic query vector and the updated instance query vector, respectively, to obtain semantic segmentation results and instance segmentation results. It also uses the updated semantic query vector as a category prototype to classify the segmented instances to obtain the category classification result of the instance.

[0039] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the general document image segmentation method based on query decoupling and prototype classification as described in the first aspect above.

[0040] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the general document image segmentation method based on query decoupling and prototype classification as described in the first aspect above.

[0041] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the general document image segmentation method based on query decoupling and prototype classification as described in the first aspect above.

[0042] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0043] This invention provides a general document image segmentation method based on query decoupling and prototype classification. It utilizes a visual skeleton network to extract multi-scale visual features from input document images, capturing different sizes and details, thus enhancing the processing capability for various document image types. By introducing a deformable attention network to optimize the extracted multi-scale visual features, it can more flexibly focus on key regions in the image, further improving feature representation. A text encoder encodes the textual descriptions of each category of regions to be segmented into semantic query vectors. This approach enables the model to understand and respond to different segmentation task prompts, supporting multiple segmentation tasks. By randomly initializing a preset number of instance query vectors, it can process multiple instances in the image, achieving instance-level segmentation. The optimized multi-scale visual features, semantic query vectors, and instance query vectors are input into a hybrid query decoder, and a cross-attention mechanism is used for multiple rounds of decoding. This process allows the model to dynamically adjust its attention, focusing on regions in the image crucial to the segmentation task, thereby improving segmentation accuracy. Simultaneously, the cross-attention mechanism effectively integrates information from different sources, enhancing the model's decoding capability. By utilizing updated semantic query vectors and instance query vectors for semantic segmentation and instance segmentation respectively, this approach can simultaneously handle semantic-level and instance-level segmentation, meeting the needs of various segmentation tasks. The updated semantic query vectors are then used as category prototypes to classify segmented instances. This method enables the model to automatically determine the instance category based on extracted features, further improving the model's versatility and flexibility.

[0044] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the general document image segmentation method based on query decoupling and prototype classification provided by the present invention.

[0048] Figure 2 This is an algorithm framework diagram of the general document image segmentation method based on query decoupling and prototype classification provided by the present invention;

[0049] Figure 3 This is a model structure diagram of an embodiment of the general document image segmentation method based on query decoupling and prototype classification provided by the present invention;

[0050] Figure 4 Example of results for the general document image segmentation method based on query decoupling and prototype classification provided by this invention on modern document layout segmentation tasks;

[0051] Figure 5 Examples of results for the general document image segmentation method based on query decoupling and prototype classification provided by this invention on ancient books and handwritten documents segmentation tasks;

[0052] Figure 6 Examples of results for the general document image segmentation method based on query decoupling and prototype classification provided by this invention on table detection and structure recognition tasks;

[0053] Figure 7 Example of results for the general document image segmentation method based on query decoupling and prototype classification provided by this invention on a scene text segmentation task;

[0054] Figure 8 A schematic diagram of the structure of the general document image segmentation device based on query decoupling and prototype classification provided by the present invention;

[0055] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] The following is combined with Figures 1-7 This invention describes a general document image segmentation method based on query decoupling and prototype classification.

[0058] This invention aims to address the limitations of existing document image segmentation methods, particularly their insufficient adaptability and generalization ability when faced with diverse segmentation tasks and labeled datasets. Existing document image segmentation methods typically focus on only a specific subtask and dataset, designing tailored algorithms and models based on the characteristics of the current task and data, lacking versatility and flexibility. Furthermore, this customization not only increases system complexity but also leads to redundant computational and storage overhead. Users need to design, develop, and maintain a corresponding algorithm and model for each task and dataset, making these methods highly inefficient for large-scale deployment in real-world scenarios.

[0059] To address the aforementioned key challenges, this invention proposes a general document image segmentation method based on query decoupling and prototype classification, focusing on overcoming the aforementioned technical obstacles. The core advantage of this method lies in its high adaptability and robustness, enabling it to handle various document segmentation tasks, such as page layout segmentation, text line extraction, character detection, table structure parsing, and scene text detection. It is applicable to various document image types, including modern documents, ancient books, natural scene images, digitally generated documents, scanned documents, and photographed documents. Furthermore, this method can directly utilize existing heterogeneous mixed datasets for learning and training, thereby minimizing the need for manual annotation of new data. This general document image segmentation method based on query decoupling and prototype classification, by introducing a query decoupling mechanism, prototype classification, multi-task learning, and hybrid supervised learning, can not only handle various segmentation tasks on multiple types of document images but also output corresponding segmentation results from any document image and segmentation task prompts as input. This improves the model's versatility, flexibility, and efficiency, providing strong technical support for subsequent fields such as text recognition, form understanding, information extraction, and document question answering. It further enhances the automation level of document processing, improves user experience, and has broad application prospects.

[0060] Query decoupling refers to breaking down the input query (i.e., the input document image and the segmentation task prompt) into multiple independent components or features. In document image segmentation, this can include decoupling image content, layout, text information, etc., into different feature vectors or representations. Through query decoupling, the model can handle different types of document images and segmentation tasks more flexibly.

[0061] Prototype classification refers to classifying or segmenting input by learning a set of prototype representations (i.e., representative or typical feature vectors). In document image segmentation, these prototypes can represent different types of document elements (such as text blocks, tables, images, etc.). Through prototype classification, the model can identify and segment multiple types of document elements, rather than being limited to a specific document type or segmentation task. This allows the model to be applied more broadly to different types of document image segmentation tasks.

[0062] Reference Figure 1 As shown, the method includes the following:

[0063] Step S110: Obtain the input document image and the text description of each category of the region to be segmented.

[0064] It receives user-provided input document images, along with textual descriptions of each category of the region to be segmented. These textual descriptions are used to subsequently convert the category information into semantic query vectors. (See reference...) Figure 2 As shown, the input document image is a heterogeneous mixed document image, which can be various types of document images such as photographed document images, modern document images, ancient book document images, natural scene images, scanned document images, and digitally generated document images.

[0065] The textual descriptions of each category of regions to be segmented are the segmentation task prompts mentioned above. These are detailed textual descriptions of each category of document regions to be segmented, which allow the model to better understand the characteristics of each category. For example, for tables, the text description is: "A table is a way of organizing data in rows and columns to display a structured collection of information. Each row represents a record, and each column corresponds to an attribute or characteristic of that record. Tables improve the readability and comparability of information through clear data arrangement, and are particularly suitable for listing, comparing, and statistical analysis. Tables may also include header rows and column labels to indicate the content type of each column. In addition, tables may have borders or background colors to enhance visual appeal and facilitate users to quickly locate and interpret data." For illustrations, the text description is: "Illustrations are visual elements in a document used to explain, supplement, or decorate the text content, including but not limited to photographs, charts, paintings, and graphics. Illustrations can intuitively convey information, making complex concepts easier to understand, while increasing the attractiveness of the document. They are often accompanied by captions or explanatory text that points out the key points of the illustration or explains its relationship with the text. The illustrations are generally placed close to the relevant text description to ensure that the connection between the two is clearly visible."

[0066] The textual descriptions of these regions to be segmented will then be encoded into semantic query vectors by a text encoder, responsible for semantic segmentation and instance classification tasks. By introducing textual descriptions of the regions to be segmented, the method can understand and utilize the semantic relationships between categories, improving the accuracy and generalization ability of segmentation.

[0067] Step S120, refer to Figure 2 As shown, a pre-built visual skeleton network is used to extract multi-scale visual features from the input document image, and a pre-built deformable attention network is used to optimize the extracted multi-scale visual features to obtain optimized multi-scale visual features.

[0068] Pre-built visual skeleton networks (such as ResNet and EfficientNet) are used to extract multi-scale visual features from input document images. Visual skeleton networks can extract hierarchical feature information from input document images. They contain multiple convolutional and pooling layers that extract feature information at different scales. During multi-scale feature extraction, the network progressively reduces the spatial resolution of the feature maps while increasing the number of channels, thereby capturing features at different scales, from low-level details to high-level semantics. Deformable attention networks are a variant of attention mechanisms that allow attention weights to have spatial flexibility.

[0069] Next, these features are optimized using a pre-built deformable attention network to enhance the ability to capture key information. Compared to traditional fixed-position attention mechanisms, deformable attention mechanisms can better adapt to irregular shapes and deformations in images, thus capturing key information more accurately. After extracting multi-scale visual features, the deformable attention network optimizes these features. Through the deformable attention mechanism, the network can dynamically adjust the position and range of the attention focus, thereby enhancing the ability to capture key information. By optimizing features, the deformable attention network enables the model to pay more attention to important regions and details in the image. This not only improves the model's ability to recognize elements such as text, tables, and images in document images, but also enhances the model's adaptability to complex scenes and deformed objects.

[0070] Multi-scale feature extraction captures image details at different scales, while deformable attention networks dynamically adjust the focus of attention, improving feature quality and adaptability to complex scenes. The combination of multi-scale visual feature extraction and deformable attention network optimization significantly improves the accuracy and robustness of document image segmentation. By capturing feature information at different scales and optimizing the ability to capture key information, the model can better adapt to different types of document images and segmentation tasks. Due to the model's stronger generalization ability, it reduces reliance on new data and annotation costs.

[0071] Step S130, refer to Figure 2 As shown, a text encoder is used to encode the text description of each category of region to be segmented into a semantic query vector, and a preset number of instance query vectors are randomly initialized.

[0072] Reference Figure 3 As shown, a text encoder (such as BERT, GPT, etc.) is used to encode the text description of each category of regions to be segmented, i.e., the semantic category to be segmented (such as paragraph, table, illustration, cell, character, word, text line, etc.), into a semantic query vector. Simultaneously, a preset number of instance query vectors are randomly initialized; these vectors will be used for subsequent instance segmentation. The number of instance query vectors can be obtained empirically or statistically, and is usually not less than the maximum number of regions to be segmented in each document. The introduction of semantic query vectors and instance query vectors enables the method to handle semantic segmentation and instance segmentation tasks simultaneously, improving the model's flexibility and multi-task processing capabilities.

[0073] Step S140: Input the optimized multi-scale visual features, the semantic query vector, and the instance query vector into the pre-constructed hybrid query decoder. Use the cross-attention mechanism to perform multiple rounds of decoding on the optimized multi-scale visual features using the semantic query vector and the instance query vector to obtain the updated semantic query vector and the updated instance query vector.

[0074] The optimized multi-scale visual features, semantic query vectors, and instance query vectors are input into a pre-built hybrid query decoder. Utilizing a cross-attention mechanism, the semantic query vectors and instance query vectors are continuously updated through multiple rounds of decoding. This multi-round decoding and cross-attention mechanism progressively refines the query vectors, better aligning them with image features and improving segmentation accuracy and detail capture capabilities.

[0075] Specifically, the hybrid query decoder receives optimized multi-scale visual features, semantic query vectors, and instance query vectors as input. The hybrid query decoder can consist of multiple decoding layers (i.e., ... Figure 3 The model consists of a multi-scale decoder, with each layer containing self-attention and cross-attention mechanisms. In each multi-scale decoder layer, the semantic query vector and the instance query vector first interact through the self-attention mechanism. Self-attention allows the query vectors to influence each other, capturing the relationships between them. Subsequently, the query vectors interact with multi-scale visual features through the cross-attention mechanism. Cross-attention allows the query vectors to extract relevant information from image features and update themselves based on image content. During multiple rounds of decoding, the semantic query vector and the instance query vector are continuously updated through the self-attention and cross-attention mechanisms at each layer. Each round of decoding makes the query vector more accurately capture the semantic and instance information in the image. As the number of decoding layers increases, the query vector gradually refines from coarse global information to specific local details. This progressive refinement process helps the model better align image features, thereby improving segmentation accuracy.

[0076] Cross-attention dynamically selects the image region most relevant to the query vector by calculating the similarity between the query vector and image features. This mechanism effectively aligns the query vector with image features, ensuring the model captures details relevant to the query. Cross-attention also captures contextual information within the image, helping the model understand relationships between objects. For example, in segmentation tasks, the model can use cross-attention to identify the relative positional relationships between objects, thereby improving segmentation accuracy.

[0077] Through multi-round decoding and cross-attention mechanisms, the hybrid query decoder progressively refines the semantic query vector and instance query vector, better aligning them with image features. This mechanism not only improves segmentation accuracy but also captures more detailed information, making it suitable for various complex visual tasks. After multiple rounds of decoding, the final semantic query vector and instance query vector are used to generate the segmentation result. The semantic query vector is used to predict the category of each pixel, while the instance query vector is used to distinguish different instances.

[0078] Step S150: Perform semantic segmentation and instance segmentation using the updated semantic query vector and the updated instance query vector respectively to obtain semantic segmentation results and instance segmentation results. Then, use the updated semantic query vector as a category prototype to classify the segmented instance to obtain the category classification result of the instance.

[0079] Semantic segmentation is performed using the updated semantic query vector to obtain the category label for each pixel. Simultaneously, instance segmentation is performed using the updated instance query vector to obtain the bounding box and mask for each instance. Furthermore, the updated semantic query vector is used as a category prototype to classify the segmented instances, yielding the instance category classification results. By combining semantic segmentation, instance segmentation, and category classification, a comprehensive understanding of the structure and content of document images can be achieved, providing strong support for subsequent tasks such as text recognition and form understanding. At the same time, the introduction of prototype classification improves the accuracy and robustness of classification.

[0080] Existing methods often require the design of tailored algorithms and models based on the characteristics of the current task and data, which increases system complexity and leads to redundant computational and storage overhead. In contrast, the method of this invention constructs a general segmentation framework (including the aforementioned visual skeleton network, deformable attention network, text encoder, and hybrid query decoder, which constitute the general document image segmentation model of this invention), and utilizes multi-scale visual feature extraction and deformable attention optimization techniques to effectively capture and process various document image features, reducing the need for algorithm customization. Users do not need to design, develop, and maintain a corresponding algorithm and model for each task and dataset, thereby reducing system complexity and maintenance costs.

[0081] This invention proposes a general document image segmentation method based on query decoupling and prototype classification. It utilizes a visual skeleton network to extract multi-scale visual features from input document images, capturing different sizes and details to enhance processing capabilities for various document image types. By introducing a deformable attention network to optimize the extracted multi-scale visual features, it can more flexibly focus on key regions in the image, further improving feature representation. A text encoder encodes the textual descriptions of each region to be segmented into semantic query vectors. This approach enables the method to understand and respond to different segmentation task prompts, thus supporting multiple segmentation tasks. By randomly initializing a preset number of instance query vectors, it can process multiple instances in the image, achieving instance-level segmentation. The optimized multi-scale visual features, semantic query vectors, and instance query vectors are input into a hybrid query decoder, and a cross-attention mechanism is used for multiple rounds of decoding. This process allows the general document image segmentation model to dynamically adjust its attention, focusing on regions in the image crucial to the segmentation task, thereby improving segmentation accuracy. Simultaneously, the cross-attention mechanism effectively integrates information from different sources, enhancing the model's decoding capabilities. This method utilizes updated semantic query vectors and instance query vectors for semantic segmentation and instance segmentation respectively, enabling simultaneous handling of semantic-level and instance-level segmentation to meet the needs of various segmentation tasks. The updated semantic query vectors are used as category prototypes to classify segmented instances. This approach allows the model to automatically determine the instance category based on extracted features, further improving the model's versatility and flexibility, reducing development and maintenance costs, and thus improving algorithm deployment efficiency. Because this method employs a text encoder and randomly initialized instance query vectors, it can handle inputs from arbitrary document images and segmentation task prompts. This gives the model extremely high versatility and flexibility. By optimizing multi-scale visual features, utilizing cross-attention mechanisms for decoding, and combining multiple steps such as semantic segmentation, instance segmentation, and classification, this method improves model efficiency while maintaining segmentation accuracy.

[0082] Thanks to the combination of query decoupling and prototype classification, the model can accept document images of any type and segmentation task cues as input. This means users can flexibly specify segmentation tasks as needed without requiring additional training or tuning of the model. The model can output corresponding segmentation results based on the input task cues and document image content. These results can be pixel-level segmentation masks, bounding boxes, or other forms of representation, depending on the requirements of the task cues. Due to the model's versatility and flexibility, the same underlying feature extraction network can be shared across multiple tasks. This reduces the cost and time of training separately for different tasks.

[0083] This invention proposes a general document image segmentation method based on query decoupling and prototype classification. By introducing semantic query vectors and instance query vectors, as well as a hybrid query decoder, it achieves comprehensive adaptability to different segmentation tasks and various types of document images. Existing methods typically focus only on specific subtasks and datasets, lacking versatility and flexibility. In contrast, the method of this invention can handle various types of document segmentation tasks, such as page layout segmentation, text line extraction, character detection, table structure parsing, and scene text detection, and is applicable to various document image types, including modern document images, ancient book images, and natural scene images. This high degree of adaptability and generalization enables the method of this invention to exhibit superior performance when facing diverse segmentation tasks and labeled datasets.

[0084] Furthermore, this general document image segmentation method based on query decoupling and prototype classification can directly utilize existing heterogeneous mixed datasets for learning and training, thereby minimizing the need for manual annotation of new data. By introducing query decoupling mechanisms, prototype classification, multi-task learning, and hybrid supervised learning, the learned model can not only handle various document segmentation tasks but also output corresponding segmentation results from arbitrary document images and segmentation task prompts as input. This improves the model's versatility, flexibility, and efficiency, providing strong technical support for subsequent fields such as text recognition, form understanding, information extraction, and document question answering. It further enhances the automation level of document processing, improves user experience, and has broad application prospects.

[0085] In an optional embodiment, to ensure consistency in scale and pixel values ​​of the input document image, facilitating subsequent feature extraction and optimization processing, refer to... Figure 3 As shown, the present invention also preprocesses the input document image, including scale and pixel value normalization. Step S120 above, which involves extracting multi-scale visual features from the input document image using a pre-constructed visual skeleton network and optimizing the extracted multi-scale visual features using a pre-constructed deformable attention network to obtain optimized multi-scale visual features, includes:

[0086] S1201. The input document image is normalized in terms of scale and pixel value to obtain a normalized document image.

[0087] The input document image is scale-normalized, that is, the image size is adjusted to meet the input requirements of the visual skeleton network. This can be achieved through scaling algorithms, such as bilinear interpolation, nearest neighbor interpolation, etc. After scale normalization, pixel value normalization is performed in this invention, adjusting the pixel values ​​of the image to a specific range (such as 0 to 1 or -1 to 1) to eliminate differences in brightness, contrast, etc. among different images.

[0088] In this invention, for the input document image, its longer side is first scaled to a specified pixel value, such as 1024 pixels, while the narrower side is scaled by the same ratio to maintain the original aspect ratio of the document image. Then, the value of each pixel in the document image is normalized to reduce the influence of differences in document styles. The specific normalization method can be either the mean-divide-variance method or the max-min normalization method. In this embodiment, the mean-divide-variance method is chosen to ensure that the mean of the normalized document image pixels is 0 and the variance is 1.

[0089] S1202. Use a pre-built visual skeleton network to extract multi-scale visual features from the normalized document image to obtain multi-scale visual features.

[0090] Reference Figure 3 As shown, the normalized document image is input into a visual skeleton network. Through the network's convolutional layers, pooling layers, and other structures, multi-scale visual features are extracted step by step. These features are represented as high-dimensional feature vectors or feature maps. For example, a multi-scale visual encoder, such as a convolutional neural network (e.g., ResNet) or a visual Transformer (e.g., Swin-Transformer), is used to encode the normalized document image, and then multiple visual features at specified scales are extracted at designated layers, such as 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size. Multi-scale features play an important role in improving the segmentation performance of multi-scale regions.

[0091] In this embodiment, the visual skeleton network employs the Swin-Transformer, an innovative visual Transformer architecture that processes image data through a hierarchical structure and a moving window mechanism. This not only preserves the locality advantage of convolutional neural networks but also possesses the long-range dependency modeling capabilities of Transformers. It segments the input into non-overlapping windows at each level and performs self-attention computation within these windows. Simultaneously, it moves the window positions in odd-numbered levels to ensure cross-information flow between different levels. This approach improves the model's flexibility and efficiency, enabling it to perform well in various computer vision tasks, such as image classification, object detection, and semantic segmentation. This invention utilizes this visual skeleton network to extract multiple feature layers at specified scales from the normalized document image, such as 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size. Features at different scales have different feature dimensions, such as 192, 384, 768, and 1536. Multi-scale features play a crucial role in improving the segmentation performance of multi-scale regions, enabling the simultaneous segmentation of large and small target regions.

[0092] S1203, reference Figure 3As shown, the multi-scale visual features are optimized using a pre-constructed deformable attention network to obtain optimized multi-scale visual features.

[0093] The structure of the aforementioned deformable attention network can be referred to in the description of existing technologies, and is not specifically limited here. For example, the deformable attention network may include multiple convolutional layers. Convolutional layers are used to transform the dimensions of the multi-scale visual features extracted by the visual skeleton network to the same value. Then, another convolutional layer is used to predict multiple coordinate offsets of each pixel position at multiple scales for each scale of visual features. These offsets represent which neighboring pixel features should interact with the feature at each pixel position. Afterward, based on the offset prediction results, a specified number of neighbor nodes and their features are extracted from the feature layers at multiple scales for each pixel position at each scale. Finally, an attention mechanism is used to interact with each pixel feature and its neighbor node features, thereby updating the feature at the current pixel position. The above calculation process can be executed repeatedly, corresponding to multiple deformable attention network layers, to further optimize the multi-scale visual features, ultimately obtaining the optimized multi-scale visual features. The kernel size of the convolutional layers, the dimensions of the transformed visual features, the number of coordinate offsets, and the number of neighbor nodes can be set according to actual needs, and are not specifically limited in this invention.

[0094] In this embodiment, firstly, a convolutional layer with a 1x1 kernel is used to convert the dimensions of the multi-scale visual features to the same value, such as 256. Then, another convolutional layer with a 1x1 kernel is used to predict multiple coordinate offsets of each pixel position at multiple scales for the visual features at each scale. The predicted offset is used to indicate which neighboring pixel features should interact with the feature at each pixel position. For example, 8 coordinate offsets are predicted for each pixel position at each scale. After that, based on the offset prediction results, a specified number of neighbor nodes and their features are extracted for each pixel position at each scale in the feature layers at multiple scales. For example, 8 neighbor nodes are extracted for each pixel at each scale. Finally, a multi-head attention mechanism is used to interact with each pixel feature and its neighbor node features to update the features at the current pixel position.

[0095] Multiscale deformable attention mechanism The specific calculation process is as follows:

[0096]

[0097] in, Indicates the currently queried pixel. This represents the characteristics of the pixel. This represents the normalized coordinates of the pixel. This represents a feature layer with L scales. Indicates the first One point of attention, Indicates the number of attention heads. Indicates the first Each scale of feature layer, Represents the current pixel's... Each neighboring sampled pixel, This indicates the number of neighboring sampled pixels. and For learnable weight parameters, and They represent the first The first one in the attention. In the feature layer of scale, the first The sampling offset and attention weight of each sampling point The function represents remapping the normalized coordinates to the first... The original coordinates of the layer. Compared to the original global multi-head attention mechanism, the deformable attention mechanism can significantly reduce computational overhead. The above calculation process can be executed multiple times, corresponding to multiple deformable attention layers, to further optimize multi-scale visual features. For example, this embodiment uses 4 deformable attention layers.

[0098] This invention eliminates differences in scale and pixel values ​​in input images through scale and pixel value normalization, improving the consistency and accuracy of feature extraction. The visual skeleton network can capture feature information at different scales, enhancing the model's adaptability to complex scenes and deformable objects. The deformable attention network optimizes multi-scale visual features, enabling the model to focus more on important regions and details in the image, improving its generalization ability. The combination of multi-scale visual feature extraction and deformable attention network optimization significantly improves the accuracy and robustness of document image segmentation. By capturing feature information at different scales and optimizing the ability to capture key information, it can better adapt to different types of document images and segmentation tasks. Due to the model's stronger generalization ability, it reduces reliance on new data and annotation costs. This method provides strong technical support for subsequent tasks such as text recognition, form understanding, and information extraction, promoting the automation and intelligentization of document processing.

[0099] In an optional embodiment, the text encoder in step S130 above comprises three main parts: a tokenizer, an embedding layer, and an encoding layer. The text encoder can be a pre-trained open-source multilingual version of the text encoder to support multilingual category text descriptions. A pre-trained multilingual version of the SentenceTransformer model paraphrase-multilingual-MiniLM-L12-v2 can be used to support multilingual category text descriptions. Of course, other types of text encoders can also be used. Step S130 above, which involves encoding the text description of each category of the region to be segmented into a semantic query vector using the text encoder and randomly initializing a preset number of instance query vectors, includes:

[0100] S1301. Use the word segmenter to segment the text description of each category of regions to be segmented (such as paragraphs, tables, illustrations) to obtain a token sequence.

[0101] The main function of a word segmenter is to divide the input text description according to certain rules (such as words, sub-words, characters, etc.) to obtain a series of tokens. For Chinese text, word segmenters can encode it according to characters, words, Wubi input method radicals, four-corner symbols, etc. For English text, it is usually encoded according to letters, words, word roots, etc. Word segmenters can also perform word segmentation based on statistical methods (such as BPE, WordPiece, etc.) or rule-based methods. Through word segmentation, the text description is converted into a series of token sequences, providing a foundation for subsequent processing.

[0102] In this embodiment, the Sentence Transformer model used is a multilingual version, and its word segmenter supports word segmentation operations for multiple languages ​​simultaneously. Word segmentation is the process of dividing continuous text into independent words according to semantic or grammatical rules. It is a fundamental step in natural language processing, helping computers understand text structure, and is especially important for languages ​​such as Chinese where word boundaries are not clearly defined. The independent words after segmentation are generally called tokens.

[0103] S1302. The token sequence is mapped into a vector sequence using the embedding layer.

[0104] The primary function of the embedding layer is to map the token sequence obtained by the token segmenter into a vector sequence. These vectors contain semantic information, ensuring that similar tokens have similar representations in the vector space. The embedding layer employs word embedding techniques, such as Word2Vec and GloVe. These techniques can map a word to a high-dimensional vector space, thereby capturing the semantic relationships between words. Through the embedding layer, the token sequence is converted into a vector sequence, providing a processable numerical representation for subsequent encoding layers.

[0105] In this embodiment, after word segmentation, the embedding layer of the Sentence Transformer model is used to map each token from one-hot encoding to a C-dimensional vector, such as 384-dimensional vectors, resulting in a vector sequence. Since the embedding layer of the Sentence Transformer model is pre-trained on a large-scale corpus, the multi-dimensional vectors obtained by mapping tokens with similar semantics also have high similarity.

[0106] S1303. Encode the vector sequence using the encoding layer to obtain the encoded token feature sequence.

[0107] The primary function of the encoding layer is to further process and encode the vector sequence obtained from the embedding layer to extract semantic information from the text. The encoding layer can employ a series of neural network layers (such as Transformer layers) to encode the vector sequence. These neural network layers can capture contextual information within the vector sequence, thereby extracting the semantic features of the text. Through the encoding layer, the vector sequence is transformed into an encoded token feature sequence with rich semantic information.

[0108] In this embodiment, the encoding layer of the Sentence Transformer model is used to encode the input token feature sequence in multiple layers to obtain the encoded token features, which still have a dimension of C. The encoding layer of the Sentence Transformer model contains multiple multi-head self-attention layers and feedforward neural network layers, which are combined and interleaved, that is, each multi-head attention layer is followed by a feedforward neural network layer. For example, the paraphrase-multilingual-MiniLM-L12-v2 model used in this embodiment contains 12 layers.

[0109] S1304. Calculate the global average of the encoded token feature sequence along the sequence dimension to obtain the semantic query vector for each type of region to be segmented.

[0110] By globally averaging the encoded token feature sequence along its sequence dimension, a semantic query vector for each category of regions to be segmented can be obtained. This vector represents the overall semantic features of that category. By performing a global averaging operation on the encoded token feature sequence, a fixed-length vector representation, i.e., the semantic query vector, can be obtained. The semantic query vector captures the overall semantic features of each category, providing a foundation for subsequent processing.

[0111] Different categories of text descriptions have varying lengths, necessitating the conversion of encoded vector sequences of different lengths into a single length. In this embodiment, a global averaging approach is used, averaging along the sequence length dimension. After averaging, each category's vector sequence is transformed into a single vector representing the semantic query vector for that category, with dimension C and length 1. If the number of semantic categories to be segmented in the current document image is N, then after the above operation, N semantic query vectors with dimension C will be generated. Subsequently, a fully connected layer is used to transform the dimensions of the semantic query vectors, ensuring that their feature dimensions match the visual feature dimensions, for example, 256 dimensions.

[0112] S1305. Randomly initialize a preset number of instance query vectors, with the dimension of each instance query vector being consistent with the dimension of the semantic query vector.

[0113] Instance query vectors are a set of randomly initialized vectors used for subsequent processing (such as attention mechanisms, decoders, etc.). (See reference...) Figure 3 As shown, learnable position embeddings are performed to generate a predetermined number of instance query vectors. The dimension of each instance query vector is consistent with the dimension of the semantic query vector. The parameters of these instance query vectors are then randomly initialized. Various initialization methods can be used, such as Gaussian distribution random initialization or uniform distribution random initialization. Random initialization of instance query vectors provides diversity for subsequent processing and helps the model better capture and process information in text data.

[0114] In this embodiment, a preset number of M instance query vectors (e.g., 500) are first generated. The dimension of each instance query vector is consistent with the dimension of the semantic query vector, for example, 256. The parameters of these instance query vectors are then randomly initialized using a Gaussian distribution with a mean of 0 and a variance of 0.02.

[0115] This invention transforms text data into a numerical representation through word segmentation, embedding, and encoding layers, making subsequent processing more efficient and accurate. The embedding and encoding layers capture semantic information within the text, enabling the model to better understand and process the text data.

[0116] In an optional embodiment, step S140 above, which involves inputting the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-constructed hybrid query decoder, and using a cross-attention mechanism to perform multiple rounds of decoding on the optimized multi-scale visual features using the semantic query vector and the instance query vector to obtain updated semantic query vectors and updated instance query vectors, includes:

[0117] S1401. Perform the first information interaction between the semantic query vector and the instance query vector to obtain the semantic query vector and the instance query vector after interaction.

[0118] The semantic query vector and the instance query vector interact for the first time through a self-attention mechanism. Self-attention allows the semantic query vector and the instance query vector to influence each other, capturing the relationships between them. For example, the semantic query vector can provide category information, while the instance query vector can provide instance-specific information. This interaction yields the interacting semantic query vector and the interacting instance query vector. Through self-attention, the semantic query vector and the instance query vector can share information, enhancing their synergy. This interaction provides richer contextual information for the subsequent decoding process.

[0119] Specifically, firstly, referring to Figure 3 As shown, the semantic query vector and instance query vector are concatenated along the sequence dimension. For example, for N semantic query vectors and M instance query vectors, the concatenation results in N+M query vectors. Next, a multi-head self-attention mechanism (V, K, Q) and a feedforward neural network are used to update the features of the concatenated query vectors (i.e., the concatenated query vectors first undergo self-attention calculation and feature update using the multi-head attention mechanism, and then feature transformation is performed using the feedforward neural network layer). Then, the updated query vectors are split into semantic query vectors and instance query vectors, with the length of the split semantic query vectors and instance query vectors remaining consistent with their original sequence lengths, for example, N interactive semantic query vectors and M interactive instance query vectors.

[0120] S1402. Using a multi-head cross-attention mechanism and a feedforward neural network, the semantic query vector after the interaction is used to decode the optimized multi-scale visual features to obtain a semantic query vector that integrates multi-scale features.

[0121] The interacting semantic query vector is then processed by a multi-head cross-attention mechanism, interacting with optimized multi-scale visual features. This mechanism calculates the similarity between the semantic query vector and the multi-scale visual features, dynamically selecting the image region most relevant to the semantic query vector. Further processing via a feed-forward network (FFN) yields a semantic query vector incorporating multi-scale features. The multi-head cross-attention mechanism captures both local and global information relevant to the semantic query vector within the image. This multi-scale feature-integrated semantic query vector more accurately represents the semantic information in the image, improving the accuracy of semantic segmentation.

[0122] Specifically, first, the two-dimensional visual features at each scale (i.e., the optimized multi-scale visual features mentioned above) are unfolded into a one-dimensional sequence. For example, the original visual features with the shape H×W×C are unfolded into HW×C, where H represents the feature layer height, W represents the feature layer width, and C represents the feature dimension; then, referring to... Figure 3 As shown, features at all scales are concatenated along the sequence dimension to obtain a visual feature sequence containing multiple scales, such as concatenating features at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32. Then, a multi-head cross-attention mechanism (including multi-head self-attention and cross-attention mechanisms) and a feedforward neural network are used to decode the semantic query vector on the multi-scale visual features, resulting in a semantic query vector that integrates multi-scale features. Two multi-head self-attention mechanisms, a cross-attention mechanism, and a feedforward neural network constitute the... Figure 3 The multi-scale decoder is used in this embodiment. The entire decoder is iteratively optimized, where T represents the number of iterations, which is 4 in this example. Each iteration produces an output result, and these intermediate results are used to calculate the loss with the labeled ground truth during the training phase. L represents the number of layers in the decoder in each iteration, which is also set to 4 in this example.

[0123] S1403. Using a multi-head cross-attention mechanism and a feedforward neural network, the optimized multi-scale visual features are decoded using the interactive instance query vector to obtain an instance query vector that integrates multi-scale features.

[0124] The interacted instance query vector interacts with optimized multi-scale visual features through a multi-head cross-attention mechanism. The multi-head cross-attention mechanism calculates the similarity between the instance query vector and the multi-scale visual features, dynamically selecting the image region most relevant to the instance query vector. Further processing via a feedforward neural network yields an instance query vector that incorporates multi-scale features. The multi-head cross-attention mechanism captures local information in the image related to the instance query vector, helping to distinguish different instances within the same category. The instance query vector incorporating multi-scale features more accurately represents instance information in the image, improving the accuracy of instance segmentation.

[0125] Specifically, firstly, referring to Figure 3 As shown, the two-dimensional visual features at each scale (i.e., the optimized multi-scale visual features mentioned above) are unfolded into a one-dimensional sequence. For example, the original visual features with the shape H×W×C are unfolded into HW×C, where H represents the feature layer height, W represents the feature layer width, and C represents the feature dimension. Then, the features of all scales are concatenated along the sequence dimension to obtain a visual feature sequence containing multiple scales. For example, the features at scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 are concatenated along the sequence dimension. Then, a multi-head cross-attention mechanism and a feedforward neural network are used to decode the instance query vector on the multi-scale visual features to obtain an updated instance query vector that incorporates the multi-scale features.

[0126] S1404. Perform a second information exchange between the semantic query vector that integrates multi-scale features and the instance query vector that integrates multi-scale features to obtain the updated semantic query vector and the updated instance query vector.

[0127] The semantic query vector and instance query vector, which integrate multi-scale features, undergo a second information interaction through a self-attention mechanism. This interaction further enhances the synergy between the semantic query vector and the instance query vector, ensuring they can jointly optimize. Through this interaction, updated semantic query vectors and updated instance query vectors are obtained. This second information interaction further refines the query vectors, ensuring they can better capture semantic and instance information in the image. Through multiple interactions, the query vectors can gradually align with image features, improving segmentation accuracy and detail capture capabilities.

[0128] Specifically, firstly, referring to Figure 3As shown, the semantic query vector and the instance query vector, which integrate multi-scale features, are concatenated along the sequence dimension. For example, for N semantic query vectors and M instance query vectors, the concatenation results in N+M query vectors. Next, a multi-head self-attention mechanism and a feedforward neural network are used to update the features of the concatenated query vectors. Then, the feature-updated query vectors are split into semantic query vectors and instance query vectors, with the split semantic query vectors and instance query vectors maintaining the same length as their original sequences, for example, N semantic query vectors and M instance query vectors. These split semantic query vectors and instance query vectors are the same as the updated semantic query vectors and updated instance query vectors described above.

[0129] The steps S1401-S1404 (first information interaction, multi-head cross-attention mechanism decoding, second information interaction) can be repeated multiple times to form a multi-round decoding process. Each round of decoding further optimizes the semantic query vector and instance query vector, making them better aligned with image features. The multi-round decoding process allows the model to progressively refine the query vector, from coarse global information to specific local details. Through iterative optimization, the model can capture more detailed information and improve the accuracy of segmentation.

[0130] This invention employs a multi-round decoding and cross-attention mechanism to progressively align semantic query vectors and instance query vectors with image features, capturing more detailed information and thus improving the accuracy of semantic and instance segmentation. The multi-head cross-attention mechanism dynamically selects the image region most relevant to the query vector, ensuring the model captures local details within the image. Through multiple information interactions and multi-round decoding, the query vector is progressively optimized to better represent the semantic and instance information in the image. This process, through multi-round decoding and cross-attention, progressively optimizes the semantic and instance query vectors, better aligning them with multi-scale visual features. This method not only improves segmentation accuracy but also captures more detailed information, making it suitable for various complex visual tasks.

[0131] In an optional embodiment, after step S1203 described above, the method further includes:

[0132] S1204, reference Figure 3 As shown, a feature pyramid network is used to fuse the optimized multi-scale visual features to obtain hybrid multi-scale visual features.

[0133] The Feature Pyramid Network (FPN) is used to fuse optimized visual features at different scales to generate hybrid multi-scale visual features with rich semantic and detail information. First, a 1x1 convolutional layer is used to transform the features at each scale of the optimized multi-scale visual features. Since the dimensions of the multi-scale features were already unified in step S1203, no further dimensionality transformation is needed here. Second, the multi-scale features are scaled to the same size, for example, 1 / 4 of the original image size. Then, the scaled multi-scale features are added pixel-by-pixel to obtain the fused multi-scale features, i.e., the aforementioned hybrid multi-scale visual features. In this embodiment, the fused multi-scale features have a dimension of 256 and a size of 1 / 4 of the original image.

[0134] The hybrid multi-scale visual features generated by FPN help the model better distinguish different instances within the same category. FPN can improve the accuracy of boundary segmentation in semantic segmentation tasks. Combining semantic segmentation and instance segmentation, FPN can simultaneously capture global semantic information and local detail information. Hybrid multi-scale visual features can simultaneously capture global semantic information and local detail information, significantly improving the model's performance in tasks such as instance segmentation and semantic segmentation.

[0135] The aforementioned visual skeleton network, deformable attention network, and feature pyramid network constitute... Figure 3 The visual encoder in the text.

[0136] In an optional embodiment, after obtaining the updated semantic query vector, the updated instance query vector, and the mixed multi-scale visual features, mask prediction and category prediction are performed based on the updated semantic query vector, the updated instance query vector, and the mixed multi-scale visual features to obtain prediction results. Step S150 above, which involves performing semantic segmentation and instance segmentation using the updated semantic query vector and the updated instance query vector respectively to obtain semantic segmentation results and instance segmentation results, and using the updated semantic query vector as a category prototype to classify the segmented instances to obtain the category classification result of the instance, includes:

[0137] S1501. Use the updated semantic query vector to perform semantic segmentation on the hybrid multi-scale visual features to obtain the semantic segmentation result.

[0138] Semantic segmentation is performed on hybrid multi-scale visual features using an updated semantic query vector. Specifically, the semantic query vector interacts with the hybrid multi-scale visual features through a cross-attention mechanism, calculating the similarity between each pixel location and the semantic query vector. Based on the similarity score, a semantic category label is assigned to each pixel, generating the semantic segmentation result. The semantic query vector captures global semantic information in the image, ensuring that the semantic segmentation result accurately reflects the object category and scene context in the image. The semantic query vector, optimized through multi-round decoding and the cross-attention mechanism, better aligns with image features, improving the accuracy of semantic segmentation.

[0139] Specifically, first, the dot product operation is used to calculate the inner product of each updated semantic query vector and the visual feature at each pixel location in the hybrid multi-scale visual features. For example, for a semantic query vector sequence of shape N×C and a visual feature of shape H×W×C, the inner product calculation yields a matrix of shape N×H×W. Then, the calculation result is normalized using the Sigmoid function to obtain the similarity between each semantic query vector and the visual feature at each pixel location, with a shape of N×H×W and a value between 0 and 1. Finally, a threshold operation is used to convert the above similarity into N binary semantic segmentation results. Each semantic query vector corresponds to one semantic segmentation result. For example, the threshold is 0.5; values ​​greater than this threshold are considered foreground, otherwise, they are considered background.

[0140] S1502. Use the updated instance query vector to perform instance segmentation on the hybrid multi-scale visual features to obtain instance segmentation results.

[0141] Instance segmentation is performed on hybrid multi-scale visual features using updated instance query vectors. Specifically, the instance query vectors interact with the hybrid multi-scale visual features through a cross-attention mechanism, calculating the similarity between each pixel location and the instance query vector. Based on the similarity score, an instance label is assigned to each pixel, generating the instance segmentation result. The instance query vectors capture instance information in the image, ensuring that the instance segmentation result can accurately distinguish different instances within the same category. The instance query vectors, optimized through multi-round decoding and the cross-attention mechanism, better align with image features, improving the segmentation accuracy of instance boundaries.

[0142] Specifically, firstly, the dot product operation is used to calculate the inner product of each updated instance query vector and the visual feature at each pixel location in the hybrid multi-scale visual features. For example, for an instance query vector sequence of shape M×C and a visual feature of shape H×W×C, the inner product calculation yields a matrix of shape M×H×W. Then, the calculation result is normalized using the Sigmoid function to obtain the similarity between each instance query vector and the visual feature at each pixel location, with a shape of M×H×W and a value between 0 and 1. Finally, a threshold operation is used to convert the above similarity into M binary instance segmentation results. Each instance query vector corresponds to one instance segmentation result; for example, a threshold of 0.5 is used, where values ​​greater than this threshold are foreground, and otherwise are background.

[0143] S1503. Calculate the inner product of each updated instance query vector and each updated semantic query vector and normalize it to obtain a similarity score. Select the semantic category corresponding to the semantic query vector with the highest similarity score for each updated instance query vector as the category classification result of the instance.

[0144] The updated semantic query vectors are used as category prototypes, with each semantic query vector corresponding to a semantic category. The dot product of each updated instance query vector and each updated semantic query vector is calculated to obtain a similarity score. The similarity scores are normalized (e.g., using the Softmax function) to obtain the similarity probability between each instance query vector and each semantic category. For each instance query vector, the semantic category corresponding to the semantic query vector with the highest similarity score is selected as the category classification result for that instance. By calculating the similarity between instance query vectors and semantic query vectors, each instance can be accurately classified into its corresponding semantic category. The instance classification results are highly consistent with the semantic segmentation results, ensuring that the category of each instance is consistent with its category in semantic segmentation.

[0145] Specifically, first, the dot product operation is used to calculate the inner product of each updated instance query vector and each updated semantic query vector. For example, for an instance query vector sequence of shape M×C and a semantic query vector of shape N×C, the inner product calculation yields a matrix of shape M×N. Then, the calculation result is normalized using the Softmax function to obtain the similarity score between each updated instance query vector and each updated semantic query vector, with a shape of M×H×W and values ​​between 0 and 1. Finally, for each updated instance query vector, the semantic category corresponding to the semantic query vector with the highest similarity score is selected as the category classification result for that instance. If the last vector among the N semantic query vectors corresponds to the category "non-target," meaning the instance segmentation result does not belong to any specified semantic category, such instance segmentation results will be filtered out and not output or saved.

[0146] Reference Figure 2 As shown, for semantic segmentation tasks, the semantic segmentation results are stored using multiple binary masks, with each semantic category corresponding to one binary mask (i.e., ...). Figure 2 The semantic mask in the model represents the background, where pixels with a value of 0 represent the background and pixels with a value of 1 represent the foreground. For instance segmentation results, the segmentation results are stored using multiple binary masks and multiple category labels, with each instance segmentation result corresponding to one binary mask (i.e., ...). Figure 2 The mask contains an instance mask and an instance category label, where pixels with a value of 0 in the mask represent the background and pixels with a value of 1 represent the foreground. Each category label corresponds to a specific semantic category.

[0147] The query vectors optimized through multi-round decoding and cross-attention mechanisms (i.e., the updated semantic query vectors and instance query vectors) can better align with image features. This invention utilizes the updated semantic query vectors and instance query vectors for semantic segmentation, instance segmentation, and instance classification, respectively, significantly improving the accuracy of these processes. The synergistic effect of the semantic query vectors and instance query vectors ensures the consistency between the segmentation and classification results. Hybrid multi-scale visual features combine high-level semantic information with low-level detail information, enabling the model to capture more local details and improve the accuracy of segmentation boundaries. The combination of hybrid multi-scale visual features and multi-round decoding mechanisms allows the model to capture more detailed information.

[0148] The general document image segmentation method based on query decoupling and prototype classification of this invention involves training the general document image segmentation model with three loss functions: semantic segmentation loss, instance segmentation loss, and instance classification loss. The semantic segmentation loss function... for:

[0149]

[0150] in, and These are the predictive semantic mask and the labeled semantic mask. and These are the focal loss function and the Dice loss function, respectively. and These are hyperparameters used to measure the weights of the two losses; in this embodiment... and Take 10 and 1 respectively.

[0151] Similarly, instance segmentation loss function for:

[0152]

[0153] in, and These are the predicted instance mask and the labeled instance mask. and These are the focal loss function and the Dice loss function, respectively. and These are hyperparameters used to measure the weights of the two losses; in this embodiment... and Take 10 and 1 respectively.

[0154] Additionally, the instance classification loss function for:

[0155]

[0156] in, and These are predicting instance categories and labeling instance categories. It is a multi-class cross-entropy loss function.

[0157] Finally, the overall loss function of the model. for:

[0158]

[0159] in, , and These are the weight coefficients for semantic segmentation loss, instance segmentation loss, and instance classification loss, respectively. In this embodiment, they are set to 5, 5, and 1, respectively.

[0160] To achieve better segmentation results, this invention performs loss calculations on the output of each layer of the hybrid query decoder. Furthermore, following Transformer-based object detection and instance segmentation methods such as DETR and Mask2Former, this invention also employs bipartite graph matching and the Hungarian algorithm to find the best-matching ground truth label for each instance segmentation result for loss calculation. It's important to note that for semantic segmentation tasks, the above matching process is unnecessary because semantic segmentation inherently involves one-to-one prediction results, i.e., each semantic category corresponds to a segmentation result mask. Unlike existing methods, the novel design proposed in this invention enables the application of heterogeneous hybrid datasets (such as...) Figure 2A general-purpose document image segmentation model can be directly trained on datasets 1, 2, ..., N, without needing to train a separate model for each task and dataset. After training, this model can be used directly as a multi-functional document image segmenter, or as a pre-trained model that can be seamlessly fine-tuned for task-specific datasets without any special modifications, such as adding or replacing linear classification layers. This design gives the model the ability to learn continuously and incrementally, making it more flexible and adaptable to new tasks.

[0161] Figures 4 to 7 Examples of segmentation results of the method of this invention on multiple public document datasets under multiple document segmentation tasks are presented. Among them, Figure 4 This is an example of the results of the general document image segmentation method based on query decoupling and prototype classification of the present invention on modern document layout segmentation tasks; Figure 5 This is an example of the results of the general document image segmentation method based on query decoupling and prototype classification of the present invention on the task of segmenting ancient books and handwritten documents; Figure 6 This is an example of the results of the general document image segmentation method based on query decoupling and prototype classification of the present invention on table detection and structure recognition tasks; Figure 7 This document provides an example of the results of the general document image segmentation method based on query decoupling and prototype classification in the present invention on a scene text segmentation task. The results show that the method of the present invention can effectively handle various document segmentation tasks in various types of document images, covering not only the segmentation of text regions of arbitrary shapes at multiple granularities from characters and text lines to paragraphs, but also the segmentation of other types of document regions including tables, formulas, and illustrations. Furthermore, this method is applicable to different types of document images, such as modern documents, ancient books, handwritten documents, scene images, scanned documents, and photographed documents. Most importantly, for the various document segmentation tasks mentioned above, the method of the present invention only requires training and maintaining a single unified model, eliminating the need to design and maintain specific models for each task. This feature significantly reduces development and maintenance costs, while also improving the convenience and efficiency of model use.

[0162] Figure 4 The samples in the middle are all from the M6Doc dataset; Figure 5 The first two samples in the first row are from the SCUT-CAB dataset, and the last three are from the MTHv2 dataset. The first three samples in the second row are from the CASIA-HWDB dataset, and the last two are from the SCUT-HCCDoc dataset. Figure 6 The first row of the samples is from the ICDAR 2019 cTDaR dataset, and the second row is from the WTW dataset; Figure 7The first row of the sample data comes from the ICDAR 2017 RCTW dataset, and the second row comes from the ICDAR 2019 LSVT dataset.

[0163] The following describes the general document image segmentation apparatus based on query decoupling and prototype classification provided by the present invention. The general document image segmentation apparatus based on query decoupling and prototype classification described below can be referred to in correspondence with the general document image segmentation method based on query decoupling and prototype classification described above.

[0164] The general document image segmentation device based on query decoupling and prototype classification provided by this invention refers to... Figure 8 As shown, it includes the following modules:

[0165] The acquisition module 210 is used to acquire the input document image and the text description of each category of the region to be segmented;

[0166] Extraction module 220 is used to extract multi-scale visual features from the input document image using a pre-built visual skeleton network, and to optimize the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features.

[0167] The encoding module 230 is used to encode the text description of each category of the region to be segmented into a semantic query vector using a text encoder, and randomly initialize a preset number of instance query vectors.

[0168] The decoding module 240 is used to input the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-constructed hybrid query decoder, and use the semantic query vector and the instance query vector to perform multiple rounds of decoding on the optimized multi-scale visual features using a cross-attention mechanism to obtain updated semantic query vectors and updated instance query vectors.

[0169] The segmentation module 250 is used to perform semantic segmentation and instance segmentation using the updated semantic query vector and the updated instance query vector respectively, to obtain semantic segmentation results and instance segmentation results, and to classify the segmented instances using the updated semantic query vector as a category prototype to obtain the category classification result of the instance.

[0170] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions from the memory 330 to execute a general document image segmentation method based on query decoupling and prototype classification.

[0171] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the general document image segmentation method based on query decoupling and prototype classification provided by the above methods.

[0173] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the general document image segmentation method based on query decoupling and prototype classification provided by the above methods.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A general document image segmentation method based on query decoupling and prototype classification, characterized in that, include: Obtain the input document image and a text description of each category of the region to be segmented; Multi-scale visual features are extracted from the input document image using a pre-built visual skeleton network, and the extracted multi-scale visual features are optimized using a pre-built deformable attention network to obtain optimized multi-scale visual features. A text encoder is used to encode the text description of each category of region to be segmented into a semantic query vector, and a preset number of instance query vectors are randomly initialized. The optimized multi-scale visual features, the semantic query vector, and the instance query vector are input into a pre-built hybrid query decoder. The optimized multi-scale visual features are decoded in multiple rounds using the semantic query vector and the instance query vector through a cross-attention mechanism to obtain the updated semantic query vector and the updated instance query vector. Semantic segmentation and instance segmentation are performed using the updated semantic query vector and the updated instance query vector, respectively, to obtain semantic segmentation results and instance segmentation results. The updated semantic query vector is used as a category prototype to classify the segmented instance to obtain the category classification result of the instance. The deformable attention network consists of multiple convolutional layers; the convolutional layers are used to convert the dimensions of the multi-scale visual features extracted by the visual skeleton network to the same value; another convolutional layer is used to predict multiple coordinate offsets of each pixel position at multiple scales for each scale of visual features, and these offsets are used to indicate which neighboring pixel features should interact with the features at each pixel position. Based on the offset prediction results, a specified number of neighboring nodes and their features are extracted on feature layers at multiple scales for each pixel location at each scale. The attention mechanism is used to interact with the features of each pixel and its neighboring nodes, thereby updating the features at the current pixel position. The calculation process is repeated multiple times, corresponding to multiple deformable attention network layers, to further optimize the multi-scale visual features, and finally obtain the above-mentioned optimized multi-scale visual features. The specific calculation process of the multi-scale deformable attention mechanism is as follows: ； in, Indicates the currently queried pixel. This represents the characteristics of the pixel. This represents the normalized coordinates of the pixel. This represents a feature layer with L scales. Indicates the first One point of attention, Indicates the number of attention heads. Indicates the first Each scale of feature layer, Represents the current pixel's... Each neighboring sampled pixel, This indicates the number of neighboring sampled pixels. and For learnable weight parameters, and They represent the first The first one in the attention. In the feature layer of scale, the first The sampling offset and attention weight of each sampling point The function represents remapping the normalized coordinates to the first... The original coordinates of the layer; The optimized multi-scale visual features, the semantic query vector, and the instance query vector are input into a pre-constructed hybrid query decoder. A cross-attention mechanism is used to decode the optimized multi-scale visual features multiple times using the semantic query vector and the instance query vector, resulting in updated semantic query vectors and updated instance query vectors, including: The semantic query vector and the instance query vector are subjected to a first information exchange to obtain the semantic query vector and the instance query vector after the exchange. By using a multi-head cross-attention mechanism and a feedforward neural network, the semantic query vector after the interaction is used to decode the optimized multi-scale visual features to obtain a semantic query vector that integrates multi-scale features. The optimized multi-scale visual features are decoded using the interactive instance query vector through a multi-head cross-attention mechanism and a feedforward neural network to obtain an instance query vector that integrates the multi-scale features. The semantic query vector that incorporates multi-scale features and the instance query vector that incorporates multi-scale features are subjected to a second information exchange to obtain the updated semantic query vector and the updated instance query vector.

2. The general document image segmentation method based on query decoupling and prototype classification according to claim 1, characterized in that, The process of extracting multi-scale visual features from the input document image using a pre-constructed visual skeleton network and optimizing the extracted multi-scale visual features using a pre-constructed deformable attention network to obtain optimized multi-scale visual features includes: The input document image is normalized in terms of scale and pixel values ​​to obtain a normalized document image; Multi-scale visual features are extracted from the normalized document image using a pre-built visual skeleton network. The multi-scale visual features are optimized using a pre-constructed deformable attention network to obtain optimized multi-scale visual features.

3. The general document image segmentation method based on query decoupling and prototype classification according to claim 1, characterized in that, The text encoder includes a word segmenter, an embedding layer, and an encoding layer; the step of using the text encoder to encode the text description of each category of regions to be segmented into a semantic query vector, and randomly initializing a preset number of instance query vectors, includes: The word segmenter is used to segment the text description of each category of regions to be segmented, resulting in a token sequence; The embedding layer is used to map the token sequence into a vector sequence; The vector sequence is encoded using the encoding layer to obtain the encoded token feature sequence; The global average of the encoded token feature sequence along the sequence dimension is calculated to obtain the semantic query vector for each category of the region to be segmented. A preset number of instance query vectors are randomly initialized, and the dimension of each instance query vector is the same as the dimension of the semantic query vector.

4. The general document image segmentation method based on query decoupling and prototype classification according to claim 1, characterized in that, The method further includes: fusing the optimized multi-scale visual features using a feature pyramid network to obtain hybrid multi-scale visual features.

5. The general document image segmentation method based on query decoupling and prototype classification according to claim 4, characterized in that, The process of performing semantic segmentation and instance segmentation using the updated semantic query vector and the updated instance query vector, respectively, to obtain semantic segmentation results and instance segmentation results, and then using the updated semantic query vector as a category prototype to classify the segmented instances to obtain the category classification result of the instance, includes: The updated semantic query vector is used to perform semantic segmentation on the hybrid multi-scale visual features to obtain the semantic segmentation result; The updated instance query vector is used to perform instance segmentation on the hybrid multi-scale visual features to obtain the instance segmentation result. Calculate the inner product of each updated instance query vector and each updated semantic query vector, and normalize it to obtain a similarity score. Select the semantic category corresponding to the semantic query vector with the highest similarity score for each updated instance query vector as the category classification result of the instance.

6. A general document image segmentation device based on query decoupling and prototype classification, characterized in that, include: The acquisition module is used to acquire the input document image and the text description of each category of the region to be segmented; The extraction module is used to extract multi-scale visual features from the input document image using a pre-built visual skeleton network, and to optimize the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features. The encoding module is used to encode the text description of each category of the region to be segmented into a semantic query vector using a text encoder, and randomly initialize a preset number of instance query vectors. The decoding module is used to input the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-constructed hybrid query decoder, and use the semantic query vector and the instance query vector to perform multiple rounds of decoding on the optimized multi-scale visual features using a cross-attention mechanism to obtain updated semantic query vectors and updated instance query vectors. The segmentation module is used to perform semantic segmentation and instance segmentation using the updated semantic query vector and the updated instance query vector respectively, to obtain semantic segmentation results and instance segmentation results, and to classify the segmented instances using the updated semantic query vector as a category prototype to obtain the category classification result of the instance. The deformable attention network consists of multiple convolutional layers; the convolutional layers are used to convert the dimensions of the multi-scale visual features extracted by the visual skeleton network to the same value; another convolutional layer is used to predict multiple coordinate offsets of each pixel position at multiple scales for each scale of visual features, and these offsets are used to indicate which neighboring pixel features should interact with the features at each pixel position. Based on the offset prediction results, a specified number of neighboring nodes and their features are extracted on feature layers at multiple scales for each pixel location at each scale. The attention mechanism is used to interact with the features of each pixel and its neighboring nodes, thereby updating the features at the current pixel position. The calculation process is repeated multiple times, corresponding to multiple deformable attention network layers, to further optimize the multi-scale visual features, and finally obtain the above-mentioned optimized multi-scale visual features. The specific calculation process of the multi-scale deformable attention mechanism is as follows: ； in, Indicates the currently queried pixel. This represents the characteristics of the pixel. This represents the normalized coordinates of the pixel. This represents a feature layer with L scales. Indicates the first One point of attention, Indicates the number of attention heads. Indicates the first Each scale of feature layer, Represents the current pixel's... Each neighboring sampled pixel, This indicates the number of neighboring sampled pixels. and For learnable weight parameters, and They represent the first The first one in the attention. In the feature layer of scale, the first The sampling offset and attention weight of each sampling point The function represents remapping the normalized coordinates to the first... The original coordinates of the layer; The optimized multi-scale visual features, the semantic query vector, and the instance query vector are input into a pre-constructed hybrid query decoder. A cross-attention mechanism is used to decode the optimized multi-scale visual features multiple times using the semantic query vector and the instance query vector, resulting in updated semantic query vectors and updated instance query vectors, including: The semantic query vector and the instance query vector are subjected to a first information exchange to obtain the semantic query vector and the instance query vector after the exchange. By using a multi-head cross-attention mechanism and a feedforward neural network, the semantic query vector after the interaction is used to decode the optimized multi-scale visual features to obtain a semantic query vector that integrates multi-scale features. The optimized multi-scale visual features are decoded using the interactive instance query vector through a multi-head cross-attention mechanism and a feedforward neural network to obtain an instance query vector that integrates the multi-scale features. The semantic query vector that incorporates multi-scale features and the instance query vector that incorporates multi-scale features are subjected to a second information exchange to obtain the updated semantic query vector and the updated instance query vector.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the general document image segmentation method based on query decoupling and prototype classification as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the general document image segmentation method based on query decoupling and prototype classification as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the general document image segmentation method based on query decoupling and prototype classification as described in any one of claims 1 to 5.

Images