General document image segmentation method based on query decoupling and prototype classification

Abstract

The invention relates to the field of image segmentation, and provides a general document image segmentation method based on query decoupling and prototype classification, and the method comprises the steps: extracting multi-scale visual features of an input document image through employing a visual skeleton network, and carrying out the optimization, and obtaining the optimized multi-scale visual features; encoding the character description of each to-be-segmented region category into a semantic query vector by using a text encoder, and randomly initializing a preset number of instance query vectors; inputting the optimized multi-scale visual features, the semantic query vector and the instance query vector into a mixed query decoder to obtain an updated semantic query vector and an updated instance query vector; and respectively carrying out semantic segmentation and instance segmentation by utilizing the updated semantic query vector and instance query vector to obtain a semantic segmentation result and an instance segmentation result. According to the method, various segmentation tasks on various types of document images can be processed, and a corresponding segmentation result can be output by taking any document image and segmentation task prompt as input.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a general document image segmentation method based on query decoupling and prototype classification. Background Art

[0002] Document Image Segmentation (DIS) is a basic task in the field of Document Analysis and Recognition (DAR). It aims to decompose complex document image content into smaller and more meaningful components, such as text regions, pictures, tables, and other elements, so as to serve downstream applications such as text recognition, information extraction (IE), and document visual question answering (DocVQA). By accurately segmenting text regions, pictures, tables, and other elements, the accuracy and efficiency of these technologies can be improved, allowing machines to better understand and parse the information in documents, which is crucial for automated document processing. Although document image segmentation plays an important role in document processing, it still faces many challenges due to the diversity of document types, the complexity of page layouts, the heterogeneity of content annotations, and the complexity of document structures.

[0003] Existing document image segmentation methods can be mainly divided into two categories: rule-based methods and machine learning-based methods. Rule-based methods segment document images through predefined rules and templates. These methods work well when processing simple and regular documents, but perform poorly when processing complex and diverse documents. Machine learning-based methods, especially deep learning-based methods, have made significant progress in recent years. These methods automatically extract features and patterns by learning from a large amount of annotated data, thereby achieving more accurate segmentation. Specifically, these methods roughly include the following categories: object detection-based methods, instance segmentation-based methods, graph model-based methods, and Transformer-based methods. Among them, the target detection-based method converts the document image segmentation task into an object detection problem, uses Faster R-CNN, Mask R-CNN, YOLO and other methods to detect different areas in the document and generate corresponding bounding boxes to achieve the segmentation of text, tables, pictures and other contents; the instance segmentation-based method converts the document image segmentation task into an image segmentation problem, uses Fully Convolutional Networks (FCN) and U-Net and other methods to classify the document pixel level, and achieves the segmentation of different areas in the document image; the graph model-based method regards the document as a hierarchical graph structure, uses Graph Neural Networks (GNN) to model the structural relationship of the document, and achieves the processing of layout analysis, table structure recognition and text detection tasks; the Transformer-based method uses Transformer-based target detection or instance segmentation methods to detect or segment areas in document images. In addition, some multimodal methods combine visual, text and layout features, such as LayoutLMv3. These methods improve the performance and generalization ability of the model by pre-training on large-scale unsupervised documents and aligning the features of different modalities.

[0004] A large number of document image segmentation datasets have been accumulated in the field of document image segmentation, which show great diversity and heterogeneity in terms of data sources, document types, written languages, processing tasks, annotation categories, annotation formats, etc. For example, PubLayNet contains digitally generated English PDF documents with region-level border annotations for layout segmentation tasks of modern documents; SCUT-CAB and MTHv2 contain scanned historical Chinese documents with region-, line-, and character-level border annotations for region segmentation, text line extraction, and character detection tasks of ancient documents; SCUT-HCCDoc contains handwritten documents with line-level border annotations for handwritten text line segmentation tasks; M6Doc contains documents of various types and languages ​​with region-level border annotations for layout segmentation tasks of multi-type documents; CTW1500 and Total-Text contain natural scene images with arbitrary-shaped text border annotations for text detection and segmentation tasks in natural scenes.

[0005] However, existing deep learning-based document image segmentation methods usually only focus on a specific segmentation subtask and dataset, converting various document image segmentation tasks into general object detection or image segmentation problems, and making some targeted modifications to the object detection and image segmentation methods in general scenarios according to the characteristics of the current tasks and data to make them more suitable for specific tasks and datasets. Although such specialized and customized solutions perform well on specific tasks, they are unable to cope with comprehensive document segmentation tasks and often lack versatility and flexibility. For example, the table detection model trained on PDF document images cannot complete the text line segmentation task on handwritten documents. In addition, such customization not only increases the complexity of the system, but also leads to redundant computing and storage overhead. Users need to design, develop and maintain a set of corresponding algorithms and models for each task and dataset, making these methods very inefficient when deployed on a large scale in real scenarios. Summary of the invention

[0006] The present invention provides a general document image segmentation method based on query decoupling and prototype classification, which is used to solve the limitations of existing document image segmentation methods, especially the defects of insufficient algorithm adaptability and generalization ability when facing diverse segmentation tasks and annotation data sets. It can not only handle various segmentation tasks on various types of document images, but also take any document image and segmentation task prompt as input, output corresponding segmentation results, and improve the versatility and flexibility of the model. The technical solutions proposed by the present invention are as follows: In a first aspect, the present invention provides a general document image segmentation method based on query decoupling and prototype classification, comprising: Obtain an input document image and a text description of each category of the region to be segmented; Extracting multi-scale visual features from the input document image using a pre-built visual skeleton network, and optimizing the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features; A text encoder is used to encode the text description of each category of the region to be segmented into a semantic query vector, and a preset number of instance query vectors are randomly initialized; Inputting the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-built hybrid query decoder, and performing multiple rounds of decoding on the optimized multi-scale visual features using the semantic query vector and the instance query vector using a cross-attention mechanism to obtain an updated semantic query vector and an updated instance query vector; The updated semantic query vector and the updated instance query vector are used to perform semantic segmentation and instance segmentation respectively to obtain semantic segmentation results and instance segmentation results, and the updated semantic query vector is used as a category prototype to classify the segmented instance to obtain a category classification result of the instance.

[0007] Optionally, the extracting multi-scale visual features from the input document image using a pre-built visual skeleton network, and optimizing the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features, comprises: Performing scale and pixel value normalization processing on the input document image to obtain a normalized document image; Using a pre-built visual skeleton network to extract multi-scale visual features from the normalized document image to obtain multi-scale visual features; The multi-scale visual features are optimized using a pre-built deformable attention network to obtain optimized multi-scale visual features.

[0008] Optionally, the text encoder includes a word segmenter, an embedding layer and an encoding layer; the text encoder is used to encode the text description of each category of the region to be segmented into a semantic query vector, and a preset number of instance query vectors are randomly initialized, including: Use the word segmenter to segment the text description of each category of the area to be segmented to obtain a token sequence; Mapping the token sequence into a vector sequence using the embedding layer; Encode the vector sequence using the encoding layer to obtain an encoded token feature sequence; Calculating the global average of the encoded token feature sequence in the sequence dimension to obtain a semantic query vector for each category of the region to be segmented; A preset number of instance query vectors are randomly initialized, and the dimension of each instance query vector is consistent with the dimension of the semantic query vector.

[0009] Optionally, the optimized multi-scale visual features, the semantic query vector, and the instance query vector are input into a pre-built hybrid query decoder, and the optimized multi-scale visual features are decoded multiple times using the semantic query vector and the instance query vector by using a cross attention mechanism to obtain an updated semantic query vector and an updated instance query vector, including: Performing a first information interaction between the semantic query vector and the instance query vector to obtain an interacted semantic query vector and an interacted instance query vector; Using the interactive semantic query vector to decode the optimized multi-scale visual features using a multi-head cross attention mechanism and a feedforward neural network, a semantic query vector integrating multi-scale features is obtained; Using the interacted instance query vector to decode the optimized multi-scale visual features using a multi-head cross attention mechanism and a feedforward neural network, an instance query vector integrating multi-scale features is obtained; The semantic query vector integrated with the multi-scale features and the instance query vector integrated with the multi-scale features are subjected to a second information exchange to obtain an updated semantic query vector and an updated instance query vector.

[0010] Optionally, the method further includes: using a feature pyramid network to fuse the optimized multi-scale visual features to obtain mixed multi-scale visual features.

[0011] Optionally, the using the updated semantic query vector and the updated instance query vector to perform semantic segmentation and instance segmentation respectively to obtain a semantic segmentation result and an instance segmentation result, and using the updated semantic query vector as a category prototype to classify the segmented instance to obtain a category classification result of the instance, includes: Using the updated semantic query vector to perform semantic segmentation on the hybrid multi-scale visual feature to obtain a semantic segmentation result; Using the updated instance query vector to perform instance segmentation on the hybrid multi-scale visual feature to obtain an instance segmentation result; The inner product of each updated instance query vector and each updated semantic query vector is calculated and normalized to obtain a similarity score, and for each updated instance query vector, the semantic category corresponding to the semantic query vector with the highest similarity score is selected as the category classification result of the instance.

[0012] In a second aspect, the present invention further provides a general document image segmentation device based on query decoupling and prototype classification, comprising the following modules: An acquisition module, used for acquiring an input document image and a text description of each category of the region to be segmented; An extraction module, configured to extract multi-scale visual features from the input document image using a pre-built visual skeleton network, and optimize the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features; An encoding module, used for encoding the text description of each category of the region to be segmented into a semantic query vector by using a text encoder, and randomly initializing a preset number of instance query vectors; A decoding module, configured to input the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-built hybrid query decoder, and use the semantic query vector and the instance query vector to perform multiple rounds of decoding on the optimized multi-scale visual features using a cross-attention mechanism to obtain an updated semantic query vector and an updated instance query vector; A segmentation module is used to use the updated semantic query vector and the updated instance query vector to perform semantic segmentation and instance segmentation respectively to obtain semantic segmentation results and instance segmentation results, and use the updated semantic query vector as a category prototype to classify the segmented instance to obtain a category classification result of the instance.

[0013] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the general document image segmentation method based on query decoupling and prototype classification as described in the first aspect above is implemented.

[0014] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the general document image segmentation method based on query decoupling and prototype classification as described in the first aspect above.

[0015] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the general document image segmentation method based on query decoupling and prototype classification as described in the first aspect above.

[0016] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows: The general document image segmentation method based on query decoupling and prototype classification provided by the present invention uses a visual skeleton network to extract multi-scale visual features of an input document image, which can capture different sizes and detail information in the image, thereby enhancing the processing capability of different types of document images. By introducing a deformable attention network to optimize the extracted multi-scale visual features, it is possible to more flexibly focus on key areas in the image, further improving the representation capability of the features. A text encoder is used to encode the text description of each category of the area to be segmented into a semantic query vector. This method enables understanding and responding to different segmentation task prompts, thereby achieving support for multiple segmentation tasks. By randomly initializing a preset number of instance query vectors, multiple instances in the image can be processed to achieve instance-level segmentation. The optimized multi-scale visual features, semantic query vectors, and instance query vectors are input into a hybrid query decoder, and multiple rounds of decoding are performed using a cross-attention mechanism. This process enables the model to dynamically adjust its attention and focus on areas in the image that are critical to the segmentation task, thereby improving the accuracy of segmentation. At the same time, the cross-attention mechanism can also effectively integrate information from different sources and enhance the decoding capability of the model. The updated semantic query vector and instance query vector are used for semantic segmentation and instance segmentation respectively, which can simultaneously handle semantic-level segmentation and instance-level segmentation to meet the needs of multiple segmentation tasks. The updated semantic query vector is used as the category prototype to classify the segmented instance. This method enables the model to automatically determine the category of the instance based on the extracted features, further improving the versatility and flexibility of the model.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic flow chart of a general document image segmentation method based on query decoupling and prototype classification provided by the present invention; Figure 2 An algorithm framework diagram of a general document image segmentation method based on query decoupling and prototype classification provided by the present invention; Figure 3 A model structure diagram of an embodiment of a general document image segmentation method based on query decoupling and prototype classification provided by the present invention; Figure 4 An example of the results of the general document image segmentation method based on query decoupling and prototype classification provided by the present invention on the task of modern document layout segmentation; Figure 5 The results of the general document image segmentation method based on query decoupling and prototype classification provided by the present invention on the ancient book and handwritten document segmentation tasks are shown in the following examples; Figure 6 The results of the general document image segmentation method based on query decoupling and prototype classification provided by the present invention on table detection and structure recognition tasks are shown in the following examples; Figure 7 An example of the results of the general document image segmentation method based on query decoupling and prototype classification provided by the present invention on the scene text segmentation task; Figure 8 A schematic diagram of the structure of a general document image segmentation device based on query decoupling and prototype classification provided by the present invention; Fig. 9 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Combine the following Figure 1-Figure 7 The general document image segmentation method based on query decoupling and prototype classification of the present invention is described.

[0023] The present invention aims to solve the limitations of existing document image segmentation methods, especially the problem of insufficient algorithm adaptability and generalization ability when facing diverse segmentation tasks and annotation data sets. Existing document image segmentation methods usually only focus on a specific subtask and data set, and design targeted algorithms and models based on the characteristics of the current task and data, which lacks versatility and flexibility. In addition, this customization not only increases the complexity of the system, but also leads to redundant computing and storage overhead. Users need to design, develop and maintain a set of corresponding algorithms and models for each task and data set, making these methods very inefficient when deployed on a large scale in real scenarios.

[0024] Focusing on the above key problems, the present invention proposes a general document image segmentation method based on query decoupling and prototype classification, focusing on overcoming the above technical obstacles. The core advantage of this method lies in its high adaptability and robustness. It can handle various types of document segmentation tasks, such as layout segmentation, text line extraction, character detection, table structure analysis and scene text detection, and is suitable for modern document images, ancient book document images, natural scene images, digitally generated document images, scanned document images and photographed document images. In addition, the method can directly use existing multiple heterogeneous mixed data sets for learning and training, thereby minimizing the need for manual annotation of new data. The general document image segmentation method based on query decoupling and prototype classification, by introducing query decoupling mechanism, prototype classification, multi-task learning and hybrid supervised learning, can not only handle multiple segmentation tasks on various types of document images, but also can take any document image and segmentation task prompt as input, output corresponding segmentation results, improve the versatility, flexibility and efficiency of the model, and provide strong technical support for subsequent text recognition, form understanding, information extraction, document question and answer and other fields, further improve the automation level of document processing, improve user experience, and have broad application prospects.

[0025] Query decoupling refers to decomposing the input query (i.e., input document image and segmentation task prompt) into multiple independent components or features. In document image segmentation, this can include decoupling image content, layout, text information, etc. into different feature vectors or representations. With query decoupling, the model is able to handle different types of document images and segmentation tasks more flexibly.

[0026] Prototype classification refers to classifying or segmenting inputs by learning a set of prototype representations (i.e., representative or typical feature vectors). In document image segmentation, these prototypes can represent different types of document elements (such as text blocks, tables, pictures, etc.). Through prototype classification, the model can recognize and segment multiple types of document elements, rather than being limited to a specific document type or segmentation task. This allows the model to be more widely applied to different types of document image segmentation tasks.

[0027] Reference Figure 1 As shown, the method includes the following: Step S110: obtaining an input document image and a text description of each category of the region to be segmented.

[0028] Receives the input document image provided by the user, and also receives the text description of each category of the region to be segmented. These text descriptions are used to convert the category information into a semantic query vector later. Figure 2 As shown, the input document image is a heterogeneous mixed document image, which can be a photographed document image, a modern document image, an ancient document image, a natural scene image, a scanned document image, a digitally generated document image, and other types of document images.

[0029] The text description of each category of the region to be segmented is the segmentation task prompt, which refers to a detailed text description of each category of the document region to be segmented that is set in advance, so that the model can better understand the characteristics of each category of the region. For example, for the table area, the text description is "A table is a way of organizing data in the form of rows and columns to display a structured collection of information. Each row represents a record, while each column corresponds to an attribute or feature of the record. Tables improve the readability and comparability of information through clear data arrangement, and are particularly suitable for enumeration, comparison, and statistical analysis. Tables can also contain header rows and column labels to indicate the content type of each column. In addition, tables may have borders or background colors to enhance the visual effect, making it easier for users to quickly locate and interpret data"; and for the illustration area, the text description is "Illustrations refer to visual elements used in documents to explain, supplement or decorate text content, including but not limited to photos, charts, paintings, graphics, etc. Illustrations can convey information intuitively, making complex concepts easier to understand while increasing the appeal of the document. They are often accompanied by captions or explanatory text to point out the key points of the illustration or explain its relationship to the text. The illustration is generally located adjacent to the relevant text description to ensure that the connection between the two is clear and visible."

[0030] The text descriptions of the categories of the regions to be segmented are then encoded into semantic query vectors by the text encoder, which is responsible for the semantic segmentation and instance classification tasks. By introducing the text descriptions of the categories of the regions to be segmented, the method can understand and utilize the semantic relationship between categories, improving the accuracy and generalization ability of segmentation.

[0031] Step S120, refer to Figure 2 As shown, a pre-built visual skeleton network is used to extract multi-scale visual features from the input document image, and a pre-built deformable attention network is used to optimize the extracted multi-scale visual features to obtain optimized multi-scale visual features.

[0032] Use pre-built visual skeleton networks (such as residual network ResNet, efficient network EfficientNet, etc.) to extract multi-scale visual features from the input document image. The visual skeleton network can extract hierarchical feature information from the input document image. The visual skeleton network contains multiple convolutional layers and pooling layers, which can extract feature information of different scales. In the multi-scale feature extraction process, the network gradually reduces the spatial resolution of the feature map while increasing the number of channels of the feature map, thereby capturing different scale features from low-level details to high-level semantics. The deformable attention network is a variant of the attention mechanism that allows attention weights to have a certain degree of flexibility in spatial position.

[0033] Next, the pre-built deformable attention network is used to optimize these features and enhance the ability to capture key information. Compared with the traditional fixed-position attention mechanism, the deformable attention mechanism can better adapt to irregular shapes and deformations in the image, thereby capturing key information more accurately. After extracting multi-scale visual features, the deformable attention network will optimize these features. Through the deformable attention mechanism, the network can dynamically adjust the position and range of the focus of attention, thereby enhancing the ability to capture key information. The deformable attention network optimizes features so that the model can pay more attention to important areas and details in the image. This not only improves the model's ability to recognize elements such as text, tables, and pictures in document images, but also enhances the model's ability to adapt to complex scenes and deformed objects.

[0034] Multi-scale feature extraction can capture image details at different scales, while the deformable attention network can dynamically adjust the focus of attention, improving the quality of features and adaptability to complex scenes. The combination of multi-scale visual feature extraction and deformable attention network optimization can significantly improve the accuracy and robustness of document image segmentation. By capturing feature information at different scales and optimizing the ability to capture key information, the model can better adapt to different types of document images and segmentation tasks. Since the model has stronger generalization ability, it can reduce the dependence on new data and the cost of annotation.

[0035] Step S130, refer to Figure 2 As shown, a text encoder is used to encode the text description of each category of the region to be segmented into a semantic query vector, and a preset number of instance query vectors are randomly initialized.

[0036] Reference Figure 3As shown, a text encoder (such as BERT, GPT, etc.) is used to encode the text description of each category of the region to be segmented, that is, the semantic category to be segmented (such as paragraphs, tables, illustrations, cells, characters, words, text lines, etc.) into a semantic query vector. At the same time, a preset number of instance query vectors are randomly initialized, and these vectors will be used for subsequent instance segmentation. The number of instance query vectors can be obtained empirically or statistically, and is usually not less than the maximum number of regions to be segmented in each document. The introduction of semantic query vectors and instance query vectors enables the method to handle semantic segmentation and instance segmentation tasks simultaneously, improving the flexibility and multi-task processing capabilities of the model.

[0037] Step S140: input the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-built hybrid query decoder, and use the semantic query vector and the instance query vector to perform multiple rounds of decoding on the optimized multi-scale visual features using a cross-attention mechanism to obtain an updated semantic query vector and an updated instance query vector.

[0038] The optimized multi-scale visual features, semantic query vectors, and instance query vectors are input into the pre-built hybrid query decoder. The semantic query vectors and instance query vectors are continuously updated through multiple rounds of decoding using the cross-attention mechanism. Multiple rounds of decoding and the cross-attention mechanism can gradually refine the query vectors to better align them with image features, improving segmentation accuracy and detail capture.

[0039] Specifically, the hybrid query decoder receives the optimized multi-scale visual features, the semantic query vector, and the instance query vector as input. The hybrid query decoder can be composed of multiple decoding layers (i.e. Figure 3 The semantic query vector and the instance query vector are composed of a multi-scale decoder in the image, each of which contains a self-attention mechanism and a cross-attention mechanism. In each layer of the multi-scale decoder, the semantic query vector and the instance query vector first interact through the self-attention mechanism. The self-attention mechanism allows the query vectors to influence each other and thus capture the relationship between them. Subsequently, the query vector interacts with the multi-scale visual features through the cross-attention mechanism. The cross-attention mechanism allows the query vector to extract relevant information from the image features and update itself based on the image content. During multiple rounds of decoding, the semantic query vector and the instance query vector are continuously updated through the self-attention and cross-attention mechanisms of each layer. Each round of decoding enables the query vector to more accurately capture the semantic and instance information in the image. As the number of decoding layers increases, the query vector gradually refines from coarse global information to specific local details. This progressive refinement process helps the model better align image features, thereby improving segmentation accuracy.

[0040] The criss-cross attention mechanism dynamically selects the image region that is most relevant to the query vector by calculating the similarity between the query vector and the image features. This mechanism can effectively align the query vector with the image features, ensuring that the model can capture details related to the query. The criss-cross attention mechanism can also capture contextual information in the image and help the model understand the relationship between objects. For example, in the segmentation task, the model can identify the relative position relationship between objects through the criss-cross attention mechanism, thereby improving the accuracy of segmentation.

[0041] Through multiple rounds of decoding and cross-attention mechanism, the hybrid query decoder can gradually refine the semantic query vector and instance query vector to better align them with image features. This mechanism not only improves the accuracy of segmentation, but also captures more detailed information and is suitable for a variety of complex visual tasks. After multiple rounds of decoding, the final semantic query vector and instance query vector will be used to generate segmentation results. The semantic query vector is used to predict the category of each pixel, while the instance query vector is used to distinguish different instances.

[0042] Step S150, using the updated semantic query vector and the updated instance query vector to perform semantic segmentation and instance segmentation respectively to obtain semantic segmentation results and instance segmentation results, and using the updated semantic query vector as a category prototype to classify the segmented instance to obtain a category classification result of the instance.

[0043] The updated semantic query vector is used to perform semantic segmentation to obtain the category label of each pixel. At the same time, the updated instance query vector is used to perform instance segmentation to obtain the bounding box and mask of each instance. In addition, the updated semantic query vector is used as the category prototype to classify the segmented instance and obtain the category classification result of the instance. By combining semantic segmentation, instance segmentation and category classification, the structure and content of the document image can be fully understood, providing strong support for subsequent tasks such as text recognition and form understanding. At the same time, the introduction of prototype classification improves the accuracy and robustness of classification.

[0044] Existing methods often need to design targeted algorithms and models according to the characteristics of the current task and data, which increases the complexity of the system and leads to redundant computing and storage overhead. The method of the present invention, by constructing a universal segmentation framework (including the above-mentioned visual skeleton network, deformable attention network, text encoder, hybrid query decoder, which constitute the universal document image segmentation model of the present invention), uses multi-scale visual feature extraction, deformable attention optimization and other technologies to achieve effective capture and processing of multiple document image features, reducing the need for algorithm customization. Users do not need to design, develop and maintain a set of corresponding algorithms and models for each task and data set, thereby reducing the complexity and maintenance cost of the system.

[0045] The general document image segmentation method based on query decoupling and prototype classification proposed by the present invention uses a visual skeleton network to extract multi-scale visual features of the input document image, which can capture different sizes and details in the image, thereby enhancing the processing ability of different types of document images. By introducing a deformable attention network to optimize the extracted multi-scale visual features, it can more flexibly focus on the key areas in the image, and further improve the representation ability of the features. The text encoder is used to encode the text description of each category of the area to be segmented into a semantic query vector. This method enables understanding and responding to different segmentation task prompts, thereby achieving support for multiple segmentation tasks. By randomly initializing a preset number of instance query vectors, multiple instances in the image can be processed to achieve instance-level segmentation. The optimized multi-scale visual features, semantic query vectors and instance query vectors are input into a hybrid query decoder, and multiple rounds of decoding are performed using a cross-attention mechanism. This process enables the above-mentioned general document image segmentation model to dynamically adjust its attention and focus on the areas in the image that are critical to the segmentation task, thereby improving the accuracy of segmentation. At the same time, the cross-attention mechanism can also effectively integrate information from different sources and enhance the decoding ability of the model. The updated semantic query vector and instance query vector are used for semantic segmentation and instance segmentation respectively, which can handle semantic-level segmentation and instance-level segmentation at the same time and meet the needs of various segmentation tasks. The updated semantic query vector is used as the category prototype to classify the segmentation instance. This method enables the model to automatically determine the category of the instance based on the extracted features, further improving the versatility and flexibility of the model, reducing the development and maintenance costs, and thus improving the efficiency of algorithm deployment. Since this method uses a text encoder and a randomly initialized instance query vector, it can handle the input of any document image and segmentation task prompt. This makes the model extremely versatile and flexible. By optimizing multi-scale visual features, using a cross-attention mechanism for decoding, and combining multiple steps such as semantic segmentation, instance segmentation, and classification, this method can improve the efficiency of the model while ensuring segmentation accuracy.

[0046] Due to the combination of query decoupling and prototype classification, the model is able to accept any type of document image and segmentation task prompt as input. This means that users can flexibly specify the segmentation task as needed without additional training or tuning of the model. The model is able to output the corresponding segmentation results based on the input task prompt and document image content. These results can be pixel-level segmentation masks, bounding boxes, or other forms of representations, depending on the requirements of the task prompt. Due to the versatility and flexibility of the model, the same underlying feature extraction network can be shared on multiple tasks. This reduces the cost and time of separate training for different tasks.

[0047] The general document image segmentation method based on query decoupling and prototype classification proposed in the present invention realizes comprehensive adaptation to different segmentation tasks and various types of document images by introducing semantic query vectors and instance query vectors, as well as a hybrid query decoder. Existing methods usually only focus on a specific subtask and data set, lacking versatility and flexibility. The method of the present invention can handle various types of document segmentation tasks, such as layout segmentation, text line extraction, character detection, table structure analysis, and scene text detection, and is applicable to various types of document images such as modern document images, ancient book document images, and natural scene images. This high degree of adaptability and generalization ability enables the method of the present invention to show more outstanding performance when facing various segmentation tasks and annotated data sets.

[0048] In addition, the general document image segmentation method based on query decoupling and prototype classification can directly use existing heterogeneous mixed data sets for learning and training, thereby minimizing the need for manual annotation of new data. By introducing query decoupling mechanism, prototype classification, multi-task learning and hybrid supervised learning, the learned model can not only handle a variety of document segmentation tasks, but also take any document image and segmentation task prompt as input and output the corresponding segmentation results, thus improving the versatility, flexibility and efficiency of the model, providing strong technical support for subsequent text recognition, form understanding, information extraction, document question and answering and other fields, further improving the level of automation of document processing, improving user experience, and having broad application prospects.

[0049] In an optional embodiment, in order to ensure the consistency of the input document image in terms of scale and pixel value, and to facilitate subsequent feature extraction and optimization processing, refer to Figure 3 As shown, the present invention also preprocesses the input document image, including scale and pixel value normalization processing. The above step S120 uses a pre-built visual skeleton network to extract multi-scale visual features from the input document image, and uses a pre-built deformable attention network to optimize the extracted multi-scale visual features to obtain optimized multi-scale visual features, including: S1201: Perform scale and pixel value normalization processing on the input document image to obtain a normalized document image.

[0050] The input document image is scale normalized, that is, the size of the image is adjusted to meet the input requirements of the visual skeleton network. This can be achieved through a scaling algorithm, such as bilinear interpolation, nearest neighbor interpolation, etc. After the scale normalization process, the present invention further performs pixel value normalization to adjust the pixel value of the image to a specific range (such as 0 to 1 or -1 to 1) to eliminate differences in brightness, contrast, etc. between different images.

[0051] In the present invention, for the input document image, first, its long side is scaled to a specified pixel value, such as 1024 pixels, and the narrow side is scaled according to the same ratio to maintain the original aspect ratio of the document image, and then the value of each pixel of the document image is normalized to reduce the influence of the difference in different document styles. The specific normalization method can select the mean subtraction and variance division method or the maximum and minimum normalization method. In this embodiment, the specific normalization method selects the mean subtraction and variance division method, so that the normalized document image pixel mean is 0 and the variance is 1.

[0052] S1202: Use a pre-built visual skeleton network to perform multi-scale visual feature extraction on the normalized document image to obtain multi-scale visual features.

[0053] Reference Figure 3 As shown in the figure, the normalized document image is input into the visual skeleton network, and multi-scale visual features are gradually extracted through the network's convolutional layer, pooling layer and other structures. These features are represented as high-dimensional feature vectors or feature maps. For example, a multi-scale visual encoder such as a convolutional neural network (such as ResNet) or a visual transformer (such as Swin-Transformer) is used to encode the normalized document image, and then multiple visual features of specified scales are extracted at the specified layer, such as 1 / 4, 1 / 8, 1 / 16, 1 / 32 of the original image size. Multi-scale features play an important role in improving the segmentation performance of multi-scale regions.

[0054] In this embodiment, the visual skeleton network adopts Swin-Transformer, which is an innovative visual Transformer architecture. It processes image data through a hierarchical structure and a moving window mechanism, which not only retains the local advantages of convolutional neural networks, but also has the ability of Transformer long-distance dependency modeling. It divides the input into non-overlapping windows at each level, performs self-attention calculations in these windows, and moves the window positions in odd levels to ensure that there is a cross-information flow between different levels. This method improves the flexibility and efficiency of the model, making it perform well in a variety of computer vision tasks, such as image classification, target detection, and semantic segmentation. The present invention uses the visual skeleton network to extract multiple feature layers of specified scales from the normalized document image, such as 1 / 4, 1 / 8, 1 / 16, 1 / 32 of the original image size, etc., and features of different scales have different feature dimensions, such as 192, 384, 768, 1536, etc. Multi-scale features play an important role in improving the segmentation performance of multi-scale regions, enabling them to simultaneously segment large target areas and small target areas.

[0055] S1203, reference Figure 3As shown, the multi-scale visual features are optimized using a pre-built deformable attention network to obtain optimized multi-scale visual features.

[0056] The structure of the above-mentioned deformable attention network can refer to the description in the prior art, and is not specifically limited here. For example, the deformable attention network may include multiple convolutional layers. The dimensions of the multi-scale visual features extracted by the visual skeleton network are converted into the same value using a convolutional layer; then, another convolutional layer is used to predict multiple coordinate offsets of each pixel position at multiple scales for the visual features of each scale, and the offset is used to indicate which pixel features of adjacent positions the features of each pixel position should interact with for attention; then, according to the offset prediction results, a specified number of neighbor nodes and their features are extracted on the feature layers of multiple scales for the features of each pixel position of each scale; finally, the attention mechanism is used to interact with each pixel feature and its neighbor node features, thereby updating the features of the current pixel position. The above-mentioned calculation process can be executed repeatedly, corresponding to multiple deformable attention network layers, to further optimize the multi-scale visual features, and finally obtain the above-mentioned optimized multi-scale visual features. The convolution kernel size of the convolutional layer, the converted visual feature dimensions, the number of coordinate offsets, and the number of neighbor nodes can be set according to actual needs, and the present invention does not make specific limitations.

[0057] In this embodiment, a convolution layer with a convolution kernel of 1x1 is first used to convert the dimensions of multi-scale visual features into the same value, such as 256; then, another convolution layer with a convolution kernel of 1x1 is used to predict multiple coordinate offsets of each pixel position at multiple scales for the visual features of each scale, and the predicted offsets are used to indicate which pixel features of adjacent positions the features of each pixel position should interact with for attention. For example, 8 coordinate offsets are predicted for each pixel position at each scale. Then, based on the offset prediction results, a specified number of neighbor nodes and their features are extracted on the feature layers of multiple scales for the features of each pixel position at each scale, for example, 8 neighbor nodes are extracted for each pixel at each scale. Finally, a multi-head attention mechanism is used to interact with each pixel feature and its neighbor node feature, thereby updating the feature of the current pixel position.

[0058] Multi-scale deformable attention mechanism The specific calculation process is as follows:

[0059] in, represents the current query pixel, Represents the characteristics of the pixel, represents the normalized coordinates of the pixel, represents the feature layer of L scales, Indicates Attention head, represents the number of attention heads, Indicates The feature layer of each scale, Indicates the current pixel Neighbor sampling pixels, Represents the number of neighbor sampling pixels, and is the learnable weight parameter, and Respectively represent The attention head In the scale feature layer The sampling offset and attention weight of the sampling points, The function represents remapping the normalized coordinates to The original coordinates of the layer. Compared with the original global multi-head attention mechanism, the deformable attention mechanism can significantly reduce the computational overhead. The above calculation process can be executed multiple times in a loop, corresponding to multiple deformable attention layers, to further optimize the multi-scale visual features. For example, 4 layers of deformable attention layers are used in this embodiment.

[0060] The present invention eliminates the differences in scale and pixel value of the input image through scale and pixel value normalization processing, and improves the consistency and accuracy of feature extraction. The visual skeleton network can capture feature information of different scales, and enhances the adaptability of the model to complex scenes and deformed objects. The deformable attention network optimizes multi-scale visual features, so that the model can pay more attention to important areas and details in the image, thereby improving the generalization ability of the model. By combining multi-scale visual feature extraction and deformable attention network optimization, the accuracy and robustness of document image segmentation can be significantly improved. By capturing feature information of different scales and optimizing the capture ability of key information, it can better adapt to different types of document images and segmentation tasks. Since the model has stronger generalization ability, it can reduce the dependence on new data and the annotation cost. This method provides strong technical support for subsequent tasks such as text recognition, form understanding, and information extraction, and promotes the automation and intelligent process of document processing.

[0061] In an optional embodiment, the text encoder in the above step S130 includes three main parts: a word segmenter, an embedding layer, and an encoding layer. The text encoder can use a pre-trained open source multilingual version of the text encoder to support multi-language category text descriptions. The pre-trained multilingual version of the SentenceTransformer model paraphrase-multilingual-MiniLM-L12-v2 can be used to support multi-language category text descriptions. Of course, the model can also use other types of text encoders. The above step S130 uses a text encoder to encode the text description of each category of the area to be segmented into a semantic query vector, and randomly initializes a preset number of instance query vectors, including: S1301, using the word segmenter to segment the text description of each category of the region to be segmented (such as paragraphs, tables, and illustrations) to obtain a token sequence.

[0062] The main function of the word segmenter is to segment the input text description according to certain rules (such as words, subwords, characters, etc.) to obtain a series of tokens. For Chinese text, the word segmenter can encode it according to characters, words, Wubi radicals, four-corner symbols, etc. For English text, it is usually encoded according to letters, words, roots, etc. The word segmenter can also segment words based on statistical methods (such as BPE, WordPiece, etc.) or rule methods. Through the word segmenter, the text description is converted into a series of token sequences, which provides a basis for subsequent processing.

[0063] In this embodiment, since the adopted Sentence Transformer model is a multilingual version model, its word segmenter supports word segmentation operations in multiple languages ​​at the same time. Word segmentation is the process of dividing continuous text into independent words according to semantic or grammatical rules. It is a basic step in natural language processing. It helps computers understand text structure and is particularly important for languages ​​such as Chinese that do not have clear word boundaries. The independent words after word segmentation are generally called tokens.

[0064] S1302: Use the embedding layer to map the token sequence into a vector sequence.

[0065] The main function of the embedding layer is to map the token sequence obtained by the tokenizer into a vector sequence. These vectors contain certain semantic information, so that similar tokens have similar representations in the vector space. The embedding layer uses word embedding technology, such as Word2Vec, GloVe, etc. These technologies can map a word into a high-dimensional vector space, thereby capturing the semantic relationship between words. Through the embedding layer, the token sequence is converted into a vector sequence, providing a processable numerical representation for the subsequent encoding layer.

[0066] In this embodiment, after word segmentation, the embedding layer of the Sentence Transformer model is used to map each token from one-hot encoding to a C-dimensional vector, for example, 384 dimensions, to obtain a vector sequence. Since the embedding layer of the Sentence Transformer model is pre-trained on a large-scale corpus, the multi-dimensional vectors obtained by mapping tokens with similar semantics also have a high degree of similarity.

[0067] S1303: Use the encoding layer to encode the vector sequence to obtain an encoded token feature sequence.

[0068] The main function of the encoding layer is to further process and encode the vector sequence obtained by the embedding layer to extract the semantic information of the text. The encoding layer can use a series of neural network layers (such as Transformer layers) to encode the vector sequence. These neural network layers can capture the contextual information in the vector sequence and thus extract the semantic features of the text. Through the encoding layer, the vector sequence is converted into an encoded token feature sequence with rich semantic information.

[0069] In this embodiment, the encoding layer of the Sentence Transformer model is used to perform multi-layer encoding on the input token feature sequence to obtain the encoded token feature, and the dimension of the encoded token feature is still C. The encoding layer of the Sentence Transformer model includes multiple multi-head self-attention layers and feedforward neural network layers, which are cross-interleaved and combined with each other, that is, each multi-head attention layer is followed by a feedforward neural network layer. For example, the paraphrase-multilingual-MiniLM-L12-v2 model used in this embodiment includes a total of 12 layers.

[0070] S1304: Calculate the global average of the encoded token feature sequence in the sequence dimension to obtain a semantic query vector for each category of the region to be segmented.

[0071] By taking the global average of the encoded token feature sequence in the sequence dimension, we can get the semantic query vector of each category of the region to be segmented. This vector represents the overall semantic features of the category. By taking the global average operation on the encoded token feature sequence, we can get a vector representation of a fixed length, namely the semantic query vector. The semantic query vector can capture the overall semantic features of each category and provide a basis for subsequent processing.

[0072] There are differences in the length of text descriptions of different categories, and the encoded vector sequences of different lengths need to be converted to the same length. In this embodiment, a global averaging method is adopted, that is, averaging is performed on the sequence length dimension. After averaging, the vector sequence of each category will be converted into a vector, which is used to represent the semantic query vector of the category, with a dimension of C and a length of 1. If the number of semantic categories to be segmented in the current document image is N, then after the above operation, N semantic query vectors with a dimension of C will be generated. Afterwards, a fully connected layer is used to transform the dimension of the semantic query vector so that its feature dimension is consistent with the visual feature dimension, for example, 256 dimensions.

[0073] S1305 . Randomly initialize a preset number of instance query vectors, where the dimension of each instance query vector is consistent with the dimension of the semantic query vector.

[0074] The instance query vector is a set of randomly initialized vectors used for subsequent processing (such as attention mechanism, decoder, etc.). Figure 3 As shown in the figure, learnable position embedding is performed to generate a preset number of instance query vectors. The dimension of each instance query vector is consistent with the dimension of the semantic query vector, and the parameters of these instance query vectors are randomly initialized. The initialization method can be adopted in a variety of ways, such as Gaussian distribution random initialization or uniform distribution random initialization. The random initialization of instance query vectors provides diversity for subsequent processing, which helps the model better capture and process information in text data.

[0075] In this embodiment, a preset number M (for example, 500) of instance query vectors are first generated, the dimension of each instance query vector is consistent with the dimension of the semantic query vector, for example, 256, and the parameters of these instance query vectors are randomly initialized using a Gaussian distribution random initialization with a mean of 0 and a variance of 0.02.

[0076] The present invention converts text data into numerical representation through the processing of word segmentation, embedding layer and encoding layer, making subsequent processing more efficient and accurate. The embedding layer and encoding layer can capture the semantic information in the text, so that the model can better understand and process text data.

[0077] In an optional embodiment, the step S140 described above inputs the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-built hybrid query decoder, and uses the semantic query vector and the instance query vector to perform multiple rounds of decoding on the optimized multi-scale visual features using a cross-attention mechanism to obtain an updated semantic query vector and an updated instance query vector, including: S1401: Perform a first information exchange between the semantic query vector and the instance query vector to obtain an interacted semantic query vector and an interacted instance query vector.

[0078] The semantic query vector and the instance query vector have their first information interaction through the self-attention mechanism. The self-attention mechanism allows the semantic query vector and the instance query vector to influence each other and capture the relationship between them. For example, the semantic query vector can provide category information, while the instance query vector can provide instance-specific information. Through this interaction, the interacted semantic query vector and the interacted instance query vector are obtained. Through the self-attention mechanism, the semantic query vector and the instance query vector can share information and enhance the synergy between them. This interaction provides richer context information for the subsequent decoding process.

[0079] Specifically, first, refer to Figure 3 As shown in the figure, the semantic query vector and the instance query vector are concatenated in the sequence dimension. For example, for N semantic query vectors and M instance query vectors, the concatenation results in N+M query vectors. Secondly, the concatenated query vector is updated with features using a multi-head self-attention mechanism (V, K, Q) and a feedforward neural network (i.e., the concatenated query vector is first self-attention calculated and feature updated using a multi-head attention mechanism, and then feature transformed using a feedforward neural network layer). Then, the updated query vector is split into a semantic query vector and an instance query vector. The split semantic query vector and instance query vector are consistent with their original sequence lengths, such as N semantic query vectors after interaction and M instance query vectors after interaction.

[0080] S1402, using a multi-head cross attention mechanism and a feedforward neural network, using the interacted semantic query vector to decode the optimized multi-scale visual features to obtain a semantic query vector that integrates the multi-scale features.

[0081] The interacted semantic query vector interacts with the optimized multi-scale visual features through a multi-head cross-attention mechanism. The multi-head cross-attention mechanism calculates the similarity between the semantic query vector and the multi-scale visual features, and dynamically selects the image region most relevant to the semantic query vector. After further processing by a feed-forward neural network (FFN), a semantic query vector that incorporates multi-scale features is obtained. The multi-head cross-attention mechanism can capture local and global information related to the semantic query vector in the image. The semantic query vector that incorporates multi-scale features can more accurately represent the semantic information in the image and improve the accuracy of semantic segmentation.

[0082] Specifically, first, the two-dimensional visual features of each scale (i.e., the optimized multi-scale visual features) are expanded into a one-dimensional sequence. For example, the visual features with the original shape of H×W×C are expanded into HW×C, where H represents the feature layer height, W represents the feature layer width, and C represents the feature dimension. Then, referring to Figure 3 As shown in the figure, features of all scales are concatenated in the sequence dimension to obtain a visual feature sequence containing multiple scales. For example, features of scales 1 / 4, 1 / 8, 1 / 16, and 1 / 32 are concatenated in the sequence dimension. Then, a multi-head cross-attention mechanism (including multi-head self-attention mechanism and cross-attention mechanism) and a feedforward neural network are used to decode the semantic query vector on the multi-scale visual features to obtain a semantic query vector that integrates multi-scale features. Figure 3 The whole decoder part is iteratively optimized. T represents the number of iterations, which is 4 in this embodiment. Each iteration will have an output result of this iteration. These intermediate results will be compared with the labeled true value to calculate the loss during the training phase. L represents the number of layers of the decoder in each iteration, which is also set to 4 in this embodiment.

[0083] S1403, using a multi-head cross attention mechanism and a feedforward neural network, using the interacted instance query vector to decode the optimized multi-scale visual features to obtain an instance query vector that integrates the multi-scale features.

[0084] The instance query vector after interaction interacts with the optimized multi-scale visual features through a multi-head cross-attention mechanism. The multi-head cross-attention mechanism calculates the similarity between the instance query vector and the multi-scale visual features, and dynamically selects the image region most relevant to the instance query vector. After further processing through a feedforward neural network, an instance query vector that incorporates multi-scale features is obtained. The multi-head cross-attention mechanism can capture local information related to the instance query vector in the image and help distinguish different instances in the same category. The instance query vector that incorporates multi-scale features can more accurately represent the instance information in the image and improve the accuracy of instance segmentation.

[0085] Specifically, first, refer to Figure 3As shown in the figure, the two-dimensional visual features of each scale (i.e., the optimized multi-scale visual features) are expanded into a one-dimensional sequence, for example, the visual features with the original shape of H×W×C are expanded into HW×C, where H represents the feature layer height, W represents the feature layer width, and C represents the feature dimension; then, the features of all scales are concatenated in the sequence dimension to obtain a visual feature sequence containing multiple scales, for example, the features of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 scales are concatenated in the sequence dimension; then, the multi-head cross-attention mechanism and the feedforward neural network are used to decode the instance query vector on the multi-scale visual features to obtain an updated instance query vector that integrates the multi-scale features.

[0086] S1404: Perform a second information exchange between the semantic query vector integrated with multi-scale features and the instance query vector integrated with multi-scale features to obtain an updated semantic query vector and an updated instance query vector.

[0087] The semantic query vector and instance query vector that incorporate multi-scale features interact with each other for the second time through the self-attention mechanism. This interaction further enhances the synergy between the semantic query vector and the instance query vector, ensuring that they can be optimized together. Through this interaction, an updated semantic query vector and an updated instance query vector are obtained. The second information interaction further refines the query vectors, ensuring that they can better capture the semantic and instance information in the image. Through multiple interactions, the query vectors can gradually align image features, improving segmentation accuracy and detail capture capabilities.

[0088] Specifically, first, refer to Figure 3 As shown, the semantic query vector that integrates multi-scale features and the instance query vector that integrates multi-scale features are spliced ​​in the sequence dimension. For example, for N semantic query vectors and M instance query vectors, the spliced ​​query vectors are N+M query vectors; secondly, the multi-head self-attention mechanism and feedforward neural network are used to update the features of the spliced ​​query vector; then, the query vector after feature update is split into a semantic query vector and an instance query vector, and the split semantic query vector and instance query vector are consistent with their original sequence lengths, such as N semantic query vectors and M instance query vectors. The split semantic query vector and instance query vector are also the updated semantic query vector and updated instance query vector mentioned above.

[0089] The above steps S1401-S1404 (first information interaction, multi-head cross attention mechanism decoding, second information interaction) can be repeated multiple times to form a multi-round decoding process. Each round of decoding will further optimize the semantic query vector and the instance query vector to better align them with the image features. The multi-round decoding process allows the model to gradually refine the query vector from rough global information to specific local details. Through iterative optimization, the model can capture more detailed information and improve the accuracy of segmentation.

[0090] The present invention uses multiple rounds of decoding and cross-attention mechanisms, and the semantic query vector and the instance query vector can gradually align image features and capture more detail information, thereby improving the accuracy of semantic segmentation and instance segmentation. The multi-head cross-attention mechanism can dynamically select the image area most relevant to the query vector to ensure that the model can capture local details in the image. Through multiple information interactions and multiple rounds of decoding, the query vector can be gradually optimized to better represent the semantic and instance information in the image. This process gradually optimizes the semantic query vector and the instance query vector through multiple rounds of decoding and cross-attention mechanisms, so that they are better aligned with multi-scale visual features. This method not only improves the accuracy of segmentation, but also can capture more detail information, and is suitable for a variety of complex visual tasks.

[0091] In an optional embodiment, after the above step S1203, the method further includes: S1204, reference Figure 3 As shown, the optimized multi-scale visual features are fused using a feature pyramid network to obtain mixed multi-scale visual features.

[0092] The Feature Pyramid Network (FPN) is used to fuse the optimized visual features of different scales to generate mixed multi-scale visual features with rich semantic information and detail information. First, a convolution layer with a convolution kernel of 1x1 is used to transform the features of each scale in the optimized multi-scale visual features. Since the dimensions of the multi-scale features have been unified in the above step S1203, there is no need to perform dimensional transformation again here; secondly, the multi-scale features are scaled to the same size, such as 1 / 4 of the size of the original image; then, the scaled multi-scale features are added pixel by pixel to obtain the fused multi-scale features, that is, the above-mentioned mixed multi-scale visual features. In this embodiment, the dimension of the fused multi-scale features is 256, and the size is 1 / 4 of the original image.

[0093] The hybrid multi-scale visual features generated by FPN can help the model better distinguish different instances in the same category. FPN can improve the accuracy of boundary segmentation in semantic segmentation tasks. Combining semantic segmentation and instance segmentation, FPN can capture global semantic information and local detail information at the same time. Hybrid multi-scale visual features can capture global semantic information and local detail information at the same time, significantly improving the performance of the model in tasks such as instance segmentation and semantic segmentation.

[0094] The above visual skeleton network, deformable attention network and feature pyramid network constitute Figure 3 The visual encoder in .

[0095] In an optional embodiment, after obtaining the updated semantic query vector and the updated instance query vector, and the hybrid multi-scale visual features, mask prediction and category prediction are performed based on the updated semantic query vector and the updated instance query vector, and the hybrid multi-scale visual features to obtain a prediction result. The above step S150 uses the updated semantic query vector and the updated instance query vector to perform semantic segmentation and instance segmentation respectively to obtain a semantic segmentation result and an instance segmentation result, and uses the updated semantic query vector as a category prototype to classify the segmented instance to obtain a category classification result of the instance, including: S1501. Use the updated semantic query vector to perform semantic segmentation on the mixed multi-scale visual feature to obtain a semantic segmentation result.

[0096] The updated semantic query vector is used to perform semantic segmentation on the hybrid multi-scale visual features. Specifically, the semantic query vector interacts with the hybrid multi-scale visual features through a cross-attention mechanism to calculate the similarity between each pixel position and the semantic query vector. Based on the similarity score, a semantic category label is assigned to each pixel to generate a semantic segmentation result. The semantic query vector can capture the global semantic information in the image and ensure that the semantic segmentation result can accurately reflect the object category and scene context in the image. The semantic query vector optimized by multiple rounds of decoding and cross-attention mechanism can better align image features and improve the accuracy of semantic segmentation.

[0097] Specifically, first, a vector dot product operation is used to calculate the inner product of each updated semantic query vector and the visual feature of each pixel position in the hybrid multi-scale visual feature. For example, for a semantic query vector sequence with a shape of N×C and a visual feature with a shape of H×W×C, after the inner product calculation, a matrix with a shape of N×H×W is obtained; then the calculation result is normalized using the Sigmoid function to obtain the similarity between each semantic query vector and the visual feature of each pixel position, with a shape of N×H×W and a value between 0 and 1; finally, a threshold operation is used to convert the above similarity into N binary semantic segmentation results. Each semantic query vector corresponds to a semantic segmentation result. For example, the threshold is 0.5, and the foreground is greater than the threshold, otherwise it is the background.

[0098] S1502: Use the updated instance query vector to perform instance segmentation on the hybrid multi-scale visual feature to obtain an instance segmentation result.

[0099] The updated instance query vector is used to perform instance segmentation on the hybrid multi-scale visual features. Specifically, the instance query vector interacts with the hybrid multi-scale visual features through a cross-attention mechanism to calculate the similarity between each pixel position and the instance query vector. Based on the similarity score, an instance label is assigned to each pixel to generate an instance segmentation result. The instance query vector can capture the instance information in the image and ensure that the instance segmentation result can accurately distinguish different instances in the same category. The instance query vector optimized by multiple rounds of decoding and cross-attention mechanism can better align image features and improve the segmentation accuracy of instance boundaries.

[0100] Specifically, first, the inner product of each updated instance query vector and the visual feature of each pixel position in the hybrid multi-scale visual feature is calculated using a vector dot product operation. For example, for an instance query vector sequence with a shape of M×C and a visual feature with a shape of H×W×C, after the inner product calculation, a matrix with a shape of M×H×W is obtained; then the calculation result is normalized using a Sigmoid function to obtain the similarity between each instance query vector and the visual feature of each pixel position, with a shape of M×H×W and a value between 0 and 1; finally, a threshold operation is used to convert the above similarity into M binary instance segmentation results. Each instance query vector corresponds to an instance segmentation result, for example, the threshold is 0.5, and the one greater than the threshold is the foreground, otherwise it is the background.

[0101] S1503: Calculate the inner product of each updated instance query vector and each updated semantic query vector and normalize them to obtain a similarity score, and select the semantic category corresponding to the semantic query vector with the highest similarity score for each updated instance query vector as the category classification result of the instance.

[0102] The updated semantic query vector is used as the category prototype, and each semantic query vector corresponds to a semantic category. The inner product of each updated instance query vector and each updated semantic query vector is calculated to obtain a similarity score. The similarity score is normalized (such as using the Softmax function) to obtain the similarity probability between each instance query vector and each semantic category. For each instance query vector, the semantic category corresponding to the semantic query vector with the highest similarity score is selected as the category classification result of the instance. By calculating the similarity between the instance query vector and the semantic query vector, each instance can be accurately classified into the corresponding semantic category. The instance classification result is highly consistent with the semantic segmentation result, ensuring that the category of each instance is consistent with its category in the semantic segmentation.

[0103] Specifically, first, the inner product of each updated instance query vector and each updated semantic query vector is calculated using the vector dot product operation; for example, for an instance query vector sequence with a shape of M×C and a semantic query vector with a shape of N×C, after the inner product calculation, a matrix with a shape of M×N is obtained. Then the calculation result is normalized using the Softmax function to obtain the similarity between each updated instance query vector and each updated semantic query vector, with a shape of M×H×W and a value between 0 and 1; finally, for each updated instance query vector, the semantic category corresponding to the semantic query vector with the highest similarity score is selected as the category classification result of the instance. The last vector in the N semantic query vectors corresponds to the category "non-target", that is, the instance segmentation result does not belong to any specified semantic category. Such instance segmentation results will be filtered out and will not be output or saved.

[0104] Reference Figure 2 As shown in Figure 2, for the semantic segmentation task, the semantic segmentation results are stored in the form of multiple binary masks, and each semantic category corresponds to a binary mask (i.e. Figure 2 The semantic mask in ), where pixels with a value of 0 represent the background and pixels with a value of 1 represent the foreground; for instance segmentation results, the instance segmentation results are stored in the form of multiple binary masks and multiple category labels, and each instance segmentation result corresponds to a binary mask (i.e. Figure 2 ) and an instance category label, where pixels with a value of 0 in the mask represent background and pixels with a value of 1 represent foreground, and each category label corresponds to a specific semantic category.

[0105] The query vector optimized by multi-round decoding and cross-attention mechanism (i.e., the updated semantic query vector and instance query vector) can better align image features. The present invention utilizes the updated semantic query vector and instance query vector to perform semantic segmentation, instance segmentation, and instance classification, respectively, significantly improving the accuracy of semantic segmentation, instance segmentation, and instance classification. The synergy of the semantic query vector and the instance query vector ensures the consistency of the segmentation results and the classification results. The hybrid multi-scale visual features combine high-level semantic information with low-level detail information, so that the model can capture more local details and improve the accuracy of the segmentation boundary. The combination of hybrid multi-scale visual features and multi-round decoding mechanism enables the model to capture more detail information.

[0106] The general document image segmentation method based on query decoupling and prototype classification of the present invention includes three loss functions for training the general document image segmentation model, namely, semantic segmentation loss, instance segmentation loss and instance classification loss. for:

[0107] in, and They are predicting semantic masks and annotating semantic masks, and They are focal loss function and dice loss function respectively. and is a hyperparameter used to measure the weight of the two losses. In this embodiment and Take 10 and 1 respectively.

[0108] Similarly, the instance segmentation loss function for:

[0109] in, and They are the predicted instance mask and the labeled instance mask, and They are focal loss function and dice loss function respectively. and is a hyperparameter used to measure the weight of the two losses. In this embodiment and Take 10 and 1 respectively.

[0110] In addition, the instance classification loss function for:

[0111] in, and They are the predicted instance category and the labeled instance category, is the multi-class cross entropy loss function.

[0112] Finally, the total loss function of the model is for:

[0113] in, , and are the weight coefficients of semantic segmentation loss, instance segmentation loss, and instance classification loss, respectively. In this embodiment, they are 5, 5, and 1, respectively.

[0114] In order to obtain better segmentation results, the present invention performs loss calculation on each layer output of the hybrid query decoder. In addition, following the Transformer-based object detection and instance segmentation methods such as DETR and Mask2Former, the present invention also uses bipartite graph matching and Hungarian algorithm to find the best matching annotation true value for each instance segmentation result to perform loss calculation. It should be noted that for semantic segmentation tasks, the above matching process is not required, because the semantic segmentation task is originally a one-to-one prediction result, that is, each semantic category corresponds to a segmentation result mask. Unlike existing methods, the novel design proposed in the present invention makes it possible to perform loss calculation on heterogeneous mixed data sets (such as Figure 2 A general document image segmentation model is trained directly on datasets 1, 2, …, N in the datasets without having to train separate models for each task and dataset. After training, the model can be used directly as a versatile document image segmentor, or as a pre-trained model that can be seamlessly fine-tuned for task-specific datasets without any specialized modifications, such as adding or replacing linear classification layers. This design gives the model the ability to learn continuously and incrementally, making it more flexible and easy to adapt to new tasks.

[0115] Figures 4 to 7 The segmentation results of the method of the present invention on multiple public document datasets under multiple document segmentation tasks are shown respectively. Figure 4 This is an example of the results of the general document image segmentation method based on query decoupling and prototype classification in the modern document layout segmentation task; Figure 5 This is an example of the results of the general document image segmentation method based on query decoupling and prototype classification in the ancient book and handwritten document segmentation tasks; Figure 6 This is an example of the results of the general document image segmentation method based on query decoupling and prototype classification in the table detection and structure recognition tasks of the present invention; Figure 7This is an example of the results of the general document image segmentation method based on query decoupling and prototype classification in the present invention on the scene text segmentation task. The results show that the method of the present invention can effectively handle a variety of document segmentation tasks in various types of document images, not only covering the segmentation of text regions of arbitrary shapes with multiple granularities such as characters, text lines to paragraphs, but also applicable to other types of document region segmentation including tables, formulas, and illustrations. In addition, the method is applicable to different types of document images, such as modern documents, ancient books, handwritten documents, scene images, scanned documents, and photographed documents. What is particularly important is that for the various types of document segmentation tasks mentioned above, the method of the present invention only needs to train and maintain a single unified model, without the need to design and maintain specific models for each task. This feature significantly reduces development and maintenance costs, while also improving the convenience and efficiency of model use.

[0116] Figure 4 All samples come from the M6Doc dataset; Figure 5 The first two samples in the first row are from the SCUT-CAB dataset, and the last three are from the MTHv2 dataset. The first three samples in the second row are from the CASIA-HWDB dataset, and the last two are from the SCUT-HCCDoc dataset. Figure 6 The first row of samples comes from the ICDAR 2019 cTDaR dataset, and the second row comes from the WTW dataset; Figure 7 The first row of samples comes from the ICDAR 2017 RCTW dataset, and the second row comes from the ICDAR 2019 LSVT dataset.

[0117] The general document image segmentation device based on query decoupling and prototype classification provided by the present invention is described below. The general document image segmentation device based on query decoupling and prototype classification described below and the general document image segmentation method based on query decoupling and prototype classification described above can be referred to each other.

[0118] The general document image segmentation device based on query decoupling and prototype classification provided by the present invention refers to Figure 8 As shown, it includes the following modules: An acquisition module 210 is used to acquire an input document image and a text description of each type of region to be segmented; An extraction module 220, configured to extract multi-scale visual features from the input document image using a pre-constructed visual skeleton network, and optimize the extracted multi-scale visual features using a pre-constructed deformable attention network to obtain optimized multi-scale visual features; An encoding module 230, used to encode the text description of each category of the region to be segmented into a semantic query vector using a text encoder, and randomly initialize a preset number of instance query vectors; A decoding module 240 is used to input the optimized multi-scale visual features, the semantic query vector and the instance query vector into a pre-built hybrid query decoder, and use the semantic query vector and the instance query vector to perform multiple rounds of decoding on the optimized multi-scale visual features using a cross-attention mechanism to obtain an updated semantic query vector and an updated instance query vector; The segmentation module 250 is used to use the updated semantic query vector and the updated instance query vector to perform semantic segmentation and instance segmentation respectively to obtain semantic segmentation results and instance segmentation results, and use the updated semantic query vector as a category prototype to classify the segmented instance to obtain a category classification result of the instance.

[0119] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the general document image segmentation method based on query decoupling and prototype classification.

[0120] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the general document image segmentation method based on query decoupling and prototype classification provided by the above-mentioned methods.

[0122] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the general document image segmentation method based on query decoupling and prototype classification provided by the above-mentioned methods.

[0123] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A general document image segmentation method based on query decoupling and prototype classification, characterized in that: include: Obtain an input document image and a text description of each category of the region to be segmented; Extracting multi-scale visual features from the input document image using a pre-built visual skeleton network, and optimizing the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features; A text encoder is used to encode the text description of each category of the region to be segmented into a semantic query vector, and a preset number of instance query vectors are randomly initialized; Inputting the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-built hybrid query decoder, and performing multiple rounds of decoding on the optimized multi-scale visual features using the semantic query vector and the instance query vector using a cross-attention mechanism to obtain an updated semantic query vector and an updated instance query vector; The updated semantic query vector and the updated instance query vector are used to perform semantic segmentation and instance segmentation respectively to obtain semantic segmentation results and instance segmentation results, and the updated semantic query vector is used as a category prototype to classify the segmented instance to obtain a category classification result of the instance.

2. The general document image segmentation method based on query decoupling and prototype classification according to claim 1, characterized in that: The method of extracting multi-scale visual features from the input document image using a pre-built visual skeleton network, and optimizing the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features, includes: Performing scale and pixel value normalization processing on the input document image to obtain a normalized document image; Using a pre-built visual skeleton network to extract multi-scale visual features from the normalized document image to obtain multi-scale visual features; The multi-scale visual features are optimized using a pre-built deformable attention network to obtain optimized multi-scale visual features.

3. The general document image segmentation method based on query decoupling and prototype classification according to claim 1, characterized in that: The text encoder includes a word segmenter, an embedding layer and an encoding layer; the text encoder is used to encode the text description of each category of the region to be segmented into a semantic query vector, and randomly initializes a preset number of instance query vectors, including: Use the word segmenter to segment the text description of each category of the area to be segmented to obtain a token sequence; Mapping the token sequence into a vector sequence using the embedding layer; Encode the vector sequence using the encoding layer to obtain an encoded token feature sequence; Calculating the global average of the encoded token feature sequence in the sequence dimension to obtain a semantic query vector for each category of the region to be segmented; A preset number of instance query vectors are randomly initialized, and the dimension of each instance query vector is consistent with the dimension of the semantic query vector.

4. The general document image segmentation method based on query decoupling and prototype classification according to claim 1, characterized in that: Inputting the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-built hybrid query decoder, and performing multiple rounds of decoding on the optimized multi-scale visual features using the semantic query vector and the instance query vector using a cross attention mechanism to obtain an updated semantic query vector and an updated instance query vector, including: Performing a first information exchange between the semantic query vector and the instance query vector to obtain an interacted semantic query vector and an interacted instance query vector; Using the interactive semantic query vector to decode the optimized multi-scale visual features using a multi-head cross attention mechanism and a feedforward neural network, a semantic query vector integrating multi-scale features is obtained; Using the interacted instance query vector to decode the optimized multi-scale visual features using a multi-head cross attention mechanism and a feedforward neural network, an instance query vector integrating multi-scale features is obtained; The semantic query vector integrated with the multi-scale features and the instance query vector integrated with the multi-scale features are subjected to a second information exchange to obtain an updated semantic query vector and an updated instance query vector.

5. The general document image segmentation method based on query decoupling and prototype classification according to claim 1, characterized in that: The method further includes: using a feature pyramid network to fuse the optimized multi-scale visual features to obtain mixed multi-scale visual features.

6. The general document image segmentation method based on query decoupling and prototype classification according to claim 5, characterized in that: The method of using the updated semantic query vector and the updated instance query vector to perform semantic segmentation and instance segmentation respectively to obtain a semantic segmentation result and an instance segmentation result, and using the updated semantic query vector as a category prototype to classify the segmented instance to obtain a category classification result of the instance includes: Using the updated semantic query vector to perform semantic segmentation on the hybrid multi-scale visual feature to obtain a semantic segmentation result; Using the updated instance query vector to perform instance segmentation on the hybrid multi-scale visual feature to obtain an instance segmentation result; The inner product of each updated instance query vector and each updated semantic query vector is calculated and normalized to obtain a similarity score, and for each updated instance query vector, the semantic category corresponding to the semantic query vector with the highest similarity score is selected as the category classification result of the instance.

7. A general document image segmentation device based on query decoupling and prototype classification, characterized in that: include: An acquisition module, used for acquiring an input document image and a text description of each category of the region to be segmented; An extraction module, configured to extract multi-scale visual features from the input document image using a pre-built visual skeleton network, and optimize the extracted multi-scale visual features using a pre-built deformable attention network to obtain optimized multi-scale visual features; An encoding module, used for encoding the text description of each category of the region to be segmented into a semantic query vector by using a text encoder, and randomly initializing a preset number of instance query vectors; A decoding module, configured to input the optimized multi-scale visual features, the semantic query vector, and the instance query vector into a pre-built hybrid query decoder, and use the semantic query vector and the instance query vector to perform multiple rounds of decoding on the optimized multi-scale visual features using a cross-attention mechanism to obtain an updated semantic query vector and an updated instance query vector; A segmentation module is used to use the updated semantic query vector and the updated instance query vector to perform semantic segmentation and instance segmentation respectively to obtain semantic segmentation results and instance segmentation results, and use the updated semantic query vector as a category prototype to classify the segmented instance to obtain a category classification result of the instance.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the general document image segmentation method based on query decoupling and prototype classification as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the general document image segmentation method based on query decoupling and prototype classification as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the general document image segmentation method based on query decoupling and prototype classification as claimed in any one of claims 1 to 6 is implemented.

Images