Layout analysis method, electronic equipment and storage medium

Abstract

The invention discloses a layout analysis method, electronic equipment and a storage medium. The method comprises the following steps: inputting a to-be-identified first document image into a layout analysis model; wherein the layout analysis model comprises a plurality of classification heads and a shared regression head, and the regression head is used for performing bounding box prediction on each layout element in the first document image, each classification head in the plurality of classification heads is used for performing category label prediction of a corresponding level on each prediction bounding box output by the regression head; processing the first document image through the regression head and the plurality of classification heads to obtain each prediction bounding box output by the layout analysis model and a category label prediction score corresponding to each prediction bounding box; and determining a final bounding box and a final category label corresponding to each layout element in the first document image based on a preset hierarchical relationship between category labels and the category label prediction score corresponding to each prediction bounding box.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a layout analysis method, electronic device, and storage medium. Background Technology

[0002] In the fields of computer vision and document image processing, layout analysis is a crucial preliminary step for achieving structured document understanding. Its main task is to identify and classify different types of layout elements in a document image, such as text, images, tables, page numbers, and seals. This task is widely used in practical scenarios such as Optical Character Recognition (OCR), intelligent document parsing, archival digitization, and contract understanding.

[0003] In related technologies, object detection models (such as Faster R-CNN, YOLO, DETR, etc.) are mainly used to perform multi-class object detection on document images. This model makes unified predictions for all categories through a single detection head, which includes a classification head and a regression head.

[0004] However, since various layout elements in document images (such as text, tables, and images) differ significantly in semantic category, geometric shape, scale, and spatial distribution, using a single-head target detection model for unified prediction results in low accuracy of document image layout analysis. Summary of the Invention

[0005] This application provides a layout analysis method, electronic device, and storage medium to solve the technical problem of low accuracy in document image layout analysis in related technologies.

[0006] According to a first aspect of this application, a layout analysis method is disclosed, the method comprising:

[0007] The first document image to be identified is input into the layout analysis model; wherein, the layout analysis model includes: multiple classification heads and a shared regression head, the regression head is used to predict the bounding boxes of each layout element in the first document image, and each of the multiple classification heads is used to predict the category label of each predicted bounding box output by the regression head at the corresponding level.

[0008] The first document image is processed using the regression head and the multiple classification heads to obtain the predicted bounding boxes and the corresponding category label prediction scores output by the layout analysis model.

[0009] Based on the hierarchical relationship between preset category labels and the category label prediction scores corresponding to each predicted bounding box, the final bounding box and final category label corresponding to each layout element in the first document image are determined.

[0010] According to a second aspect of this application, an electronic device is disclosed, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the layout analysis method as described in the first aspect.

[0011] According to a third aspect of this application, a computer-readable storage medium is disclosed having a computer program / instructions stored thereon, which, when executed by a processor, implements the layout analysis method as described in the first aspect.

[0012] According to a fourth aspect of this application, a computer program product is disclosed, comprising a computer program / instructions that, when executed by a processor, implement the layout analysis method as described in the first aspect.

[0013] In this embodiment, a first document image to be identified is input into a layout analysis model. The layout analysis model includes multiple classification heads and a shared regression head. The regression head is used to predict bounding boxes for each layout element in the first document image, and each of the multiple classification heads is used to predict the category label at the corresponding level for each predicted bounding box output by the regression head. The first document image is processed by the regression head and the multiple classification heads to obtain each predicted bounding box output by the layout analysis model and the corresponding category label prediction score. Based on the preset hierarchical relationship between category labels and the corresponding category label prediction score of each predicted bounding box, the final bounding box and final category label corresponding to each layout element in the first document image are determined.

[0014] As can be seen, the layout analysis model in this embodiment has a unique structure, comprising multiple classification heads and a shared regression head. These classification heads share a single regression head, which uniformly predicts the bounding boxes of all layout elements. This avoids conflicts between multiple regression heads, reduces redundant parameters in the model, and allows for parallel execution of classification and regression tasks, minimizing processing latency. Simultaneously, by using each of the multiple classification heads to predict different levels of category labels on the predicted bounding boxes output by the regression head, interference between different categories is reduced. This enables detailed identification of various layout elements and their fine-grained classification labels, improving the comprehensiveness and accuracy of the analysis. The final result is determined based on the hierarchical relationship between category labels and the prediction scores output by each classification head, conforming to document layout logic. This significantly improves the accuracy of layout analysis of document images while maintaining efficient computation. Attached Figure Description

[0015] Figure 1 This is an example diagram of the network structure of a target detection model in related technologies;

[0016] Figure 2 This is a flowchart of a layout analysis method provided in an embodiment of this application;

[0017] Figure 3 This is an example diagram illustrating the hierarchical relationship between category labels provided in the embodiments of this application;

[0018] Figure 4 This is one of the network structure example diagrams of the layout analysis model provided in the embodiments of this application;

[0019] Figure 5 This is the second example diagram of the network structure of the layout analysis model provided in the embodiments of this application;

[0020] Figure 6 This is a flowchart of one implementation of step 203 provided in the embodiments of this application;

[0021] Figure 7 This is a flowchart illustrating the process of constructing training data for the layout analysis model provided in this application embodiment;

[0022] Figure 8 This is a schematic diagram of the structure of a layout analysis device provided in an embodiment of this application;

[0023] Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] It should be noted that the image data obtained in this application is accessed, collected, stored, and used for subsequent analysis and processing after the user or relevant data owner has been clearly informed of the content of the data collection, the purpose of the data, the processing method, etc., and with the consent and authorization of the user or relevant data owner. Furthermore, the application can provide the user or relevant data owner with the means to access, correct, or delete the data, as well as the method to revoke consent or authorization.

[0025] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0027] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0028] Currently, related technologies mainly employ object detection models (such as Faster R-CNN, YOLO, DETR, etc.) to perform multi-class object detection on document images. Figure 1As shown, the object detection model 10 may include: a feature extraction network (Backbone) 11, a feature neck network (Neck) 12, and a detection head 13. The detection head 13 includes: a regression head 131 and a classification head 132. The feature extraction network 11 extracts image features from the input document image and passes the extracted image features to the feature neck network 12. The feature neck network 12 fuses and / or enhances the image features output by the feature extraction network 11 and passes the processed feature map to the detection head 13. The regression head 131 in the detection head 13 predicts the bounding boxes (bboxes) of each layout element in the document image based on the feature map output by the feature neck network 12 and outputs the predicted bounding boxes. The classification head 132 in the detection head 13 predicts the category label of the predicted bounding boxes output by the regression head 131 based on the feature map output by the feature neck network 12 and outputs the predicted category label score corresponding to the predicted bounding boxes.

[0029] However, the relevant technical methods have the following technical problems, which restrict their accuracy in layout analysis in real and diverse scenarios: 1) Difficulty in training with multiple datasets and inconsistent category standards: Due to the differences in the labeling standards between different public datasets, there are often problems such as missing categories (e.g., some datasets do not label stamps or page numbers) or category conflicts (e.g., in some datasets, "charts" are labeled as "natural images"), which leads to inconsistent supervision information during the training process, seriously affecting the generalization ability of the model, and thus affecting the accuracy of layout analysis.

[0030] 2) Significant differences in features between categories make unified modeling difficult: Various layout elements in document images (such as text, tables, and images) differ significantly in semantic category, geometric shape, scale, and spatial distribution. For example, page numbers are usually small and located in the footer or header; tables have complex structures and occupy a large area; stamps have diverse shapes; and charts may look similar but have different meanings. Using a single detection head uniformly can lead to interference between features of different categories, reducing the accuracy of layout analysis, especially for small targets and fine-grained categories.

[0031] In summary, the existing technologies suffer from low accuracy in document image layout analysis when faced with scenarios involving inconsistent categories, significant feature differences, and difficulties in fine-grained recognition. To address these issues, this application provides a layout analysis method, an electronic device, and a storage medium.

[0032] The following description, in conjunction with the accompanying drawings, introduces a layout analysis method provided by an embodiment of this application.

[0033] Figure 2 This is a flowchart of a layout analysis method provided in an embodiment of this application, such as... Figure 2 As shown, the method may include the following steps: step 201, step 202 and step 203.

[0034] In step 201, the first document image to be identified is input into the layout analysis model; wherein, the layout analysis model includes: multiple classification heads and a shared regression head, the regression head is used to predict the bounding boxes of each layout element in the first document image, and each of the multiple classification heads is used to predict the category label of each predicted bounding box output by the regression head at the corresponding level.

[0035] In step 202, the first document image is processed using a regression head and multiple classification heads to obtain the predicted bounding boxes and the corresponding category label prediction scores output by the layout analysis model.

[0036] In step 203, based on the preset hierarchical relationship between category labels and the category label prediction score corresponding to each predicted bounding box, the final bounding box and final category label corresponding to each layout element in the first document image are determined.

[0037] The following is a detailed description of steps 201 to 203 mentioned above.

[0038] In this embodiment of the application, the first document image refers to a two-dimensional digital image containing layout elements that can be structured and parsed. Its sources include, but are not limited to: physical document digitization (such as images acquired by flatbed / sheet scanners, document photos taken by mobile devices, special imaging of ancient books / archives), digitally generated documents (such as bitmaps generated by PDF rendering, PNG / JPGs exported by public software, web page screenshots, etc.).

[0039] In this embodiment, the first document image can be preprocessed, such as adjusting the image size to the size required by the model, or performing normalization, so that it can be subsequently input into the layout analysis model.

[0040] In this embodiment, a hierarchical relationship between preset category labels is introduced. By classifying the layout elements in the document image into multiple levels, the layout elements can be classified more meticulously, avoiding vague and general classification methods, achieving a refined understanding of the layout structure, and making the classification results more accurate and specific.

[0041] In this embodiment, to adapt to the hierarchical relationship between the category labels, multiple independent classification heads are deployed in the layout analysis model. Structurally, these classification heads operate in parallel, each responsible for handling a specific category prediction task or a specific hierarchical category prediction. They are independent of each other, each possessing its own parameters and computational logic. This independence allows the layout analysis model to be optimized specifically according to the characteristics of different classification tasks or hierarchical categories. Logically, the category labels output by each classification head may have a certain hierarchical relationship in the layout structure.

[0042] In this embodiment of the application, to address the issue of significant differences between page elements, multiple classification heads are used to predict different levels of categories, avoiding the interference effect caused by the traditional shared detection head processing all categories uniformly, and significantly improving the detection performance of small targets and fine-grained categories.

[0043] In this embodiment of the application, the category label may include: a primary category label and a subordinate category label.

[0044] In this embodiment, the primary-level category label is the most basic and broadest category label in the entire category label system. It represents the most core and general classification, and is usually used to make a preliminary, macro-level distinction between layout elements. For example, the following primary-level category labels for layout elements in document images are: body text, title, description, image, and table.

[0045] In this embodiment, one or more subordinate category labels can be set under the main category label. Subordinate category labels are further subdivisions and refinements of the main category labels, enabling more precise descriptions of the characteristics and attributes of layout elements. As the number of subordinate levels increases, the accuracy of the classification also improves. For example, under the "Text" main category label, first-level subordinate category labels such as "Paragraph" and "Summary" can be set. Under the "Image" main category label, first-level subordinate category labels such as "Natural Image," "Chart," and "Smart Graphic" can be set; and under the "Chart" first-level subordinate category label, second-level subordinate category labels such as "Pie Chart," "Line Chart," and "Bar Chart" can be further set.

[0046] In some embodiments, where the category labels include primary category labels and subordinate category labels, the primary category labels may include: first-level category labels and independent category labels; the subordinate category labels may include: second-level category labels and third-level category labels; the primary category labels may include at least one of the following: body text labels, heading labels, description labels, image labels, and table labels; the independent category labels may include at least one of the following: header labels, footer labels, page number labels, and stamp labels;

[0047] Second-level category tags may include at least one of the following: paragraph tags under the body text tag, summary tags under the body text tag, natural image description tags under the description tag, chart description tags under the description tag, smart graphic description tags under the description tag, table description tags under the description tag, natural image tags under the image tag, chart tags under the image tag, smart graphic tags under the image tag, full solid line table tags under the table tag, half solid line table tags under the table tag, table tags without solid lines under the table tag, circular stamp tags under the stamp tag, triangular stamp tags under the stamp tag, square stamp tags under the stamp tag, and elliptical stamp tags under the stamp tag; third-level category tags include at least one of the following: pie chart tags under the chart tag, line chart tags under the chart tag, and bar chart tags under the chart tag.

[0048] For example, such as Figure 3 As shown, the first-level category tags include: body text tags, heading tags, description tags, image tags, and table tags; independent category tags include: header tags, footer tags, page number tags, and stamp tags; second-level category tags include: paragraph tags under body text tags, summary tags under body text tags, natural image description tags under description tags, chart description tags under description tags, smart graphic description tags under description tags, table description tags under description tags, natural image tags under image tags, chart tags under image tags, smart graphic tags under image tags, full solid line table tags under table tags, half solid line table tags under table tags, no solid line table tags under table tags, circular stamp tags under stamp tags, triangular stamp tags under stamp tags, square stamp tags under stamp tags, and elliptical stamp tags under stamp tags; third-level category tags include: pie chart tags under chart tags, line chart tags under chart tags, and bar chart tags under chart tags.

[0049] In this embodiment, the document layout is divided into first-level category labels, independent category labels, second-level category labels, and third-level category labels. This not only covers common layout elements (such as body text, titles, images, and tables), but also provides detailed annotations for descriptive content (such as chart descriptions), structural elements (such as headers and footers, page numbers, and stamps), and chart types (such as pie charts and line charts). This achieves a refined understanding of the layout structure and improves the accuracy of structure recognition and semantic expression capabilities.

[0050] In this embodiment of the application, a shared regression head means that all classification heads use the same regression head for regression prediction.

[0051] In this embodiment, by using multiple classification heads, the layout analysis model can better capture the differences and relationships between different levels of categories, improving classification accuracy. Simultaneously, the shared regression head can fully utilize the common information between different category prediction tasks, improving regression prediction accuracy. The shared regression head avoids designing separate regression heads for each category prediction task, thereby reducing the number of model parameters, lowering model complexity, and reducing the risk of overfitting. The method of learning and sharing features across multiple category prediction tasks enables the model to learn richer feature representations, improving its generalization ability on unseen data.

[0052] In this embodiment, the bounding box coordinate information output by the regression head constitutes a set of predicted bounding boxes. These bounding boxes identify possible layout element regions on the image, such as text regions, image regions, and table regions in a document image. The category label prediction scores output by multiple classification heads are associated with the predicted bounding boxes output by the regression head. Each predicted bounding box typically has a set of category label prediction scores from different classification heads, and these scores collectively describe the classification status of the predicted bounding box at different levels.

[0053] In some embodiments, such as Figure 4 As shown, the layout analysis model 40 may include: a feature extraction network 41 and a detection head 42. The detection head 42 includes: a regression head 421 and N classification heads 422_1, ..., 422_N, where N is an integer greater than 1.

[0054] After the first document image is input into the layout analysis model, image features are first extracted through the feature extraction network 41 (such as a convolutional neural network, CNN) in the layout analysis model. This feature extraction network consists of multiple convolutional layers, pooling layers, etc. Through the layer-by-layer calculation of these layers, representative features are extracted from the image to obtain a feature map. This feature map contains information such as the position, shape, and texture of different regions in the first document image, which is the basis for subsequent regression and classification tasks.

[0055] Then, the feature maps extracted by the feature extraction network 41 are input into the regression head 421 and the classification head 422_1, ..., 422_N in the detection head 42, respectively.

[0056] The regression head 421 is typically composed of fully connected layers or other layers suitable for regression tasks. It receives the feature map output by the feature extraction network 41 as input, calculates the coordinate information of the bounding box for each possible position in the feature map, and thus obtains the predicted bounding box. The aforementioned coordinate information generally includes the coordinates (x, y) of the top left corner of the bounding box, as well as the width (w) and height (h) of the bounding box. These coordinate information define the position and size of the bounding box in the image, and each predicted bounding box corresponds to a potential object region in the image.

[0057] The classification heads 422_1, ..., 422_N are designed based on the possible hierarchical categories of layout elements in the document image. For example, in document layout analysis, some classification heads can distinguish major categories such as body text, title, description, image, and table; some classification heads further distinguish natural images, charts, and intelligent graphics under the "image" category; and some classification heads further distinguish pie charts, line charts, and bar charts under the "chart" category. Each classification head has its own parameters and calculation logic, focusing on the classification of specific hierarchical categories. Each classification head 422_1, ..., 422_N predicts the corresponding hierarchical category label based on the feature map output by the feature extraction network 41 and the predicted bounding box output by the regression head 421, outputting a category label prediction score. The category label prediction score indicates the probability that the predicted bounding box belongs to each category label; the higher the score, the greater the probability of belonging to that category label.

[0058] In some embodiments, the layout analysis model may include, in addition to multiple classification heads and a shared regression head, a feature extraction network and a feature neck network; the feature extraction network is used to extract multi-level features of the first document image; the feature neck network is used to fuse and / or enhance the multi-level features output by the feature extraction network; wherein, the feature map output by the feature neck network is used by the regression head and multiple independent heads for corresponding prediction processing.

[0059] Specifically, such as Figure 5 As shown, the layout analysis model 50 may include: a feature extraction network 51, a feature neck network 52, and a detection head 53. The detection head 53 may include: a regression head 531 and N classification heads 532_1, ..., 532_N, where N is an integer greater than 1.

[0060] After the first document image is input into the layout analysis model, image features are first extracted through the feature extraction network 51 (such as convolutional neural network, CNN) in the layout analysis model. This feature extraction network consists of multiple convolutional layers, pooling layers, etc. Through the layer-by-layer calculation of these layers, multi-level features of the first document image are extracted, which is the basis for subsequent regression and classification tasks.

[0061] Next, the feature extraction network 51 inputs the extracted multi-level features into the feature neck network 52. The feature neck network 52 fuses and / or enhances the multi-level features output by the feature extraction network 51 to obtain a feature map.

[0062] Finally, the feature maps output by the feature neck network 52 are input into the regression head 531 and the classification head 532_1, ..., 532_N in the detection head 53, respectively.

[0063] The regression head 531 typically consists of fully connected layers or other layers suitable for regression tasks. It receives the feature map output by the feature neck network 52 as input, calculates the coordinate information of the bounding box for each possible position in the feature map, and thus obtains the predicted bounding box. The aforementioned coordinate information generally includes the coordinates (x, y) of the top left corner of the bounding box, as well as the width (w) and height (h) of the bounding box. These coordinate information define the position and size of the bounding box in the image, and each predicted bounding box corresponds to a potential object region in the image.

[0064] The classification heads 532_1, ..., 532_N are designed based on the possible hierarchical categories of layout elements in document images. For example, in document layout analysis, some classification heads can distinguish major categories such as body text, title, description, image, and table; some classification heads further distinguish natural images, charts, and intelligent graphics under the "image" category; and some classification heads further distinguish pie charts, line charts, and bar charts under the "chart" category. Each classification head has its own parameters and calculation logic, focusing on the classification of specific hierarchical categories. Each classification head 532_1, ..., 532_N predicts the corresponding hierarchical category label based on the feature map output by the feature neck network 52 and the predicted bounding box output by the regression head 531, outputting a category label prediction score. The category label prediction score indicates the probability that the predicted bounding box belongs to each category label; the higher the score, the greater the probability of belonging to that category label.

[0065] As can be seen, the layout analysis model based on the above network structure in this embodiment has the following advantages: 1) The feature extraction network can quickly and effectively extract representative features from the input document image. These features include key information such as the location, shape, and texture of different regions in the image, providing a solid foundation for subsequent regression and classification tasks. 2) The feature neck network further optimizes and fuses the features output by the feature extraction network, which can enhance the expressive power of the features, reduce redundant information, and make the features more suitable for subsequent regression and classification tasks, thereby improving the performance and accuracy of the model. 3) A regression head and multiple levels of classification heads are used to handle regression tasks (predicting bounding box positions) and classification tasks (predicting different level category labels), respectively. This processing method can improve the computational efficiency of the model, reduce processing time, and enable the model to obtain results faster. 4) Multiple classification heads can perform classification prediction for different level categories, enabling more detailed and accurate identification of various layout elements in the document, such as distinguishing between the major categories (text, images, tables, etc.) and minor categories (paragraphs, summaries, etc. in the text) of layout elements, thus improving the accuracy and depth of classification. 5) The non-classification head can be flexibly adjusted and expanded according to specific application needs, such as adding or removing classification levels, to adapt to the analysis tasks of different types of documents, enhancing the model's versatility and adaptability. 6) By sharing the feature extraction network and the feature neck network, the number of model parameters is reduced, lowering the model's complexity and avoiding overfitting caused by excessive model complexity. This also facilitates model training and deployment. 7) The above structure enables the model to learn richer and more general feature representations, improving its generalization ability on unseen document images and allowing it to better handle documents of various styles and types.

[0066] In object detection tasks, the predicted bounding boxes generated by the model often suffer from numerous problems, such as overlapping, redundancy, and excessive background information. Step 203, a post-processing step, effectively addresses these issues. It precisely filters through numerous predicted bounding boxes, eliminating inaccurate, redundant, or irrelevant ones, ultimately determining the final bounding boxes and category labels that accurately identify the location and extent of page elements. This process not only improves the accuracy of page element localization and reduces errors caused by inaccurate bounding boxes but also optimizes the analysis results, making the final page element information clearer and more reliable. This provides high-quality foundational data for subsequent document processing and analysis, improving the efficiency and accuracy of the entire document processing workflow.

[0067] In this embodiment of the application, the final category label corresponding to a layout element may include multiple hierarchical category labels. For example, for the layout element of a pie chart, its final category label may be "image label → chart label → pie chart label"; or, the final category label corresponding to a layout element may only include the most granular category label. For example, for the layout element of a pie chart, its final category label may be "pie chart label".

[0068] In this embodiment, the post-processing process is guided by a pre-defined hierarchical relationship between category labels, enabling more detailed and accurate identification of various layout elements in document images. This avoids the ambiguity and misclassification that may result from a single-level classification. This hierarchical relationship helps the layout analysis model better handle documents with complex layouts and multiple layout elements, meeting the needs of different application scenarios. Furthermore, the hierarchical relationship and classification criteria between category labels can be dynamically adjusted according to actual application requirements, enhancing the flexibility and adaptability of the parsing process.

[0069] In this embodiment, the prediction score-based filtering mechanism can effectively remove low-confidence prediction results, reduce redundant results, and improve the simplicity, usability, accuracy, and reliability of layout analysis results.

[0070] In this embodiment, based on the hierarchical relationship between preset category labels, the prediction results output by the model can be effectively transformed into structured layout analysis results with hierarchical category labels, providing reliable basic data support for applications such as document automation processing and information extraction.

[0071] As can be seen from the above embodiments, in this embodiment, the first document image to be identified is input into the layout analysis model; wherein, the layout analysis model includes: multiple classification heads and a shared regression head, the regression head is used to predict bounding boxes for each layout element in the first document image, and each of the multiple classification heads is used to predict the category label at the corresponding level for each predicted bounding box output by the regression head; the first document image is processed by the regression head and the multiple classification heads to obtain each predicted bounding box output by the layout analysis model and the category label prediction score corresponding to each predicted bounding box; based on the preset hierarchical relationship between category labels and the category label prediction score corresponding to each predicted bounding box, the final bounding box and the final category label corresponding to each layout element in the first document image are determined.

[0072] As can be seen, the layout analysis model in this embodiment has a unique structure, comprising multiple classification heads and a shared regression head. These classification heads share a single regression head, which uniformly predicts the bounding boxes of all layout elements. This avoids conflicts between multiple regression heads, reduces redundant parameters in the model, and allows for parallel execution of classification and regression tasks, minimizing processing latency. Simultaneously, by using each of the multiple classification heads to predict different levels of category labels on the predicted bounding boxes output by the regression head, interference between different categories is reduced. This enables detailed identification of various layout elements and their fine-grained classification labels, improving the comprehensiveness and accuracy of the analysis. The final result is determined based on the hierarchical relationship between category labels and the prediction scores output by each classification head, conforming to document layout logic. This significantly improves the accuracy of layout analysis of document images while maintaining efficient computation.

[0073] In some embodiments provided in this application, such as Figure 6 As shown, step 203 above may specifically include the following steps:

[0074] Steps 2031, 2032 and 2033.

[0075] In step 2031, for a layout element with only one predicted bounding box, the final bounding box corresponding to the layout element is determined based on the predicted category label score of the predicted bounding box corresponding to the layout element.

[0076] In step 2032, for a layout element with multiple predicted bounding boxes, the final bounding box corresponding to the layout element is determined based on the prediction scores of the primary category labels of each predicted bounding box.

[0077] In step 2033, the final category label corresponding to each final bounding box is determined based on the preset hierarchical relationship between category labels, the predicted scores of the primary category labels and the predicted scores of the subordinate category labels corresponding to each final bounding box.

[0078] The following is a detailed description of steps 2031 to 2033 mentioned above.

[0079] In this embodiment, for layout elements with only one predicted bounding box, the final bounding box is determined directly based on the predicted category label score of the predicted bounding box. This method avoids the complex multi-bounding box filtering process and uses the prediction score as a key indicator in a simple and efficient manner to quickly and accurately determine the final bounding box of the layout element, ensuring the accuracy of the positioning of such simple layout elements and providing a reliable foundation for subsequent processing.

[0080] In this embodiment, the category label prediction score typically reflects the confidence level of the layout analysis model in that the predicted bounding box belongs to a certain category; the higher the score, the greater the confidence level. Since there is only one predicted bounding box, and under normal circumstances, the predicted bounding box generated by the model has a certain degree of confidence, it can be directly determined as the final bounding box of the layout element.

[0081] In this embodiment, to ensure classification accuracy, a second threshold can be preset. If the predicted category label score of the predicted bounding box corresponding to a page element is greater than the second threshold, then the predicted bounding box is determined as the final bounding box corresponding to the page element. The second threshold can be determined based on factors such as the actual performance of the model and the accuracy requirements of the application scenario. For example, if high classification accuracy is required, a relatively high threshold can be set; if recall is more important, the threshold can be set slightly lower.

[0082] In this embodiment, for layout elements with multiple predicted bounding boxes, the final bounding box is determined by the prediction score of the primary category label of each predicted bounding box. Since the primary category label typically reflects the core features and main category of a layout element (e.g., in a document, body text, title, and description belong to different primary categories), filtering based on the primary category label prediction score can select the bounding box that best represents the essential features of the layout element from multiple potentially overlapping and redundant predicted bounding boxes. This effectively removes inaccurate or redundant bounding boxes, improves the accuracy and rationality of layout element positioning, and reduces information interference.

[0083] In this embodiment of the application, the subordinate category label is usually a further subdivision of the main category, for example... Figure 3 As shown, under the "Body Text" category label, it can be further divided into the following category labels: "Paragraph", "Abstract", etc.

[0084] In the embodiments of this application, the hierarchical relationship reflects the logical connection and subordinate relationship between different categories, such as the main text level being higher than the paragraph level, and the main text level being higher than the abstract level, etc.

[0085] In this embodiment, the category of each final bounding box is evaluated by comprehensively considering the predicted scores of the primary level category label, the predicted scores of the subordinate level category labels, and the hierarchical relationship between the category labels. For example, if the predicted score of the primary level category label of a certain final bounding box is high, and the predicted scores of the subordinate level category labels are also logical and meet the requirements of the hierarchical relationship, then the subordinate level category label is more likely to be the final category label.

[0086] For example, the primary category label of a final bounding box is "Text", with a predicted score of 0.85. The subordinate category labels are "Paragraph" (predicted score 0.7) and "Summary" (predicted score 0.6). Based on the preset hierarchical relationship and the predicted scores, the final category label of the final bounding box is determined to be "Paragraph".

[0087] In this embodiment, the final category label is determined by comprehensively considering the hierarchical relationship between preset category labels, the predicted scores of the primary-level category labels corresponding to each final bounding box, and the predicted scores of the subordinate-level category labels. This approach, which comprehensively considers multi-level information, enables a more comprehensive and in-depth understanding of the category characteristics of layout elements. The hierarchical relationship between category labels reflects the logical connection between different categories, while the predicted scores of the primary and subordinate-level category labels provide classification criteria at different granularities. Through this fusion of multi-dimensional information, the final category label corresponding to the final bounding box can be determined more accurately, avoiding classification bias that may be caused by single pieces of information. This improves the accuracy and reliability of category label determination and provides high-quality layout element information for subsequent document processing, analysis, and understanding.

[0088] As can be seen, in this embodiment, the above three steps constitute a complete and reasonable post-processing workflow, capable of adapting to page element recognition situations of varying complexity. This ensures that whether it's a page element with a single predicted bounding box or multiple predicted bounding boxes, accurate final bounding boxes and final category labels can be obtained through the corresponding steps. This flexibility enables the page analysis model to better handle various page structures in real-world documents, enhancing the model's robustness and practicality, allowing it to perform well in different scenarios, and providing strong support for related applications in the document processing field.

[0089] In some embodiments provided in this application, step 2032 may specifically include the following steps: step 20321 and step 20322.

[0090] In step 20321, candidate boxes are determined in each predicted bounding box based on the main-level category label prediction score of each predicted bounding box corresponding to the layout element; wherein the main-level category label prediction score of the candidate box is greater than the first threshold.

[0091] In step 20322, for all candidate boxes with the same primary category label, non-maximum suppression is performed on all candidate boxes based on the predicted scores of the primary category labels corresponding to all candidate boxes to obtain the final bounding boxes corresponding to the layout elements.

[0092] The following is a detailed description of steps 20321 to 20322 mentioned above.

[0093] In this embodiment, a first threshold can be preset based on factors such as the actual application scenario, model performance, and document layout complexity. This first threshold is used to filter out predicted bounding boxes with high confidence, i.e., candidate boxes. For example, if the model is well-trained and accuracy requirements are high, the first threshold can be set relatively high to ignore those less certain predicted bounding boxes.

[0094] In this embodiment of the application, the primary level category label prediction score of each predicted bounding box is compared with a first threshold. If the primary level category label prediction score of a predicted bounding box is greater than the first threshold, the predicted bounding box is determined as a candidate box; otherwise, the predicted bounding box is discarded.

[0095] For example, in document layout analysis, for a layout element that is suspected to be main text, the model generates 5 predicted bounding boxes with primary category label prediction scores of 0.8, 0.6, 0.9, 0.4, and 0.7, respectively. If the first threshold is set to 0.65, the predicted bounding boxes with primary category label prediction scores of 0.8, 0.9, and 0.7 will be identified as candidate boxes, while the predicted bounding boxes with prediction scores of 0.6 and 0.4 will be discarded.

[0096] In this embodiment, by setting a first threshold, only predicted bounding boxes with a primary-level category label prediction score greater than the threshold are retained as candidate boxes. This operation effectively filters out bounding boxes with low model prediction confidence, preventing low-quality bounding boxes from interfering with subsequent processing. For example, in a document layout, some blurry, unclear, or misjudged layout elements may generate bounding boxes with low prediction scores. Threshold filtering can exclude these irrelevant or erroneous bounding boxes, thus focusing on those bounding boxes that are more likely to accurately identify layout elements, improving the quality and reliability of candidate boxes. Furthermore, selecting only candidate boxes that meet the conditions from among many predicted bounding boxes greatly reduces the amount of data that needs to be processed in subsequent steps. This not only saves computing resources but also speeds up the entire layout analysis process. For example, when processing large documents or documents containing a large number of layout elements, reducing the number of bounding boxes processed can significantly shorten processing time and improve the real-time performance of the system. Since the candidate boxes are selected bounding boxes with high prediction scores for the primary category labels, they are more likely to correspond to the correct category. This lays a good foundation for further determining the final category labels based on these candidate boxes, which helps to improve the accuracy of category classification in the overall layout analysis.

[0097] In this embodiment, candidate boxes with the same main-level category label are further processed to remove redundant bounding boxes, resulting in final bounding boxes that can accurately identify layout elements.

[0098] In this embodiment of the application, all candidate boxes are grouped according to the main level category label of the candidate boxes. Candidate boxes in the same group have the same main level category label, indicating that they may correspond to the same layout element.

[0099] In this embodiment of the application, for each group of candidate boxes, they are sorted from high to low according to their main hierarchical category label prediction scores. The higher the prediction score, the more likely the model believes that the candidate box is an accurate bounding box.

[0100] In this embodiment, the candidate box with the highest score after sorting is selected as the baseline box. The Intersection over Union (IOU) ratio between the baseline box and other candidate boxes in the group is calculated. IOU is an indicator that measures the degree of overlap between two bounding boxes, and the calculation formula is the area of ​​the intersection of two bounding boxes divided by the area of ​​their union. An IOU threshold is set (usually determined according to the actual application scenario, such as 0.5). If the IOU of a candidate box with the baseline box is greater than the threshold, the candidate box is considered to have a high degree of overlap with the baseline box and may be a redundant bounding box, and is discarded; otherwise, the candidate box is retained. The above process is repeated until all candidate boxes in the group have been processed. The final retained candidate box is the final bounding box corresponding to the layout element.

[0101] For example, in step 20321, three candidate boxes were selected, all with the primary category label "body text," and predicted scores of 0.9, 0.8, and 0.7, respectively. After sorting by predicted scores from highest to lowest, they are 0.9, 0.8, and 0.7. The candidate box with a predicted score of 0.9 is selected as the baseline box. The Interchange of Union (IOU) between the baseline box and the candidate box with a predicted score of 0.8 is calculated, assuming it is 0.6. This is greater than the set IOU threshold of 0.5, so the candidate box with a predicted score of 0.8 is discarded. The IOU between the baseline box and the candidate box with a predicted score of 0.7 is calculated, assuming it is 0.3. This is less than the set IOU threshold of 0.5, so the candidate box with a predicted score of 0.7 is retained. Ultimately, the candidate boxes with predicted scores of 0.9 and 0.7 are retained. However, since only one final bounding box is selected, and the score of 0.9 is higher, the candidate box with a predicted score of 0.9 is determined as the final bounding box of the "body text" layout element.

[0102] In this embodiment, Non-Maximum Suppression (NMS) filtering is performed on all candidate boxes with the same primary-level category label based on their predicted scores. In document layout, the same element may generate multiple overlapping candidate boxes. These overlapping boxes not only increase data redundancy but can also cause confusion in subsequent processing. NMS removes low-scoring candidate boxes that highly overlap with high-scoring candidate boxes, retaining only the most representative bounding boxes as the final bounding boxes. This optimizes the localization of layout elements, making the final bounding boxes more accurately identify the actual position and extent of the elements. Furthermore, NMS prioritizes retaining candidate boxes with higher predicted scores, as these generally better reflect the model's accurate identification of the element. By removing redundant low-scoring candidate boxes, the final bounding boxes better represent the element, improving their accuracy and reliability. Furthermore, in document processing, consistent bounding box identification facilitates subsequent tasks such as document structure analysis and information extraction. Through NMS processing, the inconsistency of analysis results caused by multiple overlapping bounding boxes is avoided, making the layout analysis results of different documents or different parts of the same document more comparable and consistent, thus improving the stability and reliability of the entire document processing workflow.

[0103] In summary, in this embodiment, by setting a first threshold to filter candidate boxes, the number of bounding boxes that need to be processed subsequently can be significantly reduced, thereby reducing computational complexity and improving processing efficiency. Ignoring bounding boxes with low prediction scores avoids interference from these bounding boxes in subsequent non-maximum suppression and other processing steps, which helps improve the accuracy of the final bounding boxes. In document layout analysis, there may be multiple bounding boxes that are predicted to be of the same category and overlap with each other. Non-maximum suppression can effectively remove these redundant bounding boxes, retaining only the bounding boxes that are most likely to represent the object of that category, making the results more concise and accurate. By selecting the bounding boxes with the highest prediction scores and those that overlap with other bounding boxes for suppression processing, the final bounding boxes obtained can better locate the actual object regions, improving the localization accuracy of the bounding boxes.

[0104] In some embodiments provided in this application, step 2033 above may specifically include the following steps: step 20331 and step 20332.

[0105] In step 20331, for each final bounding box, the presence or absence of a subordinate category label is determined based on the predicted score of the subordinate category label corresponding to the final bounding box.

[0106] In step 20332, if the final bounding box has subordinate category labels, the last subordinate category label with the highest predicted category label score is determined as the final category label of the final bounding box; if the final bounding box does not have subordinate category labels, the main category label of the final bounding box is determined as the final category label of the final bounding box.

[0107] The following is a detailed description of steps 20331 to 20332 mentioned above.

[0108] In this embodiment of the application, after determining the primary level category label of the final bounding box, it is further determined whether the final bounding box has a subordinate level category label. The subordinate level category label is usually a further subdivision of the primary level category label. For example, in document layout analysis, when the primary level category label is "body text", its subordinate level category labels may be "paragraph", "abstract", etc.

[0109] In this embodiment, a rule for determining the existence of subordinate category labels can be set. For example, it can be checked whether the predicted score of the subordinate category label is greater than a preset threshold. If at least one subordinate category label predicted score is greater than the threshold, the final bounding box is considered to have a subordinate category label; otherwise, it is considered not to have one. Based on the predicted score of the subordinate category label corresponding to the final bounding box and the set judgment rule, each final bounding box is judged to determine whether it has a subordinate category label.

[0110] In this embodiment of the application, the terminal subordinate level category label refers to the category label at the lowest level in the hierarchical structure. For example... Figure 3 The third-level category labels in the hierarchy shown are the end-level subordinate category labels.

[0111] In this embodiment, when the final bounding box has a subordinate category label, the category label with the highest prediction score among the final subordinate categories (i.e., the most detailed / fine-grained category among the subordinate categories) is selected as the final category label corresponding to the final bounding box. For example, in the text subordinate category, the category label "paragraph" has the highest prediction score, so the category label "paragraph" is determined as the final category label. If the final bounding box does not have a subordinate category label, the primary category label corresponding to the final bounding box is directly determined as the final category label.

[0112] As can be seen, in this embodiment, by determining whether a subordinate category label exists, it is possible to clarify whether a more detailed classification of the bounding box is needed, providing a basis for subsequent refined classification and thus helping to improve the accuracy and depth of classification. For documents of varying complexity, simple documents may only require primary category classification, while complex documents require further differentiation of subordinate categories, which can well adapt to such document differences and offers high flexibility. Based on the existence of subordinate categories and the predicted scores, the final category label of the final bounding box can be accurately determined, making the classification results more consistent with reality and improving classification accuracy. Regardless of whether subordinate categories exist, there are clear processing rules, ensuring the consistency and repeatability of the processing flow, facilitating large-scale processing in practical applications.

[0113] In some embodiments provided in this application, in order to meet the user's personalized customization needs, an ignore map strategy can also be introduced in the layout analysis scenario. The ignore map strategy is a commonly used technique in tasks such as object detection and semantic segmentation. It is used to control the degree of attention the model pays to specific regions. It usually improves the robustness of the model or avoids interference by ignoring certain regions (such as difficult-to-label boundaries, occluded parts or background). Accordingly, the above step 203 can specifically include the following steps: step 2034.

[0114] In step 2034, based on the hierarchical relationship between preset category labels, the first label processing rule, and the category label prediction score corresponding to each predicted bounding box, the final bounding box and final category label corresponding to each layout element in the first document image are determined; wherein, the first label processing rule is used to indicate that the preset category labels are ignored.

[0115] In this embodiment of the application, the first tag processing rule can be a rule constructed based on the user's actual needs and the Ignore Map strategy.

[0116] In this embodiment of the application, during the inference stage, the first label processing rule can be selectively used according to business needs. For example, if certain categories are not defined or do not need to be identified in the first document image, the corresponding category prediction is automatically masked.

[0117] For example, if the first label processing rule ignores the "paragraph" and "description" categories during the inference process, and the first-level label of the predicted bounding box is "body text", then the category label of the bounding box is "body text".

[0118] As can be seen, in this embodiment of the application, users can choose which type of layout element to predict for the document image according to their actual needs, which can meet the personalized customization needs of users.

[0119] In some embodiments provided in this application, such as Figure 7 As shown, the training data in the training process of the layout analysis model is constructed through the following steps: Step 701, Step 702 and Step 703.

[0120] In step 701, at least two sample document images are acquired.

[0121] In step 702, bounding box annotations and end-level category label annotations are performed on each layout element in each sample document image to obtain the sample bounding box and end-level category label corresponding to each layout element in each sample document image.

[0122] In step 703, based on the preset hierarchical relationship between category labels, the sample bounding boxes corresponding to each page element in each sample document image, and the end-level category labels, the sample category labels of each level corresponding to each sample bounding box are determined; wherein, the training data of the page analysis model includes: each sample document image, the sample bounding boxes of each page element in each sample document image, and the sample category labels of each level corresponding to each sample bounding box.

[0123] The following is a detailed description of steps 701 to 703 mentioned above.

[0124] In this embodiment of the application, sample document images are the basic data for training the layout analysis model. Through these images, the model can learn various features and patterns of the document layout, providing a basis for subsequent bounding box prediction and category classification.

[0125] Given that publicly available datasets typically contain a rich variety of document types and layouts—for example, the PubLayNet dataset contains a large number of academic document images covering different journal formats, while the ICDAR dataset has document images from various scenarios, such as invoices and letters—this embodiment of the application can select sample document images from multiple publicly available datasets. Selecting multiple datasets increases data diversity, enabling the model to learn a wider range of document features.

[0126] In this embodiment of the application, in practical applications, in order to train a layout analysis model with good generalization ability, it is usually necessary to acquire a large number of sample document images, which may range from several thousand to hundreds of thousands, depending on the complexity of the model, task requirements and limitations of computing resources.

[0127] In this embodiment, professional annotation tools (such as LabelImg, CVAT, etc.) can be used to annotate the bounding boxes of each layout element in each sample document image. Annotators need to accurately delineate the bounding box of each layout element (such as "body text", "image", "table", etc.) to ensure that the bounding box can tightly surround the target element, while avoiding excessive overlap with the bounding boxes of other elements.

[0128] In this embodiment of the application, while annotating the bounding box, each layout element within the bounding box is labeled with its end-level category label. The end-level category label is the most detailed classification of the layout element. For example, under the "body text" category, the end categories may include "paragraph" and "abstract".

[0129] In this embodiment, the annotator needs to select the most suitable end-level category label for each bounding box based on the semantics and structure of the document. Through bounding box annotation and end-level category label annotation, the sample bounding boxes and end-level category labels corresponding to each layout element in each sample document image are obtained. This sample data is the basis for subsequently determining the sample category labels at each level and is also a key input for model training.

[0130] In this embodiment, the hierarchical relationship between category labels defines the hierarchical relationship from the main-level category label to the end-level category label. For example, the main-level category label may include "body text," "title," "description," "image," and "table." Subordinate-level category labels may be further subdivided under "image," such as "natural image," "chart," and "body text." End-level category labels are further subdivisions of the subordinate-level category labels, such as "pie chart," "line chart," and "bar chart" under the "chart" category label. Based on the sample bounding boxes of the layout elements of each sample document image and the sample category labels at each level corresponding to the sample bounding boxes, combined with the hierarchical relationship between category labels, the sample category labels at each level corresponding to each sample bounding box are determined from bottom to top. For example, referring to... Figure 3 As shown in the hierarchy, if the end-level category label of a bounding box is "pie chart", then it can be classified as "chart" in the subordinate level category label and "image" in the main level category label.

[0131] As can be seen, in this embodiment, after determining the sample category labels at each level corresponding to each sample bounding box, the training data for the layout analysis model is complete. This includes each sample document image, the sample bounding boxes of each layout element in each sample document image, and the sample category labels at each level corresponding to each sample bounding box. This data provides the model with rich, accurate, and hierarchically structured comprehensive supervision information, enabling the model to learn the mapping relationship from low-level features to high-level semantics. This helps the model better understand the structure and semantics of the document layout, thereby improving the accuracy of layout analysis, such as the accuracy of bounding box prediction and the precision of category classification.

[0132] In some embodiments provided in this application, the loss value during the training process of the layout analysis model is calculated based on the sample bounding box, predicted bounding box, and sample category label and predicted category label of each level corresponding to each layout element in the sample document image.

[0133] In this embodiment, a regression loss function can be used to calculate the difference between the sample bounding box and the predicted bounding box. The regression loss function includes, but is not limited to, Smooth L1 Loss and Intersection over Union (IOU) related loss function. Using the Smooth L1 Loss function can balance sensitivity to small errors and robustness to large errors. Using the IOU loss function can more intuitively reflect the localization accuracy of the bounding box.

[0134] In this embodiment, a classification loss function can be used to calculate the difference between the sample class label and the predicted class label. This classification loss function can be a cross-entropy loss. In multi-class classification tasks, cross-entropy loss measures the difference between the probability distribution predicted by the model and the true probability distribution. For each sample bounding box, the cross-entropy loss between its predicted class probability and the true class label is calculated. The cross-entropy losses of all sample bounding boxes are averaged to obtain the final class label loss.

[0135] In this embodiment, the bounding box loss and the category label loss can be weighted and summed to obtain the final comprehensive loss value. The weights can be adjusted according to the actual task requirements. For example, if more emphasis is placed on the accuracy of bounding box localization, the weight of the bounding box loss can be appropriately increased; if more emphasis is placed on the accuracy of category classification, the weight of the category label loss can be appropriately increased.

[0136] As can be seen, in this embodiment, considering that layout analysis typically includes two key tasks—bounding box localization and category classification—the comprehensive loss combines the bounding box loss (such as regression loss) and the category label loss (such as cross-entropy loss), enabling the model to simultaneously optimize on both tasks, thereby improving the multi-task learning effect. Furthermore, the comprehensive loss considers multiple aspects of information, allowing the model to access more comprehensive data features during the learning process. This helps the model reduce its dependence on specific data patterns, improves its adaptability to different document types, and thus enhances the model's generalization ability.

[0137] In some embodiments provided in this application, for scenarios where categories are missing or conflict, a preset second label processing rule can be used during the training process of the layout analysis model; wherein, the second label processing rule is used to set the loss value corresponding to the layout elements of the sample document image whose unlabeled category labels or whose labeled category labels have semantic conflicts to 0.

[0138] In this embodiment of the application, an Ignore Map strategy can also be introduced during the training phase of the layout analysis model, wherein the second label processing rule is created based on the Ignore Map strategy.

[0139] Considering the differences in annotation standards between different public datasets, issues such as missing categories (e.g., some datasets lacking stamps or page numbers) or category conflicts (e.g., "charts" being labeled as "natural images" in some datasets) often arise, leading to inconsistent supervision information during training and severely impacting the model's generalization ability. To address these issues, this application employs a second labeling rule to guide loss calculation during model training. Specifically, when a category is missing, the prediction for that category is marked as ignored and not included in the loss calculation for that category. In cases of category conflicts, the loss weights are dynamically adjusted to prevent backpropagation of errors to the model, thereby enhancing robustness and generalization ability during cross-dataset training and improving the model's robustness and transferability across multi-source data.

[0140] In summary, the embodiments of this application can effectively adapt to training on multiple datasets, enhance cross-task and cross-scenario transfer capabilities, thereby improving the model's generalization ability; accurately detect subdivided elements such as chart types and seal shapes, thereby enhancing fine-grained category recognition capabilities; multiple classification heads reduce interference between highly similar categories, improving overall detection accuracy and thus reducing inter-category false recognition rates; and support for label system expansion allows for rapid adaptation to different business needs, thereby improving system flexibility and scalability. The aforementioned layout analysis method, which balances generalization ability and recognition accuracy, can cope with the complex needs of multiple scenarios and multiple datasets.

[0141] Figure 8This is a schematic diagram of the structure of a layout analysis device provided in an embodiment of this application, as shown below. Figure 8 As shown, the layout analysis device 800 may include: an input module 801, a processing module 802, and a determination module 803.

[0142] The input module 801 is used to input the first document image to be identified into the layout analysis model; wherein, the layout analysis model includes: multiple classification heads and a shared regression head, the regression head is used to predict the bounding boxes of each layout element in the first document image, and each of the multiple classification heads is used to predict the category label of each predicted bounding box output by the regression head at the corresponding level.

[0143] Processing module 802 is used to process the first document image through the regression head and the multiple classification heads to obtain the predicted bounding boxes and the category label prediction scores corresponding to each predicted bounding box output by the layout analysis model.

[0144] The determination module 803 is used to determine the final bounding box and final category label corresponding to each layout element in the first document image based on the preset hierarchical relationship between category labels and the category label prediction score corresponding to each predicted bounding box.

[0145] As can be seen from the above embodiments, in this embodiment, the first document image to be identified is input into the layout analysis model; wherein, the layout analysis model includes: multiple classification heads and a shared regression head, the regression head is used to predict bounding boxes for each layout element in the first document image, and each of the multiple classification heads is used to predict the category label at the corresponding level for each predicted bounding box output by the regression head; the first document image is processed by the regression head and the multiple classification heads to obtain each predicted bounding box output by the layout analysis model and the category label prediction score corresponding to each predicted bounding box; based on the preset hierarchical relationship between category labels and the category label prediction score corresponding to each predicted bounding box, the final bounding box and the final category label corresponding to each layout element in the first document image are determined.

[0146] As can be seen, the layout analysis model in this embodiment has a unique structure, comprising multiple classification heads and a shared regression head. These classification heads share a single regression head, which uniformly predicts the bounding boxes of all layout elements. This avoids conflicts between multiple regression heads, reduces redundant parameters in the model, and allows for parallel execution of classification and regression tasks, minimizing processing latency. Simultaneously, by using each of the multiple classification heads to predict different levels of category labels on the predicted bounding boxes output by the regression head, interference between different categories is reduced. This enables detailed identification of various layout elements and their fine-grained classification labels, improving the comprehensiveness and accuracy of the analysis. The final result is determined based on the hierarchical relationship between category labels and the prediction scores output by each classification head, conforming to document layout logic. This significantly improves the accuracy of layout analysis of document images while maintaining efficient computation.

[0147] Optionally, as an embodiment, the determining module 803 is specifically used to: for a layout element with only one predicted bounding box, determine the final bounding box corresponding to the layout element based on the category label prediction score of the predicted bounding box corresponding to the layout element; for a layout element with multiple predicted bounding boxes, determine the final bounding box corresponding to the layout element based on the primary level category label prediction score of each predicted bounding box corresponding to the layout element; and determine the final category label corresponding to each final bounding box based on the preset hierarchical relationship between category labels, the primary level category label prediction score of each final bounding box, and the subordinate level category label prediction score.

[0148] Optionally, as an embodiment, the determining module 803 is specifically used to determine candidate boxes in each predicted bounding box based on the primary level category label prediction score of each predicted bounding box corresponding to the layout element; wherein, the primary level category label prediction score of the candidate box is greater than a first threshold; for all candidate boxes with the same primary level category label, non-maximum suppression processing is performed on all candidate boxes based on the primary level category label prediction scores of all candidate boxes to obtain the final bounding box corresponding to the layout element.

[0149] Optionally, as an embodiment, the determining module 803 is specifically used to determine whether a final bounding box has a subordinate category label based on the predicted score of the subordinate category label corresponding to the final bounding box; if the final bounding box has a subordinate category label, then the last subordinate category label with the highest predicted score of the category label corresponding to the final bounding box is determined as the final category label corresponding to the final bounding box; if the final bounding box does not have a subordinate category label, then the main category label corresponding to the final bounding box is determined as the final category label corresponding to the final bounding box.

[0150] Optionally, as an embodiment, the determining module 803 is specifically used to determine the final bounding box and final category label corresponding to each layout element in the first document image based on the hierarchical relationship between preset category labels, the first label processing rule, and the category label prediction score corresponding to each predicted bounding box; wherein, the first label processing rule is used to indicate that preset category labels are ignored.

[0151] Optionally, as an embodiment, the layout analysis model may further include: a feature extraction network and a feature neck network; the feature extraction network is used to extract multi-level features of the first document image; the feature neck network is used to fuse and / or enhance the multi-level features output by the feature extraction network; wherein, the feature map output by the feature neck network is used for corresponding prediction processing of the regression head and the multiple independent heads.

[0152] Optionally, as an embodiment, the main hierarchical category label may include: a primary category label and independent category labels; wherein, the primary category label may include at least one of the following: body text label, title label, description label, image label, and table label; the independent category label may include at least one of the following: header label, footer label, page number label, and stamp label;

[0153] The subordinate hierarchical category labels may include: second-level category labels and third-level category labels; wherein, the second-level category labels may include at least one of the following: paragraph labels under the body text label, summary labels under the body text label, natural image description labels under the description label, chart description labels under the description label, smart graphic description labels under the description label, table description labels under the description label, natural image labels under the image label, chart labels under the image label, smart graphic labels under the image label, full solid line table labels under the table label, half solid line table labels under the table label, no solid line table labels under the table label, circular stamp labels under the stamp label, triangular stamp labels under the stamp label, square stamp labels under the stamp label, and elliptical stamp labels under the stamp label; the third-level category labels may include at least one of the following: pie chart labels under the chart label, line chart labels under the chart label, and bar chart labels under the chart label.

[0154] Optionally, as an embodiment, the training data in the training process of the layout analysis model can be constructed in the following way: acquiring at least two sample document images; annotating each layout element in each of the sample document images with bounding boxes and end-level category labels to obtain the sample bounding boxes and end-level category labels corresponding to each layout element in each of the sample document images; determining the sample category labels of each level corresponding to each sample bounding box based on the preset hierarchical relationship between the category labels, the sample bounding boxes and end-level category labels corresponding to each layout element in each of the sample document images; wherein, the training data of the layout analysis model includes: each of the sample document images, the sample bounding boxes of each layout element in each of the sample document images, and the sample category labels of each level corresponding to each sample bounding box.

[0155] Optionally, as an embodiment, the loss value during the training process of the layout analysis model is calculated based on the sample bounding boxes, predicted bounding boxes, and sample category labels and predicted category labels at each level corresponding to each layout element in the sample document image.

[0156] Optionally, as an embodiment, the layout analysis model uses a preset second label processing rule during training; wherein, the second label processing rule is used to set the loss value corresponding to the layout elements of the sample document image whose unlabeled category labels or whose labeled category labels have semantic conflicts to 0.

[0157] Any step and specific operation in any step of the layout analysis method provided in this application can be completed by the corresponding module in the layout analysis device. The process of the corresponding operation completed by each module in the layout analysis device is described in the corresponding operation process in the implementation of the layout analysis method.

[0158] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0159] Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of this application, such as... Figure 9 As shown, the electronic device includes a processing component 922, which further includes one or more processors, and memory resources represented by memory 932 for storing instructions executable by the processing component 922, such as application programs. The application programs stored in memory 932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 922 is configured to execute instructions to perform the layout analysis method described above.

[0160] The electronic device may also include a power supply component 926 configured to perform power management of the electronic device, a wired or wireless network interface 950 configured to connect the electronic device to a network, and an input / output (I / O) interface 958. The electronic device may operate on an operating system stored in memory 932, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, or similar.

[0161] According to yet another embodiment provided in this application, this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps in the layout analysis method as described in any of the above embodiments.

[0162] According to yet another embodiment provided in this application, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps in the layout analysis method as described in any of the above embodiments.

[0163] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0164] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0165] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0167] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0168] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0169] The foregoing has provided a detailed description of the layout analysis method, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A layout analysis method, characterized in that, The method includes: The first document image to be identified is input into the layout analysis model; wherein, the layout analysis model includes: multiple classification heads and a shared regression head, the regression head is used to predict the bounding boxes of each layout element in the first document image, and each of the multiple classification heads is used to predict the category label of each predicted bounding box output by the regression head at the corresponding level. The first document image is processed using the regression head and the multiple classification heads to obtain the predicted bounding boxes and the corresponding category label prediction scores output by the layout analysis model. Based on the hierarchical relationship between preset category labels and the category label prediction scores corresponding to each predicted bounding box, the final bounding box and final category label corresponding to each layout element in the first document image are determined.

2. The method according to claim 1, characterized in that, The determination of the final bounding box and final category label corresponding to each layout element in the first document image based on the preset hierarchical relationship between category labels and the category label prediction score corresponding to each predicted bounding box includes: For a layout element with only one predicted bounding box, the final bounding box corresponding to the layout element is determined based on the predicted score of the category label of the predicted bounding box corresponding to the layout element. For a layout element with multiple predicted bounding boxes, the final bounding box corresponding to the layout element is determined based on the prediction scores of the main-level category labels of each predicted bounding box corresponding to the layout element. Based on the pre-defined hierarchical relationship between category labels, the predicted scores of the primary category labels and the predicted scores of the subordinate category labels corresponding to each final bounding box, the final category label corresponding to each final bounding box is determined.

3. The method according to claim 2, characterized in that, The step of determining the final bounding box corresponding to the layout element based on the predicted scores of the primary category labels of each predicted bounding box corresponding to the layout element includes: Based on the main-level category label prediction scores of each predicted bounding box corresponding to the layout element, candidate boxes are determined in each predicted bounding box; wherein, the main-level category label prediction score of the candidate box is greater than a first threshold. For all candidate boxes with the same primary category label, non-maximum suppression is performed on all candidate boxes based on the predicted scores of the primary category labels corresponding to all candidate boxes to obtain the final bounding boxes corresponding to the layout elements.

4. The method according to claim 2, characterized in that, The step of determining the final category label corresponding to each final bounding box based on the preset hierarchical relationship between category labels, the predicted scores of the primary-level category labels and the predicted scores of the subordinate-level category labels corresponding to each final bounding box includes: For each of the final bounding boxes, it is determined whether the final bounding box has a subordinate category label based on the predicted score of the subordinate category label corresponding to the final bounding box; If the final bounding box has subordinate category labels, then the last subordinate category label with the highest predicted category label score corresponding to the final bounding box is determined as the final category label corresponding to the final bounding box. If the final bounding box does not have a subordinate category label, then the primary category label corresponding to the final bounding box is determined as the final category label corresponding to the final bounding box.

5. The method according to any one of claims 1-4, characterized in that, The determination of the final bounding box and final category label corresponding to each layout element in the first document image based on the preset hierarchical relationship between category labels and the category label prediction score corresponding to each predicted bounding box includes: Based on the hierarchical relationship between the preset category labels, the first label processing rules, and the category label prediction scores corresponding to each predicted bounding box, the final bounding box and final category label corresponding to each layout element in the first document image are determined. The first label processing rule is used to indicate that preset category labels should be ignored.

6. The method according to any one of claims 1-4, characterized in that, The layout analysis model also includes: a feature extraction network and a feature neck network; The feature extraction network is used to extract multi-level features from the first document image; The feature neck network is used to fuse and / or enhance the multi-level features output by the feature extraction network; The feature map output by the feature neck network is used for corresponding prediction processing of the regression head and the multiple independent heads.

7. The method according to claim 2, characterized in that, The main hierarchical category labels include: primary category labels and independent category labels; wherein, the primary category labels include at least one of the following: body text label, title label, description label, image label, and table label; the independent category labels include at least one of the following: header label, footer label, page number label, and stamp label; The subordinate category labels include: second-level category labels and third-level category labels; The second-level category tags include at least one of the following: paragraph tags under the main text tag, summary tags under the main text tag, natural image description tags under the description tag, chart description tags under the description tag, smart graphic description tags under the description tag, table description tags under the description tag, natural image tags under the image tag, chart tags under the image tag, smart graphic tags under the image tag, full solid line table tags under the table tag, half solid line table tags under the table tag, no solid line table tags under the table tag, circular stamp tags under the stamp tag, triangular stamp tags under the stamp tag, square stamp tags under the stamp tag, and elliptical stamp tags under the stamp tag; The three-level category labels include at least one of the following: pie chart label under the chart label, line chart label under the chart label, and bar chart label under the chart label.

8. The method according to claim 6, characterized in that, The training data for the layout analysis model is constructed in the following way: Obtain at least two sample document images; Boundary box annotation and end-level category label annotation are performed on each page element in each of the sample document images to obtain the sample boundary box and end-level category label corresponding to each page element in each of the sample document images; Based on the hierarchical relationship between the preset category labels, the sample bounding boxes corresponding to each layout element in each sample document image, and the end-level category labels, the sample category labels of each level corresponding to each sample bounding box are determined. The training data for the layout analysis model includes: each of the sample document images, sample bounding boxes of each layout element in each of the sample document images, and sample category labels at each level corresponding to each of the sample bounding boxes.

9. The method according to claim 8, characterized in that, The loss value during the training process of the layout analysis model is calculated based on the sample bounding boxes, predicted bounding boxes, and sample category labels and predicted category labels at each level corresponding to each layout element in the sample document image.

10. The method according to claim 9, characterized in that, The layout analysis model uses a preset second label processing rule during training; The second label processing rule is used to set the loss value of layout elements in the sample document image that have unlabeled category labels or whose labeled category labels have semantic conflicts to 0.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the layout analysis method according to any one of claims 1-10.

12. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the layout analysis method according to any one of claims 1-10.

13. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the layout analysis method according to any one of claims 1-10.

Images