Layout analysis method and device for customs declaration attached document, computer equipment and computer program product

Abstract

The invention provides a customs declaration attached document-oriented layout analysis method and device, computer equipment and a computer program product. The method comprises the following steps: acquiring a to-be-analyzed customs declaration attached document image; performing layout analysis on the document image, detecting and identifying a plurality of layout element areas in the image, and generating an initial labeling box with category labels and confidence for each layout element area; combining and optimizing the initial annotation boxes to obtain an optimized annotation box set; performing optical character recognition on the document image to obtain a plurality of OCR textboxes and text contents thereof; the OCR textbox is associated with the optimized annotation box set, and corresponding text content is distributed to each optimized annotation box; based on the category labels and the position information of the optimized labeling boxes, sorting the labeling boxes to which the text contents are distributed, and constructing a hierarchical structure of the document; and finally, outputting structured data according to the hierarchical structure.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a method, apparatus, computer equipment and computer program product for layout analysis of documents accompanying customs declarations. Background Technology

[0002] Document Layout Analysis (DLA) aims to automatically identify various layout elements (such as text, tables, and headings) and their layout structure in document images, providing a foundation for subsequent information processing. Existing technical solutions mainly fall into two categories: In existing technologies, single-modal methods based on computer vision are mainstream. For example, object detection models such as Faster R-CNN and YOLO are used, taking document images as input to directly detect and locate layout units. In addition, multimodal methods that integrate visual and textual features (such as LayoutLMv3) exist to improve the accuracy of the analysis. For preparing model training data, manual annotation tools (such as LabelMe) or semi-automatic annotation tools (such as SegBuilder based on the SAM model) are typically relied upon to generate labeled data.

[0003] Despite the progress made by the aforementioned existing technical methods, the following obvious drawbacks still exist when applied to the specific scenario of accompanying documents for customs declarations: Poor adaptability to specific layout elements: Existing models are not optimized for non-text elements such as fixed-position electronic signatures, watermarks / stamps, etc., which are common in customs documents. They are easily misidentified as text or tables, introducing noise. At the same time, for "implicit tables" (without physical borders) that are commonly found in documents and are separated by spaces, traditional methods will split them into independent text blocks, failing to restore their inherent row and column structure, resulting in chaotic information extraction.

[0004] Data dependency and insufficient generalization ability: Deep learning-based methods require a large amount of labeled data for training or fine-tuning. However, customs documents are mostly classified documents, and there are few anonymized samples available for the model. This makes the model prone to overfitting, making it difficult to cover document variations from different companies and in different formats, resulting in limited generalization ability.

[0005] The efficiency and effectiveness of annotation tools are poor: purely manual annotation is time-consuming and labor-intensive. The annotation results generated by semi-automatic annotation tools (such as masks) usually only contain location information and lack the category labels and hierarchical relationship information between areas (such as the subordinate relationship between table notes and tables) necessary for layout analysis. They still require a lot of manual intervention for correction and classification, which is inefficient. Summary of the Invention

[0006] To address the technical problem mentioned in the background art of effectively identifying and processing special elements such as signatures and hidden forms, and to overcome the weak generalization ability of existing technologies due to the scarcity of training data, this application provides a method, apparatus, computer equipment, and computer program product for layout analysis of documents accompanying customs declarations. The aim is to provide an efficient and accurate layout analysis solution for documents accompanying customs declarations, which can significantly improve the automation and accuracy of layout analysis and structure reconstruction.

[0007] In a first aspect, embodiments of this application provide a method for layout analysis of documents accompanying a customs declaration, the method comprising: S1: Obtain images of the documents accompanying the customs declaration to be analyzed; S2: Utilize the pre-trained DocLayout-YOLO model to perform layout analysis on the document image, detect and identify multiple layout element regions in the document image, and generate an initial label box with category labels and confidence levels for each layout element region; the category labels include text boxes, tables, implicit tables, and customs stamps; S3: Based on the overlap value (lap), the initial bounding boxes are merged and optimized to eliminate redundant boxes, resulting in an optimized set of bounding boxes; S4: Perform optical character recognition (OCR) on the document image to obtain multiple OCR text boxes and their text content; S5: Match the OCR text box with the optimized annotation box set by coordinate matching, assign corresponding text content to each optimized annotation box, and obtain the target annotation box; S6: Based on the category labels and position information of the optimized annotation boxes, sort the target annotation boxes that have been assigned text content to construct a hierarchical structure of the document; S7: Output structured data based on the hierarchical structure.

[0008] In this embodiment of the application, the complete process from image input in step S1 to structured output in step S7 realizes automated and structured analysis of the document layout accompanying the customs declaration, solving the problems of poor adaptability to customs declaration format, redundant and chaotic analysis results, low efficiency of manual review, and confusion caused by inconsistent formats in the existing technology.

[0009] In one embodiment of the first aspect, the step of merging and optimizing the initial bounding boxes in S3 includes: S31: Assign an index to each of the initial bounding boxes, the index being generated based on its confidence level; S32: Calculate the lap value between any two initial annotation boxes; S33: When the lap value of two initial annotation boxes is greater than the preset lap threshold, the two initial annotation boxes are merged into a new annotation box, and the index of the new annotation box is set to the index of the one with the earlier index among the two initial annotation boxes being merged. S34: Repeat steps S32 and S33 until there are no bounding box pairs with a lap value greater than the lap threshold, and obtain the optimized set of labeled bounding boxes.

[0010] In this embodiment of the application, when merging the initial annotation boxes, since the arrangement order of the original annotation information is based on the confidence level, an index is set for each annotation information in order to retain the confidence level information. The index of the merged new annotation box is set to the one with the first index in the merged box. After the merging is completed, the merged box is deleted from the list. The merging continues until there are no more boxes that can be merged. The optimized annotation box set is sorted by index, which can solve the problem of overlapping box redundancy and conflict caused by low confidence threshold.

[0011] In one embodiment of the first aspect, the step of associating the OCR text box with the optimized set of annotation boxes in S5 by coordinate matching includes: Using the optimized annotation boxes as a reference, calculate the lap value between each OCR text box and each optimized annotation box; Associate an OCR text box with all optimized annotation boxes whose lap value is greater than a preset value.

[0012] In this embodiment of the application, by fusing structural information, the problem of accurately mapping OCR text content to page element areas can be solved.

[0013] In one embodiment of the first aspect, the step of assigning corresponding text content to each optimized annotation box in S5 specifically includes: For the optimized annotation box with the category of title block, the text content of all its associated OCR text boxes is combined to form a complete title text; For optimized annotation boxes categorized as table blocks, the row and column data of the table are generated using the text content of the associated OCR text boxes.

[0014] In this embodiment, text content can be integrated according to page layout category for subsequent processing and analysis.

[0015] In one embodiment of the first aspect, the step of sorting the target annotation boxes assigned text content in S6 includes: Determine the center point of each target annotation box; Sort the target annotation boxes in ascending order according to the horizontal coordinates of the center point of the annotation box; While ensuring the order of the horizontal coordinates of the center points of each target annotation box, sort them in ascending order according to the vertical coordinates of the center points of the target annotation boxes.

[0016] In this embodiment, the problem of how to restore the correct reading order of documents, especially the order of horizontally formatted customs declarations, is solved.

[0017] In one embodiment of the first aspect, the step of constructing the hierarchical structure of the document in S6 includes: From the optimized set of annotation boxes, identify the annotation boxes whose category is title and whose associated text content is empty, and mark them as top-level main titles; For non-table page portions that do not have a top-level heading with a category as the title, assign a "Untitled" title and mark it as an "Untitled" heading; Based on the top-level main title and / or the "Untitled" title (i.e., the top-level title of the Untitled title), the subordinate annotation boxes of the category of title or body text are grouped to form a hierarchical document structure.

[0018] In this embodiment, not only is the reading order of the document restored, but its internal logical organization can also be understood, laying a solid foundation for subsequent information extraction and data analysis.

[0019] Secondly, this application also proposes a layout analysis device for documents accompanying a customs declaration, comprising: The upload unit is used to obtain images of the documents attached to the customs declaration to be analyzed; The analysis unit is used to perform layout analysis on the document image using a pre-trained DocLayout-YOLO model, detect and identify multiple layout element regions in the document image, and generate an initial label box with category labels and confidence scores for each layout element region; the category labels include text boxes, tables, implicit tables, and customs stamps; An optimization unit is used to merge and optimize each of the initial bounding boxes based on the overlap value (lap) to eliminate redundant boxes and obtain an optimized set of bounding boxes. The recognition unit is used to perform optical character recognition (OCR) on the document image to obtain multiple OCR text boxes and their text content. The fusion unit is used to perform coordinate matching and association between the OCR text box and the optimized annotation box set, and to assign corresponding text content to each optimized annotation box to obtain the target annotation box; The construction unit is used to sort the target annotation boxes with assigned text content based on the category labels and position information of the optimized annotation boxes, and to construct the hierarchical structure of the document. The output unit is used to output structured data according to the hierarchical structure.

[0020] Thirdly, this application also proposes a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0021] Fourthly, this application also proposes a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0022] It should be noted that the technical effects of the second to fourth aspects mentioned above are explained in the description of the technical effects of the first aspect, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a main embodiment of a layout analysis method for documents accompanying a customs declaration provided in this application; Figure 2 A flowchart illustrating an embodiment of a layout analysis method for accompanying documents of a customs declaration provided in this application, which involves merging and optimizing initial annotation boxes; Figure 3 A comparative schematic diagram showing the original image a of the accompanying documents for the hazardous chemical customs declaration provided as an application example of this application and the analysis result b obtained by the method of this invention. Figure 4 A comparative diagram showing the analysis results before optimization (a) and after optimization (b) provided as an application example of this application; Figure 5 A comparative diagram showing the original image a obtained from layout analysis and the title box result b obtained after OCR combined with layout analysis, provided as an application example of this application; Figure 6 A schematic diagram illustrating component information provided as an application example in this application; Figure 7The original table a and the table content b extracted by OCR are provided as an application example for this application; Figure 8 To obtain a formatted table through the layout analysis method for documents accompanying customs declarations as described in this application; Figure 9 A structural block diagram of a layout analysis device for documents accompanying a customs declaration provided in this application; Figure 10 This application provides a schematic diagram of an embodiment of a computer device for performing a layout analysis method for documents accompanying a customs declaration. Detailed Implementation

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0029] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), unless otherwise expressly and specifically defined.

[0030] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0032] The terms “including,” “comprising,” “having,” and variations thereof all mean “including but not limited to,” unless otherwise specifically emphasized.

[0033] It should be noted that the purpose of this invention is to address the problem of inconsistent formats and chaotic layouts in accompanying documents to customs declarations (especially hazardous chemical safety data sheets), leading to low efficiency and high error rates in manual review. This invention aims to develop a layout analysis method, apparatus, computer equipment, and computer program product for accompanying documents to customs declarations. The layout analysis method for accompanying documents to customs declarations can be classified as a computer analysis program, and the layout analysis apparatus can be a web layout analysis tool based on the DocLayout-YOLO model. This tool integrates functional modules such as document upload and management, automated annotation generation (object detection), interactive annotation adjustment, text recognition and layout information fusion, and model upload and fine-tuning to achieve efficient layout analysis and key information extraction of accompanying documents to customs declarations. This improves the efficiency and accuracy of customs' compliance review of import and export documents and reduces the burden of manual review.

[0034] To address the aforementioned technical issues, this application provides a layout analysis method for documents accompanying customs declarations. Please refer to [link / reference]. Figure 1 , Figure 1 A flowchart illustrating the first embodiment of a layout analysis method for documents accompanying a customs declaration provided in this application; The layout analysis method for accompanying documents of customs declarations provided in this embodiment of the invention mainly includes the following steps S1 to S7: Step S1: Obtain images of the documents accompanying the customs declaration to be analyzed; Understandably, the analytical method in this application is implemented by the aforementioned web layout analysis tool based on the DocLayout-YOLO model. Through a technical path of "model optimization - functional modularization - system integration," it achieves efficient processing of documents with inconsistent formats and chaotic layouts. This system integrates image uploading and management functions, supporting batch uploading and format conversion of ZIP compressed packages.

[0035] In its implementation, the system provides users with a convenient document input interface through an interactive front-end page, allowing users to process a large number of documents at once. The format conversion function ensures that subsequent processing modules can receive standardized image input, which is the starting point and data preparation stage of the entire analysis methodology process.

[0036] Considering the layout characteristics of documents accompanying customs declarations, this invention defines corresponding layout categories and utilizes the DocLayout-YOLO model for initial annotation, which inherently includes category information. Through an interactive interface, users can quickly adjust and categorize these annotation areas and, based on the layout characteristics, efficiently complete annotations by dragging and dropping square annotation boxes with the mouse.

[0037] Step S2: Use the pre-trained DocLayout-YOLO model to perform layout analysis on the document image, detect and identify multiple layout element regions in the document image, and generate an initial bounding box with category label and confidence for each layout element region; For example, the category labels may include, but are not limited to, text boxes, tables, implicit tables and customs stamps, GHS diagrams, and obsolete parts (such as page numbers and page footers).

[0038] It should be noted that this step uses the DocLayout-YOLO model as the core, integrating object detection and layout structure analysis capabilities, and is optimized for the characteristics of the accompanying documents of customs declarations: the DocLayout-YOLO model based on the YOLOv10 algorithm is fine-tuned using customs document images (including scanned copies and electronic documents), the object detection head and loss function are optimized, the recognition accuracy of special elements such as text boxes, tables, implicit tables (space-separated), and customs stamps is improved, and the overlapping box problem is solved (by merging redundant boxes through a custom lap overlap degree algorithm).

[0039] Understandably, this step is the cornerstone of intelligent analysis; for example, the "pre-trained DocLayout-YOLO model" can be a model "based on the YOLOv10 algorithm and fine-tuned using customs document images." Its output, the "initial bounding boxes with category labels and confidence scores," is precisely the result of the model recognizing elements such as "text boxes, tables, implicit tables, and customs stamps."

[0040] Step S3: Based on the overlap value (lap), merge and optimize the initial bounding boxes to eliminate redundant boxes, obtaining an optimized set of bounding boxes; the formula for calculating the lap value is: lap = (A ∩ B) / min(A, B), where A and B are the areas of the two bounding boxes to be merged and optimized, respectively; Understandably, the bounding box in step S3 can be a detection box or a bounding box; It should be noted that the confidence threshold mentioned in this application refers to a score assigned by the DocLayout-YOLO model to each bounding box it detects. This score represents the model's confidence in two aspects: (location confidence) how accurately the model believes the box is located; and (category confidence) how likely the model believes the content within the box belongs to its predicted category (such as text, table, etc.).

[0041] To obtain more complete layout information, a lower confidence threshold needs to be selected during detection. This often leads to overlapping bounding boxes in the image, especially when processing documents containing a large amount of text and other information, such as customs declarations and accompanying documents. Overlapping boxes cause redundancy and conflicts in layout information, making it difficult to obtain text location information and subsequent analysis. To address the problem of overlapping bounding boxes caused by dense text and intersecting tables in documents, this embodiment proposes a merging strategy based on lap (degree of overlap), calculating the degree of overlap using the minimum area of ​​the two boxes as the denominator.

[0042] The lap value is between 0 and 1. A value of 1 indicates that the larger box completely encloses the smaller box or the two boxes completely overlap. Since the denominator becomes smaller, lap will be larger than IoU in value. In actual use, the threshold can be set slightly larger. In this embodiment of the invention, the default threshold lap_threshold can be 0.8.

[0043] In some embodiments, reference Figure 2 The merging and optimization of the initial annotation boxes in step S3 may specifically include: S31: Assign an index to each of the initial bounding boxes, the index being generated based on its confidence level; S32: Calculate the lap value between any two initial annotation boxes; S33: When the lap value of two initial annotation boxes is greater than the preset lap threshold, the two initial annotation boxes are merged into a new annotation box, and the index of the new annotation box is set to the index of the one with the earlier index among the two initial annotation boxes being merged. S34: Repeat steps S32 and S33 until there are no bounding box pairs with a lap value greater than the lap threshold, and obtain the optimized set of labeled bounding boxes.

[0044] It should be noted that during the initial merging of annotation boxes, since the original annotation information is arranged according to the confidence level, an index is set for each annotation information in order to preserve the confidence level information. The index of the merged new annotation box is set to the index of the first one in the merged box. After the merging is completed, the merged box is deleted from the list. The merging is continued until there are no more boxes that can be merged. The remaining boxes in the list ("i.e., the optimized annotation box set") are sorted by index and saved into a new annotation txt file.

[0045] Understandably, this step is crucial for improving the cleanliness of the analysis results. From calculating the lap value and comparing it with the threshold, to the merging strategy (inheriting the first index) and iterative loop, the final output is the "optimized set of bounding boxes" and it is saved. This can solve the problem of overlapping box redundancy and conflict caused by low confidence threshold.

[0046] Step S4: Perform optical character recognition (OCR) on the document image to obtain multiple OCR text boxes and their text content; It should be noted that step S4 is a key technical step in achieving text extraction. For example, text recognition can be performed by combining OCR engines such as PPOCRv3.

[0047] Understandably, this step is performed in parallel with or after the aforementioned S3 layout analysis steps, focusing on extracting raw text information from the image; In this embodiment, the "Optical Character Recognition (OCR)" in step S4 can use the "PPOCRv3" engine; its output "multiple OCR text boxes and their text content" is the unstructured raw recognition result, which prepares for the next step S5 of merging with the layout structure.

[0048] Step S5: Match the coordinates of the OCR text box with the optimized annotation box set, and assign corresponding text content to each optimized annotation box to obtain the target annotation box; It should be noted that step S5 is the core of information fusion. In the layout analysis process of this invention, in addition to optimizing the detection results to handle the overlapping box problem, it is also necessary to effectively fuse the text recognition results with the layout structure information to construct a complete and accurate document structure. This process is crucial for understanding the content hierarchy and semantic relationships of the document, especially when processing documents such as customs declaration documents that contain a large amount of text and multiple information types.

[0049] It is understandable that after completing layout detection and text recognition, it is necessary to associate the text boxes recognized by OCR with the annotation boxes obtained from layout detection. Each annotation box represents a specific area in the document, such as a title, table, or body text, while the OCR text box contains the specific text content within that area. To achieve this association, this embodiment employs a "coordinate matching association" method: First, the coordinates of the YOLO format annotation boxes are converted to image coordinates for unified comparison with the coordinates of the OCR text boxes. Then, the overlap between each OCR text box and the annotation box is calculated to determine if they belong to the same region. In some embodiments, step S5, which involves coordinate matching and association between the OCR text boxes and the optimized annotation box set, may specifically include: calculating the lap value between each OCR text box and each optimized annotation box, using the optimized annotation boxes as a reference; associating an OCR text box with all optimized annotation boxes whose lap value is a preset value (preferably 0.005), thereby accurately mapping the OCR text content to the page element area. It can be understood that the overlap calculation method used in step S5 is the same as the lap method mentioned in the aforementioned S3 detection result optimization step, which calculates the overlap value. Using the annotation boxes as a reference, each annotation box is associated with all OCR text boxes that overlap with it; one text box can correspond to multiple annotation boxes. This situation applies when a page can serve as multiple categories, such as a section that can be both a title and a table, or both text and a table.

[0050] Then, after determining the association between the OCR text boxes and the annotation boxes, it is necessary to assign the text content recognized by the OCR to the corresponding annotation boxes. In some embodiments, the assignment of corresponding text content to each optimized annotation box in step S5 may specifically include: for optimized annotation boxes of the category of title blocks, combining the text content of all associated OCR text boxes to form a complete title text; for optimized annotation boxes of the category of table blocks, generating table row and column data through the text content of associated OCR text boxes.

[0051] As you can understand, step S5 divides the text content into different sections based on the category of the annotation boxes; here, it's divided into title blocks and table blocks. For title blocks, the text content in the associated OCR text boxes is combined into a complete title and displayed according to a specific format. For table blocks, the content in the OCR text boxes within each table is organized to generate the table's row and column information for subsequent table processing and analysis.

[0052] Step S6: Based on the category labels and position information of the optimized annotation boxes, sort the target annotation boxes that have been assigned text content to construct a hierarchical structure of the document; In this embodiment, after the text content is allocated, the overall structure of the document is generated based on the category and hierarchical relationship of the annotation boxes. For documents containing multiple headings and tables, a hierarchical structure is constructed to clearly display the document's content organization and logical relationships. For example, for a document containing multiple chapters and tables, the headings and corresponding body text of each chapter are grouped, and the tables are saved separately as an Excel file.

[0053] To ensure that the generated document structure accurately reflects the original content and layout, it is necessary to process the annotation box position information and text box position information.

[0054] Since the annotation boxes were used as the reference when matching the text boxes previously, and the annotation boxes were sorted first, in one embodiment, the step of sorting the target annotation boxes assigned text content in step S6 may specifically include: Determine the center point of each target annotation box; sort the target annotation boxes in ascending order according to the horizontal coordinates of their center points; while ensuring the order of the horizontal coordinates of the center points of each target annotation box, sort them in ascending order according to the vertical coordinates of their center points. Understandably, in computer image processing, y-coordinate values ​​increase from top to bottom, while x-coordinate values ​​increase from left to right. Customs declaration documents are horizontally formatted. Therefore, the center point of each annotation box is sorted in multiple ascending orders: first, the horizontal position (x-coordinate) is sorted, and then the vertical position (y-coordinate) is sorted without disrupting this dimension. Without considering hidden tables, the text arrangement of the accompanying documents on the customs declaration is relatively regular, and horizontal or vertical reversal is usually not observed. Therefore, for the title boxes, only the relative positions of the annotation boxes need to be considered. All title boxes are sorted according to their vertical position (y_center). If two title boxes have similar vertical positions (the difference is less than a certain threshold), they are sorted according to their horizontal position (x_center). This ensures that the title boxes are arranged in a top-to-bottom, left-to-right order.

[0055] Then, in some embodiments, the step of "building a hierarchical structure of the document" in S6 may specifically include: From the optimized set of annotation boxes, identify the annotation boxes whose category is title and whose associated text content is empty, and mark them as top-level main titles; For non-table page portions that do not have a top-level heading with a category as the title, assign a "Untitled" title and mark it as an "Untitled" heading; Based on the top-level main title and / or the "Untitled" title, the subordinate annotation boxes categorized as headings or body text are grouped to form a hierarchical document structure.

[0056] Understandably, for hierarchical structures with nested subheadings, the system will identify from the optimized set of annotation boxes (from step S3) those that are categorized as "heading" and whose associated text content (from step S5) is empty. These empty heading annotation boxes are usually because the main heading itself may be detected by the model through visual features such as font, size, or position, but its text content may be identified as a separate OCR text box. The system identifies and marks these annotation boxes as top-level main headings.

[0057] For non-table layout sections of a document that do not have a heading box at the top (such as the header area of ​​some documents or the body text that begins directly), the system will use a default "Untitled" as its top-level heading. This ensures that every logical part of the document has a corresponding heading node, maintaining structural integrity.

[0058] After identifying and completing the titles, the system groups the subordinate headings categorized as "title" or "body text" based on the top-level main title and / or "untitled" headings. For example, all subheadings and body paragraphs under a main title will be grouped under that main title node, thus clearly defining the hierarchical relationship between chapters and content, ultimately forming a hierarchical document structure. This process mechanism in this embodiment not only restores the reading order of the document but also enables a deeper understanding of its internal logical organization, laying a solid foundation for subsequent information extraction and data analysis.

[0059] Step S7: Output structured data according to the hierarchical structure.

[0060] It should be noted that step 7 ultimately outputs structured data, which can be in JSON format for data exchange, or in an Excel spreadsheet or a TXT document. In this embodiment, the system utilizes the Flask framework to build a RESTful API, encompassing key functions such as document upload processing, analysis result feedback, text box positioning information provision, saving user-modified annotation information, and OCR recognition execution. The backend routing efficiently handles document upload requests, analyzes document images using the DocLayout-YOLO model, optimizes them with appropriate algorithms, and outputs the analysis results in JSON format for further frontend processing. Simultaneously, the system includes five core modules: model management (upload, fine-tuning, and testing interface), automated annotation (DocLayout-YOLO generates initial annotation boxes), interactive adjustment (mouse dragging to modify box position / category), and text extraction (combining PPOCRv3 with layout information to output Excel / txt). Ultimately, this achieves efficient layout analysis and key information extraction for the entire customs declaration and its accompanying documents.

[0061] The technical advantage of this application's embodiments lies in the following: When processing documents containing a large amount of text and information, such as customs declaration accompanying documents, low confidence thresholds can easily lead to text box overlap, resulting in redundancy and positional information conflicts, which makes subsequent analysis difficult. To address this, a merging strategy based on overlap degree is proposed. This strategy optimizes box merging by calculating the overlap degree (lap value), retains confidence information, and saves the merged boxes in sequence. Next, a coordinate matching method is used to associate the text boxes recognized by OCR with the annotation boxes, assigning text content to the annotation boxes and simultaneously constructing the document's hierarchical structure to ensure accurate reflection of the original content. The boxes are sorted first by horizontal position and then by vertical position to ensure that the title and body text are arranged in a reasonable order, ultimately forming a document structure with clear logical relationships.

[0062] The advantages of this application compared to existing technologies are: Existing manual annotation tools like LabelMe and LabelStudio, which require clicking point by point to determine bounding boxes and select categories, significantly increase annotation time, averaging about 3 minutes per image. Semi-automatic tools like SegBuilder generate complex semantic segmentation (SAM) masks with redundant mask position information, resulting in massive computational demands. In contrast, the layout analysis method for customs declaration documents based on this invention requires only 40 milliseconds for single-image recognition, and the time for manual adjustment of annotation results is reduced to half a minute. Furthermore, this invention can generate COCO format label files, facilitating adjustments to the Doclayout-YOLO model and subsequent layout analysis, significantly improving work efficiency.

[0063] In practical applications, one can choose an accompanying document to a hazardous chemicals customs declaration, such as the original image. Figure 3As shown in 'a', by applying the layout analysis method for customs declaration documents of this invention for preliminary layout identification, the document layout structure can be obtained, and the visualization results are displayed in... Figure 3 b in the text.

[0064] In practical applications, during the analysis process, if the detection model is simply applied to a large amount of text and other content, it will lead to problems such as... Figure 4 The situation shown in 'a' is as described in the figure. However, after optimization and adjustment by referring to the lap value, it is possible to obtain the following: Figure 4 The improvement shown in b is the effect of the improvement.

[0065] In practical applications, the content under each subheading in the document is clearly distinguished and integrated with the text content obtained from OCR recognition. After integrating the structural information, the resulting text is as follows: Figure 5 As shown.

[0066] In practical applications, there are two types of image information extracted from accompanying documents of customs declarations: one is... Figure 6 The stamp watermark in the blue box below is another type, such as... Figure 6 The blue box above shows the GHS label pictogram. For watermarks, processing using a watermark removal algorithm is required; this invention only provides annotation. Similarly, the GHS label pictogram needs to be accurately extracted for customs verification. The difference between these two types of images lies in whether they contain GHS label elements or pictogram captions.

[0067] In practical applications, the original forms (including ordinary forms and implicit forms) in the accompanying documents of the customs declaration are directly identified by PPOCRv3, and the result is as follows: Figure 7 The content is rather chaotic, and the semantics of the implicit table cannot be correctly understood through the language model, nor can the data information of the ordinary table be obtained.

[0068] In practical applications, the layout analysis method for customs declaration documents (combining layout analysis information with OCR recognition) according to the embodiments of this application can generate formatted tables; such as... Figure 8 As shown, text boxes are generated and categorized by column, then saved row-wise to an Excel file. For each text box, its column index is determined based on the x-coordinate of its center point, and the corresponding OCR-recognized text is filled into the corresponding cell in the Excel file. If a column does not have a corresponding text box, an empty string is filled into the corresponding cell in that column. This ensures that the row and column information of the original table is preserved. The extracted Excel format table is shown below. Figure 8 As can be seen, Excel stores the structural and textual information of the original document image. Figure 8The results include: ordinary tables and hidden tables, extracted by combining layout analysis information with OCR.

[0069] Please see Figure 10 , Figure 10 A schematic diagram illustrating an embodiment of a computer device for a layout analysis method for documents accompanying a customs declaration, as provided in this application, is shown below. Figure 10 As shown, the computer device 1 of this embodiment includes: at least one processor 10 ( Figure 10 (Only one is shown in the image) a processor 10, a memory 11, and a computer program 12 stored in the memory 11 and executable on the at least one processor 10, wherein the processor 10 executes the computer program 12 to implement the steps in the embodiment of the layout analysis method for customs declaration accompanying documents of this application.

[0070] The computer device may be a desktop computer, laptop, smartphone, PDA, or other computing device.

[0071] Figure 10 The computer device shown may include, but is not limited to, processor 10 and memory 11. Those skilled in the art will understand that... Figure 10 The computer device 1 is merely an example and does not constitute a limitation on the computer device 1. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0072] The processor 10 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0073] In some embodiments, the memory 11 may be an internal storage unit of the computer device 1, such as a hard disk or memory of the computer device 1. In other embodiments, the memory 11 may be an external storage device of the computer device 1, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 1. Furthermore, the memory 11 may include both internal and external storage units of the computer device 1. The memory 11 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 11 can also be used to temporarily store data that has been output or will be output.

[0074] Furthermore, in one embodiment, as Figure 9 As shown, the present invention also provides a layout analysis device for documents accompanying customs declarations, comprising: Upload unit 10 is used to acquire images of the documents attached to the customs declaration to be analyzed; Analysis unit 20 is used to perform layout analysis on the document image using a pre-trained DocLayout-YOLO model, detect and identify multiple layout element regions in the document image, and generate an initial label box with category label and confidence for each layout element region; the category label includes text box, table, implicit table and customs stamp; The optimization unit 30 is used to merge and optimize each of the initial bounding boxes based on the overlap value (lap) to eliminate redundant boxes and obtain an optimized set of bounding boxes. The recognition unit 40 is used to perform optical character recognition (OCR) on the document image to obtain multiple OCR text boxes and their text content; The fusion unit 50 is used to perform coordinate matching and association between the OCR text box and the optimized annotation box set, and to assign corresponding text content to each optimized annotation box to obtain the target annotation box. The construction unit 60 is used to sort the target annotation boxes with assigned text content based on the category labels and position information of the optimized annotation boxes, and to construct a hierarchical structure of the document. Output unit 70 is used to output structured data according to the hierarchical structure.

[0075] It should be noted that the layout analysis device for customs declaration accompanying documents in this embodiment can be understood as a web layout analysis tool based on the DocLayout-YOLO model. It can be installed in the computer device of the aforementioned embodiment. The computer device calls the layout analysis device for customs declaration accompanying documents through the processor, and then runs the specific implementation scheme in the above embodiment of the layout analysis method for customs declaration accompanying documents.

[0076] In its implementation, the upload unit 10 represents a front-end interactive interface. For the front-end technology selection, this device primarily relies on basic technologies such as HTML, CSS, and JavaScript. HTML is responsible for constructing the page structure, defining the position and hierarchical relationship of each element. CSS is responsible for the page's style design, enhancing the visual effect and user experience by defining attributes such as color, font, and layout. JavaScript implements the page's interactive functions, including event listening, data processing, and asynchronous requests. Furthermore, this device can use the Flask framework's template engine and leverage Jinja2 syntax to render dynamic content, enabling the page to update in real-time based on backend data. This device utilizes the Flask framework to build a RESTful API, covering key functions such as document upload processing, analysis result feedback, text box positioning information provision, saving user-modified annotation information, and OCR recognition execution. The backend routing efficiently handles document upload requests. The analysis unit 20 and optimization unit 30 analyze the document images using the DocLayout-YOLO model, optimize them using corresponding algorithms, and output the analysis results in JSON format for further front-end processing.

[0077] The information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and the resulting technical effects, please refer to the method embodiments section.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0079] This application also provides a computer program product, which, when executed by a processor, implements the steps in the various method embodiments described above.

[0080] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0082] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0083] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for layout analysis of a customs declaration accompanying document, characterized in that, The method comprises: S1: obtaining a customs declaration form attached document image to be analyzed; S2: using a pre-trained DocLayout-YOLO model to perform layout analysis on the document image, detecting and identifying a plurality of layout element regions in the document image, and generating an initial labeling box with a category label and a confidence for each layout element region; the category label at least includes a text box, a table, an implicit table and a customs seal; S3: based on the overlap degree lap value, each initial labeling box is merged and optimized to eliminate redundant boxes to obtain an optimized labeling box set; S4: performing optical character recognition on the document image to obtain a plurality of OCR text boxes and their text contents; S5: coordinate matching and association of the OCR text box and the optimized labeling box set is performed, and the corresponding text content is assigned to each optimized labeling box to obtain a target labeling box; S6: based on the category label and position information of the optimized labeling box, the target labeling box to which the text content is assigned is sorted to construct a hierarchical structure of the document; S7: outputting structured data according to the hierarchical structure.

2. The method of claim 1, wherein, The step of merging and optimizing each initial labeling box in S3 comprises: S31: setting an index for each initial labeling box, which is generated based on the confidence sorting; S32: calculating the lap value between any two initial labeling boxes; S33: when the lap value of two initial labeling boxes is greater than a preset lap threshold, the two initial labeling boxes are merged into a new labeling box, and the index of the new labeling box is set as the index of the initial labeling box with the earlier index among the two initial labeling boxes; S34: repeating steps S32 and S33 until there is no box pair with a lap value greater than the lap threshold to obtain an optimized labeling box set.

3. The method of claim 1, wherein, The step of coordinate matching and association of the OCR text box and the optimized labeling box set in S5 comprises: calculating the lap value between each OCR text box and each optimized labeling box based on the optimized labeling box; associating an OCR text box to all optimized labeling boxes with a lap value greater than a preset value.

4. The method of claim 3, wherein, The step of assigning corresponding text content to each optimized labeling box in S5 specifically comprises: for the optimized labeling box with a category of title block, combining the text contents of all OCR text boxes associated therewith to form a complete title text; for the optimized labeling box with a category of table block, generating row and column data of the table through the text contents of the OCR text boxes associated therewith.

5. The method according to any one of claims 1 to 4, characterized in that, The step of sorting the target labeling box to which the text content is assigned in S6 comprises: determining the center point of each target labeling box; ascendingly sorting each target labeling box according to the horizontal coordinate of the center point of the labeling box; ascendingly sorting the target labeling boxes according to the vertical coordinate of the center point of the target labeling box under the premise of ensuring the order of the horizontal coordinate of the center point of each target labeling box.

6. The method of claim 5, wherein, The step of constructing the hierarchical structure of the document in S6 comprises: From the optimized set of bounding boxes, identify the bounding boxes with the category of title and the associated text content as empty, and mark them as top-level large titles; For the non-table layout part at the top without bounding boxes with the category of title, assign a top-level title without title; According to the top-level large title and / or the top-level title without title, group the subordinate bounding boxes with the category of title or text to form a hierarchical document structure.

7. The method according to any one of claims 1 to 4, characterized in that, The formula for calculating the lap value in S3 is: lap = (A ∩ B) / min(A, B) Where A and B are the areas of the two bounding boxes to be merged and optimized.

8. A layout analysis apparatus for a customs declaration form attached document, characterized by comprising: The device comprises: An uploading unit configured to obtain an image of a customs declaration form and attached documents to be analyzed; An analysis unit configured to perform layout analysis on the image of the documents by using a pre-trained DocLayout-YOLO model, detect and identify a plurality of layout element regions in the image of the documents, and generate an initial bounding box with a category label and a confidence for each layout element region; the category label includes a text box, a table, an implicit table, and a customs seal; An optimization unit configured to merge and optimize each initial bounding box based on the overlap degree lap value to eliminate redundant boxes and obtain an optimized set of bounding boxes; An identification unit configured to perform optical character recognition (OCR) on the image of the documents to obtain a plurality of OCR text boxes and their text content; A fusion unit configured to coordinate match the OCR text boxes with the optimized set of bounding boxes, assign corresponding text content to each optimized bounding box, and obtain target bounding boxes; A construction unit configured to sort the target bounding boxes to which text content is assigned based on the category label and position information of the optimized bounding boxes, and construct a hierarchical structure of the document; An output unit configured to output structured data according to the hierarchical structure.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer program product, characterised in that, The computer program product comprises a computer program that, when executed by a processor, implements the steps of the method of any one of claims 1 to 7.

Images