Page analysis method and device for customs declaration accompanying documents, computer device and computer program product

Abstract

The application provides a page analysis method and device for customs declaration forms and attached documents, computer equipment and computer program products, which include: obtaining an image of a customs declaration form and attached documents to be analyzed; performing page analysis on the document image, detecting and identifying a plurality of page element regions in the image, and generating an initial labeling box with a category label and a confidence for each page element region; merging and optimizing each initial labeling box to obtain an optimized labeling box set; performing optical character recognition on the document image to obtain a plurality of OCR text boxes and their text contents; associating the OCR text boxes with the optimized labeling box set, and assigning corresponding text contents to each optimized labeling box; sorting the labeling boxes to which the text contents are assigned based on the category labels and position information of the optimized labeling boxes, and constructing a hierarchical structure of the document; and finally outputting structured data according to the hierarchical structure.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computers, and particularly relates to a page analysis method and device for customs declaration forms and attached documents, a computer device, and a computer program product. BACKGROUND

[0002] Document Layout Analysis (DLA) aims to automatically identify various layout elements (such as text, tables, titles, etc.) and their layout structures in document images, providing a basis for subsequent information processing. Existing technical solutions mainly include two categories:

[0003] In existing technologies, single-modal methods based on computer vision are mainstream, such as using Faster R-CNN, YOLO, etc. target detection models, taking document images as input, directly detecting and positioning the layout units therein. In addition, there are also multi-modal methods (such as LayoutLMv3) that fuse visual and text features to improve the accuracy of analysis. In the preparation of model training data, it is usually necessary to rely on manual annotation tools (such as LabelMe) or semi-automatic annotation tools (such as SegBuilder based on the SAM model) to generate annotation data.

[0004] Although the above-mentioned existing technical methods have made progress, when applied to the specific scenario of customs declaration forms and attached documents, there are still the following obvious defects:

[0005] Poor adaptability to specific layout elements: existing models are not optimized for non-text elements such as fixed-position electronic seals, watermarks / stamps, etc. that are common in customs documents, which are prone to misidentification as text or tables, introducing noise. At the same time, for the "implicit tables" (without entity borders) that are ubiquitous in documents and separated by spaces, traditional methods will split them into independent text blocks, which cannot restore their inherent row-column structure, leading to confusion in information extraction.

[0006] Data dependence and insufficient generalization ability: deep learning-based methods require a large amount of annotated data for training or fine-tuning. However, customs documents are mostly classified files, and there are few desensitized samples available for model use, which makes the model prone to overfitting and difficult to cover different enterprise and format variations of documents, with limited generalization ability.

[0007] Annotation tool efficiency and effectiveness are poor: pure manual annotation is time-consuming and labor-intensive. The annotation results (such as masks) generated by semi-automatic annotation tools usually only contain position information, lack the necessary class labels for layout analysis, and lack the hierarchical relationship information between regions (such as the subordinate relationship between table notes and tables), still requiring a lot of manual intervention for correction and classification, which is inefficient. SUMMARY

[0008] To solve the problem of how to effectively identify and process special elements such as seals and implicit forms mentioned in the background, and overcome the technical problem of weak generalization ability caused by the lack of training data in the prior art, the embodiments of the present application provide a page analysis method and device for customs declaration forms and attached documents, computer equipment and computer program products, aiming to provide an efficient and accurate page analysis scheme for customs declaration forms and attached documents, which can significantly improve the automation and accuracy of page analysis and structure reconstruction.

[0009] In a first aspect, the embodiments of the present application provide a page analysis method for customs declaration forms and attached documents, which comprises:

[0010] S1: obtaining a customs declaration form and attached document image to be analyzed;

[0011] S2: using a pre-trained DocLayout-YOLO model to perform page analysis on the document image, detecting and identifying a plurality of page element regions in the document image, and generating an initial annotation box with a class label and a confidence for each page element region; the class label includes a text box, a table, an implicit table, and a customs seal;

[0012] S3: based on the overlap value lap, merging and optimizing each initial annotation box to eliminate redundant boxes, and obtaining an optimized annotation box set;

[0013] S4: performing optical character recognition (OCR) on the document image to obtain a plurality of OCR text boxes and their text content;

[0014] S5: matching and associating the OCR text box with the optimized annotation box set, assigning corresponding text content to each optimized annotation box, and obtaining a target annotation box;

[0015] S6: based on the class label and position information of the optimized annotation box, sorting the target annotation box to which the text content is assigned, and constructing a hierarchical structure of the document;

[0016] S7: outputting structured data according to the hierarchical structure.

[0017] In the embodiments of the present application, the complete process from image input in step S1 to structured output in step S7 realizes the automatic and structured analysis of the page of the customs declaration form and attached documents, and solves the problems of poor format adaptation, redundant and chaotic analysis results, low efficiency of manual review, and chaos caused by different formats in the prior art.

[0018] In one embodiment of the first aspect, the step of merging and optimizing each initial annotation box in S3 comprises:

[0019] S31: setting an index for each of the initial annotation boxes, the index being generated based on sorting of the confidence thereof;

[0020] S32: calculating a lap value between any two initial annotation boxes;

[0021] S33: when the lap value of two initial annotation boxes is greater than a preset lap threshold, merging the two initial annotation boxes into a new annotation box, and setting the index of the new annotation box as the index of the one with a higher index among the two initial annotation boxes;

[0022] S34: repeating steps S32 and S33 until no box pair with a lap value greater than the lap threshold exists, to obtain an optimized annotation box set.

[0023] In the embodiment of the present application, when the initial annotation boxes are merged, since the arrangement order of the original annotation information is based on the confidence, in order to preserve the confidence information, an index is set for each annotation information, the index of the new annotation box after merging is set as the one with a higher index among the merged boxes, the merged boxes are deleted from the list after merging, and the merging is repeated until there is no box that can be merged, and then the process is ended. The optimized annotation box set is sorted according to the index, which can solve the problems of overlapping box redundancy and conflict caused by a low confidence threshold.

[0024] In one embodiment of the first aspect, the step of coordinate matching and associating the OCR text box with the optimized annotation box set in S5 comprises:

[0025] calculating a lap value between each OCR text box and each optimized annotation box based on the optimized annotation box;

[0026] associating an OCR text box to all the optimized annotation boxes with a lap value greater than a preset value.

[0027] In the embodiment of the present application, by fusing the structural information, the problem that the OCR text content can be accurately corresponded to the layout element region can be solved.

[0028] In one embodiment of the first aspect, the step of assigning a corresponding text content to each optimized annotation box in S5 specifically comprises:

[0029] for the optimized annotation box of the title block type, combining the text content of all the OCR text boxes associated therewith to form a complete title text;

[0030] for the optimized annotation box of the table block type, generating row and column data of the table through the text content of the OCR text boxes associated therewith.

[0031] In the embodiments of the present application, the text content can be integrated according to the layout category for subsequent processing and analysis.

[0032] In one embodiment of the first aspect, the step of sorting the target bounding boxes to which the text content is allocated in S6 comprises:

[0033] determining the center point of each target bounding box;

[0034] sorting each target bounding box in ascending order according to the horizontal coordinate of the center point of the bounding box;

[0035] sorting in ascending order according to the vertical coordinate of the center point of the target bounding box under the premise of ensuring the order of the horizontal coordinate of the center point of each target bounding box.

[0036] In the embodiments of the present application, the problem of how to restore the correct reading order of the document, especially the order of the horizontally laid out customs declaration form, is solved.

[0037] In one embodiment of the first aspect, the step of constructing the hierarchical structure of the document in S6 comprises:

[0038] From the optimized bounding box set, identifying a bounding box with a category of title and associated text content as empty, and marking it as a top-level major title;

[0039] For a non-table layout part at the top without a bounding box with a category of title, assigning a top-level title of "no title", and marking it as a "no title" title;

[0040] According to the top-level major title and / or the "no title" title (i.e. the top-level title without title), grouping the subordinate bounding boxes with a category of title or text to form a hierarchical document structure.

[0041] In the embodiments of the present application, not only is the reading order of the document restored, but also the internal logical organization is understood, laying a solid foundation for subsequent information extraction and data analysis.

[0042] In a second aspect, the present application further provides a layout analysis device for customs declaration form attached documents, comprising:

[0043] An uploading unit configured to obtain a customs declaration form attached document image to be analyzed;

[0044] An analysis unit configured to perform layout analysis on the document image by using a pre-trained DocLayout-YOLO model, detect and identify a plurality of layout element regions in the document image, and generate an initial bounding box with a category label and a confidence for each layout element region; the category label includes a text box, a table, an implicit table, and a customs seal.

[0045] an optimization unit, configured to perform merging optimization on each of the initial annotation boxes based on a lap value of the overlap degree, so as to eliminate redundant boxes and obtain an optimized annotation box set;

[0046] an identification unit, configured to perform optical character recognition (OCR) on the document image to obtain a plurality of OCR text boxes and text content thereof;

[0047] a fusion unit, configured to perform coordinate matching and association between the OCR text boxes and the optimized annotation box set, to assign corresponding text content to each of the optimized annotation boxes, and obtain target annotation boxes;

[0048] a construction unit, configured to sort the target annotation boxes to which the text content is assigned based on a class label and position information of the optimized annotation boxes, and construct a hierarchical structure of the document;

[0049] an output unit, configured to output structured data according to the hierarchical structure.

[0050] In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method in the first aspect when executing the computer program.

[0051] In a fourth aspect, a computer program product is provided, which includes a computer program, and the computer program implements the steps of the method in the first aspect when executed by a processor.

[0052] It should be noted that the technical effects of the second aspect to the fourth aspect are described in the technical effect of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0054] Figure 1 A flowchart of a main embodiment of a page analysis method for customs declaration documents provided by the present application;

[0055] Figure 2 A flowchart of an embodiment of a step of performing merging optimization on initial annotation boxes in a page analysis method for customs declaration documents provided by the present application;

[0056] Figure 3 The comparative diagram of the original image a of the attached document of the dangerous chemical declaration form and the analysis result b obtained by the method of the present application for the application example of the present application;

[0057] Figure 4 The comparative diagram of the original image a of the attached document of the dangerous chemical declaration form and the analysis result b obtained by the method of the present application for the application example of the present application;

[0058] Figure 5 The comparative diagram of the original image a of the attached document of the dangerous chemical declaration form and the analysis result b obtained by the method of the present application for the application example of the present application;

[0059] Figure 6 The diagram of the icon annotation component information provided for the application example of the present application;

[0060] Figure 7 The original table a and the table content b extracted by the OCR provided for the application example of the present application;

[0061] Figure 8 The formatted table obtained by the method of the present application for the layout analysis of the attached document of the declaration form;

[0062] Figure 9 The structural block diagram of the layout analysis device for the attached document of the declaration form provided by the present application;

[0063] Figure 10 The structural diagram of the computer device for performing the layout analysis method for the attached document of the declaration form provided by the present application. DETAILED DESCRIPTION

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application; the description and the drawings of the specification and the appended claims should be considered in conjunction with the terms used in the description and the claims themselves as well as the foregoing recitations to support the disclosure; the terms "comprise" and "have" and any variations thereof, used in the specification and the claims, are intended to cover both the inclusive and the exclusive.

[0065] It should be understood that, when used in the specification and the appended claims of the present application, the term "comprise" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or sets thereof.

[0066] It should also be understood that, in the description of the application and the appended claims, the term "and / or" is used to mean one or more of the items in the list joined by "and / or". It is further noted that the terms "comprising", "including", "containing", "having" and the like are to be construed to be open-ended terms (i.e., comprising, including, containing, having). Thus, use of such terms is intended to enable a statement that something comprises, includes, contains, and / or has one or more items, and is not a limitation that the something must consist of one or more of the items.

[0067] As used in the description of the application and the appended claims, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon [the described condition or event] being detected" or "in response to [the described condition or event] being detected," depending on the context.

[0068] In the description of the embodiments of the application, the term "plurality" refers to two or more (including two), unless otherwise expressly and specifically limited.

[0069] In addition, the terms "first", "second", "third", etc. are used herein only to describe different instances, and are not intended to imply or suggest relative importance of the described elements.

[0070] The terms "one embodiment", "some embodiments", "an embodiment", "one specific embodiment", "some specific embodiments", and "an specific embodiment" described in the present application mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, the appearances of the phrases "in one embodiment" or "in some embodiments" or "in an embodiment" or "in some specific embodiments" or "in an specific embodiment" in various places in the specification are not necessarily all referring to the same embodiment, unless otherwise are otherwise be specifically stated

[0071] The terms "comprising", "including", "containing", "having" and the like are not intended to exclude possible incorporation of one or more other items that are not recited.

[0072] It should be noted that the purpose of the present application is to solve the problem of different formats and chaotic layout of the attached documents of the customs declaration form (especially the dangerous chemical safety data sheet), which leads to low efficiency and high error rate of manual review. A layout analysis method, device, computer equipment and computer program product for the attached documents of the customs declaration form are developed. The layout analysis method for the attached documents of the customs declaration form can be a computer analysis program in a computer program product, and the layout analysis device for the attached documents of the customs declaration form can be a Web layout analysis tool based on a DocLayout-YOLO model. The tool realizes efficient layout analysis and key information extraction of the attached documents of the customs declaration form by integrating document uploading and management, automatic annotation generation (target detection), interactive annotation adjustment, text recognition and layout information fusion, model uploading and fine-tuning, etc. Functional modules, thereby improving the efficiency and accuracy of the customs compliance review of import and export documents, and reducing the pressure of manual review.

[0073] To solve the above-mentioned technical problems, the present application provides a layout analysis method for the attached documents of the customs declaration form, please refer to Figure 1 , Figure 1 The flowchart of a first embodiment of the layout analysis method for the attached documents of the customs declaration form provided by the present application is shown in the figure.

[0074] The layout analysis method for the attached documents of the customs declaration form provided by the present application mainly includes the following steps S1 to S7:

[0075] Step S1: obtaining the image of the attached document of the customs declaration form to be analyzed;

[0076] It can be understood that the execution subject of the analysis method of the present application is the aforementioned Web layout analysis tool based on the DocLayout-YOLO model. Through the technical path of "model optimization-function modularization-system integration", efficient processing of documents with different formats and chaotic layout is realized. The system integrates image uploading and management functions, supports ZIP compression package batch uploading and format conversion.

[0077] In specific implementation, the system can provide a convenient document input interface for users through a front-end interactive page, allowing users to process a large number of documents at a time. The format conversion function ensures that the subsequent processing module can receive standardized image input, which is the starting point and data preparation stage of the entire analysis method process.

[0078] According to the page characteristics of the accompanying documents of the customs declaration form, an embodiment of the present application defines corresponding page categories, and can perform primary labeling by using a DocLayout-YOLO model, and the labeling is self-provided with category information. Through an interactive interface, a user can quickly adjust and classify the labeling regions, and can create a square labeling box by using a mouse dragging method to efficiently complete labeling in combination with page information characteristics.

[0079] Step S2: performing page analysis on the document image by using the pre-trained DocLayout-YOLO model, detecting and identifying a plurality of page element regions in the document image, and generating an initial labeling box with a category label and a confidence for each page element region;

[0080] Exemplarily, the category label can include but is not limited to a text box, a table, an implicit table, and a customs seal, a GHS diagram, and a waste part (such as a page code and a page footer).

[0081] It should be noted that this step takes the DocLayout-YOLO model as the core, integrates target detection and page structure analysis capabilities, and optimizes the characteristics of the accompanying documents of the customs declaration form: the DocLayout-YOLO model based on the YOLOv10 algorithm is fine-tuned by using the customs document images (including scanned copies and electronic documents), the target detection head and the loss function are optimized, the recognition accuracy of special elements such as text boxes, tables, implicit tables (space separation), and customs seals is improved, and the problem of overlapping boxes is solved (redundant boxes are merged by using a self-defined lap overlap algorithm).

[0082] It can be understood that this step is the cornerstone of intelligent analysis; exemplarily, the pre-trained DocLayout-YOLO model can be a model fine-tuned based on the YOLOv10 algorithm and using customs document images. The initial labeling box with the category label and the confidence output by the model is the result of the model identifying elements such as text boxes, tables, implicit tables, and customs seals.

[0083] Step S3: merging and optimizing each initial labeling box based on a lap value to eliminate redundant boxes, to obtain an optimized labeling box set; the calculation formula of the lap value is:

[0084] lap = (A ∩ B) / min(A, B), where A and B are the areas of two labeling boxes to be merged and optimized;

[0085] It can be understood that the labeling box (Bounding Box) in step S3 can be a detection box or a bounding box.

[0086] It should be noted that the confidence threshold in the present application refers to a score assigned by the DocLayout-YOLO model to each bounding box it detects. This score represents the model's confidence in two aspects: (position confidence) the model's belief about the accuracy of the position of the box; (category confidence) the model's belief about the likelihood that the content in the box belongs to the category (such as text, table, etc.) it predicts.

[0087] In order to obtain more complete layout information, a lower confidence threshold needs to be selected during detection, which leads to the phenomenon of overlapping bounding boxes in the image, especially when dealing with documents such as customs declaration accompanying documents that contain a large amount of text and other information, the problem of overlapping text boxes is particularly evident. Overlapping boxes will lead to redundancy and conflict of layout information, making it difficult to obtain position information of text and subsequent analysis. To solve the problem of overlapping bounding boxes caused by dense text and overlapping tables in single documents, the present embodiment proposes a merging strategy based on lap (overlap degree), which calculates the overlap degree with the minimum area of the two boxes as the denominator:

[0088]

[0089] The lap value is between 0 and 1, and a value of 1 represents a larger box completely wrapping a smaller box or two boxes completely overlapping. Since the denominator becomes smaller, lap will be larger in numerical value than IoU. In actual use, the threshold can be set slightly larger. The default threshold lap_threshold used in the present embodiment is 0.8.

[0090] In some embodiments, with reference to Figure 2 The merging and optimization of each initial bounding box in step S3 can specifically include:

[0091] S31: Set an index for each initial bounding box, which is generated based on the confidence sorting;

[0092] S32: Calculate the lap value between any two initial bounding boxes;

[0093] S33: When the lap value of two initial bounding boxes is greater than a preset lap threshold, merge the two initial bounding boxes into a new bounding box, and set the index of the new bounding box as the index of the one with the earlier index among the two merged initial bounding boxes;

[0094] S34: Repeat steps S32 and S33 until there are no bounding box pairs with a lap value greater than the lap threshold, and obtain the optimized bounding box set.

[0095] It should be noted that, in the initial annotation box merging, since the arrangement order of the original annotation information is based on the confidence, in order to retain the confidence information, an index is set for each annotation information, the index of the new annotation box after merging is set as the one with the earlier index in the merged box, the merged box is deleted from the list after merging, and the merging is ended after the iteration is ended without the box that can be merged, the remaining boxes in the list (i.e. the optimized annotation box set) are sorted according to the index, and saved into a new annotation txt file.

[0096] It can be understood that this step is the key to improve the cleanliness of the analysis result, from the calculation of the lap value, the comparison with the threshold, to the merging strategy (inherit the earlier index) and the iteration loop, finally outputting the "optimized annotation box set" and saving; the problem of redundant and conflict overlapping boxes caused by the low confidence threshold can be solved.

[0097] Step S4: performing optical character recognition (OCR) on the single document image to obtain a plurality of OCR text boxes and text contents thereof;

[0098] It should be noted that this step S4 is a key technical link for realizing text extraction, and the OCR engine such as PPOCRv3 can be used for text recognition.

[0099] It can be understood that this step is executed in parallel with or after the step of the aforementioned S3 layout analysis, and focuses on extracting original text information from the image.

[0100] In this embodiment, the "optical character recognition (OCR)" in step S4 can use the "PPOCRv3" engine; the output "a plurality of OCR text boxes and text contents thereof" is the original recognition result without structuring, and prepares for the next step S5 of fusing with the layout structure.

[0101] Step S5: performing coordinate matching and association on the OCR text box and the optimized annotation box set, assigning corresponding text contents to each of the optimized annotation boxes, and obtaining a target annotation box;

[0102] It should be noted that this step S5 is the core of information fusion; in the layout analysis process of the present application, in addition to optimizing the detection result to process the overlapping box problem, it is also necessary to effectively fuse the text recognition result with the layout structure information, so as to construct a complete and accurate document structure. This process is crucial for understanding the content hierarchy and semantic relationship of the document, especially in processing the customs declaration single document and the like which contains a large amount of text and multiple information types.

[0103] It can be understood that after completing the layout detection and character recognition, it is necessary to associate the OCR recognized text box with the label box obtained by layout detection. Each label box represents a specific area in the document, such as title, table, text, etc., and the OCR text box contains the specific text content in the area. In order to achieve this association, a "coordinate matching association" method can be used in this embodiment:

[0104] Firstly, the label box coordinates in YOLO format are converted into image coordinates for unified comparison with the coordinates of the OCR text box. Then, by calculating the overlap between each OCR text box and the label box, it is determined whether they belong to the same area; in some embodiments, the step S5 of coordinate matching association between the OCR text box and the optimized label box set can specifically include: taking the optimized label box as a reference, calculating the lap value between each OCR text box and each optimized label box; an OCR text box is associated with all optimized label boxes with a preset value (the preset value is preferably 0.005) of lap value, and the OCR text content can be accurately corresponded to the layout element area. It can be understood that the overlap calculation method used in this step S5 is the same as the lap method mentioned in the aforementioned S3 detection result optimization step, i.e. calculating the overlap lap value. Taking the label box as a reference, each label box is associated with all OCR text boxes overlapping with it, and one text box can correspond to multiple label boxes. This situation is for a layout that can serve as multiple categories, such as a part that can be both a title and a table, or both text and a table.

[0105] Then, after determining the association between the OCR text box and the label box, the text content recognized by OCR needs to be assigned to the corresponding label box. In some embodiments, the step S5 of assigning corresponding text content to each optimized label box can specifically include: for the optimized label box of the title block category, combining the text content of all OCR text boxes associated with it to form a complete title text; for the optimized label box of the table block category, generating row and column data of the table through the text content of the OCR text boxes associated with it.

[0106] It can be understood that in this step S5, the text content is divided into different blocks according to the category of the label box, which is divided into title block and table block here. For the title block, the text content in the OCR text box associated with it is combined into a complete title and displayed in a certain format. For the table block, the OCR text box content in each table is organized to generate row and column information of the table, so as to facilitate subsequent table processing and analysis.

[0107] Step S6: based on the class label and position information of the optimized annotation box, sorting the target annotation box to which the text content is assigned, and constructing the hierarchical structure of the document;

[0108] In this embodiment, after completing the assignment of text content, the overall structure of the document is generated according to the class and hierarchical relationship of the annotation box. For a document containing multiple titles and tables, a hierarchical structure is constructed to clearly show the content organization and logical relationship of the document. For example, for a document containing multiple chapters and tables, the title of each chapter and the corresponding body content are grouped, and the table is saved separately as an excel file.

[0109] In order to ensure that the generated document structure can accurately reflect the original content and layout of the document, the position information of the annotation box and the position information of the text box need to be processed.

[0110] Since the annotation box is used as the reference when matching the annotation box and the text box, the annotation box is sorted first in an embodiment. In step S6 of sorting the target annotation box to which the text content is assigned, the step can specifically include:

[0111] determining the center point of each target annotation box; sorting each target annotation box in ascending order according to the horizontal coordinate of the center point of the annotation box; and sorting the target annotation box in ascending order according to the vertical coordinate of the center point of the target annotation box on the premise of ensuring the order of the horizontal coordinate of the center point of each target annotation box.

[0112] It can be understood that in computer image processing, the y coordinate value is increasing from top to bottom, and the x coordinate value is increasing from left to right. The accompanying documents of the customs declaration form belong to horizontally laid documents, so the center point position of each annotation box is sorted in multiple ascending orders, first sorting the horizontal position (x coordinate), and then sorting the vertical position (y coordinate) without disturbing this dimension. Without considering the implicit table, the text arrangement order of the accompanying documents of the customs declaration form is relatively regular, and usually does not appear left-right reversed and upside-down. Therefore, only the relative position of the annotation box needs to be considered for the title box. All title boxes are sorted according to the vertical position (y_center) of the title box. If the vertical positions of two title boxes are similar (the difference is less than a certain threshold), they are sorted according to the horizontal position (x_center). In this way, the title boxes can be arranged in the order from top to bottom and from left to right.

[0113] Then, in some embodiments, the step of "constructing the hierarchical structure of the document" in S6 can specifically include:

[0114] From the optimized set of bounding boxes, identify the bounding boxes with the category of title and the associated text content as empty, and mark them as top-level large titles;

[0115] For the non-table layout part at the top of the document without a bounding box with the category of title, assign a top-level title of "no title", and mark it as a "no title" title;

[0116] According to the top-level large title and / or the "no title" title, group the subordinate bounding boxes with the category of title or text to form a hierarchical document structure.

[0117] It can be understood that for the hierarchical structure of large title nested small title, the system identifies the bounding boxes with the category of "title" and the associated text content (from step S5) as empty from the optimized set of bounding boxes (from step S3). Such empty title bounding boxes are usually because the large title itself may be detected by the model through visual features such as font, font size or position, but its text content may be identified as another OCR text box. The system identifies and marks such bounding boxes as top-level large titles.

[0118] For the non-table layout part at the top of the document without a bounding box with the category of "title" (for example, the header area of some certificates or directly started text), the system uses a default "no title" as its top-level title. This ensures that each logical part of the document has a title node corresponding to it, maintaining the integrity of the structure.

[0119] After completing the identification and completion of the title, the system groups the subordinate bounding boxes with the category of "title" or "text" according to the top-level large title and / or "no title" title. For example, all the small titles and text paragraphs under a large title will be grouped under the large title node, thereby clearly defining the subordinate relationship between chapters and content, and finally forming a hierarchical document structure. This set of flow mechanisms in the embodiment makes the application not only restore the reading order of the document, but also understand its internal logical organization, laying a solid foundation for subsequent information extraction and data analysis.

[0120] Step S7: Output structured data according to the hierarchical structure.

[0121] It should be noted that this step 7 finally outputs structured data, and the output structured data can be in JSON format for data exchange, or in Excel table or txt document;

[0122] In this embodiment, the system can use the Flask framework to build a RESTful API, covering key functions such as document upload processing, analysis result feedback, text box positioning information provision, user modified annotation information saving, and OCR recognition execution. The backend routing can efficiently handle document upload requests, analyze document images using the DocLayout-YOLO model, optimize using corresponding algorithms, and output analysis results in JSON format for further processing by the front end. At the same time, the system includes five core modules: model management (upload, fine-tuning, test interface), automatic annotation (DocLayout-YOLO generates initial annotation boxes), interactive adjustment (mouse drag to modify box position / category), and text extraction (combined with PPOCRv3 and layout information, output Excel / txt). Finally, efficient layout analysis and key information extraction of the entire customs declaration single accompanying document are realized.

[0123] The technical effect of the embodiments of the present application is that when processing documents such as customs declaration single accompanying documents containing a large amount of text and information, a low confidence threshold can easily cause text box overlap problems, resulting in redundancy and position information conflicts, making subsequent analysis difficult. Therefore, a merging strategy based on overlap degree is proposed, which optimizes the merging of boxes by calculating the overlap degree (lap value), preserves the confidence information, and saves the merged boxes in order. Then, the text box and the annotation box are associated through the coordinate matching method, the text content is assigned to the annotation box, and the hierarchical structure of the document is constructed to ensure accurate reflection of the original content. The sorting of the boxes is in the order of horizontal position first and vertical position second to ensure that the title and the main text are arranged in a reasonable order, and finally a document structure with clear logical relationship is formed.

[0124] The advantage of the present application compared with the prior art is that:

[0125] For the existing manual annotation tools such as LabelMe and LabelStudio, the annotation method of determining a rectangular box and selecting a category by clicking point by point significantly increases the annotation time, with an average annotation time of about 3 minutes per picture. The semi-automatic tool SegBuilder generates a fine semantic segmentation SAM mask, but the redundant mask position information leads to a large amount of calculation. In contrast, the layout analysis method for customs declaration single accompanying documents based on the present application takes only 40 milliseconds to recognize a single picture, and the time for manual adjustment of the annotation result is shortened to half a minute. In addition, the present application can generate a COCO format label file, which is convenient for adjusting the Doclayout-YOLO model and subsequent layout analysis processing, significantly improving work efficiency.

[0126] In specific applications, a hazardous chemical customs declaration single accompanying document can be selected, and the original image is as followsFigure 3 The layout structure of the document can be obtained by applying the layout analysis method of the present application to the preliminary layout identification of the accompanying documents of the customs declaration form, and the visual result is shown in Figure 3 b.

[0127] In specific applications, if only the detection model is applied to detect the large amount of text and other content during the analysis process, the situation shown in Figure 4 a will occur. However, after optimization adjustment by referring to the lap value, the improved effect shown in Figure 4 b can be obtained.

[0128] In specific applications, according to each subheading in the document, the content under each heading is clearly distinguished and combined with the text content obtained by OCR recognition. After the integration processing of the structure information, the obtained text result is shown in Figure 5

[0129] In specific applications, there are two types of image information extracted on the accompanying documents of the customs declaration form: one is the seal watermark in the blue box below Figure 6 , and the other is the GHS label pictograph in the blue box above Figure 6 . For the seal watermark, it needs to be processed by a seal watermark removal algorithm, and the present application only performs labeling; for the GHS label pictograph, it also needs to be accurately extracted so that the customs can make a judgment. The difference between the two types of images lies in whether the GHS label element or the pictograph annotation is included.

[0130] In specific applications, the original table (including ordinary table and implicit table) in the accompanying documents of the customs declaration form is directly recognized by PPOCRv3, and the result is shown in Figure 7 The content is chaotic, and the language model cannot correctly understand the semantics of the implicit table, nor can it obtain the data information of the ordinary table.

[0131] In specific applications, the method of the present application for layout analysis of the accompanying documents of the customs declaration form (layout analysis information combined with OCR recognition) can obtain a formatted table; as shown in Figure 8 , a text box is generated according to the column classification, and then stored in an Excel file according to the row. For each table-related text box, the column index to which it belongs is determined according to the x-coordinate of its center point, and the corresponding OCR recognized text is filled into the corresponding cell of the Excel file. If there is no corresponding text box in a column, an empty string is filled in the corresponding cell of the column to ensure that the row and column information of the original table is preserved, and the extracted Excel formatted table is shown in Figure 8 It can be seen that the excel saves the structure information and text information of the original document image.​Figure 8 In some embodiments, the method comprises: combining the common table and the implicit table with the OCR extraction result through layout analysis information.

[0132] Referring to Figure 10 , Figure 10 An embodiment structure schematic diagram of a computer device for the layout analysis method for the customs declaration form attached documents provided by the present application is shown in Figure 10 The computer device 1 of the embodiment includes at least one processor 10 (only one is shown in the figure), a memory 11, and a computer program 12 stored in the memory 11 and executable on the at least one processor 10, wherein the processor 10 implements the steps in the layout analysis method for the customs declaration form attached documents in the present application when executing the computer program 12. Figure 10

[0133] The computer device can be a desktop computer, a notebook, a smart phone, a palm computer, etc.

[0134] Figure 10 The computer device shown in the figure can include, but is not limited to, the processor 10 and the memory 11. Those skilled in the art can understand that Figure 10 The computer device 1 shown in the figure is only an example and does not constitute a limitation on the computer device 1, and can include more or fewer components than those shown in the figure, or combine certain components, or different components, for example, can also include an input / output device, a network access device, etc.

[0135] The processor 10 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0136] ​The memory 11 can be an internal storage unit of the computer device 1 in some embodiments, for example, a hard disk or a memory of the computer device 1. The memory 11 can also be an external storage device of the computer device 1 in other embodiments, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 1. Further, the memory 11 can include both the internal storage unit and the external storage device of the computer device 1. The memory 11 is used to store an operating system, an application program, a boot loader, data, and other programs, etc., for example, program codes of the computer program, etc. The memory 11 can also be used to temporarily store data that has been output or will be output.

[0137] Further, in an embodiment, as shown in Figure 9 The application further provides a page layout analysis device for a customs declaration form attached document, comprising:

[0138] An uploading unit 10 is configured to acquire an image of a customs declaration form attached document to be analyzed;

[0139] An analysis unit 20 is configured to perform page layout analysis on the document image by using a pre-trained DocLayout-YOLO model, detect and identify a plurality of page element regions in the document image, and generate an initial annotation box with a category label and a confidence for each page element region; the category label includes a text box, a table, an implicit table, and a customs seal;

[0140] An optimization unit 30 is configured to perform merging and optimization on each initial annotation box based on a lap value of overlap degree, so as to eliminate redundant boxes and obtain an optimized annotation box set;

[0141] An identification unit 40 is configured to perform optical character recognition (OCR) on the document image to acquire a plurality of OCR text boxes and text contents thereof;

[0142] A fusion unit 50 is configured to perform coordinate matching and association between the OCR text boxes and the optimized annotation box set, assign corresponding text contents to each optimized annotation box, and obtain a target annotation box;

[0143] A construction unit 60 is configured to sort the target annotation box to which the text contents are assigned based on the category label and position information of the optimized annotation box, and construct a hierarchical structure of a document;

[0144] An output unit 70 is configured to output structured data according to the hierarchical structure.

[0145] It should be noted that the page analysis device for the customs declaration accompanying documents of the embodiment can be understood as a Web page analysis tool based on the DocLayout-YOLO model, which can be loaded in the computer device of the foregoing embodiment. The computer device invokes the page analysis device for the customs declaration accompanying documents through the processor, and then runs the specific implementation solutions in the foregoing page analysis method embodiments for the customs declaration accompanying documents.

[0146] In a specific implementation, the uploading unit 10 can represent a front-end interactive interface. In the selection of front-end technology, the device mainly relies on the basic technologies of HTML, CSS and JavaScript. HTML is responsible for building the structure of the page, defining the position and hierarchical relationship of each element. CSS is responsible for the style design of the page, which enhances the visual effect and user experience of the page by defining color, font, layout and other attributes. JavaScript implements the interactive functions of the page, including event listening, data processing and asynchronous requests, etc. In addition, the device can use the template engine of the Flask framework and use the Jinja2 syntax to realize the rendering of dynamic content, so that the page can be updated in real time according to the back-end data. The device uses the Flask framework to build RESTful API, which covers document uploading processing, analysis result feedback, text box positioning information provision, user modification annotation information saving and OCR recognition execution and other key functions. The back-end routing can efficiently handle document uploading requests. The analysis unit 20 and the optimization unit 30 use the DocLayout-YOLO model to analyze the document image, use the corresponding algorithm for optimization, and output the analysis result in JSON format, which is convenient for the front-end to further process.

[0147] The information interaction, execution process and other contents between the above devices / units can be based on the same concept as the method embodiments of the present application. For specific functions and technical effects brought by them, please refer to the method embodiment part.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure is divided into different functional units or modules to complete all or part of the above-described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software functional unit. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the application. The specific working process of the unit and module in the system can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0149] The embodiment of the application further provides a computer program product, wherein the computer program is executed by a processor to realize the steps in each method embodiment.

[0150] The embodiment of the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in each method embodiment.

[0151] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the embodiment of the application realizes all or part of the processes in the above-mentioned method, which can be completed by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium. The computer program is executed by a processor to realize the steps in each method embodiment. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0152] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0153] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for layout analysis of a customs declaration accompanying document, characterized in that, The method comprises: S1: obtaining a customs declaration form attached document image to be analyzed; S2: using a pre-trained DocLayout-YOLO model to perform layout analysis on the document image, detecting and identifying a plurality of layout element regions in the document image, and generating an initial annotation box with a class label and a confidence for each layout element region; the class label at least includes a text box, a table, an implicit table and a customs seal; S3: based on the overlap degree lap value, each initial annotation box is merged and optimized to eliminate redundant boxes to obtain an optimized annotation box set; wherein the calculation formula of the lap value is: lap = (A ∩ B) / min(A, B); wherein A and B are the areas of two annotation boxes to be merged and optimized; wherein the step of merging and optimizing each initial annotation box in S3 comprises: S31: setting an index for each initial annotation box, which is generated based on the confidence sorting; S32: calculating the lap value between any two initial annotation boxes; S33: when the lap value of two initial annotation boxes is greater than a preset lap threshold, the two initial annotation boxes are merged into a new annotation box, and the index of the new annotation box is set as the index of the initial annotation box with the earlier index; S34: repeat steps S32 and S33 until there is no box pair with a lap value greater than the lap threshold, and obtain an optimized annotation box set; S4: performing optical character recognition on the document image to obtain a plurality of OCR text boxes and their text contents; S5: coordinate matching and association of the OCR text boxes and the optimized annotation box set, assigning corresponding text contents to each optimized annotation box to obtain a target annotation box; wherein the step of coordinate matching and association of the OCR text boxes and the optimized annotation box set comprises: calculating the lap value between each OCR text box and each optimized annotation box based on the optimized annotation box; associate an OCR text box to all optimized annotation boxes with a lap value greater than a preset value; S6: based on the class label and position information of the optimized annotation box, sorting the target annotation box to which the text content is assigned to construct a hierarchical structure of the document; wherein the step of constructing a hierarchical structure of the document in S6 comprises: identifying an annotation box with a class label of title and an associated text content of null from the optimized annotation box set, and marking it as a top-level main title; assigning a top-level title without a title to a non-table layout part without a class label of title at the top; grouping the annotation boxes with a class label of title or body under the top-level main title and / or the top-level title without a title to form a hierarchical document structure; S7: outputting structured data according to the hierarchical structure.

2. The method of claim 1, wherein, The step of assigning corresponding text contents to each optimized annotation box in S5 comprises: For the optimized mark box of the category of title block, the text contents of all the OCR text boxes associated with the mark box are combined to form a complete title text; For the optimized mark box of the category of table block, the row and column data of the table are generated through the text contents of the OCR text boxes associated with the mark box.

3. The method according to claim 1 or 2, characterized in that, The step of sorting the target mark boxes to which the text contents are assigned in S6 comprises: determining the center points of the target mark boxes; sorting the target mark boxes in ascending order according to the horizontal coordinates of the center points of the mark boxes; sorting the target mark boxes in ascending order according to the vertical coordinates of the center points of the target mark boxes under the premise of ensuring the order of the horizontal coordinates of the center points of the target mark boxes.

4. A layout analysis apparatus for a customs declaration form attached document, characterized by comprising: The device comprises: an uploading unit configured to acquire a single certificate image attached to a customs declaration form to be analyzed; an analysis unit configured to perform layout analysis on the single certificate image by using a pre-trained DocLayout-YOLO model, detect and identify a plurality of layout element regions in the single certificate image, and generate an initial mark box with a category label and a confidence for each layout element region; the category label comprises a text box, a table, an implicit table, and a customs seal; an optimization unit configured to merge and optimize each initial mark box based on a lap value to eliminate redundant boxes and obtain an optimized mark box set; a calculation formula of the lap value is lap = (A ∩ B) / min(A,B), where A and B are the areas of two mark boxes to be merged and optimized; the step of merging and optimizing each initial mark box comprises: S31: setting an index for each initial mark box, the index being generated based on the confidence sorting; S32: calculating the lap value between any two initial mark boxes; S33: when the lap value between two initial mark boxes is greater than a preset lap threshold, merging the two initial mark boxes into a new mark box, and setting the index of the new mark box as the index of the initial mark box with the earlier index; S34: repeating steps S32 and S33 until there is no box pair with a lap value greater than the lap threshold, and obtaining the optimized mark box set; an identification unit configured to perform optical character recognition (OCR) on the single certificate image to obtain a plurality of OCR text boxes and their text contents; a fusion unit configured to coordinate match the OCR text boxes and the optimized mark box set, assign corresponding text contents to each optimized mark box, and obtain a target mark box; the step of coordinate matching the OCR text boxes and the optimized mark box set comprises: calculating the lap value between each OCR text box and each optimized mark box based on the optimized mark box as a reference; and associating an OCR text box to all the optimized mark boxes with a lap value greater than a preset value. The constructing unit is configured to sort the target annotation boxes to which the text content is assigned based on the category label and the position information of the optimized annotation boxes, and construct a hierarchical structure of the document; wherein the step of constructing the hierarchical structure of the document comprises: identifying an annotation box with a category of title and associated text content being empty from the set of optimized annotation boxes, and marking the annotation box as a top-level major title; assigning a top-level title without a title to a non-table layout part without an annotation box with a category of title at the top; and grouping annotation boxes with a category of title or text under the top-level major title and / or the top-level title without a title to form a hierarchical document structure. The output unit is configured to output structured data according to the hierarchical structure.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the steps of the method according to any one of claims 1 to 3 when executing the computer program.

6. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program implements the steps of the method according to any one of claims 1 to 3 when executed by a processor.

Images