A method and device for automatically identifying a bibliographical list

Abstract

The present invention discloses a method and device for automatically identifying a document catalog, which includes: obtaining documents to be processed; converting the documents to be processed into a plurality of images; performing layout analysis on the images based on a layout analysis model to obtain the categories and bounding box coordinates of each text area in the image; the categories at least include chapter titles and ordinary text; inputting the image and the bounding box coordinates of each text area in the image into a text recognition model to obtain the text in the bounding box; only retaining the contents of the categories of chapter titles and ordinary text; performing confidence analysis on the retained chapter titles and the ordinary text corresponding to the chapter titles based on a large language model, screening out the chapter titles with confidence greater than a preset threshold, and generating a catalog. The present invention can improve the accuracy of document catalog recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for automatically identifying a document catalog. Background Art

[0002] The automatic recognition technology of document catalog structure is an important branch of computer vision and pattern recognition, which involves the automatic detection and recognition of text lines, paragraphs, titles, lists, tables, images and other layout elements from scanned or photographed document images. This technology is crucial for the fields of document automation processing, information retrieval, digital library construction, and ancient book restoration.

[0003] At present, machine learning-based methods, especially deep learning technologies such as convolutional neural networks (CNNs), recurrent neural networks (RNNs) and transformer models, have been widely used in layout analysis, significantly improving the accuracy and robustness of analysis, and can handle complex document structures and multilingual environments, including priority analysis of non-text images, and adapting to different typesetting and printing styles. Combining this technology with a multimodal large language model enables AI to "read" and classify documents, improving the efficiency of information processing.

[0004] The field is still facing the following problems:

[0005] 1. Complex layout structure

[0006] Complex layout structures such as multi-column layouts, irregular text blocks, nested lists and tables remain a challenge, especially in handwritten or artistically typeset documents.

[0007] 2. Diversity and Noise

[0008] Different types of documents (e.g., academic papers, legal documents, news reports, etc.) may have unique layout characteristics, while physical damage such as scan quality, shadows, stains, or fold marks can also interfere with analysis.

[0009] 3. Differences across languages ​​and cultures

[0010] Documents may contain multiple languages ​​or use different writing systems, which requires algorithms to be highly flexible and adaptable.

[0011] 4. Real-time performance and resource consumption:

[0012] High-precision analysis often requires a lot of computing resources. For large-scale document sets or real-time processing scenarios, how to balance accuracy and efficiency is a difficult problem. Summary of the invention

[0013] The purpose of the present invention is to provide a method and device for automatically identifying a document catalogue in order to solve the problem of low accuracy of catalogue structure recognition results caused by the complex layout structure of the document in the prior art.

[0014] The object of the present invention is achieved through the following technical scheme: a method for automatically identifying a document catalogue, comprising:

[0015] Get documents to be processed;

[0016] Converting the document to be processed into a plurality of images;

[0017] Performing layout analysis on the image based on the layout analysis model to obtain categories and bounding box coordinates of each text area in the image; the categories at least include chapter titles and ordinary text;

[0018] Input the image and the bounding box coordinates of each text area in the image into a text recognition model to obtain the text in the bounding box; only retain the content of the category of chapter title and ordinary text;

[0019] Based on the large language model, confidence analysis is performed on the retained chapter titles and the ordinary text corresponding to the chapter titles, and the chapter titles with confidence greater than the preset threshold are screened out to generate a directory.

[0020] Furthermore, the confidence analysis of the retained chapter titles and the ordinary texts corresponding to the chapter titles is performed based on the large language model, and the chapter titles and the ordinary texts corresponding to the chapter titles whose confidences are greater than a preset threshold are first screened out;

[0021] Outputting the chapter titles and the ordinary texts corresponding to the chapter titles of the document to be processed based on the multimodal large language model; performing confidence analysis on the chapter titles and the ordinary texts corresponding to the chapter titles based on the large language model, and screening out the chapter titles and the text contents corresponding to the chapter titles whose confidences are greater than a preset threshold for the second time;

[0022] Based on the large language model, the chapter titles and the ordinary texts corresponding to the chapter titles selected for the first time and the chapter titles and the ordinary texts corresponding to the chapter titles selected for the second time are integrated to generate a literature catalog.

[0023] Furthermore, the multimodal large language model includes Gemini, LLaMA-2, Claude 3, Grok and Flamingo.

[0024] Furthermore, the large language model is a large language model dedicated to document structure understanding, including Llama3.1, Claude3, Qwen2 and Mistral.

[0025] Furthermore, the layout analysis models include LayoutLM, DocFormer, Donut and PublayNet.

[0026] Further, the categories include chapter title, table, table title, image, image title, normal text, header, footer, formula and footnote.

[0027] The present invention also provides a document catalog automatic recognition device, comprising:

[0028] Document acquisition module, used to acquire documents to be processed;

[0029] A document conversion module, used for converting the document to be processed into a plurality of images;

[0030] A layout analysis module, used for performing layout analysis on the image based on a layout analysis model to obtain the category and bounding box coordinates of each text area in the image; the category includes at least chapter titles and ordinary text;

[0031] A text recognition module, used for inputting the coordinates of the bounding boxes of the image and each text area in the image into a text recognition model to obtain the text in the bounding box; only retaining the content of the category of chapter title and ordinary text;

[0032] The catalog generation module is used to perform confidence analysis on the retained chapter titles and the ordinary text corresponding to the chapter titles based on the large language model, screen out the chapter titles with confidence greater than a preset threshold, and generate a document catalog.

[0033] The present invention also provides a document catalog automatic recognition device, comprising:

[0034] The multimodal large language model processing link module is used to output the chapter titles and the text content corresponding to the chapter titles of the document to be processed based on the multimodal large language model; perform confidence analysis on the chapter titles and the text content corresponding to the chapter titles, and first screen out the chapter titles and the text content corresponding to the chapter titles whose confidence is greater than a preset threshold;

[0035] A computer vision processing link module is used to convert the document to be processed into a plurality of images; perform layout analysis on the image using a layout analysis model to obtain the category and bounding box coordinates of each text area in the image; the category includes at least chapter title and ordinary text; input the image and the bounding box coordinates of each text area in the image into a text recognition model to obtain the text in the bounding box; only retain the content of the category "chapter title" and "ordinary text"; perform confidence analysis on the retained "chapter title" and "ordinary text" content, and screen out the chapter titles and text content corresponding to the chapter titles whose confidence is greater than a preset threshold for the second time;

[0036] The fusion module is used to fuse the chapter titles and text contents corresponding to the chapter titles screened out for the first time and the chapter titles and text contents corresponding to the chapter titles screened out for the second time based on the large language model to generate a bibliography.

[0037] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned method for automatic identification of a document catalog.

[0038] The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method for automatically identifying a document catalog is implemented.

[0039] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0040] 1. The embodiment of the present invention combines layout analysis and confidence analysis to improve the accuracy of document catalog recognition;

[0041] 2. Another embodiment of the present invention greatly reduces the proportion of missed detection and false detection in document layout recognition through the complementary advantages of computer vision and large language model technology, further improves the accuracy of document catalog recognition, and provides strong support for document automation processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0043] Figure 1 It is a schematic diagram of the overall processing flow of the present invention.

[0044] Figure 2 Schematic diagram of the computer vision processing link flow of the present invention.

[0045] Figure 3 It is a schematic diagram of the flow of the layout element filtering module in the computer vision processing link of the present invention.

[0046] Figure 4 Schematic diagram of the multimodal large language model processing link of the present invention.

[0047] Figure 5 It is a schematic diagram of the fusion module flow of the present invention.

[0048] Figure 6A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0051] A method for automatically identifying a bibliography of a document of the present invention comprises the following steps:

[0052] (1) Obtain documents to be processed;

[0053] (2) converting the document to be processed into a plurality of images;

[0054] (3) performing layout analysis on the image based on a layout analysis model to obtain categories and bounding box coordinates of each text area in the image; the categories at least include chapter titles and ordinary text;

[0055] (4) inputting the coordinates of the bounding boxes of the image and each text area in the image into a text recognition model to obtain the text within the bounding boxes; only the content classified as chapter titles and ordinary text is retained;

[0056] (5) Based on the large language model, the confidence analysis of the retained chapter titles and the ordinary text corresponding to the chapter titles is performed, and the chapter titles with confidence greater than the preset threshold are screened out to generate a table of contents.

[0057] In one embodiment, see Figure 2 , taking a document in PDF format as an example, the document is split into several images, and the layout analysis model is used to analyze the layout of each image to obtain the category and bounding box coordinates of each text area in each image; and the text recognition model is used to recognize the text in the bounding box; the categories include at least chapter titles and ordinary text, see Figure 3 , the categories also include tables, table titles, images, image titles, headers, footers, formulas and footnotes. In this embodiment, only the contents classified as chapter titles and ordinary text need to be retained, so other categories are filtered out; the chapter title and the ordinary text corresponding to the chapter title are represented as <title, text>, and the <title, text> is input into the large language model. According to the matching degree between the chapter title and its content, the confidence of each <title, text> is given, and the <title, text> with inconsistent semantics is removed, so as to generate a document catalog.

[0058] In one embodiment, the confidence analysis of the retained chapter titles and the ordinary texts corresponding to the chapter titles is performed based on the large language model, and the chapter titles and the ordinary texts corresponding to the chapter titles whose confidences are greater than a preset threshold are screened out for the first time; specifically, the large language model is a large language model (also called a document structure model) dedicated to document structure understanding, such as Llama3.1, Claude3, Qwen2, Mistral, etc., and outputs semantically consistent <title, text> (i.e., <title, text> with a confidence greater than a preset threshold) in combination with a prompt word, wherein the prompt word is: "You are a scientific researcher with rich experience in reading and summarizing papers. The following content comes from the same document, and its structure is <title, text>. Please give the confidence of each <title, text> according to the degree of matching between the title and the content, and remove the <title, text> with inconsistent semantics."

[0059] Based on the multimodal large language model, the chapter titles and the ordinary texts corresponding to the chapter titles of the document to be processed are output; based on the large language model, the confidence analysis of the chapter titles and the ordinary texts corresponding to the chapter titles is performed, and the chapter titles and the text contents corresponding to the chapter titles whose confidences are greater than a preset threshold are screened out for the second time; specifically, the multimodal large language model includes Gemini, LLaMA-2, Claude 3, Grok, Flamingo, etc. This large language model is consistent with the above and will not be repeated here.

[0060] See also Figure 4 , the document and the prompt word are simultaneously input into the multimodal large language model, where the prompt word is "You are a scientific researcher with rich experience in reading and summarizing papers. What is the directory structure of this article? Note that the names of the directories at all levels and the corresponding original text are given at the same time." The output is the document parsing result with the following structure:

[0061] Title 1: Title 1 content

[0062] Title 1.1: Title 1.1 Contents

[0063] Title 1.2: Contents of Title 1.2

[0064] Title 2: Title 2 content

[0065] Title 2.1: Contents of Title 2.1

[0066] Title 3: Title 3 content

[0067] …

[0068] Based on the large language model, the chapter titles and the corresponding ordinary texts of the chapter titles are fused with the chapter titles and the corresponding ordinary texts of the chapter titles that are screened out for the first time, and the chapter titles and the corresponding ordinary texts that are screened out for the second time, to generate a bibliography. Specifically, the <title, text> screened out for the first time, the <title, text> screened out for the second time, and the prompt words are input into the large language model, and a sorted bibliography is output, where the prompt words are: "You are a scientific researcher with rich experience in reading and summarizing papers. The following subsections are from the same document, and their structure is <title, text>. Please sort the subsections according to the logical relationship of the content, and remove unreasonable subsections."

[0069] The embodiment of the present invention also provides a document catalog automatic recognition device, comprising:

[0070] Document acquisition module, used to acquire documents to be processed;

[0071] A document conversion module, used for converting the document to be processed into a plurality of images;

[0072] A layout analysis module, used for performing layout analysis on the image based on a layout analysis model to obtain the category and bounding box coordinates of each text area in the image; the category includes at least chapter titles and ordinary text;

[0073] A text recognition module, used for inputting the coordinates of the bounding boxes of the image and each text area in the image into a text recognition model to obtain the text in the bounding box; only retaining the content of the category of chapter title and ordinary text;

[0074] The catalog generation module is used to perform confidence analysis on the retained chapter titles and the ordinary text corresponding to the chapter titles based on the large language model, screen out the chapter titles with confidence greater than a preset threshold, and generate a document catalog.

[0075] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0076] The embodiment of the present invention also provides another automatic identification device for document catalogs, such as Figure 5 As shown, including:

[0077] The multimodal large language model processing link module is used to output the chapter titles and the text content corresponding to the chapter titles of the document to be processed based on the multimodal large language model; perform confidence analysis on the chapter titles and the text content corresponding to the chapter titles, and first screen out the chapter titles and the text content corresponding to the chapter titles whose confidence is greater than a preset threshold;

[0078] A computer vision processing link module is used to convert the document to be processed into a plurality of images; perform layout analysis on the image using a layout analysis model to obtain the category and bounding box coordinates of each text area in the image; the category includes at least chapter title and ordinary text; input the image and the bounding box coordinates of each text area in the image into a text recognition model to obtain the text in the bounding box; only retain the content of the category "chapter title" and "ordinary text"; perform confidence analysis on the retained "chapter title" and "ordinary text" content, and screen out the chapter titles and text content corresponding to the chapter titles whose confidence is greater than a preset threshold for the second time;

[0079] The fusion module is used to fuse the chapter titles and text contents corresponding to the chapter titles screened out for the first time and the chapter titles and text contents corresponding to the chapter titles screened out for the second time based on the large language model to generate a bibliography.

[0080] like Figure 1 As shown, the input document passes through the computer vision processing link and the multimodal large language model processing link in parallel to obtain two independent outputs, which are then processed by the fusion module to output the final document catalog.

[0081] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0082] Figure 6 This is a schematic diagram of an electronic device structure provided by an embodiment of the present invention. Figure 6 The electronic device provided in this embodiment includes: a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, a method for automatic identification of a document catalog of the present invention is implemented.

[0083] It should be noted that, in addition to Figure 6 In addition to the memory and processor shown, the electronic device may also include other hardware according to its actual functions, which will not be described in detail.

[0084] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a method for automatically identifying a bibliography in the above embodiment is implemented.

[0085] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0086] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for automatically identifying a bibliography, characterized in that: include: Get documents to be processed; Converting the document to be processed into a plurality of images; Performing layout analysis on the image based on the layout analysis model to obtain categories and bounding box coordinates of each text area in the image; the categories at least include chapter titles and ordinary text; Input the image and the bounding box coordinates of each text area in the image into a text recognition model to obtain the text in the bounding box; only retain the content of the category of chapter title and ordinary text; Based on the large language model, the confidence analysis is performed on the retained chapter titles and the ordinary texts corresponding to the chapter titles, and the chapter titles and the ordinary texts corresponding to the chapter titles with confidence greater than a preset threshold are screened out for the first time; Outputting the chapter titles and the ordinary texts corresponding to the chapter titles of the document to be processed based on the multimodal large language model; performing confidence analysis on the chapter titles and the ordinary texts corresponding to the chapter titles based on the large language model, and screening out the chapter titles and the text contents corresponding to the chapter titles whose confidences are greater than a preset threshold for the second time; Based on the large language model, the chapter titles and the ordinary texts corresponding to the chapter titles selected for the first time and the chapter titles and the ordinary texts corresponding to the chapter titles selected for the second time are integrated to generate a literature catalog.

2. A method for automatic identification of a document catalogue according to claim 1, characterized in that: The multimodal large language model is any one of Gemini, LLaMA-2, Claude 3, Grok and Flamingo.

3. The method for automatic identification of a document catalogue according to claim 1, characterized in that: The large language model is a large language model dedicated to document structure understanding, and is any one of Llama3.1, Claude3, Qwen2 and Mistral.

4. The method for automatic identification of a bibliography according to claim 1, characterized in that: The layout analysis model is any one of LayoutLM, DocFormer, Donut and PublayNet.

5. The method for automatic identification of a bibliography according to claim 1, characterized in that: The categories include section title, table, table title, image, image title, normal text, header, footer, formula, and footnote.

6. A document catalog automatic recognition device, characterized in that: include: Document acquisition module, used to acquire documents to be processed; A document conversion module, used for converting the document to be processed into a plurality of images; A layout analysis module, used for performing layout analysis on the image based on a layout analysis model to obtain the category and bounding box coordinates of each text area in the image; the category includes at least chapter titles and ordinary text; A text recognition module, used for inputting the coordinates of the bounding boxes of the image and each text area in the image into a text recognition model to obtain the text in the bounding box; only retaining the content of the category of chapter title and ordinary text; A directory generation module is used to perform confidence analysis on the retained chapter titles and the ordinary texts corresponding to the chapter titles based on the large language model, and first screen out the chapter titles and the ordinary texts corresponding to the chapter titles whose confidences are greater than a preset threshold; Outputting the chapter titles and the ordinary texts corresponding to the chapter titles of the document to be processed based on the multimodal large language model; performing confidence analysis on the chapter titles and the ordinary texts corresponding to the chapter titles based on the large language model, and screening out the chapter titles and the text contents corresponding to the chapter titles whose confidences are greater than a preset threshold for the second time; Based on the large language model, the chapter titles and the ordinary texts corresponding to the chapter titles selected for the first time and the chapter titles and the ordinary texts corresponding to the chapter titles selected for the second time are integrated to generate a literature catalog.

7. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement a method for automatic identification of a document catalog as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a method for automatically identifying a bibliography as described in any one of claims 1 to 5 is implemented.

Images