A document parsing reasoning method, device and medium based on large language model

Through the combination of preset layout analysis model and OCR technology, document label information is reorganized and large language models are used for reasoning, which solves the problem of slow parsing complex documents and achieves efficient and accurate document analysis.

CN118734831BActive Publication Date: 2024-11-12SHANDONG INSPUR SCI RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411230351.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-11-12
Estimated Expiration
2044-09-04

AI Technical Summary

Technical Problem

When parsing complex documents, especially PDF documents containing multiple layout elements, the inference speed is slow, resulting in high computing resources consumption and affecting user experience.

Method used

The preset layout analysis model is used for layout recognition, the tag information is reorganized in logical order, combined with OCR recognition and large language model for reasoning, and the preset draft large language model is used to accelerate the analysis process.

Benefits of technology

It improves the efficiency and accuracy of document parsing, reduces the calculation load of large language models, and ensures the quality and speed of analysis results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118734831B_ABST
    Figure CN118734831B_ABST
Patent Text Reader

Abstract

The embodiment of this specification discloses a document parsing reasoning method, device and medium based on a large language model, which relates to the field of natural language processing technology and is used to solve the problem that the existing large language model is difficult to parse complex structure documents. The method includes: performing layout recognition on the document to be parsed based on a preset layout analysis model to obtain the layout analysis results of each page; determining the label information of each page based on the layout analysis results to reorganize and sort the target boxes corresponding to the label information according to the reading order of the document; performing image recognition on the target box to obtain the target text content, combining the target text content with the label name to obtain structured data; generating designated prompt words according to the structured data and parsing requirements, and inputting the structured data and the designated prompt words as input content into the preset target large language model and the preset draft large language model for inference decoding to obtain the reasoning result of the document to be parsed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present specification relates to the field of natural language processing technology, and in particular to a document parsing and reasoning method, device and medium based on a large language model. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, document data, as an important carrier of information exchange, has become a key link in improving data processing efficiency and value through parsing and understanding. Against this background, document parsing technology has emerged and has quickly demonstrated its broad application potential and demand in multiple industries such as finance, medical care, law, and education. Large language models have injected new vitality into the field of document parsing in the field of natural language processing with their powerful language understanding and generation capabilities. These large language models can not only handle complex and diverse document structures and contents, but also deeply understand the semantic information in the text, thereby accurately extracting key information and performing structured processing.

[0003] However, when currently parsing and reasoning about documents based on a large language model, if it is necessary to parse documents in file formats such as PDF documents, since such documents contain more complex layout elements such as tables, lists, and images compared to documents in formats such as Word and TXT, the complex hierarchical structure of such documents increases the difficulty of the large language model in recognizing such documents to be parsed, and there are higher parsing requirements for document parsing in professional fields. In order to meet high parsing requirements, the large language model needs to have a large parameter scale, which results in a large amount of computing resources being consumed in the reasoning process based on the large language model. For large-scale documents or real-time parsing tasks, this may lead to slow reasoning speed and affect user experience. Therefore, how to optimize the reasoning efficiency and accuracy of the large language model has become a demanding problem that needs to be solved in current document parsing. Summary of the invention

[0004] In order to solve the above technical problems, one or more embodiments of this specification provide a document parsing reasoning method, device and medium based on a large language model.

[0005] One or more embodiments of this specification adopt the following technical solutions:

[0006] One or more embodiments of this specification provide a document parsing reasoning method based on a large language model, the method comprising:

[0007] Performing layout recognition on the document to be parsed based on a preset layout analysis model to obtain layout analysis results of each page in the document to be parsed;

[0008] Determine the tag information of each page based on the layout analysis result, so as to reorganize and sort the target boxes corresponding to the tag information according to the reading order of the document to be parsed; wherein the tag information includes: tag name and tag position;

[0009] Inputting the target frames corresponding to the label positions into a preset OCR text detection and recognition model in sequence based on the sorting order, performing text detection and recognition on the corresponding target frames, and obtaining target text content;

[0010] Combining the target text content with the tag name to obtain structured data corresponding to the target text content;

[0011] Receive designated prompt words corresponding to the structured data, and input the structured data and the designated prompt words as input contents into a preset target large language model and a preset draft large language model for inference decoding to obtain an inference result of the document to be parsed.

[0012] Optionally, in one or more embodiments of the present specification, determining the tag information of each page based on the layout analysis result so as to reorganize and sort the target boxes corresponding to the tag information according to the reading order of the document to be parsed specifically includes:

[0013] Obtaining the layout analysis result to determine the tag information of each page, so as to obtain the horizontal coordinate of the target frame based on the tag position;

[0014] Determine the range value between the horizontal coordinate of the target frame and the horizontal coordinate of the center point of the page where the target frame is located, so as to classify the target frame corresponding to the label information according to the range value to obtain a single-column target frame and a double-column target frame;

[0015] Divide the double-column target frame into a left target frame and a right target frame according to the horizontal coordinate of the double-column target frame and the horizontal coordinate of the center point of the page where the double-column target frame is located; and sort all the double-column target frames from top to bottom and from left to right according to the coordinates of the upper left corners;

[0016] The single-column target frames are sorted based on the vertical coordinates of the single-column target frames, and are inserted into the double-column target frames based on the corresponding vertical coordinates, so as to realize the order reorganization of the target frames corresponding to the label information.

[0017] Optionally, in one or more embodiments of the present specification, the target boxes corresponding to the label positions are sequentially input into a preset OCR text detection and recognition model based on the sorting order, and text detection and recognition are performed on the corresponding target boxes to obtain target text content, specifically including:

[0018] Determine a fixed size corresponding to the preset OCR text detection and recognition model, so as to sequentially resize and scale each of the target frames based on the reorganized order to obtain a target frame to be processed with a fixed size;

[0019] Detecting the target frame to be processed based on the preset OCR text detection and recognition model, determining a text area of ​​the target frame to be processed, and generating a text target frame corresponding to the text area;

[0020] Rearranging the text target frames according to the coordinates of the upper left corners of the text target frames to restore the normal reading order and obtain a text target frame sequence;

[0021] According to the fixed size corresponding to the preset OCR text detection and recognition model, each of the text target frames is resized, so that the resized text target frames are input into the preset OCR text detection and recognition model based on the order of the text target frame sequence, and the character feature vector of each of the text target frames is extracted;

[0022] The target text content is obtained by selecting a character that best matches the character feature vector from a preset font library.

[0023] Optionally, in one or more embodiments of the present specification, combining the target text content with the tag name to obtain structured data corresponding to the target text content specifically includes:

[0024] Based on the tag level corresponding to each tag name and the nesting relationship between each tag name, determine the structured order corresponding to each tag name in the document to be parsed, and create an empty dictionary corresponding to the nesting relationship;

[0025] Traversing the target text contents, determining the tag names corresponding to the target text contents, and sequentially constructing key-value pairs of the target text contents and the tag names;

[0026] The key-value pair is used as structured data corresponding to the target content, so as to synchronize the structured data to a position corresponding to the tag name in a preset empty dictionary for storage.

[0027] Optionally, in one or more embodiments of the present specification, receiving a specified prompt word corresponding to the structured data specifically includes:

[0028] All structured data dictionaries are traversed and combined with prompt word templates respectively; wherein the structured data dictionary is used to describe and define data structure and relationship.

[0029] Optionally, in one or more embodiments of the present specification, the structured data and the specified prompt word are combined as input content and input into a preset target large language model and a preset draft large language model for inference decoding to obtain an inference result of the document to be parsed, specifically including:

[0030] Inputting the structured data and the designated prompt word as input content into a preset target large language model and a preset draft large language model for pre-filling, so as to obtain a first text segment corresponding to the preset target large language model and a second text segment corresponding to the preset draft large language model; wherein the magnitude of the preset target large language model is greater than the magnitude of the preset draft large language model;

[0031] Replacing the second text segment based on the first text segment, so that the preset draft large language model iteratively acquires a preset number of third text segments based on the first text segment;

[0032] splicing the third text segments to obtain a spliced ​​text segment and inputting the spliced ​​text segment into the preset target large language model to determine whether the third text segment at each position in the spliced ​​text segment is accepted;

[0033] An initial inference result is determined based on the acceptance result to iteratively obtain the inference result of the document to be parsed.

[0034] Optionally, in one or more embodiments of the present specification, determining the initial reasoning result based on the acceptance result to iteratively obtain the reasoning result of the document to be parsed specifically includes:

[0035] If the acceptance result is not accepted, the third text segment iteratively generated by the preset draft large language model is deleted, and the target large language continues to generate the first text segment of the current round and passes it to the preset draft large language model to generate a preset number of third text segments of the current round;

[0036] If the acceptance result is accepted, the concatenated text segment is used as an initial reasoning result;

[0037] The initial reasoning result is spliced ​​to the first text segment to obtain the third text segment of the current round. When judging whether the third text segment is accepted, the target large language model generates a corresponding text segment for each position. If it is consistent with the third text segment, it is accepted; otherwise, it is rejected;

[0038] A new text segment is generated at the end of the text after the target large language model completes the judgment. If all the third text segments are accepted, the new text segment is passed to the draft large language model for further iteration to iteratively obtain the inference result of the document to be parsed.

[0039] Optionally, in one or more embodiments of the present specification, before performing layout recognition on the document to be parsed based on a preset layout analysis model and obtaining layout analysis results of each page in the document to be parsed, the method further includes:

[0040] Based on the YOLOv8 network architecture as the benchmark model of the preset layout analysis model, a dual-link attention mechanism is introduced into the backbone network of the benchmark model to obtain an initial layout analysis model;

[0041] Acquire a document dataset corresponding to the layout analysis result based on a public channel, and annotate document samples in the document dataset based on preset tags to obtain a document recognition dataset;

[0042] Dividing the document recognition data set according to a preset ratio to obtain a document recognition training sample set and a document recognition verification sample set;

[0043] Inputting the document recognition training sample set into the initial layout analysis model to obtain a training layout analysis result, and determining recognition error information of the initial layout analysis model based on the training layout analysis result and the annotation of the document recognition training sample set, so as to provide feedback to the dual-link attention mechanism to correct the attention weight of each path based on the recognition error information;

[0044] The dual links of the initial layout analysis model are iteratively trained based on the document recognition sample set, and the trained initial layout analysis model is verified based on the document recognition verification sample set to obtain a preset layout analysis model that meets the requirements.

[0045] One or more embodiments of this specification provide a document parsing and reasoning device based on a large language model, the device comprising:

[0046] at least one processor; and,

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned methods.

[0049] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute any of the above-described methods.

[0050] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0051] According to the preset layout analysis model, the layout of the document to be parsed is identified, and the layout analysis results of each page in the document to be parsed are obtained, which solves the problem of different document layouts in different industries and different application scenarios, and the difficulty of identifying documents with non-fixed layouts. Through sequential reorganization, the input order of the target box is consistent with the reading order of the document, which enables the OCR recognition process to proceed in a logical order, avoiding additional processing time and resource consumption caused by disordered input. At the same time, since each target box is processed independently, this also provides the possibility for parallel processing. Based on the preset draft large language model and the preset target large language model, the target large language model can perform a deeper analysis based on the output of the draft large language model, thereby ensuring the accuracy and quality of the final parsing result while ensuring the parsing speed and reducing the computing power of the target large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0053] Figure 1 A flowchart of a document parsing and reasoning method based on a large language model provided in an embodiment of this specification;

[0054] Figure 2 A schematic diagram of document parsing reasoning results based on a large language model provided in an embodiment of this specification;

[0055] Figure 3 A schematic diagram of the structure of a document parsing and reasoning device based on a large language model provided in an embodiment of this specification;

[0056] Figure 4 A schematic diagram of a non-volatile storage medium structure provided in an embodiment of this specification. DETAILED DESCRIPTION

[0057] The embodiments of this specification provide a document parsing reasoning method, device and medium based on a large language model.

[0058] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0059] like Figure 1 As shown, the embodiment of this specification provides a schematic diagram of the structure of a document parsing reasoning method based on a large language model, which is composed of Figure 1 It can be seen that in one or more embodiments of this specification, a document parsing reasoning method based on a large language model includes the following steps:

[0060] S101: Performing layout recognition on the document to be parsed based on a preset layout analysis model to obtain layout analysis results of each page in the document to be parsed.

[0061] Traditional document processing systems often focus on text processing, and have limited parsing capabilities for complex layouts containing multiple elements such as figures, tables, and formulas. Therefore, in order to achieve the parsing of documents to be parsed with complex layouts, the comprehensiveness and accuracy of document parsing are improved. The embodiment of this specification performs layout recognition on the document to be parsed according to a pre-set layout analysis model, obtains the layout analysis results of each page in the document to be parsed, and solves the problem of different document layouts in different industries and different application scenarios, and the difficulty of identifying documents with non-fixed layouts. It should be noted here that in order to ensure that the preset layout analysis model can be quickly inferred, YOLOv8 can be selected as the basic framework, and a layout analysis model specifically for Chinese documents is trained on the annotated data set. The model contains 12 tags, including first-level titles, second-level titles, third-level titles, text (paragraphs), pictures, picture titles, tables, table titles, headers, footers, formulas, and references.

[0062] Furthermore, in one or more embodiments of the present specification, before performing layout recognition on the document to be parsed based on the preset layout analysis model and obtaining the layout analysis results of each page in the document to be parsed, the method further includes:

[0063] First, YOLOv8 is an advanced real-time target detection network, known for its high speed and accuracy. In order to quickly identify multiple elements in a document, this specification selects the YOLOv8 network architecture as the benchmark model of the preset layout analysis model, and introduces a dual-link attention mechanism in the backbone network of the benchmark model to obtain an initial layout analysis model, thereby allowing the model to focus on the characteristics of global and local features at the same time with the help of the dual-link attention mechanism, enhancing the model's ability to capture key information in the document layout so that the subsequent preset layout analysis model can more accurately identify elements in a complex layout. Then, a document dataset corresponding to the layout analysis results is collected through public channels such as the Internet or academic databases, and then the document samples in the document dataset are annotated according to preset labels to obtain a document recognition dataset; wherein the preset labels include: first-level title, second-level title, third-level title, text (paragraph), picture, picture title, table, table title, header, footer, formula and reference. The document recognition dataset is divided according to a preset ratio (such as 80% training set, 20% verification set) to obtain a document recognition training sample set and a document recognition verification sample set. The document recognition training sample set is input into the initial layout analysis model to obtain the training layout analysis results, and then the recognition error information of the initial layout analysis model is determined according to the training layout analysis results and the annotations of the document recognition training sample set, and then the dual-link attention mechanism is fed back to correct the attention weights of each path according to the recognition error information.

[0064] For example, in a scenario, we have a document recognition dataset containing 1,000 document samples. These documents cover various layout structures, such as newspaper articles, academic papers, financial statements, etc. The goal is to build a preset layout analysis model that can accurately identify the layout structure of these documents. First, the dataset is divided according to the preset ratio (80% training set, 20% validation set) to obtain a training set sample containing 800 document samples for training the initial layout analysis model. The validation set sample containing 200 document samples is used to verify the performance of the model during the training process and adjust the model parameters accordingly. The training set is input into the initial layout analysis model based on YOLOv8 and introduced with a dual-link attention mechanism. The model starts to process these documents, identifies the text blocks, pictures, tables and other elements, and outputs the training layout analysis results, and then compares the training layout analysis results with the annotations of each document sample in the training set. Through comparison, we can calculate the errors of the model in identifying each element, such as position deviation, type misjudgment, etc. These error information will be used as feedback to guide the adjustment of model parameters. Then, according to the recognition error information, the dual-link attention mechanism is fed back and corrected. For example, for elements with large errors, the attention weight of the corresponding path is increased. If the error information indicates that the attention mechanism of some paths is not effective, the network structure is adjusted, such as adding convolution layers, changing activation functions, etc., to improve the recognition ability of the model. Specifically, if we find that the model's attention allocation is insufficient when processing the edge of the table by comparing the analysis results of the training layout with the annotations, we increase the attention weight of the path that processes the edge of the table, so that the model pays more attention to the edge features when recognizing the table. At the same time, adjust the convolution kernel size and step size on this path to better capture the edge information of the table.

[0065] After the attention weights of each path are corrected by feedback, the initial layout analysis model is iteratively trained multiple times using the document recognition training sample set, and the model is verified using the document recognition verification sample set after each iteration. By continuously adjusting the model parameters and attention weights until the performance of the model on the verification set meets the preset requirements, the model obtained at this time is a preset layout analysis model that meets the requirements. Based on the above process, during the training process, the recognition error information is determined according to the comparison between the training layout analysis results and the annotations, and the dual-link attention mechanism is feedback-corrected accordingly. This mechanism can adjust the model parameters and attention weights in real time, so that the model pays more attention to those elements that are prone to misjudgment in subsequent training, thereby improving the overall recognition performance. By comprehensively using the YOLOv8 network architecture, dual-link attention mechanism, detailed data set annotation and effective feedback correction mechanism, an efficient, accurate and robust preset layout analysis model is constructed. This is of great significance for improving the level of document processing automation and reducing labor costs.

[0066] S102: Determine the tag information of each page based on the layout analysis result, so as to reorganize and sort the target boxes corresponding to the tag information according to the reading order of the document to be parsed; wherein the tag information includes: tag name and tag position.

[0067] After obtaining the layout analysis result based on the above step S101, in order to maintain the logical structure of the document and facilitate subsequent processing and understanding, it is necessary to regroup all the marked contents according to the reading order of the original document. Therefore, in the embodiment of this specification, the label information contained in each page is determined based on the layout analysis result, and then the target boxes corresponding to the label information are reorganized and sorted according to the reading order of the document to be parsed. Among them, it should be noted that the label information includes: label name, label position. By regrouping all the marked contents based on the reading order of the original document, it is convenient to detect and recognize the text in the image based on the subsequent OCR technology.

[0068] Specifically, in one or more embodiments of the present specification, the tag information of each page is determined based on the layout analysis result, so as to reorganize and sort the target boxes corresponding to the tag information according to the reading order of the document to be parsed, which specifically includes the following process:

[0069] First, based on the layout analysis results, the label information of each page is determined, so that the horizontal coordinate of the target frame is obtained according to the label position. After accurately locating each content area in the document, that is, the target frame corresponding to the label information, the extreme difference between the horizontal coordinate of the target frame and the horizontal coordinate of the center point of the page where the target frame is located is determined, so that the target frame corresponding to the label information is classified according to the extreme difference value to obtain a single-column target frame and a double-column target frame. For example: suppose we have a document page with a width of W. The horizontal coordinate of the center point of the page is naturally W / 2. Through the layout analysis results, it is determined that there are target frames A, B, and C in the page; among them, target frame A: the horizontal coordinate range is [100, 300], target frame B: the horizontal coordinate range is [400, 600], and target frame C: the horizontal coordinate range is [W / 2 - 50, W / 2 + 50]. Then, for each target frame, we calculate the extreme difference between its horizontal coordinate range and the horizontal coordinate of the center point of the page (W / 2). The range value can be understood as the maximum horizontal distance from the leftmost or rightmost end of the target frame to the center of the page. The range value of target frame A: max(|100- W / 2|, |300 - W / 2|), the range value of target frame B: max(|400 - W / 2|, |600 - W / 2|), the range value of target frame C: Since it spans the center of the page, the range value is relatively small, 50. Determine the corresponding range value threshold according to the actual situation and the page width W. If the range value is greater than the range value threshold, it is part of a double-column or multi-column target frame. If the range values ​​of target frames A and B are both greater than the range value threshold, and their horizontal axis ranges do not overlap, then they may belong to two columns in a double-column layout. The range value of target frame C is less than the range value threshold and spans the center of the page, so it may be a special single-column target frame.

[0070] Based on the above process, the target frames are classified, and after obtaining the single-column target frame and the double-column target frame, the double-column target frame is divided into the left target frame and the right target frame according to the horizontal coordinate of the double-column target frame and the horizontal coordinate of the center point of the page where the double-column target frame is located. Then, all the double-column target frames are sorted from top to bottom and from left to right according to the coordinates of the upper left corners. At the same time, the single-column target frames are sorted according to the vertical coordinates of each single-column target frame, and the corresponding vertical coordinates are inserted into the double-column target frame, so that the order of the target frames corresponding to the label information is reorganized. By flexibly adjusting the classification and sorting rules of the target frames, it can adapt to documents with different layout formats and realize the order reorganization of the target frames. This sorting and reorganization method not only conforms to people's reading habits, but also facilitates the subsequent further processing and analysis of the document content. Therefore, based on the above process, the accuracy of document parsing can be significantly improved by accurately locating, classifying, sorting and reorganizing the target frames. The various content areas originally scattered in the document are organized in an orderly manner, forming a document view with a clear structure and easy to understand.

[0071] S103: Inputting the target frames corresponding to the label positions into a preset OCR text detection and recognition model in sequence based on the sorting order, performing text detection and recognition on the corresponding target frames, and obtaining target text content.

[0072] After the order is reorganized through the above step S102, in order to detect and recognize the text of the content of each target box. In the embodiment of this specification, the target boxes corresponding to each label position are input into the preset OCR text detection and recognition model in sequence according to the sorting order, and the image recognition is performed on each corresponding target box to obtain the target text content. Among them, the preset OCR text detection and recognition model can be a recognition model of the currently commonly used OCR technology. In this process, since the target boxes have been accurately located and classified, and the target boxes in the above steps have also been reorganized according to the reading order, the input order of the target boxes is consistent with the reading order of the document, which enables the OCR recognition process to be carried out in a logical order, avoiding the extra processing time and resource consumption caused by disordered input, and greatly reducing the interference factors in the OCR recognition process. The content in each target box is usually more concentrated and clear, which helps the OCR model to more accurately identify the text content and reduce the error recognition rate and missed recognition rate.

[0073] Specifically, in one or more embodiments of the present specification, the target frames corresponding to the label positions are sequentially input into the preset OCR text detection and recognition model based on the sorting order, and the corresponding target frames are image recognized to obtain the target text content, which specifically includes the following process:

[0074] In order to reduce the recognition error caused by inconsistent image size, thereby improving the recognition accuracy. In the embodiment of this specification, after determining the fixed size corresponding to the preset OCR text detection and recognition model, each target frame is scaled and adjusted in turn based on the reorganized order to obtain a target frame to be processed with a uniform fixed size. Then, the target frame to be processed is detected according to the preset OCR text detection and recognition model, the text area of ​​the target frame to be processed is determined, and a text target frame corresponding to the text area is generated. By realizing a more precise positioning of the text area, that is, the text target frame, the interference of background noise can be reduced, and the recognition effect can be further improved. Then, each text target frame is reordered according to the upper left corner coordinates of each text target frame to restore the normal reading order to obtain a text target frame sequence. Similarly, according to the fixed size corresponding to the preset OCR text detection and recognition model, each text target frame is scaled, so that the scaled text target frame is input into the preset OCR text detection and recognition model according to the order of the text target frame sequence, and the character feature vector of each text target frame is extracted. Then, the target text content is obtained by selecting the character that best matches the character feature vector from the preset font library. The character feature vector extraction and character matching process is usually more efficient and can quickly complete the recognition task of a large number of characters.

[0075] S104: Combine the target text content with the tag name to obtain structured data corresponding to the target text content.

[0076] In order to convert unstructured documents into machine-readable form, the target text content and tag name are combined in the embodiment of this specification to obtain structured data corresponding to the target text content. Through structured data with clear attributes and relationships, search engines can locate key information more accurately. In large-scale text data, users can quickly find the information they need, greatly improving the efficiency of data use.

[0077] Specifically, in one or more embodiments of the present specification, combining the target text content with the tag name to obtain structured data corresponding to the target text content specifically includes the following process:

[0078] First, based on the tag level corresponding to each tag name and the nested relationship between each tag name, the structured order corresponding to each tag name in the document to be parsed is determined, and an empty dictionary corresponding to the nested relationship is created. In document processing, complex nested structures such as chapters, paragraphs, lists, etc. are often encountered. By defining the nested relationship of tags, these complex structures can be accurately captured and reflected in the data structure. This is particularly important for application scenarios that need to process complex document structures. Then the creation of the empty dictionary provides a framework or container for the data to be filled. As the data is gradually filled, this framework will gradually become richer to form a complete data structure. Then, the target text content is traversed to determine the tag name corresponding to each target text content, so as to sequentially construct the key-value pairs of each target text content and each tag name. The key-value pairs are used as the structured data corresponding to the target content, so as to synchronize the structured data to the position corresponding to the tag name in the preset empty dictionary for storage. By storing the key-value pairs as structured data in the preset empty dictionary, the data structure characteristics of the dictionary can be fully utilized to achieve efficient data access operations. And because the data structure is predefined and each data item is explicitly assigned a tag name, the entire data set can maintain a high degree of consistency during storage. Data with multi-level hierarchical relationships can be flexibly represented by nesting complex data structures such as dictionaries or lists. This capability enables the method to handle complex document structures such as nested chapters, lists, etc. without losing data integrity and accuracy.

[0079] In a certain application scenario, the target text content is combined with the tag name to obtain the structured data corresponding to the target text content, which specifically includes the following process: First, create an empty dictionary to store structured data. Then determine whether the tag information is a first-level title. If it is confirmed to be a first-level title, use it as the new key of the dictionary and start collecting the content after the title as the corresponding value. Continue traversing until the next first-level title is encountered. The collected content during this period is stored in the dictionary as the value of the previous title. When a new title is encountered, repeat the above process to create a new key-value pair for the new title. After traversing the entire document, all content will be structured and stored in the dictionary.

[0080] S105: receiving designated prompt words corresponding to the structured data, and inputting the structured data and the designated prompt words as input contents into a preset target large language model and a preset draft large language model for inference decoding to obtain an inference result of the document to be parsed.

[0081] In order to avoid the document parsing in the existing professional field having higher parsing requirements, and in order to meet the high parsing requirements, the large language model needs to have a huge parameter scale, which leads to the need to consume a large amount of computing resources in the reasoning process based on the large language model. For large-scale documents or real-time parsing tasks, this may lead to slow reasoning speed and affect the user experience. In order to improve the processing efficiency, the following measures can be taken: First, the document is divided according to the first-level title, so that each part represents an independent theme paragraph or chapter, making the processing more modular and easy to manage. The second is to use the method of speculative decoding to further accelerate the reasoning process of the model. The method of speculative decoding is further elaborated in the embodiment of this specification: according to the structured data and the parsing requirements, the specified prompt words are generated, and the structured data and the specified prompt words are input as input content into the preset target large language model and the preset draft large language model for speculative decoding to obtain the reasoning result of the document to be parsed. The preset draft large language model and the preset target large language model improve the reasoning speed while ensuring the parsing requirements. By introducing the preset draft large language model, the model has a smaller parameter scale or is optimized for fast response. In the preliminary parsing stage, the draft large language model can quickly perform preliminary processing on the structured data and prompt words to generate a relatively rough but sufficiently fast parsing result. Since the parameter scale of the draft large language model is relatively small, its computing requirements are also reduced accordingly. Using the draft large language model in the preliminary parsing stage can reduce the demand for computing resources, and then the structured data, prompt words, and the output of the draft large language model are passed as input to the preset target large language model for further speculative decoding. The target large language model usually has a larger parameter scale and stronger reasoning ability, and can perform a deeper parsing based on the output of the draft large language model, thereby ensuring the accuracy and quality of the final parsing result while ensuring the parsing speed. In addition, by generating specified prompt words and making customized adjustments according to the parsing requirements, the solution can flexibly cope with document parsing tasks of different fields and different complexities. The introduction of prompt words provides additional contextual information for the parsing process, which helps the model to understand the document content more accurately and generate parsing results that meet the requirements.

[0082] Specifically, in one or more embodiments of the present specification, receiving a specified prompt word corresponding to the structured data specifically includes: traversing all structured data dictionaries and combining them with prompt word templates respectively, wherein it should be noted that the structured data dictionary is used to describe and define data structures and relationships.

[0083] Specifically, in one or more embodiments of the present specification, the structured data and the specified prompt words are input as input content into the preset target large language model and the preset draft large language model for inference decoding to obtain the inference result of the document to be parsed, which specifically includes the following process:

[0084] First, the structured data and the specified prompt words are input as input content into the preset target large language model and the preset draft large language model for pre-filling, so as to obtain the first text segment corresponding to the preset target large language model and the second text segment corresponding to the preset draft large language model. Among them, it can be understood that the magnitude of the preset target large language model is greater than the magnitude of the preset draft large language model. Then, the second text segment is replaced based on the first text segment, so that the preset draft large language model can iteratively obtain a preset number of third text segments according to the first text segment. Then, the third text segment is spliced ​​to obtain the spliced ​​text segment and input the spliced ​​text segment into the preset target large language model to determine whether the third text segment at each position in the spliced ​​text segment is accepted. Thus, the initial reasoning result is determined based on the acceptance result to iteratively obtain the reasoning result of the document to be parsed. In this process, by first using the preset draft large language model for pre-filling and using its smaller magnitude for preliminary reasoning, the computing resources required for directly using the large-scale target language model can be reduced. This hierarchical processing method significantly improves the processing efficiency while ensuring the parsing quality. Although the size of the preset draft large language model is relatively small, its preliminary reasoning results provide an important basis for subsequent iterations. By replacing the second text segment with the first text segment and using the target large language model to judge the acceptance of the spliced ​​text segments, we can gradually filter out reasoning results that are more in line with the actual content of the document, thereby improving the accuracy of the final reasoning. The third text segment is obtained through iteration and spliced, and then the acceptance of the spliced ​​text segment is judged. This process can reduce the time required for a single reasoning to a certain extent.

[0085] Further, in one or more embodiments of the present specification, determining an initial reasoning result based on the acceptance result to iteratively obtain the reasoning result of the document to be parsed specifically includes the following process:

[0086] If the acceptance result is not accepted, then the third text segment iteratively generated by the draft large language model is deleted, and then the target large language continues to generate the first text segment of the current round and passes it to the preset draft large language model to generate a preset number of third text segments for the current round. If the reception result is accepted, then the spliced ​​text segment is used as the initial reasoning result, and then the initial reasoning result is spliced ​​to the first text segment to obtain the third text segment of the current round. When judging whether the third text segment is accepted, the target large language model will generate corresponding text segments for each position. If it is consistent with the third text segment, it will be accepted, otherwise it will be rejected. The new text segment generated at the end of the text after the target large language model has completed the judgment is obtained. If the third text segment is fully accepted, then the new text segment is passed to the draft large language model to continue iterating to obtain the reasoning result of the document to be parsed. By inputting the first text segment of the current round into the preset draft large language model again for iteration, the learning and reasoning capabilities of the model can be fully utilized to further explore the potential information and associations in the document. This efficient use of model capabilities helps to improve the depth and breadth of reasoning results. Among them, for example Figure 2 Shown is a schematic diagram of an analysis result provided in an embodiment of this specification.

[0087] Specifically, in a certain application scenario of the present specification, a lightweight draft large language model is used to pre-generate some text fragments (tokens), and then a more powerful target large language model is used to evaluate in parallel whether these fragments meet the expected output standards, thereby significantly reducing unnecessary calculations and improving the overall processing speed, that is, the structured data and the specified prompt words are input as input content into the preset target large language model and the preset draft large language model for inference decoding to obtain the reasoning result of the document to be parsed, which specifically includes the following process: First, the received content is input into the target large language model and the draft large language model for pre-filling, and each generates the first text fragment token. The token generated by the target large language model replaces the token of the draft large language model, so that the draft large language model continues to generate 5 tokens based on the token. These 5 tokens are spliced ​​together and sent to the target large language model for parallel reasoning to determine whether to accept the token at each position. If an unacceptable token appears, delete the kV cache of the draft large language model and the target large language model for the token and the token after it. The token at the position where the target large language model starts to delete is passed to the draft large language model for generation. The target large language model generates a new token and passes it to the draft large language model again, and the draft large language model continues to generate 5 tokens. Repeat the above process until the generation is completed.

[0088] like Figure 3As shown, the embodiment of this specification provides a schematic diagram of the structure of a document parsing and reasoning device based on a large language model. Figure 3 It can be seen that in one or more embodiments of this specification, a document parsing and reasoning device based on a large language model includes:

[0089] at least one processor; and,

[0090] a memory communicatively connected to the at least one processor; wherein,

[0091] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned methods.

[0092] like Figure 4 As shown, an embodiment of this specification provides a non-volatile storage medium storing computer executable instructions 401, wherein the computer executable instructions can execute any of the above-described methods.

[0093] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0094] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0095] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A document parsing reasoning method based on a large language model, characterized in that: The method comprises: Performing layout recognition on the document to be parsed based on a preset layout analysis model to obtain layout analysis results of each page in the document to be parsed; Determine the tag information of each page based on the layout analysis result, so as to reorganize and sort the target boxes corresponding to the tag information according to the reading order of the document to be parsed; wherein the tag information includes: tag name and tag position; Inputting the target frames corresponding to the label positions into a preset OCR text detection and recognition model in sequence based on the sorting order, performing text detection and recognition on the corresponding target frames, and obtaining target text content; Combining the target text content with the tag name to obtain structured data corresponding to the target text content; Receive designated prompt words corresponding to the structured data, and input the structured data and the designated prompt words as input contents into a preset target large language model and a preset draft large language model for inference decoding to obtain an inference result of the document to be parsed.

2. According to the document parsing and reasoning method based on a large language model according to claim 1, it is characterized in that: Determining the tag information of each page based on the layout analysis result, so as to reorganize and sort the target boxes corresponding to the tag information according to the reading order of the document to be parsed, specifically includes: Obtaining the layout analysis result to determine the tag information of each page, so as to obtain the horizontal coordinate of the target frame based on the tag position; Determine the range value between the horizontal coordinate of the target frame and the horizontal coordinate of the center point of the page where the target frame is located, so as to classify the target frame corresponding to the label information according to the range value to obtain a single-column target frame and a double-column target frame; Divide the double-column target frame into a left target frame and a right target frame according to the horizontal coordinate of the double-column target frame and the horizontal coordinate of the center point of the page where the double-column target frame is located; and sort all the double-column target frames from top to bottom and from left to right according to the coordinates of the upper left corners; The single-column target frames are sorted based on the vertical coordinates of the single-column target frames, and are inserted into the double-column target frames based on the corresponding vertical coordinates, so as to realize the order reorganization of the target frames corresponding to the label information.

3. According to the document parsing reasoning method based on a large language model as described in claim 1, it is characterized in that: Inputting the target frames corresponding to the label positions into a preset OCR text detection and recognition model in sequence based on the sorting order, performing text detection and recognition on the corresponding target frames, and obtaining target text content specifically includes: Determine a fixed size corresponding to the preset OCR text detection and recognition model, so as to sequentially resize the target frames based on the reorganized order to obtain a target frame to be processed with a fixed size; Detecting the target frame to be processed based on the preset OCR text detection and recognition model, determining a text area of ​​the target frame to be processed, and generating a text target frame corresponding to the text area; Rearranging the text target frames according to the coordinates of the upper left corners of the text target frames to restore the normal reading order and obtain a text target frame sequence; According to the fixed size corresponding to the preset OCR text detection and recognition model, each of the text target frames is resized, so that the resized text target frames are input into the preset OCR text detection and recognition model based on the order of the text target frame sequence, and the character feature vector of each of the text target frames is extracted; The target text content is obtained by selecting a character that best matches the character feature vector from a preset font library.

4. According to claim 1, a document parsing and reasoning method based on a large language model is characterized in that: Combining the target text content with the tag name to obtain structured data corresponding to the target text content specifically includes: Based on the tag level corresponding to each tag name and the nesting relationship between each tag name, determine the structured order corresponding to each tag name in the document to be parsed, and create an empty dictionary corresponding to the nesting relationship; Traversing the target text contents, determining the tag names corresponding to the target text contents, and sequentially constructing key-value pairs of the target text contents and the tag names; The key-value pair is used as structured data corresponding to the target content, so as to synchronize the structured data to a position corresponding to the tag name in a preset empty dictionary for storage.

5. According to claim 1, a document parsing and reasoning method based on a large language model is characterized in that: Receiving a specified prompt word corresponding to the structured data specifically includes: All structured data dictionaries are traversed and combined with prompt word templates respectively; wherein the structured data dictionary is used to describe and define data structure and relationship.

6. The document parsing and reasoning method based on a large language model according to claim 1, characterized in that: The structured data and the designated prompt word are input as input content into a preset target large language model and a preset draft large language model for inference decoding to obtain an inference result of the document to be parsed, specifically including: Inputting the structured data and the designated prompt word as input content into a preset target large language model and a preset draft large language model for pre-filling, so as to obtain a first text segment corresponding to the preset target large language model and a second text segment corresponding to the preset draft large language model; wherein the magnitude of the preset target large language model is greater than the magnitude of the preset draft large language model; Replacing the second text segment based on the first text segment, so that the preset draft large language model iteratively acquires a preset number of third text segments based on the first text segment; splicing the third text segments to obtain a spliced ​​text segment and inputting the spliced ​​text segment into the preset target large language model to determine whether the third text segment at each position in the spliced ​​text segment is accepted; An initial inference result is determined based on the acceptance result to iteratively obtain the inference result of the document to be parsed.

7. The document parsing and reasoning method based on a large language model according to claim 6 is characterized in that: The determining of the initial reasoning result based on the acceptance result to iteratively obtain the reasoning result of the document to be parsed specifically includes: If the acceptance result is not accepted, the third text segment iteratively generated by the preset draft large language model is deleted, and the target large language continues to generate the first text segment of the current round and passes it to the preset draft large language model to generate a preset number of third text segments of the current round; If the acceptance result is accepted, the concatenated text segment is used as an initial reasoning result; The initial reasoning result is spliced ​​to the first text segment to obtain the third text segment of the current round. When judging whether the third text segment is accepted, the target large language model generates a corresponding text segment for each position. If it is consistent with the third text segment, it is accepted; otherwise, it is rejected; A new text segment is generated at the end of the text after the target large language model completes the judgment. If all the third text segments are accepted, the new text segment is passed to the draft large language model for further iteration to obtain the inference result of the document to be parsed.

8. The document parsing and reasoning method based on a large language model according to claim 1, characterized in that: Before performing layout recognition on the document to be parsed based on the preset layout analysis model and obtaining the layout analysis results of each page in the document to be parsed, the method further includes: Based on the YOLOv8 network architecture as the benchmark model of the preset layout analysis model, a dual-link attention mechanism is introduced into the backbone network of the benchmark model to obtain an initial layout analysis model; Acquire a document dataset corresponding to the layout analysis result based on a public channel, and annotate document samples in the document dataset based on preset tags to obtain a document recognition dataset; Dividing the document recognition data set according to a preset ratio to obtain a document recognition training sample set and a document recognition verification sample set; Inputting the document recognition training sample set into the initial layout analysis model to obtain a training layout analysis result, and determining recognition error information of the initial layout analysis model based on the training layout analysis result and the annotation of the document recognition training sample set, so as to provide feedback to the dual-link attention mechanism to correct the attention weight of each path based on the recognition error information; The dual links of the initial layout analysis model are iteratively trained based on the document recognition sample set, and the trained initial layout analysis model is verified based on the document recognition verification sample set to obtain a preset layout analysis model that meets the requirements.

9. A document parsing and reasoning device based on a large language model, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: execute any of the methods described in claims 1-8.

10. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions can: execute the method described in any one of claims 1 to 8.