Document analysis method and device, medium and computer program product

Abstract

The invention discloses a document parsing method and device, a medium and a computer program product, and relates to the technical field of data processing.The document parsing method comprises the steps that a to-be-parsed document is obtained, layout detection is conducted on the to-be-parsed document, and element blocks are obtained; determining a content extraction rule corresponding to the element block according to the layout element of the element block; processing the element block through a content extraction rule to obtain initial content of the element block; and correcting the preliminary content by using a large language model to obtain a target document. According to the method, different element blocks are divided, different content extraction methods are adopted for the element blocks, and the extracted content is corrected by utilizing the large language model, so that the correctness of document analysis is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a document parsing method, device, medium and computer program product. Background Art

[0002] With the continuous evolution of information technology, document formats have gradually developed from early plain text and rich text to a variety of document formats, such as the currently widely used PDF (Portable Document Format), Word format and Markdown format.

[0003] With the help of OCR (Optical Character Recognition) technology, text information in documents can be effectively extracted. However, since documents may contain complex layout designs, various fonts, and image elements, it is easy to encounter challenges when applying OCR technology to document parsing, and it is difficult to accurately extract all elements in the document. Specifically, the inaccuracy of OCR text recognition and errors in text segmentation will reduce the accuracy of document parsing.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a document parsing method, device, medium and computer program product, aiming to solve the technical problem of inaccurate document parsing.

[0006] To achieve the above objectives, the present application proposes a document parsing method, which includes: Obtaining a document to be parsed, performing layout detection on the document to be parsed, and obtaining element blocks; Determining a content extraction rule corresponding to the element block according to the layout elements of the element block; Processing the element block according to the content extraction rule to obtain preliminary content of the element block; The preliminary content is corrected using a large language model to obtain a target document.

[0007] In one embodiment, before the step of using the large language model to correct each of the preliminary contents, the method further includes: In the case where the layout element is text, a segmented correction model is used to perform segmented correction on the preliminary content of the element block.

[0008] In one embodiment, the step of performing segmented correction on the preliminary content of the element block using the segmented correction model comprises: Deleting line break marks in the preliminary content and extracting adjacent sentence pairs of preset symbols; Using the segmentation correction model, determining whether the adjacent sentence pairs belong to the same paragraph; In the case where the adjacent sentence pairs do not belong to the same paragraph, a line break mark is added between the adjacent sentence pairs.

[0009] In one embodiment, the segment correction model includes a bidirectional encoder representation BERT model from a converter, and the step of using the segment correction model to determine whether the adjacent sentence pairs belong to the same paragraph includes: Using the BERT model, extracting semantic features of the adjacent sentence pairs; Mapping the semantic features into probability values, wherein the probability values ​​are probabilities that the adjacent sentence pairs belong to the same paragraph; When the probability value is greater than a preset probability threshold, determining that the adjacent sentence pairs belong to the same paragraph; When the probability value is not greater than a preset probability threshold, it is determined that the adjacent sentence pairs do not belong to the same paragraph.

[0010] In one embodiment, the step of performing layout detection on the document to be parsed to obtain element blocks includes: Convert the document to be parsed into a picture to be parsed; Determine the layout elements contained in the image to be parsed and the element positions of the layout elements by using the layout detection model; The image to be parsed is split according to the element positions to obtain the element blocks.

[0011] In one embodiment, the step of processing each element block according to each content extraction rule to obtain preliminary content of each element block includes: In the case where the layout element is a title, using optical character recognition technology to determine preliminary content and style features of each element block; Determining the title level of the element block according to the style feature; Before the step of obtaining the target document, the method further includes: According to the title level of the element block, the document display style in the element block is adjusted.

[0012] In one embodiment, the preliminary content includes image content and text content, and the step of processing the element block according to the content extraction rule to obtain the preliminary content of the element block includes: In the case where the layout element is a picture, determining the element block as the picture content; The step of correcting the preliminary content by using a large language model comprises: Based on the image content, the text content in the preliminary content is corrected using the large language model.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a document parsing device, which includes: A detection module is used to obtain a document to be parsed, perform layout detection on the document to be parsed, and obtain element blocks; A determination module, used to determine a content extraction rule corresponding to the element block according to the layout elements of the element block; An extraction module, used to process the element block according to the content extraction rule to obtain preliminary content of the element block; The correction module is used to correct the preliminary content by using a large language model to obtain a target document.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a document parsing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the document parsing method described above.

[0015] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the document parsing method described above are implemented.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the document parsing method described above are implemented.

[0017] One or more technical solutions proposed in the present application have at least the following technical effects: obtaining a document to be parsed, performing layout detection on the document to be parsed, obtaining element blocks, and identifying and distinguishing different content areas in the document through layout detection, which helps to more accurately extract and process information in the document and improve the accuracy and efficiency of content extraction; then, according to the layout elements of the element blocks, determining the content extraction rules corresponding to the element blocks, making the extraction process more customized and accurate, and improving the quality of content extraction; then, the element blocks can be processed according to the content extraction rules corresponding to the element blocks to obtain preliminary content of the element blocks, providing basic data for subsequent content correction and optimization, and ensuring the originality and integrity of the information; after obtaining the preliminary content, using a large language model to correct it, obtaining the target document of each element block, improving the accuracy and readability of the text content, thereby jointly improving the accuracy of document parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of the first embodiment of the document parsing method of the present application is provided; Figure 2 A schematic diagram of layout detection provided in Example 1 of the document parsing method of this application; Figure 3 A schematic diagram of content extraction provided in Example 1 of the document parsing method of this application; Figure 4 A flow chart of training and application of a segmented correction model provided in Example 1 of the document parsing method of this application; Figure 5 This is a schematic diagram of the module structure of the document parsing device according to an embodiment of the present application; Figure 6 A schematic diagram of the device structure of the hardware operating environment involved in the document parsing method in the embodiment of the present application. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] In this embodiment, for the convenience of description, the following description is made with the terminal as the execution subject.

[0024] Conventional technology is first responsible for identifying and accurately locating various elements in the document, including text, images, tables, etc., and then uses specially designed parsing strategies for each specific element for in-depth processing. However, although this method can achieve good results in most cases, there are still some problems and challenges. For example, OCR technology may have inaccurate recognition in actual applications, resulting in deviations or omissions of text information; in addition, errors may also occur in the text segmentation process, incorrectly cutting originally coherent paragraphs, or incorrectly merging texts from different paragraphs together. These problems will not only reduce the readability of the document, but also affect the accuracy of the parsing of the original document, resulting in inaccurate document parsing.

[0025] The present application provides a solution, which includes obtaining a document to be parsed, performing layout detection on the document to be parsed, obtaining element blocks, and identifying and distinguishing different content areas in the document, such as text, images, tables, etc., through layout detection, which helps to more accurately extract and process information in the document and improve the accuracy of content extraction; then, according to the layout elements of the element blocks, determining the content extraction rules corresponding to the element blocks, making the extraction process more customized and accurate, and improving the quality of content extraction; then, through the content extraction rules corresponding to the element blocks, processing the element blocks, and obtaining preliminary content of the element blocks, such as saving pictures and recording file information, identifying the style and content of titles, etc., providing basic data for subsequent content correction and optimization, and ensuring the originality and integrity of the information; after extracting the text content, a segmented correction model can be used to perform paragraph recognition on it, trying to maintain the logic and readability of the text content, and improving the quality of document parsing; then, using a large language model to correct the content of the element blocks, obtaining the target document of each element block, improving the accuracy and readability of the text content, and thus jointly improving the accuracy of document parsing.

[0026] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes the document parsing device as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0027] Based on this, the present application embodiment provides a document parsing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the document parsing method of the present application.

[0028] In this embodiment, the document parsing method includes steps S10 to S40: Step S10, obtaining a document to be parsed, performing layout detection on the document to be parsed, and obtaining element blocks; It should be noted that the document to be parsed refers to the digital document to be processed, analyzed or parsed, which can be a file in the format of PDF, Word, HTML, etc. The document can contain various types of data such as text, pictures, tables, etc.; the element block refers to the area or block divided according to the layout elements and their locations. Each element block contains only one layout element, such as a paragraph of text or a table.

[0029] In addition, it should be noted that layout detection refers to the use of algorithms and technical means to analyze the structure and layout of a document and identify the location areas and arrangement methods of different parts of the document. Common layout detection models include the PicoDet model and the LayoutParser model.

[0030] For example, please refer to Figure 2 , Figure 2 A layout detection schematic diagram is provided. For a PDF document containing a variety of different layout elements, layout detection is used to identify layout elements in the document to be parsed, including tables, pictures related to a large language model, a title "large language model", and an introduction text to the large language model. Then, element splitting is performed according to each layout element and its position to obtain table element blocks, picture element blocks, title element blocks, and text element blocks, so that different content extraction rules can be adopted for different layout elements in the future to perform targeted content parsing.

[0031] It is understandable that dividing the document into element blocks facilitates subsequent steps such as content extraction and text analysis.

[0032] Step S20, determining a content extraction rule corresponding to the element block according to the layout element of the element block; It should be noted that content extraction rules refer to algorithms or logical rules for extracting internal information or content based on the layout elements of element blocks (such as text format, image features, table format, etc.). For example, when the layout element is a table, a table structure recognition model, such as the TableAtten model, is used to determine the table structure and corresponding location information; the OCR model is used to identify the table content and record the corresponding location of the content.

[0033] Exemplarily, for a PDF document containing layout elements such as a title, body, and table, the extraction rule for the title element block can be determined as extracting plain text and converting English letters to uppercase; the extraction rule for the body element block is to extract text content and remove formatting; and the extraction rule for the table element block is to extract table data and store it by column.

[0034] It is understandable that different content extraction rules are provided for element blocks for different layout elements. This customized extraction rule helps to improve the accuracy of content extraction, thereby improving the accuracy of document parsing.

[0035] Step S30, processing the element block according to the content extraction rule to obtain the preliminary content of the element block; It should be noted that processing refers to applying content extraction rules to each element block to extract text information, image information, etc. therein.

[0036] For example, please refer to Figure 3 , Figure 3 A content extraction schematic diagram is provided. For the element blocks obtained after element splitting, content extraction is completed according to the content extraction rules corresponding to its layout elements. For element blocks whose layout elements are tables, the TableAtten model is mainly used to identify the table structure and the PPOCRv4 model is used to identify the table content; for element blocks whose layout elements are pictures, the pictures are saved and the file information is recorded. The file information includes the file name, file storage path, etc.; for element blocks whose layout elements are titles or texts, the PPOCRv4 model is directly used for content recognition to obtain the title content and text content. The output of the above models is the preliminary content of each element block. After obtaining the preliminary content, the results are aggregated, including using the BERT model to perform preliminary correction on the text content, using the Qwen large model to perform text correction on all preliminary content, layout sorting, and outputting the target document.

[0037] Step S40, using the large language model to correct the preliminary content and obtain the target document; It should be noted that the big language model is a natural language processing model based on deep learning technology. Through training with a large-scale corpus, the model can understand and generate human language. The model has powerful text generation, understanding and analysis capabilities and can handle complex language structures and semantic relationships.

[0038] In addition, it should be noted that the target document refers to the document that ultimately contains the content of all element blocks. The format of the target document can be determined according to user needs, such as a text format suitable for model training, Markdown format, etc. This implementation does not make specific limitations on this.

[0039] For example, a large language model is used to identify typos in text content, and candidate replacement words are generated for the typos based on the context; then, based on the contextual semantics of the typos, a target replacement word for the typos is determined from the candidate replacement words, and the target replacement word is used to replace the typos in the text content to complete the content correction. For example, the sentence "Please ensure the correctness of the algorithm" in the document to be parsed may be recognized by OCR technology as "clearly promote the correctness of the algorithm". In this regard, the large language model's ability to understand the context can be used to correct the text content, thereby obtaining the correct result.

[0040] For example, for table content, the column information in the table is the production month, and the production month is also mentioned in the previous text when introducing the table, but the table column name in the table recognition result is "production day". The large language model modifies the table column name by understanding the table column information and the previous text information, and changes "production day" to "production month". Similarly, for formula information in the document, there may be situations where the superscript and subscript recognition is unclear. The context understanding ability of the large model can also be used to understand the introduction information of the formula in the document, so as to modify the formula, or correct the relevant introduction content in the document based on the formula information.

[0041] Exemplarily, when there are multiple element blocks, the content layout in the target document can be adjusted according to the order of appearance of the element blocks to ensure that the layout is consistent with the original document, thereby enhancing the parsing accuracy and readability of the target document.

[0042] It can be understood that through the correction of the large language model, the accuracy and readability of the preliminary content are improved, thereby improving the accuracy of document parsing.

[0043] This embodiment provides a document parsing method, which performs layout detection on the document to be parsed, divides it into small element blocks, and then applies content extraction rules corresponding to the layout elements to each element block to ensure the accuracy of content recognition; then uses a large language model to further correct it, further improving the correctness of content extraction, thereby improving the accuracy of document parsing.

[0044] In a feasible implementation manner, before step S40, the method further includes: Step A10: When the layout element is text, the preliminary content of the element block is corrected in sections using the section correction model.

[0045] It should be noted that the segment correction model refers to an algorithm or program used to correct and optimize text paragraphs. Through a series of rules and algorithms, the model can identify and adjust paragraph boundaries in the text to ensure that the paragraph division of the text is more accurate and reasonable. Common segment correction models include correction models based on natural language processing, such as the BERT (Bidirectional Encoder Representations from Transformers) model, the Seq2Seq (sequence to sequence) model, and the joint model based on CRF (Conditional Random Field) and n-gram.

[0046] Exemplarily, the Transform model is used to segment the text content, and the self-attention mechanism of the model is used to capture the association between various parts of the text, thereby generating segmentation suggestions and dividing the original text into multiple paragraphs with coherent content.

[0047] In this embodiment, by introducing a segmentation correction model, the preliminary content of the text is intelligently segmented and corrected, the text structure is optimized, and the logic and coherence of the content are ensured, thereby effectively improving the accuracy and readability of the text content and improving the accuracy of document parsing.

[0048] In a feasible implementation manner, step A10 includes: Step A20, deleting line break marks in the preliminary content and extracting adjacent sentence pairs of preset symbols; It should be noted that the line break mark refers to a specific character or code used in the document format to indicate the beginning of a new line or a new paragraph. For example, in web page coding, it is , which may be a carriage return in a Word document. A preset symbol refers to a mark or symbol used to identify a specific position, structure, or content in a text. It can be a punctuation mark or a special character, etc., depending on the text content.

[0049] In addition, it should be noted that adjacent sentence pairs refer to two or more sentences that are connected in sequence in the text. They may belong to the same paragraph or may need to be divided into different paragraphs. The number of adjacent sentence pairs extracted can be 1, that is, 1 sentence is extracted forward and backward respectively. It can also be set according to the capabilities of the model. This embodiment does not make any specific limitations on this.

[0050] It can be understood that by selectively extracting text content as input to the segment correction model, unnecessary text processing operations can be reduced and the efficiency of text parsing can be improved.

[0051] Step A30, using the segmentation correction model to determine whether adjacent sentence pairs belong to the same paragraph; After the segmentation correction model receives pairs of adjacent sentences, it performs feature extraction, semantic analysis and other processing on the input text to identify the relationship and logical structure between the sentences. Based on the structural relationship, it outputs a judgment conclusion on whether each pair of adjacent sentences belongs to the same paragraph.

[0052] Step A40: When the adjacent sentence pairs do not belong to the same paragraph, a line break mark is added between the adjacent sentence pairs.

[0053] The segmentation correction model is used to analyze the text content of adjacent sentence pairs and automatically determine whether the adjacent sentence pairs belong to the same paragraph. If they do not belong to the same paragraph, appropriate line break marks are automatically added between the adjacent sentence pairs according to the format requirements of the target document to separate different paragraphs.

[0054] In this implementation, by accurately determining the attribution of paragraphs, the coherence and structural rationality of the document are enhanced, and the accuracy of document parsing is improved.

[0055] In a feasible implementation, the segment correction model includes a BERT model, and step A20 includes: Step A21, using the BERT model to extract semantic features of adjacent sentence pairs; It should be noted that the BERT model is a pre-trained language representation model based on the Transform architecture. This model can capture rich language features, including vocabulary, syntax, semantics and other levels, through unsupervised learning on a large-scale text corpus. Semantic features are used to reflect the meaning or information expressed by the text, including the semantics of vocabulary, the semantic relations of sentences, the contextual information of the text, etc.

[0056] It is understandable that the fine-tuning cost of using the BERT model is relatively small, and compared with the generative large language model, it has a clear advantage in the speed of paragraph recognition.

[0057] Step A22, mapping the semantic features into probability values, wherein the probability value is the probability that the adjacent sentence pairs belong to the same paragraph; It should be noted that mapping refers to inputting the extracted semantic features into a certain algorithm or model through the algorithm or model, and the algorithm or model outputs a probability value that identifies the possibility that adjacent sentence pairs belong to the same paragraph.

[0058] Exemplarily, MLP (MultiLayer Perceptron) is used as the classification head of the paragraph correction model, the features extracted by the BERT model are converted into a real value, and then an activation function (such as a sigmoid function) is used to map the real value into a probability, and based on the probability value, it is determined whether adjacent sentence pairs belong to the same paragraph.

[0059] It can be understood that by mapping semantic features into probability values, it is possible to more accurately determine whether adjacent sentence pairs belong to the same paragraph based on the logical relationship between the adjacent sentence pairs, thereby improving the accuracy of document parsing.

[0060] Step A23, when the probability value is greater than a preset probability threshold, determining that the adjacent sentence pair belongs to the same paragraph; Step A24: When the probability value is not greater than the preset probability threshold, it is determined that the adjacent sentence pairs do not belong to the same paragraph.

[0061] For example, please refer to Figure 4 , Figure 4 The invention provides a training and application process of a segmented correction model based on a BERT model. The model training process includes: firstly obtaining a standard document, extracting text content from the standard document to construct a training data set, adding a label "0" to sentences from different paragraphs, and adding a label "1" to sentences from the same paragraph; then using the BERT model to extract features of the sentences in the training data set to obtain a feature vector of each pair of sentences, and mapping the feature vector to a prediction result through a classification head, that is, predicting whether the pair of sentences belong to the same paragraph; then reading the label content, performing loss calculation and gradient return, and calculating the error by comparing the prediction result with the label and using a binary cross entropy loss function to guide the optimization direction of the classification head, and returning the error to each parameter in the model layer by layer through a gradient return algorithm, and calculating the gradient of the parameter at the same time. These gradients are used to update the model parameters to minimize the loss function value, and the model training ends when the loss function value tends to be stable or reaches a preset number of times.

[0062] Figure 4 The application process of the model is also provided, including: obtaining preliminary content obtained after content extraction of element blocks, extracting the text before and after the line break (preset symbol) (adjacent sentence pairs), using the BERT model to perform feature extraction, obtaining feature vectors for each pair of adjacent sentences, using MLP as the classification head to map the feature vectors into probability values, and when the probability value is greater than a preset probability threshold, determining that the adjacent sentence pairs belong to the same paragraph; when the probability value is not greater than the preset probability threshold, determining that the adjacent sentence pairs do not belong to the same paragraph, thereby obtaining a prediction result.

[0063] In this implementation, the accuracy of paragraph judgment is improved by using the deep semantic features extracted by the BERT model; the readability of the document content is improved through reasonable text segmentation, and the accuracy of document parsing is improved.

[0064] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, the step of performing layout detection on the document to be parsed in step S10 to obtain element blocks includes: Step S11, converting the document to be parsed into a picture to be parsed; It should be noted that the image to be parsed refers to an image file that requires layout detection and element extraction. It may be a scanned paper document, a picture taken, or a screenshot of a web page with a complex layout. The image to be parsed contains various layout elements. The position and arrangement of these elements constitute the layout structure of the image.

[0065] Step S12, using the layout detection model to determine the layout elements and element positions of the layout elements contained in the image to be analyzed; It should be noted that the layout detection model can identify and locate different layout elements in an image by analyzing the visual features of the image; the element position refers to the coordinate information of the layout element in the image to be parsed, which describes the specific position of the element in the image. The specific position information provides a basis for subsequent splitting operations.

[0066] Existing layout detection models are basically visual models, so the document to be parsed needs to be converted into an image first, and then the image is used as the input of the layout detection model to output the location information of the layout elements.

[0067] Step S13, splitting the image to be parsed according to the element positions to obtain element blocks.

[0068] According to the given coordinate position, the image to be parsed is divided into one or more image fragments (element blocks) containing independent layout elements.

[0069] Exemplarily, after the layout detection is completed, it is identified that the upper left corner of the element position of the table is at (50, 30) and the lower right corner is at (250, 80), so the split layout element is that the element block of the table is the area from the (50, 30) position to the (250, 80) position of the image.

[0070] In this embodiment, by accurately positioning and splitting elements, interference between different layout elements is reduced, and the accuracy of subsequent text content extraction is improved, thereby improving the accuracy of document parsing.

[0071] In a feasible implementation manner, step S30 includes: Step B10, when the layout element is a title, using optical character recognition technology to determine preliminary content and style features of the element block; It should be noted that OCR technology refers to a series of image processing technologies that convert text in an image into an editable and searchable text format; style features refer to the visual attributes of text such as font, size, and thickness.

[0072] Step B20, determining the title level of the element block according to the style features; It should be noted that the title level refers to the hierarchical relationship between different titles in a document, which is usually distinguished by different fonts, sizes, bold, etc. For example, the first-level title may represent the chapter name of the document, and the second-level title represents the subsection name under the chapter.

[0073] For example, an element block with style features of size 2 font and boldness may be determined as a first-level title.

[0074] Understandably, determining the heading level through OCR technology helps to better understand the importance and hierarchy of the text content.

[0075] Before obtaining the target document in step S40, the method further includes: Step B30, adjusting the document display style of the element block according to the title level of the element block.

[0076] It should be noted that the document display style refers to the corresponding presentation form of the layout elements in the target document, including visual attributes such as font and size. By adjusting the document display style, different layout elements can be distinguished and the key content in the document can be highlighted.

[0077] For example, in the target document, a series of title levels and corresponding document display styles are predefined. Before writing the title text content into the target document, the corresponding document display style needs to be determined first, so as to facilitate adjustment of the title style.

[0078] In this implementation, by displaying titles of different levels separately, the readability of the document is enhanced; applying corresponding display style rules according to the title level of the element block is also a restoration of the original document to be parsed, thereby improving the accuracy of document parsing.

[0079] In a feasible implementation manner, the preliminary content includes picture content and text content, and step S30 includes: Step E10, when the layout element is a picture, determining the element block as picture content; It should be noted that image content refers to the image itself contained in the document to be parsed, which is usually a combination of visual elements such as text, graphics, and symbols. In the process of document processing or information extraction, image content is usually regarded as an important source of information. Text content refers to all text content (except pictures), including titles, text, formulas, and tables. Since the large language model can actually understand the structure and content of the table, it can understand and correct all text content except pictures.

[0080] In step S40, the step of correcting the preliminary content using the large language model includes: Step E20, based on the image content, using a large language model, correct the text content in the preliminary content.

[0081] Exemplarily, a document to be parsed containing a layout element of a picture is obtained, and a layout detection is performed on it to obtain multiple element blocks; then, each element block is processed using a content extraction rule corresponding to the layout element in the element block, and the preliminary content obtained after processing for an element block whose layout element is a picture is the picture itself (picture content), and the preliminary content obtained after processing for an element block whose layout element is not a picture is text content; then, a multimodal large language model with image recognition capability is used to learn and understand the semantics, context and other information in the picture content and the text content, and typos in the text content are corrected to improve the accuracy and readability of the text content; finally, the processed content is presented to the user in the original order for further reading and analysis by the user.

[0082] In this implementation, by retaining the image content, effective reference information is provided for subsequent content correction; at the same time, based on the image recognition ability and context understanding ability of the multimodal large language model, the accuracy of the text content is improved, thereby improving the accuracy of document parsing.

[0083] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the document parsing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0084] The present application also provides a document parsing device. Figure 5 , the document parsing device comprises: A detection module 10 is used to obtain a document to be parsed, perform layout detection on the document to be parsed, and obtain element blocks; A determination module 20, configured to determine a content extraction rule corresponding to the element block according to the layout elements of the element block; An extraction module 30, used to process the element block according to the content extraction rule to obtain the preliminary content of the element block; The correction module 40 uses the large language model to correct the preliminary content and obtain the target document.

[0085] The document parsing device provided in the embodiment of the present application adopts the document parsing method in the above embodiment, which can solve the technical problem of inaccurate document parsing. Compared with the prior art, the beneficial effects of the document parsing device provided in the present application are the same as the beneficial effects of the document parsing method provided in the above embodiment, and other technical features in the document parsing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0086] An embodiment of the present application provides a document parsing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the document parsing method in the above-mentioned embodiment one.

[0087] Reference below Figure 6 , which shows a schematic diagram of the structure of a document parsing device suitable for implementing the embodiment of the present application. The document parsing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The document parsing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0088] like Figure 6As shown, the document parsing device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the document parsing device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the document parsing device to communicate with other devices wirelessly or by wire to exchange data. Although the document parsing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0089] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0090] The document parsing device provided in the embodiment of the present application adopts the document parsing method in the above embodiment, which can solve the technical problem of inaccurate document parsing. Compared with the prior art, the beneficial effects of the document parsing device provided in the present application are the same as the beneficial effects of the document parsing method provided in the above embodiment, and other technical features in the document parsing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0091] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0092] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0093] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the document parsing method in the above embodiment.

[0094] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0095] The computer-readable storage medium may be included in the document parsing device; or may exist independently without being assembled into the document parsing device.

[0096] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the document parsing device, the document parsing device: obtains the document to be parsed, performs layout detection on the document to be parsed, and obtains element blocks; determines content extraction rules corresponding to the element blocks according to the layout elements of the element blocks; processes the element blocks according to the content extraction rules to obtain preliminary content of the element blocks; and uses a large language model to correct the preliminary content to obtain a target document.

[0097] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0100] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned document parsing method, and can solve the technical problem of inaccurate document parsing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the document parsing method provided in the above-mentioned embodiment, and will not be elaborated here.

[0101] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the document parsing method as described above when executed by a processor.

[0102] The computer program product provided in the embodiment of the present application can solve the technical problem of inaccurate document parsing. Compared with the prior art, the beneficial effects of the computer program product provided in the present application are the same as the beneficial effects of the document parsing method provided in the above embodiment, which will not be repeated here.

[0103] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A document parsing method, characterized in that: The document parsing method comprises: Obtaining a document to be parsed, performing layout detection on the document to be parsed, and obtaining element blocks; Determining a content extraction rule corresponding to the element block according to the layout elements of the element block; Processing the element block according to the content extraction rule to obtain preliminary content of the element block; The preliminary content is corrected using a large language model to obtain a target document.

2. The document parsing method according to claim 1, characterized in that: Before the step of using the large language model to correct each of the preliminary contents, the method further includes: In the case where the layout element is text, a segmented correction model is used to perform segmented correction on the preliminary content of the element block.

3. The document parsing method according to claim 2, characterized in that: The step of performing segmented correction on the preliminary content of the element block by using the segmented correction model comprises: Deleting line break marks in the preliminary content and extracting adjacent sentence pairs of preset symbols; Using the segmentation correction model, determining whether the adjacent sentence pairs belong to the same paragraph; In the case where the adjacent sentence pairs do not belong to the same paragraph, a line break mark is added between the adjacent sentence pairs.

4. The document parsing method according to claim 3, characterized in that: The segment correction model includes a bidirectional encoder representation BERT model from a converter, and the step of using the segment correction model to determine whether the adjacent sentence pairs belong to the same paragraph includes: Using the BERT model, extracting semantic features of the adjacent sentence pairs; Mapping the semantic features into probability values, wherein the probability values ​​are probabilities that the adjacent sentence pairs belong to the same paragraph; When the probability value is greater than a preset probability threshold, determining that the adjacent sentence pairs belong to the same paragraph; When the probability value is not greater than a preset probability threshold, it is determined that the adjacent sentence pairs do not belong to the same paragraph.

5. The document parsing method according to claim 1, characterized in that: The step of performing layout detection on the document to be parsed to obtain element blocks comprises: Convert the document to be parsed into a picture to be parsed; Determine the layout elements contained in the image to be parsed and the element positions of the layout elements by using the layout detection model; The image to be parsed is split according to the element positions to obtain the element blocks.

6. The document parsing method according to claim 1, characterized in that: The step of processing the element block by using the content extraction rule to obtain the preliminary content of the element block comprises: In the case where the layout element is a title, using optical character recognition technology to determine preliminary content and style features of the element block; Determining the title level of the element block according to the style feature; Before the step of obtaining the target document, the method further includes: According to the title level of the element block, the document display style of the element block is adjusted.

7. The document parsing method according to claim 1, characterized in that: The preliminary content includes picture content and text content. The step of processing the element block according to the content extraction rule to obtain the preliminary content of the element block includes: In the case where the layout element is a picture, determining the element block as the picture content; The step of correcting the preliminary content by using a large language model comprises: Based on the image content, the text content in the preliminary content is corrected using the large language model.

8. A document parsing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the document parsing method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the document parsing method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the document parsing method according to any one of claims 1 to 7 are implemented.

Images