Document processing method and device, equipment and storage medium

Abstract

The invention discloses a document processing method and device, equipment and a storage medium, and relates to the technical field of data processing, the disclosed document processing method comprises the following steps: obtaining a to-be-processed document, and identifying a text region and an image region in the to-be-processed document; performing text extraction on the text area to obtain vector text content; performing character recognition on the image area through a plurality of optical character recognition models and obtaining a recognition result; determining a confidence coefficient weight in the recognition result according to the character recognition result corresponding to each optical character recognition model; fusing each character recognition result with the vector text content according to each confidence coefficient weight to obtain a fused text content; and outputting the structured data or the semi-structured data according to the fused text content, thereby being compatible with the input of the mixed type document, realizing the processing of the mixed type document containing the text region and the image region, and effectively meeting the user requirements.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a document processing method, apparatus, device and storage medium. Background Art

[0002] Normative documents such as standards, specifications, laws, and regulations are crucial resources used to guide and regulate behavior across various industries. These documents are typically published in electronic formats such as Portable Document Format (PDF) and Word, and some are even stored as scanned copies (i.e., PDF or images in image format). Existing standards and regulatory documents are mostly in unstructured formats, making it difficult for computers to directly understand their hierarchical structure and semantic content. For example, a standards document may contain multiple levels of headings and clauses, but if it exists only in plain text, retrieving specific clauses or comparing the contents of different documents is inefficient. Efficient retrieval, comparison, and analysis of document content can only be achieved by converting unstructured documents into structured or semi-structured formats through data processing.

[0003] Currently, when converting standards and regulatory documents into structured or semi-structured formats, single strategies are often used to extract text, such as relying solely on direct text extraction or optical character recognition (OCR) models. However, for documents containing both text regions (i.e., extractable text) and image regions (i.e., scanned images), relying solely on direct text extraction is unable to process pure scans, while relying solely on OCR models can lead to unnecessary time overhead and potential recognition errors for text-heavy PDFs. Consequently, existing strategies typically target a single document type and are incompatible with mixed document input, thus failing to meet user needs.

[0004] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a document processing method, device, equipment and storage medium, aiming to solve the technical problem that the existing technology is not compatible with mixed types of document input and thus cannot meet user needs.

[0006] To achieve the above objectives, the present application proposes a document processing method, which includes:

[0007] Acquire a document to be processed, and identify a text area and an image area in the document to be processed;

[0008] Performing text extraction on the text area to obtain vector text content;

[0009] Performing character recognition on the image area using multiple optical character recognition models and obtaining recognition results;

[0010] Determining a confidence weight in the recognition result according to the character recognition result corresponding to each of the optical character recognition models;

[0011] fusing the character recognition results with the vector text content according to the confidence weights to obtain fused text content;

[0012] Output structured data or semi-structured data according to the fused text content.

[0013] In one embodiment, the step of fusing the character recognition results with the vector text content according to the confidence weights to obtain fused text content includes:

[0014] Comparing each of the character recognition results with the vector text content;

[0015] When the character recognition results are inconsistent with the vector text content, fusing the character recognition results according to the confidence weights to obtain the first scanned text content;

[0016] The first scanned text content is fused with the vector text content to obtain fused text content.

[0017] In one embodiment, when each of the character recognition results is inconsistent with the vector text content, the step of fusing the character recognition results according to the confidence weights to obtain the first scanned text content includes:

[0018] When each of the character recognition results is inconsistent with the vector text content, determining the confidence weights of different character recognition results at any position in the image area;

[0019] Selecting the character recognition result with the highest confidence weight as the recognized character at any position;

[0020] The recognized characters at various positions in the image region are merged to obtain first scanned text content.

[0021] In one embodiment, after the step of comparing each of the character recognition results with the vector text content, the method further includes:

[0022] When there is a target character recognition result that is consistent with the vector text content in each of the character recognition results, determining each other character recognition result other than the target character recognition result in each of the character recognition results;

[0023] fusing the other character recognition results according to their confidence weights to obtain a second scanned text content;

[0024] The second scanned text content is fused with the vector text content to obtain fused text content.

[0025] In one embodiment, the step of outputting structured data or semi-structured data according to the fused text content includes:

[0026] Determining hierarchical structure information of the document to be processed;

[0027] Constructing a document tree structure model according to the hierarchical structure information;

[0028] Generating a standard document structure model according to the document tree structure model and the fused text content;

[0029] The structured data or semi-structured data is output in a preset format based on the standard file structuring model.

[0030] In one embodiment, the step of constructing a document tree structure model based on the hierarchical structure information includes:

[0031] Determining hierarchical information in the hierarchical structure information;

[0032] Obtaining font information and typesetting information of the document to be processed;

[0033] A document tree structure model is constructed according to the hierarchical information, font information and typesetting information.

[0034] In one embodiment, the step of generating a standard document structure model based on the document tree structure model and the fused text content includes:

[0035] Identifying the ordered list data of the documents to be processed;

[0036] When table structure information exists in the document to be processed, converting the table structure into nested cell structure information;

[0037] When formula data exists in the document to be processed, converting the formula data into a preset expression;

[0038] Inserting the ordered list data, the nested cell structure information, and the preset expression into the document tree structure model according to the layout position information of the document to be processed to obtain an initial document structure model;

[0039] The fused text content is filled into the corresponding content of the initial document structured model to generate a standard document structured model.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a document processing device, wherein the document processing includes:

[0041] A text input module, configured to obtain a document to be processed and identify text areas and image areas in the document to be processed;

[0042] A text extraction module, configured to extract text from the text area to obtain vector text content;

[0043] an optical character recognition module, configured to perform character recognition on the image area using multiple optical character recognition models and obtain recognition results;

[0044] a confidence fusion module, configured to determine confidence weights in the recognition results according to the character recognition results corresponding to the optical character recognition models;

[0045] The confidence fusion module is further configured to fuse the character recognition results with the vector text content according to the confidence weights to obtain fused text content;

[0046] A data output module is used to output structured data or semi-structured data according to the fused text content.

[0047] In addition, to achieve the above-mentioned purpose, the present application also proposes a document processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the document processing method described above.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the document processing method described above are implemented.

[0049] One or more technical solutions proposed in this application have at least the following technical effects:

[0050] This application obtains a document to be processed and identifies the text area and image area in the document to be processed; performs text extraction on the text area to obtain vector text content; performs character recognition on the image area through multiple optical character recognition models and obtains recognition results; determines confidence weights in the recognition results according to the character recognition results corresponding to each optical character recognition model; fuses each character recognition result with the vector text content according to each confidence weight to obtain fused text content; and outputs structured data or semi-structured data according to the fused text content. This application extracts text from the text area by text extraction, performs character recognition on the image area through multiple optical character recognition models, and then fuses each character recognition result with the vector text content according to the confidence weight, so that it can be compatible with mixed-type document input and realizes the processing of mixed-type documents containing text areas and image areas, effectively meeting user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 This is a flowchart of the first embodiment of the document processing method of this application;

[0054] Figure 2 This is a flowchart of the second embodiment of the document processing method of this application;

[0055] Figure 3 This is a flowchart of the third embodiment of the document processing method of this application;

[0056] Figure 4 This is a schematic diagram of the module structure of the document processing device of this application;

[0057] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the document processing method in the embodiment of the present application.

[0058] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0060] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0061] The main solution of the embodiment of the present application is: obtaining a document to be processed and identifying text areas and image areas in the document to be processed; performing text extraction on the text area to obtain vector text content; performing character recognition on the image area through multiple optical character recognition models and obtaining recognition results; determining confidence weights in the recognition results according to the character recognition results corresponding to each optical character recognition model; fusing each character recognition result with the vector text content according to each confidence weight to obtain fused text content; and outputting structured data or semi-structured data according to the fused text content.

[0062] Since existing technologies are usually targeted at a single type of document, they are not compatible with mixed-type document input and thus cannot meet user needs.

[0063] This application provides a solution that extracts text from text areas through text extraction, performs character recognition on image areas through multiple optical character recognition models, and then fuses each character recognition result with the vector text content based on the confidence weight, thereby being compatible with mixed-type document input and realizing the processing of mixed-type documents containing text areas and image areas, effectively meeting user needs.

[0064] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or document processing device capable of implementing the above functions. This embodiment and the following embodiments will be described below using a document processing device as an example.

[0065] Based on this, the embodiment of the present application provides a document processing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the document processing method of this application.

[0066] In this embodiment, the document processing method includes steps S10 to S60:

[0067] Step S10: obtaining a document to be processed, and identifying a text area and an image area in the document to be processed.

[0068] It should be noted that the aforementioned document to be processed can be an initial document that has undergone preprocessing. This initial document can be a normative document to be processed, such as a standard document, a regulatory document, or a specification document. This initial document can contain text areas and image areas, and supports a variety of file formats, such as PDF, Word documents, and scanned images. The text area is vector text from which text can be directly extracted, and the image area can be a raster image containing an image that requires OCR model recognition.

[0069] In a specific implementation, the above-mentioned document processing device can obtain the initial document to be processed, which supports multiple file formats, and then preprocess the initial document to obtain the document to be processed. In the preprocessing stage, the initial document can be format converted and segmented. For example, the PDF or Word in the initial document can be split into units for page processing, and the pictures embedded in the PDF can be extracted and processed separately. For the image part in the initial document, image preprocessing such as denoising, binarization, and tilt correction can be performed to improve the recognition rate of the subsequent OCR model.

[0070] Step S20: extract text from the text area to obtain vector text content.

[0071] In a specific implementation, the document processing device may determine whether each page or each part of the document to be processed is a text area or an image area by detecting the existence of a text layer in the page algorithm of the document to be processed or by pixel features.

[0072] Exemplarily, when the initial document contains PDF content, the PDF parsing library can be used to detect whether there is a text object. If so, the area where the text object is located is determined to be a text area. For the entire initial document, it can be judged based on file metadata (such as information such as the document's author, creation date, modification date, subject, etc.) or information provided by the user (such as the user can enter corresponding information before document processing to specify or mark certain parts of the document as text or images). If extractable text is detected, the part containing the extractable text will be marked as a text area. If a part containing an image is detected, the part containing the image will be marked as an image area.

[0073] In addition, the above-mentioned document processing device can directly perform text extraction on the identified text area to obtain vector text content. Specifically, the above-mentioned document processing device can also use PDF or Text parsing tools to read the text, paragraphs, font style and position information in the text area, and can preliminarily identify elements such as titles, text, headers, footers, and footnotes by analyzing the font size, thickness and layout position. The obtained text part can be the vector text content. The above strategy is also adopted for formats such as Word to obtain the corresponding vector text content and style information. This process is a lossless extraction, ensuring that the content obtained from the text area is accurate.

[0074] Step S30: performing character recognition on the image area using multiple optical character recognition models and obtaining a recognition result.

[0075] In a specific implementation, the document processing device may integrate at least two deep learning-based OCR models to improve the recognition coverage and accuracy of the initial document. The device may call upon each OCR model to perform character recognition on an image region, with the resulting recognition results including the character recognition results corresponding to each OCR model.

[0076] Step S40: Recognize the character corresponding to each of the optical character recognition models in the recognition results.

[0077] It should be noted that a large number of known text images and their corresponding correct text can be used as training data to train the OCR model. During training, the difference between the OCR model's recognition result and the correct result for each character can be recorded, and the number of correct and incorrect recognitions can be counted. For each character, an initial confidence weight is calculated based on the ratio of the number of correct recognitions by the model to the total number of recognitions in the training data.

[0078] In a specific implementation, when the above-mentioned document processing device performs character recognition on an image area through each OCR model, each OCR model can analyze image features, such as clarity, contrast, noise level, etc., and output corresponding confidence weights according to the parameter sizes corresponding to the image features. For example, the recognition result with high clarity has a higher confidence weight. Based on this, the confidence weights corresponding to each character recognition result can be obtained.

[0079] Step S50 , fusing the character recognition results with the vector text content according to the confidence weights to obtain fused text content.

[0080] In a specific implementation, the character recognition results corresponding to the same character have confidence weights corresponding to different OCR models. The above-mentioned document processing device can select the character recognition result with the highest confidence weight as the final character recognition result of the same character, and re-combine the final character recognition results corresponding to the selected characters according to the position information of each part in the image area, and then fuse the combined character recognition results with the vector text content according to the position information of each part of the document to be processed to obtain the fused text content.

[0081] In addition, the document processing device can perform spelling and format consistency checks on the fused text content, for example, verifying the continuity and correctness of the standard or regulatory clause numbers. If it is detected that the recognized clause numbers are discontinuous, it is determined that a certain position in the image area is incorrectly recognized and the position is marked for further processing.

[0082] Step S60: outputting structured data or semi-structured data according to the fused text content.

[0083] In a specific implementation, the above-mentioned document processing device can perform structural analysis on the fused text content, extract the hierarchical structure and format elements of the document to be processed, restore the document structure of the document to be processed based on the hierarchical structure and format elements, and then fill the fused text content into the restored document structure, and output structured data or semi-structured data according to user needs.

[0084] This embodiment obtains a document to be processed and identifies the text area and image area in the document to be processed; performs text extraction on the text area to obtain vector text content; performs character recognition on the image area through multiple optical character recognition models and obtains character recognition results; determines confidence weights in the recognition results according to the character recognition results corresponding to each optical character recognition model; fuses each character recognition result with the vector text content according to each confidence weight to obtain fused text content; and outputs structured data or semi-structured data according to the fused text content. This embodiment extracts text from the text area by text extraction, performs character recognition on the image area through multiple optical character recognition models, and then fuses each character recognition result with the vector text content according to the confidence weight, so that it is compatible with mixed-type document input and realizes the processing of mixed-type documents containing text areas and image areas, effectively meeting user needs.

[0085] Based on the first embodiment of the present application, the second embodiment of the present application is proposed. In the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction and will not be repeated hereafter. Figure 2 , Figure 2 This is a flowchart of the second embodiment of the document processing method of this application.

[0086] In this embodiment, step S50 includes steps S51 to S53:

[0087] Step S51 : comparing each of the character recognition results with the vector text content.

[0088] Step S52: When the character recognition results are inconsistent with the vector text content, the character recognition results are fused according to the confidence weights to obtain the first scanned text content.

[0089] In a specific implementation, the document processing device may compare each character recognition result with the vector text content, that is, determine whether each character recognition result is consistent with the character corresponding to the vector text content. When each character recognition result is inconsistent with the vector text content, the document processing device determines that the text obtained by the OCR model is inconsistent with the directly extracted text. In this case, the character recognition results may be selected and merged based on the confidence weight of each character recognition result, that is, the character recognition result with the highest confidence weight is selected for the same character, and then the selected character recognition results are merged based on the position information of the original character in the image area, thereby obtaining the first scanned text content.

[0090] In a feasible implementation, step S52 includes steps S521 to S523:

[0091] Step S521 : When all the character recognition results are inconsistent with the vector text content, determine the confidence weights of different character recognition results at any position in the image area.

[0092] In a specific implementation, when the character recognition results of the above-mentioned document processing device are inconsistent with the vector text content, the confidence weight of the character at any position in the image area can be obtained, and the process can be repeated to obtain the confidence weight of the character recognition results of the characters at each position in the image area.

[0093] Step S522: Select the character recognition result with the highest confidence weight as the recognized character at any position.

[0094] In a specific implementation, for any character, there are character recognition results of various OCR models and corresponding confidence weights. If there are at least two character recognition results with the same confidence weight, any character recognition result can be randomly selected as the final character recognition result of the character, and the final character recognition result is the outputtable recognized character; for any character with different confidence weights of the character recognition results, the character recognition result with the highest confidence weight can be selected as the recognized character of the character. The above-mentioned document processing device can obtain the recognized characters of the image area based on the above-mentioned strategy. For example, taking Engine A model and Engine B model as examples of different OCR models, Engine A model recognizes a word as "Clause" with a confidence of 80%; Engine B recognizes it as "Clausee" with a confidence of 75%, then the result of Engine A is taken.

[0095] Furthermore, in Chinese scenarios, the system can compare characters by word or line, selecting the most reliable result character by character. Furthermore, character recognition results with confidence weights lower than a preset weight can be corrected using a language model or by using a dictionary of specialized terms to infer the correct character.

[0096] Step S523: Merge the recognized characters at various positions in the image area to obtain a first scanned text content.

[0097] In a specific implementation, the above-mentioned document processing device can obtain character recognition results through various OCR models while also identifying the position information of each part in the image area (such as text box coordinates). Then, the position of each recognized character in the original image area can be determined through the position information, and the recognized characters can be merged again based on the position to obtain the first scanned text content.

[0098] In addition, for table-like image areas, relevant table recognition algorithms can also be used to extract cell content and change row and column structures.

[0099] Step S53: Fusing the first scanned text content with the vector text content to obtain fused text content.

[0100] In a specific implementation, the document processing device may identify position information of each portion of the document to be processed, and then fuse the first scanned text content with the vector text content according to the position information of each portion of the document to be processed to obtain fused text content.

[0101] In this embodiment, step S51 is followed by steps S54 to S56:

[0102] Step S54 : when there is a target character recognition result that is consistent with the vector text content in each of the character recognition results, determining each other character recognition result other than the target character recognition result in each of the character recognition results.

[0103] In a specific implementation, when the above-mentioned document processing device detects the existence of a target character recognition result consistent with the vector text content in each character recognition result, it can determine each other character recognition result other than the target character recognition result in each character recognition result, thereby dividing the character recognition structure into the target character recognition structure and each other character recognition structure.

[0104] Step S55 , fusing the other character recognition results according to the confidence weights of the other character recognition results to obtain the second scanned text content.

[0105] In a specific implementation, in order to avoid text duplication, the above-mentioned document processing device may delete the target character device structure in each character recognition structure, retain the other character recognition structures, and then fuse the other character recognition structures according to the confidence weights of the other character recognitions. The specific fusion process may be consistent with the steps of fusing the above-mentioned character recognition results according to the confidence weights to obtain the first scanned text content, that is, for other character recognition structures at the same position in the image area, the character recognition result with the highest confidence weight may be selected as the other recognized character at that position, and the other recognized characters at each position in the image area may be merged to obtain the second scanned text content.

[0106] Step S56: Fusing the second scanned text content with the vector text content to obtain fused text content.

[0107] In a specific implementation, the document processing device may fuse the second scanned text content with the vector text content according to the position information of each part of the document to be processed to obtain fused text content.

[0108] This embodiment compares each character recognition result with the vector text content; when each character recognition result is inconsistent with the vector text content, the character recognition results are fused according to each confidence weight to obtain the first scanned text content; the first scanned text content is fused with the vector text content to obtain the fused text content; when there is a target character recognition result that is consistent with the vector text content in each character recognition result, each other character recognition result other than the target character recognition result in each character recognition result is determined; each other character recognition result is fused according to the confidence weight of each other character recognition result to obtain the second scanned text content; the second scanned text content is fused with the vector text content to obtain the fused text content. This embodiment fuses each character recognition result through confidence weight, which not only fully utilizes the accuracy of the original digital text, but also compensates for the possible omission or misrecognition of the OCR model, ensuring that all content is extracted and as accurate and reliable as possible, effectively reducing the impact of the OCR model's misrecognition on the final result, and improving the accuracy of text recognition.

[0109] Based on the first and second embodiments of the present application, the third embodiment of the present application is proposed. In the third embodiment of the present application, the same or similar contents as those of the first and second embodiments can be referred to above and will not be described in detail later. Figure 3 , Figure 3 This is a flowchart of the third embodiment of the document processing method of this application.

[0110] In this embodiment, step S60 includes steps S61 to S64:

[0111] Step S61: Determine the hierarchical structure information of the document to be processed.

[0112] It should be noted that the above-mentioned hierarchical information can represent the organization and arrangement of the document content of the document to be processed, including the layout hierarchy and content hierarchy. The layout hierarchy includes pages, chapters, etc., and the content hierarchy includes titles, clause numbers, paragraph levels, etc.

[0113] In a specific implementation, the document processing device can identify hierarchical structure information composed of chapter titles, section titles, clause numbers, paragraph levels, etc. according to pre-set normative document formatting rules. For example, when detecting the presence of numbers such as "1., 1.1, 1.1.1" or "Chapter 1" or "Article 1" in Chinese in a document to be processed, the device can divide the document into chapters, sections, clauses, and paragraphs based on these numbers.

[0114] Step S62: constructing a document tree structure model according to the hierarchical structure information.

[0115] In a specific implementation, the document processing device may determine the starting and ending positions of each level based on the hierarchical structure information, and convert the document to be processed into a tree-like hierarchical structure model based on the starting and ending positions of each level, thereby obtaining a document tree structure model. The document tree structure model reflects the inclusion and subordination relationships between different elements in the document to be processed.

[0116] In a feasible implementation, step S62 includes steps S621 to S623:

[0117] Step S621: Determine the level information in the hierarchical structure information.

[0118] In a specific implementation, the document processing device may extract elements such as titles, chapters, paragraphs, lists, etc. from the hierarchical structure information, and determine the hierarchical information according to the subordinate relationships of the elements.

[0119] Step S622: Obtain the font information and typesetting information of the document to be processed.

[0120] In a specific implementation, the above-mentioned document processing device can obtain the font size, thickness, and style information of each text element in the document to be processed during text extraction of the text area or character recognition through the OCR model, and use the font size, thickness, and style information of each text element as font information, and can perform page layout analysis on the document to be processed to determine the page layout of the page and the positional relationship of the text elements in the page. The page layout and positional relationship can be used as typesetting information.

[0121] Step S623: constructing a document tree structure model according to the hierarchical information, font information, and typesetting information.

[0122] In a specific implementation, the document processing device can determine, based on hierarchical information, a root node (e.g., the entire document), branch nodes at each level (e.g., chapters, sub-chapters, etc.), and each branch node including its subordinate child nodes, thereby forming an initial tree structure based on each node. Furthermore, the initial tree structure can be optimized using font and layout information, such as by adding attributes to the title node, to obtain a document tree structure model that more accurately reflects the actual structure and style information of the document to be processed.

[0123] Step S63: Generate a standard document structure model based on the document tree structure model and the fused text content.

[0124] In a specific implementation, the document processing device may fill the fused text content into the corresponding nodes of the document tree structure model to generate a standard document structured model.

[0125] In a feasible implementation, step S63 includes:

[0126] Step S631 : identifying data with a clear sequence relationship in the document to be processed, which may appear in the document to be processed as list elements such as numbers and letters.

[0127] In a specific implementation, the above-mentioned document processing device can identify list elements such as bulleted lists and numbered lists that appear in the document to be processed, and retain the sequential structure of the list elements, such as the sub-items a), b), c) or (i), (ii), etc. that appear under the legal provisions, and then convert the list elements into ordered list data.

[0128] Step S632: When table structure information exists in the document to be processed, convert the table structure into nested cell structure information.

[0129] In a specific implementation, when the above-mentioned document processing device detects the presence of a table structure in the document to be processed during text parsing or OCR recognition, it can reconstruct the table structure according to rows and columns, represent the table as nested cell structure information, and represent the table as other formats that can reflect the relationship between rows and columns, so that the table content and its structure are clear and usable.

[0130] Step S633: When there is formula data in the document to be processed, convert the formula data into a preset expression.

[0131] In a specific implementation, when the above-mentioned document processing device detects the presence of formula data in the document to be processed during text parsing or OCR recognition, it can convert the formula data into a corresponding preset expression, such as a LaTex expression, based on pre-set rules, and can record the formula number, the location where the formula appears, the meaning and unit of each symbol in the formula, and other information through labels. If an operation example of the formula is identified, it is associated with the formula through an index label.

[0132] Furthermore, if the document processing device detects references, footnotes, or appendix numbers in the document being processed during text parsing or OCR recognition, it can also mark them accordingly, ensuring that these special parts are distinguished within the structured data. Labeled figures and figure captions can also be extracted and numbered.

[0133] Step S634: insert the ordered list data, the nested cell structure information and the preset expression into the document tree structure model according to the layout position information of the document to be processed to obtain an initial file structured model.

[0134] In a specific implementation, the above-mentioned document processing device can determine the position of the ordered list data, nested cell structure information and preset expression in the document to be processed based on the layout position information of the document to be processed, thereby determining the node position of the ordered list data, nested cell structure information and preset expression in the document tree structure model. Based on this, the ordered list data, nested cell structure information and preset expression can be inserted into the corresponding node position of the tree structure model to generate an initial file structured model.

[0135] Step S635: Fill the fused text content into the corresponding content of the initial document structured model to generate a standard document structured model.

[0136] In a specific implementation, the document processing device can determine the location of the fused text content within the document to be processed based on the hierarchical structure information of the document to be processed, thereby determining the node location of the fused text content within the initial file structure model. Based on this, the fused text content can be populated into the corresponding content, i.e., the corresponding node, of the initial file structure model, thereby generating a standard file structure model. This standard file structure model can be hierarchical data in XML / JSON, or a multi-table design in a relational database, covering the hierarchical organization of document metadata (title, publication number, date, etc.) and the text content, but this embodiment does not limit this.

[0137] Step S64: outputting structured data or semi-structured data in a preset format based on the standard file structuring model.

[0138] It should be noted that the above-mentioned preset format can be the format in which the document to be processed is output in advance, which is set by the user according to the needs.

[0139] In a specific implementation, the document processing device may convert the format of a standard document structure model into a pre-set format to generate structured or semi-structured data, and then output the structured or semi-structured data. During the output process, the device may retain references to the document being processed, such as mapping information such as page numbers and paragraph numbers, to facilitate verification or tracing between the structured data and the document being processed.

[0140] For example, structured or semi-structured data can be output as JSON or XML files based on user needs, facilitating further processing or exchange across different systems. Furthermore, the text content within the structured or semi-structured data is stored directly within the JSON or XML file using a pre-defined chapter structure. Extracted images, tables, formulas, and the like can be written into the JSON or XML document using relative positional links. The output structured or semi-structured data can also be written into a relational database or knowledge base system, storing each chapter and clause as an independent record and indexed for easy retrieval. Furthermore, the structured or semi-structured data can be displayed on the front-end system for user review and verification, such as by displaying it in a tree-like directory and content. Users can browse the entire file and check to ensure that the format and content are consistent with the document being processed. Users or other systems can directly utilize this structured data for fine-grained content retrieval, statistical analysis, and comparison operations. For example, it can quickly locate the content of a specific clause in a standard or automatically align and compare similar clauses across multiple regulations.

[0141] This embodiment determines the hierarchical structure information of the document to be processed; constructs a document tree structure model based on the hierarchical structure information; generates a standard file structuring model based on the document tree structure model and the fused text content; outputs structured data or semi-structured data in a preset format based on the standard file structuring model, ensuring that the output structured data or semi-structured data corresponds to the content and structure of the document to be processed, thereby retaining and structuring the logical hierarchy of the original document, thereby improving the accuracy of document processing and facilitating the reuse of document content.

[0142] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the document processing method of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0143] This application also provides a document processing device, please refer to Figure 4 , Figure 4 This is a schematic diagram of the module structure of the document processing device of the present application, which includes:

[0144] The text input module 10 is used to obtain a document to be processed and identify a text area and an image area in the document to be processed.

[0145] The text extraction module 20 is used to extract text from the text area to obtain vector text content.

[0146] The optical character recognition module 30 is configured to perform character recognition on the image area using multiple optical character recognition models and obtain recognition results.

[0147] The confidence fusion module 40 is configured to determine confidence weights in the recognition results according to the character recognition results corresponding to the optical character recognition models.

[0148] The confidence fusion module 40 is further configured to fuse the character recognition results with the vector text content according to the confidence weights to obtain fused text content.

[0149] The data output module 50 is configured to output structured data or semi-structured data according to the fused text content.

[0150] The document processing device provided in this application, utilizing the document processing method of the aforementioned embodiment, can resolve the technical issue of existing technologies being incompatible with mixed-type document input and, consequently, failing to meet user needs. Compared to existing technologies, the document processing device provided in this application achieves the same beneficial effects as the document processing method of the aforementioned embodiment. Other technical features of the document processing device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0151] The present application provides a document processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the document processing method in the above-mentioned embodiment one.

[0152] Reference below Figure 5 , Figure 5The figure is a schematic diagram of the device structure of the hardware operating environment involved in the document processing method in the embodiment of the present application, which shows a schematic diagram of the structure of a document processing device suitable for implementing the embodiment of the present application. The document processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The document processing device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0153] like Figure 5 As shown, the document processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the document processing device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. Communication device 1009 can allow the document processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a document processing device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or provided instead.

[0154] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0155] The document processing device provided by this application, utilizing the document processing method of the aforementioned embodiment, can resolve the technical issue of the prior art being incompatible with mixed-type document input and, consequently, unable to meet user requirements m. Compared to the prior art, the beneficial effects of the document processing device provided by this application are the same as those of the document processing method provided by the aforementioned embodiment. Other technical features of this document processing device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0156] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0157] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0158] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the document processing method in the above embodiment.

[0159] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0160] The computer-readable storage medium may be included in the document processing device, or may exist independently without being assembled into the document processing device.

[0161] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a document processing device, the document processing device: obtains a document to be processed and identifies text areas and image areas in the document to be processed; performs text extraction on the text area to obtain vector text content; performs character recognition on the image area through multiple optical character recognition models and obtains recognition results; determines confidence weights in the recognition results according to the character recognition results corresponding to each optical character recognition model; fuses each character recognition result with the vector text content according to each confidence weight to obtain fused text content; and outputs structured data or semi-structured data according to the fused text content.

[0162] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0163] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0164] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0165] The computer-readable storage medium provided in this application is a computer-readable storage medium storing computer-readable program instructions (i.e., a computer program) for executing the document processing method described above. This computer-readable storage medium can address the technical issues of the prior art, which are incompatible with mixed-type document input and, therefore, cannot meet user needs. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the document processing method provided in the above-described embodiments, and are not further elaborated here.

[0166] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A document processing method, characterized in that: The method includes: Acquire a document to be processed, and identify a text area and an image area in the document to be processed; Performing text extraction on the text area to obtain vector text content; Performing character recognition on the image area using multiple optical character recognition models and obtaining recognition results; Determining a confidence weight in the recognition result according to the character recognition result corresponding to each of the optical character recognition models; fusing the character recognition results with the vector text content according to the confidence weights to obtain fused text content; Output structured data or semi-structured data according to the fused text content.

2. The document processing method according to claim 1, wherein: The step of fusing the character recognition results with the vector text content according to the confidence weights to obtain fused text content includes: Comparing each of the character recognition results with the vector text content; When the character recognition results are inconsistent with the vector text content, fusing the character recognition results according to the confidence weights to obtain the first scanned text content; The first scanned text content is fused with the vector text content to obtain fused text content.

3. The document processing method according to claim 2, wherein: When the character recognition results are inconsistent with the vector text content, the step of fusing the character recognition results according to the confidence weights to obtain the first scanned text content includes: When each of the character recognition results is inconsistent with the vector text content, determining the confidence weights of different character recognition results at any position in the image area; Selecting the character recognition result with the highest confidence weight as the recognized character at any position; The recognized characters at various positions in the image region are merged to obtain first scanned text content.

4. The document processing method according to claim 3, wherein: After the step of comparing each of the character recognition results with the vector text content, the method further includes: When there is a target character recognition result that is consistent with the vector text content in each of the character recognition results, determining each other character recognition result other than the target character recognition result in each of the character recognition results; fusing the other character recognition results according to their confidence weights to obtain a second scanned text content; The second scanned text content is fused with the vector text content to obtain fused text content.

5. The document processing method according to any one of claims 1 to 4, characterized in that: The step of outputting structured data or semi-structured data according to the fused text content includes: Determining hierarchical structure information of the document to be processed; Constructing a document tree structure model according to the hierarchical structure information; Generating a standard document structure model according to the document tree structure model and the fused text content; The structured data or semi-structured data is output in a preset format based on the standard file structuring model.

6. The document processing method according to claim 5, wherein: The step of constructing a document tree structure model according to the hierarchical structure information includes: Determining hierarchical information in the hierarchical structure information; Obtaining font information and typesetting information of the document to be processed; A document tree structure model is constructed according to the hierarchical information, font information and typesetting information.

7. The document processing method according to claim 6, wherein: The step of generating a standard document structure model based on the document tree structure model and the fused text content includes: Identifying the ordered list data of the documents to be processed; When table structure information exists in the document to be processed, converting the table structure into nested cell structure information; When formula data exists in the document to be processed, converting the formula data into a preset expression; Inserting the ordered list data, the nested cell structure information, and the preset expression into the document tree structure model according to the layout position information of the document to be processed to obtain an initial document structure model; The fused text content is filled into the corresponding content of the initial document structured model to generate a standard document structured model.

8. A document processing device, characterized in that: The document processing includes: A text input module, configured to obtain a document to be processed and identify text areas and image areas in the document to be processed; A text extraction module, configured to extract text from the text area to obtain vector text content; an optical character recognition module, configured to perform character recognition on the image area using multiple optical character recognition models and obtain recognition results; a confidence fusion module, configured to determine confidence weights in the recognition results according to the character recognition results corresponding to the optical character recognition models; The confidence fusion module is further configured to fuse the character recognition results with the vector text content according to the confidence weights to obtain fused text content; A data output module is used to output structured data or semi-structured data according to the fused text content.

9. A document processing device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the document processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the document processing method according to any one of claims 1 to 7 are implemented.

Images