Method and framework for optimizing scanning copy content recognition quality by using large model

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a method and a framework for optimizing scanning copy content recognition quality by using a large model. The method comprises the steps of document image analysis and processing, character extraction and formatting and context-based OCR correction by using a large model. The invention aims to provide the method and the framework for optimizing the identification quality of the scanned copy content by using the large model, the powerful functions of large language models such as a visual model and a text model are combined, the deep understanding of the document content and layout is realized, the document layout is accurately analyzed, different elements such as text blocks, tables and images are identified, and the identification quality of the scanned copy content is improved. And in combination with an analysis result of the visual model, converting the document content into a graceful and smooth Markdown format, and retaining an original layout of the document.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and framework for optimizing the quality of content recognition of scanned documents using a large model. Background Art

[0002] With the continuous advancement of digital technology, the digital transformation of historical documents has become a key means of protecting and inheriting cultural heritage. As the field of document design continues to evolve, document structure, semantic content, and layout are showing an unprecedented trend of diversification. This diversity not only enriches the expression of information, but also poses new challenges and opportunities for document processing, retrieval, and comprehension technologies.

[0003] Traditional content recognition of scanned documents primarily relies on simple character recognition through OCR. This lacks understanding of document structure and semantics, neglecting key information like tables and charts. The resulting output is disorganized and difficult to use. This is particularly true for documents with complex layouts, such as newspapers and periodicals, where recognition accuracy and formatting are often unsatisfactory. The complexity of layouts, varying layouts, and design styles place higher demands on OCR processing capabilities.

[0004] Currently, content recognition of scanned documents mainly uses OCR technology to convert scanned documents into editable electronic text. This technology is mainly based on image processing and statistical machine learning to extract text content from images. It mainly includes image preprocessing, layout analysis, character segmentation, character recognition, layout recovery, and post-processing. However, it has certain limitations in complex scenarios, mainly manifested in the following problems: 1. High requirements for scanning quality and limited recognition accuracy: The recognition effect depends largely on the quality of the input image. When the image is blurred, tilted, unevenly illuminated, or has a complex background, it may lead to image recognition errors, which requires higher requirements and standards for the scanning quality and shooting conditions of the scanned documents.

[0005] 2. Diverse fonts and complex layouts make parsing difficult: Different languages ​​have different character sets, writing rules, and grammatical structures. Recognition accuracy can drop significantly, especially when multiple languages ​​are mixed in a document. Complex elements like tables, images, and annotations, as well as non-standard typesetting, can affect OCR recognition. Variations in text line curvature, font size, and color can also increase recognition difficulty. Traditional OCR is often inaccurate in character segmentation and recognition, leading to errors in recognition results.

[0006] 3. Insufficient understanding of contextual semantics, making it difficult to use: Currently, content recognition mainly performs simple text character recognition and lacks understanding of document structure and semantics. If a character is blurred in an image, it may not be accurately recognized. At the same time, key information such as tables and charts is ignored, and the output text format is chaotic and difficult to use directly.

[0007] In summary, to address the common problem of scanned document content recognition quality, we proposed a method and framework for optimizing the scanned document content recognition quality using a large model. Summary of the Invention

[0008] In order to solve the above problems, the purpose of the present invention is to provide a method and framework for optimizing the quality of scanned document content recognition using large models. This method combines the powerful functions of large language models such as visual models and text models to achieve a deep understanding of document content and layout, accurately analyze document layout, identify different elements such as text blocks, tables, images, etc., and combine the analysis results of the visual model to convert the document content into an elegant and fluent Markdown format while retaining the original layout of the document.

[0009] To achieve the above object, the present invention adopts the following technical solution: comprising the following steps: Step S1: Using document parsing technology in combination with the file format, extracting the image content of the scanned document, reconstructing it using image super-resolution, and preprocessing the image; Step S2: Based on the pre-processing results of the image scan, key areas in the document are identified through positioning recognition technology, and the logical structure of the document is analyzed and converted into structured data; Step S3: Based on the above structured data, identify the noise data and use the LayoutParser model to perform noise filtering; Step S4: Based on the noise filtering results of the document scan, use the PaddleOCR model to extract the text in the document, integrate the text information and format the result into a JSON structure; Step S5: Combine the integrated text information and use the large model to perform context-based OCR correction.

[0010] Furthermore, the pre-processing is specifically as follows: Combining image processing algorithms with deep learning technology to repair images; Adjust the hue, saturation, and brightness of an image through stretch correction and HSV color separation to perform background replacement and / or background decolorization; Preserve image edge details through smoothing filters, non-local mean denoising, and pre-trained denoising models; The texture of the surrounding area of ​​the image is generated through mask repair and GAN network, replacing the dirty blocks to perform image dirty block repair; Use edge detection and / or Sobel operator to locate the black edge range, analyze the edge pixel intensity, and automatically remove continuous black edges.

[0011] Furthermore, the noise processing process includes: The bounding box is calculated using polygon information of the line, and noise data is determined by comparing an inclination angle of the bounding box relative to the horizontal direction with an angle threshold set by the device relative to the horizontal direction; Zero-shot classification is performed through LLM: mPlug-DocOwl and LLaVa1.6, and the LayoutParser document analysis model is used to judge noisy data. Bagging is used to filter out negative samples and reduce the input to the object detector of open-Vocab.

[0012] Furthermore, in step S4, integrating text information specifically includes: using the PaddleOCR model to extract text from the document, parsing the graphic layout information, dividing the content into blocks, marking the contextual relationship and merging the text information and its layout information.

[0013] Furthermore, the step S5 is specifically as follows: Step S51: Select candidate regions of text that need to be corrected, collect text windows of predefined block size from the text set, identify candidate text regions of the locally encapsulated generated text based on fuzzy string matching based on Levenshtein distance, find windows with candidate text region similarity greater than a threshold, and fill in the highest candidate text; Step S52: Use the standard template to perform LIM query on each text segment and take hallucination mitigation measures to obtain LIM answers; Step S53: extracting a suitable cropping window from the above LIM answer, and performing an operation using an algorithm to obtain the cropped corrected text; Step S54: performing spelling correction on the corrected text in combination with the LIM in the context.

[0014] Furthermore, the algorithm execution operation of step S53 is specifically as follows: Step S531: splitting the original text and the LIM answer text into a single split; Step S532: Slide a window with the same length as the original text ±1 at the single level to cover the LLM answer; Step S533: Calculate the similarity ratio of each window using Levenshtein distance; Step S534: Select the best matching window with the highest candidate similarity.

[0015] The present invention has the following beneficial effects: This invention adapts to various scan quality and complex scene content recognition requirements, significantly improving the ability to process low-quality scans and analyze complex layouts. By leveraging the deep feature extraction and region segmentation capabilities of the large visual model, it can accurately distinguish and process a variety of layout contents, significantly improving recognition accuracy on low-quality scans and those with complex backgrounds.

[0016] 2. This invention leverages the contextual understanding capabilities of a large language model to intelligently correct spelling errors and, combined with grammatical analysis, reconstruct text logic, ensuring that the final recognition results are more consistent with human reading habits and semantic logic. When processing specialized terminology and domain-specific vocabulary, a customized vocabulary library further enhances semantic accuracy.

[0017] 3-When processing scanned documents with complex layouts, the present invention combines the powerful functions of large language models such as visual models and text models. It proposes to use large models for semantic analysis and feature optimization on the basis of the original OCR, and improves the OCR quality by accurately analyzing the document layout through the visual model and utilizing the filling and context adaptation capabilities of the language model.

[0018] 4-The present invention provides multimodal collaborative optimization, language correction and semantic consistency capabilities. Through deep understanding of contextual semantics combined with a large language model to analyze semantics, contextual understanding can effectively complete missed content and improve text integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the present invention. DETAILED DESCRIPTION

[0020] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments: refer to Figure 1 As shown, in this embodiment, a method and framework for optimizing the quality of scanned content recognition using a large model is provided, including the following steps: Step S1: Using document parsing technology in combination with the file format, extracting the image content of the scanned document, reconstructing it using image super-resolution, and pre-processing the image to improve the clarity of the image; Step S2: Based on the preprocessing results of the image scan, key areas in the document are identified through positioning and recognition technology, and the logical structure of the document is analyzed and converted into structured data. The image scan is mainly subjected to visual layout analysis, and the key areas of various visual elements in the document are identified through positioning. The logical structure of the document is analyzed, that is, the types and semantic relationships of various semantic elements in the document are identified and converted into machine-readable structured data. Based on the feature fusion method of image and text, the layout detection model is optimized from the aspects of diverse document pre-training and model structure optimization through end-to-end training of the DocLayout-YOLO model. Step S3: Based on the above structured data, identify the noise data and use the LayoutParser model to perform noise filtering; Step S4: Based on the noise filtering results of the document scan, use the PaddleOCR model to extract the text in the document, integrate the text information and format the result into a JSON structure; Step S5: Combine the integrated text information and use the large model to perform context-based OCR correction.

[0021] Furthermore, the pre-processing is specifically as follows: Image restoration is achieved by combining image processing algorithms with deep learning technology. Image processing technology is used to remove yellow background, noise, stains, black edges and other stains from scanned images, correct tilted images, etc., eliminate irrelevant information in the image, enhance the detectability of relevant information and simplify data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition.

[0022] Adjust the hue, saturation, and brightness of an image through stretch correction and HSV color separation to perform background replacement and / or background decolorization; Preserve image edge details and reduce blur through smoothing filters, non-local mean denoising, and pre-trained denoising models; The texture of the surrounding area of ​​the image is generated through mask repair and GAN network, replacing the dirty blocks to perform image dirty block repair; Use edge detection and / or Sobel operator to locate the black edge range, analyze the edge pixel intensity, and automatically remove continuous black edges.

[0023] Furthermore, the noise processing process includes: The bounding box is calculated using polygon information of the line, and noise data is determined by comparing an inclination angle of the bounding box relative to the horizontal direction with an angle threshold set by the device relative to the horizontal direction; Zero-shot classification is performed through LLM: mPlug-DocOwl and LLaVa1.6, and the LayoutParser document analysis model is used to judge noisy data. Bagging is used to filter out negative samples and reduce the input to the object detector of open-Vocab.

[0024] Furthermore, in step S4, integrating text information specifically includes: using the PaddleOCR model to extract text from the document, parsing the graphic layout information, dividing the content into blocks, marking the contextual relationship and merging the text information and its layout information.

[0025] Furthermore, the step S5 is specifically as follows: Step S51: Select candidate regions of text that need to be corrected, collect text windows of predefined block size from the text set, identify candidate text regions of the locally encapsulated generated text based on fuzzy string matching based on Levenshtein distance, find windows with candidate text region similarity greater than a threshold, and fill in the highest candidate text; Step S52: Use the standard template to perform LIM query on each text segment and take hallucination mitigation measures to obtain LIM answers; Step S53: extracting a suitable cropping window from the above LIM answer, and performing an operation using an algorithm to obtain the cropped corrected text; Step S54: performing spelling correction on the corrected text in combination with the LIM in the context.

[0026] The above standard template is:

[0027] As the model tries to produce the most contextually appropriate continuation, it often hallucinates or generates text that was not part of the original input. This can lead to unexpected completions when the input is cropped or incomplete.

[0028] Optimally, some texts are very short or cannot be found anywhere else, and a minor spelling correction is usually made by the LLM, as the preceding and following texts (if any) are on the same page. The following is another standard template for spelling corrections based on the LLM:

[0029] To mitigate hallucinations in this step, we implemented a strict policy: the LLM’s answers must have high similarity, as we expect minimal changes, small differences in text length, and no changes in word count. If the LLM returns a result that does not meet these conditions, we reject the correction and keep the corrected text as the original text.

[0030] Furthermore, the algorithm execution operation of step S53 is specifically as follows: Step S531: splitting the original text and the LIM answer text into a single split; Step S532: Slide a window with the same length as the original text ±1 at the single level to cover the LLM answer; Step S533: Calculate the similarity ratio of each window using Levenshtein distance; Step S534: Select the best matching window with the highest similarity among the candidates.

[0031] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0032] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0033] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0034] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0035] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A method and framework for optimizing the quality of scanned content recognition using a large model, characterized in that: The following steps are involved: Step S1: Using document parsing technology in combination with the file format, extracting the image content of the scanned document, reconstructing it using image super-resolution, and preprocessing the image; Step S2: Based on the pre-processing results of the image scan, key areas in the document are identified through positioning recognition technology, and the logical structure of the document is analyzed and converted into structured data; Step S3: Based on the above structured data, identify the noise data and use the LayoutParser model to perform noise filtering; Step S4: Based on the noise filtering results of the document scan, use the PaddleOCR model to extract the text in the document, integrate the text information and format the result into a JSON structure; Step S5: Combine the integrated text information and use the large model to perform context-based OCR correction.

2. The method and framework for optimizing the quality of scanned content recognition using a large model according to claim 1, characterized in that: The pre-processing is as follows: Combining image processing algorithms with deep learning technology to repair images; Adjust the hue, saturation, and brightness of an image through stretch correction and HSV color separation to perform background replacement and / or background decolorization; Preserve image edge details through smoothing filters, non-local mean denoising, and pre-trained denoising models; The texture of the surrounding area of ​​the image is generated through mask repair and GAN network, replacing the dirty blocks to perform image dirty block repair; Use edge detection and / or Sobel operator to locate the black edge range, analyze the edge pixel intensity, and automatically remove continuous black edges.

3. The method and framework for optimizing the quality of scanned content recognition using a large model according to claim 1, characterized in that: The noise processing process includes: The bounding box is calculated using polygon information of the line, and noise data is determined by comparing an inclination angle of the bounding box relative to the horizontal direction with an angle threshold set by the device relative to the horizontal direction; Zero-shot classification is performed through LLM: mPlug-DocOwl and LLaVa1.6, and the LayoutParser document analysis model is used to judge noisy data. Bagging is used to filter out negative samples and reduce the input to the object detector of open-Vocab.

4. The method and framework for optimizing the quality of scanned content recognition using a large model according to claim 1, characterized in that: In step S4, integrating text information specifically includes: using the PaddleOCR model to extract text from the document, parsing the graphic layout information, dividing the content into blocks, marking the contextual relationship and merging the text information and its layout information.

5. The method and framework for optimizing the quality of scanned content recognition using a large model according to claim 1, characterized in that: The step S5 is specifically as follows: Step S51: Select candidate regions of text that need to be corrected, collect text windows of predefined block size from the text set, identify candidate text regions of the locally encapsulated generated text based on fuzzy string matching based on Levenshtein distance, find windows with candidate text region similarity greater than a threshold, and fill in the highest candidate text; Step S52: Use the standard template to perform LIM query on each text segment and take hallucination mitigation measures to obtain LIM answers; Step S53: extracting a suitable cropping window from the above LIM answer, and performing an operation using an algorithm to obtain the cropped corrected text; Step S54: performing spelling correction on the corrected text in combination with the LIM in the context.

6. The method and framework for optimizing the quality of scanned content recognition using a large model according to claim 5, characterized in that: The algorithm execution operation of step S53 is specifically as follows: Step S531: splitting the original text and the LIM answer text into a single split; Step S532: Slide a window with the same length as the original text ±1 at the single level to cover the LLM answer; Step S533: Calculate the similarity ratio of each window using Levenshtein distance; Step S534: Select the best matching window with the highest candidate similarity.

Images