Document image content identification method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a document image content identification method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence and document processing. The method comprises the following steps: acquiring an original document image; performing classification processing on candidate elements in the original document image to obtain an element category of each candidate element; performing image enhancement processing on the original document image to obtain a target document image; inputting the target document image into a preset content identification expert model for content identification to obtain verification content identification information; performing content identification on a target element in the target document image through a preset target content identification model, the element category and the verification content identification information to obtain target content identification information; wherein the candidate elements comprise target elements. The method can be applied to business systems needing a large amount of data, such as financial science and technology and health medical treatment, and the recognition accuracy of the document image can be improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and document processing technology, and in particular to a method and apparatus for content recognition of document images, a computer device, and a storage medium. Background Technology

[0002] In digital document content recognition scenarios, specifically those involving document images with multiple elements such as text, tables, formulas, and graphics, accurately identifying the content of document images is a core task. Related technologies primarily employ OCR recognition combined with layout analysis and rule combinations. However, these methods suffer from poor accuracy when affected by interfering elements such as watermarks, seals, and background textures. Furthermore, they neglect the contextual relationships between text, formulas, and tables within the document image, resulting in fragmented and disjointed text structure, further contributing to low accuracy. Therefore, improving the accuracy of document image content recognition has become a pressing technical challenge. Summary of the Invention

[0003] The main objective of this application is to provide a method and apparatus for content recognition of document images, as well as a computer device and storage medium, in order to improve the accuracy of document image recognition.

[0004] To achieve the above objectives, a first aspect of this application proposes a method for content recognition of document images, the method comprising: Obtain the original document image; The candidate elements in the original document image are classified to obtain the element category of each candidate element; The original document image is subjected to image enhancement processing to obtain the target document image; The target document image is input into a preset content recognition expert model for content recognition to obtain verification content recognition information; The target elements in the target document image are identified by using a preset target content recognition model, the element category, and the verification content recognition information to obtain target content recognition information; wherein, the candidate elements include the target elements.

[0005] In some embodiments, performing image enhancement processing on the original document image to obtain the target document image includes: The original document image is masked to obtain the original image mask data; Based on the original image mask data, noise regions are removed from the original document image to obtain a preliminary document image; Feature extraction is performed on each layer of the preliminary document image to obtain layer features; Based on the layer features, layers in the preliminary document image that are different from the target element are deleted to obtain candidate document images; The candidate document image is reconstructed using super-resolution to obtain the target document image.

[0006] In some embodiments, the step of deleting layers in the preliminary document image that are different from the target feature based on the layer features to obtain candidate document images includes: A map is constructed based on the layer features to obtain a layer association map; wherein, the layer association map includes candidate nodes and target nodes, the candidate nodes represent candidate layers, and the target nodes represent layers of the target features; The similarity between the candidate nodes and the target nodes is calculated based on the layer association graph to obtain the node similarity. The candidate layers whose node similarity is lower than a preset first similarity are selected as the selected layers; The selected layer in the preliminary document image is deleted to obtain the candidate document image.

[0007] In some embodiments, before performing content recognition on target elements in the target document image using a preset target content recognition model, the element category, and the verification content recognition information to obtain target content recognition information, the method further includes: Constructing the target content recognition model specifically includes: Obtain a training dataset; wherein the training dataset includes: training document images, recognition task categories of the training document images, expert content recognition information, and reference content recognition information, wherein the expert content recognition information is obtained by the content recognition expert model based on the training document images for content recognition; The training document image is subjected to content recognition by a preset large language model and the recognition task category to obtain model content recognition information; The large language model is fine-tuned based on the recognition task category, the model content recognition information, the expert content recognition information, and the reference content recognition information to obtain the target content recognition model.

[0008] In some embodiments, the step of fine-tuning the large language model based on the recognition task category, the model content recognition information, the expert content recognition information, and the reference content recognition information to obtain the target content recognition model includes: The task weight is determined based on the identified task category; The matching degree of the model content recognition information, the expert content recognition information, and the reference content recognition information is evaluated to obtain the recognition matching degree. The model is optimized based on the task weights and the recognition matching degree. The large language model is fine-tuned based on the model optimization data to obtain the target content recognition model.

[0009] In some embodiments, the step of performing content recognition on target elements in the target document image using a preset target content recognition model, the element category, and the verification content recognition information to obtain target content recognition information includes: Based on the target content recognition model and the element category, the target elements in the target document image are identified to obtain preliminary content recognition information; A similarity assessment is performed based on the verified content identification information and the preliminary content identification information to obtain the content similarity. The target content recognition model is optimized based on the content similarity and the verification content recognition information to obtain the target content recognition information.

[0010] In some embodiments, optimizing the output of the target content recognition model based on the content similarity and the verification content recognition information to obtain the target content recognition information includes: If the content similarity is greater than or equal to a preset second similarity, the preliminary content recognition information output by the target content recognition model shall be used as the target content recognition information; If the content similarity is less than a preset second similarity, the preliminary content identification information is located based on the verification content identification information to obtain the difference content, and the target content identification model is output corrected based on the difference content to obtain the target content identification information.

[0011] To achieve the above objectives, a second aspect of this application provides a document image content recognition device, the device comprising: The acquisition module is used to acquire the original document image; The classification module is used to classify the candidate elements in the original document image to obtain the element category of each candidate element. The image enhancement module is used to perform image enhancement processing on the original document image to obtain the target document image; The expert recognition module is used to input the target document image into a preset content recognition expert model for content recognition and obtain verification content recognition information. The model recognition module is used to perform content recognition on target elements in the target document image by using a preset target content recognition model, the element category, and the verification content recognition information to obtain target content recognition information; wherein, the candidate elements include the target elements.

[0012] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0014] The document image content recognition method, apparatus, computer device, and storage medium proposed in this application, after acquiring the original document image, classify each candidate element within the original document image to determine the element category of each candidate element, and then enhance the original document image into a target document image. Simultaneously, a content recognition expert model is set up to generate verification content recognition information for the target document image. Then, the element category, the verification content recognition model, and the target content recognition model are combined to perform content recognition on the target document image, thereby achieving content recognition of complex document images and improving the accuracy of document image content recognition. Attached Figure Description

[0015] Figure 1 This is a flowchart of the document image content recognition method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S103 in the process; Figure 3 yes Figure 2 The flowchart of step S204 in the process; Figure 4 This is a flowchart of a document image content recognition method provided in another embodiment of this application; Figure 5 yes Figure 4 The flowchart of step S403 in the process; Figure 6 yes Figure 1 The flowchart of step S106 in the process; Figure 7 yes Figure 6 The flowchart of step S603 in the process; Figure 8 This is a system framework diagram of the document image content recognition method provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the document image content recognition device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar elements and are not necessarily used to describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] First, let's analyze some of the terms used in this application: Large Language Models (LLMs): With technological advancements, the concept has expanded to include large visual models, multimodal large models, and fundamental scientific models. Large language models can simultaneously process and understand multiple types of data, such as text, images, and audio, enabling cross-modal information fusion and generation. These models excel in tasks such as image and text generation and video generation, breaking the limitations of single-modality models and enabling richer interactions and creations.

[0020] Semantic segmentation is one of the key problems in the field of computer vision today. From a macro perspective, semantic segmentation is a high-level task that paves the way for a complete understanding of scenes. Scene understanding, as a core computer vision problem, is important because an increasing number of applications rely on inferring knowledge from images. Some of these applications include autonomous vehicles, human-computer interaction, and virtual reality. In recent years, with the popularization of deep learning, many semantic segmentation problems are being solved using deep structures, most commonly convolutional neural networks, which have significantly surpassed other methods in accuracy.

[0021] A mask is an array or matrix with the same shape as the original data (usually an image or matrix), whose elements are used to control operations on the original data. Essentially, it's a filtering mechanism that determines whether an element is retained, modified, or discarded by combining operations on each element of the original data with the corresponding element in the mask.

[0022] Retrieval Augmentation (RAG) combines traditional information retrieval systems (such as search and databases) with generative large language models (LLMs). By integrating external knowledge sources with the language skills of the large language model, AI applications can generate more accurate answers. The core of RAG lies in allowing the large language model to retrieve relevant information from external knowledge bases before generating a response, thereby expanding the knowledge boundaries of the large language model and improving the accuracy and usability of the answers.

[0023] Morphological operations are an image analysis discipline built upon graph theory and topology, and represent the fundamental theory of mathematical morphological image processing. Its basic operations include erosion and dilation, opening and closing operations, skeleton extraction, limit erosion, hit-and-miss transformation, morphological gradient, Top-hat transformation, particle analysis, and watershed transformation. Image opening is the process of sequentially processing an image through erosion and dilation. Erosion removes noise but also compresses the image; subsequent dilation of the eroded image removes noise while preserving the original image quality.

[0024] Graph Attention Network (GAT) is a neural network specifically designed for processing graph-structured data. It automatically learns the relationships between nodes by introducing an attention mechanism, eliminating the need for manually preset weights. Its core advantage lies in the weighted aggregation of neighboring nodes, improving the model's robustness and interpretability.

[0025] The diffusion model is a generative model inspired by the diffusion process in thermodynamics. It generates samples by simulating the gradual denoising process of data from noise. Its core idea is to progressively add noise (forward process) and gradually denoise (reverse process). In the forward process, Gaussian noise is gradually added to the data, eventually transforming it into pure noise; in the reverse process, it learns how to gradually denoise from the noise to recover the original data distribution.

[0026] In digital document processing scenarios, extracting structured information from document images containing diverse elements such as text, tables, formulas, and graphics is a core step in achieving intelligent document understanding and enhanced retrieval. Traditional document image content recognition mainly employs a pipeline model of OCR recognition + layout analysis + rule combination, which has the following drawbacks: First, lack of element correlation: processing each element independently ignores the semantic correlation between text and tables, leading to fragmented structured results. Second, sensitivity to noise interference: insufficient robustness to interfering elements such as watermarks and seals, especially with a sharp drop in recognition rate in low-contrast scenes. Third, error accumulation effect: multi-stage serial processing leads to a gradual amplification of initial errors, ultimately affecting end-to-end parsing accuracy. With the application of multimodal large models to end-to-end document image parsing scenarios, image quality enhancement improves image feature recognition accuracy. However, two problems still exist when dealing with complex layout document images: First, in document images with mixed text and images, background textures, signatures, watermarks, and body text spatially overlap, resulting in low text content recognition accuracy. Second, multimodal large models are prone to generating fictitious content in specific task scenarios such as formula symbols and table structures, which affects the quality of document image recognition.

[0027] Based on this, embodiments of this application provide a method and apparatus for content recognition of document images, a computer device, and a storage medium. The aim is to acquire an original document image, classify each candidate element within the original document image to determine the element category of each candidate element, and then enhance the original document image into a target document image. Simultaneously, a content recognition expert model is set to generate verification content recognition information for the target document image. Then, by combining the element category, the verification content recognition model, and the target content recognition model, content recognition of the target document image is performed. This enables content recognition of complex document images and improves the accuracy of document image content recognition.

[0028] The document image content recognition method, apparatus, computer equipment, and storage medium provided in this application are specifically described through the following embodiments. First, the document image content recognition method in this application is described.

[0029] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0030] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0031] The document image content recognition method provided in this application relates to the fields of artificial intelligence and document processing technology. The document image content recognition method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the document image content recognition method, but is not limited to the above forms.

[0032] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, elements, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0033] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0034] Figure 1 This is an optional flowchart of the document image content recognition method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0035] Step S101: Obtain the original document image; Step S102: Classify the candidate elements in the original document image to obtain the element category of each candidate element; Step S103: Perform image enhancement processing on the original document image to obtain the target document image; Step S104: Input the target document image into a preset content recognition expert model for content recognition to obtain verification content recognition information; Step S105: The target elements in the target document image are identified by using a preset target content recognition model, element category and verification content recognition information to obtain target content recognition information; wherein, candidate elements include target elements.

[0036] Steps S101 to S105, as illustrated in this embodiment, firstly, an original document image is acquired, and the element category of each candidate element in the original document image is determined. Then, the original document image is enhanced to obtain a target document image. Simultaneously, a content recognition expert model is set to recognize the target document image to obtain verification content recognition information. This verification information guides the target content recognition model to complete the content recognition of the target document image. Furthermore, the target content recognition model also identifies target elements in the target document image based on element categories, outputting more accurate target content recognition information. Therefore, when setting up content recognition for document images, this application introduces a content recognition expert model and element categories when the model infers the content of target elements in the target document image. This can improve the target content recognition model's recognition capability in specific cross-domain applications. Moreover, the advantages of the target content recognition model and the content recognition expert model are complementary, effectively suppressing the illusion problem of the target content recognition model and improving the content recognition of document images.

[0037] In step S101 of some embodiments, the original document image contains at least one candidate element, which can be any of the following: background element, text element, watermark element, seal element, table element, formula element, and graphic element. To accurately recognize the content of the original document image, each candidate element needs to be segmented and then the content of the target element needs to be recognized. It should be noted that the target element is a type of candidate element; in this embodiment, the target element is a text element, a table element, etc., to achieve text and table recognition.

[0038] In step S102 of some embodiments, classifying candidate elements in the original document image mainly involves determining the category of each candidate element to obtain an element category. The element category can be any of the following: background category, text category, watermark category, stamp category, table category, formula category, and image category.

[0039] Specifically, in this embodiment, the element category of candidate elements is determined by processing the original document image into a grayscale image. A pre-trained image segmentation model is then used to segment the grayscale image to obtain at least one candidate element. These candidate elements are represented as mask data, thus generating mask data labeled with six categories of candidate elements. Therefore, the element category of each candidate element is determined based on these six categories of mask data. For example, if the background pixel value is set to 0, the text pixel value to 1, the watermark pixel value to 2, the seal pixel value to 3, the table pixel value to 4, and the graphic pixel value to 5, then the element category of each candidate element can be determined through the pixel values.

[0040] Please see Figure 2 In some embodiments, step S103 may include, but is not limited to, steps S201 to S205: Step S201: Perform masking on the original document image to obtain the original image mask data; Step S202: Remove noise regions from the original document image based on the original image mask data to obtain a preliminary document image; Step S203: Extract features from each layer of the initial document image to obtain layer features; Step S204: Based on layer features, delete layers in the preliminary document image that are different from the target features to obtain candidate document images; Step S205: Perform super-resolution reconstruction on the candidate document image to obtain the target document image.

[0041] In step S201 of some embodiments, a pre-trained image segmentation model is used to directly mask the original document image, which is represented as feature segmentation processing. Candidate features in the original document image are split off, and the candidate features are labeled with mask data to obtain the original image mask data.

[0042] In step S202 of some embodiments, noise region removal mainly involves removing regions in the original document image that affect the recognition of target elements, thereby improving the accuracy of content recognition of target elements. To more accurately remove noise regions from the original document image, this embodiment uses morphological opening operations to optimize the original image mask data to obtain optimized image mask data. Connected component detection is then performed based on the optimized image mask data, and the areas of all connected components are counted. Connected components with areas smaller than a preset threshold are selected as noise regions, and the pixel values ​​of these noise regions are set to background values ​​(e.g., 0), thus achieving noise region removal. For example, if the preset threshold is 0.5%, regions with areas smaller than 0.5% are considered noise regions, and these noise regions are deleted from the initial document image to obtain the initial document image.

[0043] In steps S203 and S204 of some embodiments, the preliminary document image is input into a feature extraction network to extract layer features. Then, a graph attention neural network and layer features are used to remove layer features in the preliminary document image that are different from the target element, resulting in a candidate document image. It should be noted that the graph attention neural network and layer features can construct a graph of relationships between layers, assisting in layer filtering and retaining layers related to the target element.

[0044] In step S205 of some embodiments, super-resolution reconstruction refers to the technique of reconstructing a high-resolution image from a low-resolution image. The candidate document image has undergone noise region removal and layer removal, resulting in a loss of image resolution and missing regions. Therefore, super-resolution reconstruction is performed on the candidate document image to enhance image edge sharpness and obtain the target document image.

[0045] In some embodiments, before performing super-resolution reconstruction on the candidate document image, it is necessary to fill in the missing regions in the candidate document image. Specifically, for the missing regions in the candidate document image, this embodiment uses a diffusion model to fill in the pixel content of the missing regions. It should be noted that pixel-level detection is performed on the candidate document image, and the missing regions of the candidate document image are marked by methods such as threshold segmentation and edge detection, generating mask data for the missing regions. The contextual features around the missing regions are extracted as the basic reference information for the diffusion model filling. The missing regions of the candidate document image are filled by the diffusion model and the contextual features. The filling process incorporates the core semantic features extracted by the CLIP model as guiding coefficients into the sampling process of the diffusion model, so that the diffusion model gradually generates missing pixels based on the mask data of the missing regions and the contextual features, and fills the missing regions of the candidate document image with the missing pixels to construct a complete candidate document image.

[0046] After filling the candidate document image, a 4x super-resolution reconstruction is performed, and a high-frequency boosting filter is used to enhance the edge sharpness of the candidate document image to obtain a clearer target document image. In this embodiment, an efficient super-resolution model (such as ESRGAN, Real-ESRGAN, or SwinIR) is used to complete the super-resolution reconstruction of the candidate document image, which can adapt to the 4x magnification scene and effectively restore image details. The high-frequency boosting filter uses an enhancement coefficient α=0.6 to weight and enhance the high-frequency components of the candidate document image, and then fuses the enhanced high-frequency components with the original low-frequency components to form the target document image, thereby improving edge sharpness.

[0047] In steps S201 to S205 of this embodiment, the original document image is first masked to obtain original image mask data. Then, noise regions in the original document image are removed based on the original mask data to obtain a preliminary document image. Next, layers different from the target element in the preliminary document image are deleted using layer features to obtain candidate document images. Finally, super-resolution reconstruction is performed on the candidate document images to obtain the target document image. Therefore, in the image enhancement process of the original document image, noise regions are filtered out first, layers different from the target element are removed, and then super-resolution reconstruction is performed to obtain the target document image, improving the quality of the document image and thus enhancing the accuracy of document image content recognition.

[0048] Please see Figure 3 In some embodiments, step S204 may include, but is not limited to, steps S301 to S304: Step S301: Construct a map based on layer features to obtain a layer association map; wherein, the layer association map includes candidate nodes and target nodes, candidate nodes represent candidate layers, and target nodes represent layers that are target features; Step S302: Calculate the similarity between candidate nodes and target nodes based on the layer association graph to obtain the node similarity. Step S303: Select candidate layers whose node similarity is lower than the preset first similarity as selected layers; Step S304: Delete the selected layer in the initial document image to obtain the candidate document image.

[0049] In step S301 of some embodiments, the graph construction mainly uses a graph attention network to treat candidate layers as candidate nodes and the layer features of the candidate layers as node features. The node features include the semantic vector of the candidate layer content and layer attributes (such as type, size, and use for labels). Then, the edges between candidate nodes are determined based on the spatial overlap, content association, and reference relationship between candidate layers, and the edge weights characterize the association strength between candidate layers. Therefore, a layer association graph is constructed using candidate layers and their layer features.

[0050] In step S302 of some embodiments, node similarity represents the similarity between candidate layers and target layers. Specifically, the similarity is calculated by comparing the node feature matrix of each candidate node in the layer association graph with the node feature matrix of the target node. A multi-head attention mechanism is used to complete the semantic similarity calculation of nodes, highlighting the association weight between target nodes and candidate nodes to obtain node similarity.

[0051] In steps S303 and S304 of some embodiments, the first similarity is customized. If it is set to 0.2, then candidate layers with node similarity less than 0.2 are selected as the selected layers, and the selected layers of the initial document image are deleted to obtain candidate document images.

[0052] In steps S301 to S304 of this embodiment, a layer association map is constructed through layer features. The similarity between each candidate layer and the target element layer is calculated using the layer association map. Then, candidate layers with node similarity lower than the first similarity are selected layers, and the selected layers in the initial document image are deleted to obtain candidate document images. This filters out layers that are irrelevant to the target element and retains layers that are relevant to the target element, thereby improving the content recognition of the document image.

[0053] In step S104 of some embodiments, the content recognition expert model can be a model running on an expert user terminal, or a model trained based on expert training data. Specifically, if the content recognition expert model is a model running on an expert user terminal, each expert user terminal is equipped with a domain of expertise label. A suitable expert user terminal can be selected as the target user terminal based on the task category and domain of expertise label of the target document image. The target document image is sent to the target user terminal, and the verification content recognition information fed back by the content recognition expert model on the target user terminal based on the target document image is received. It should be noted that the verification content recognition information serves as reference recognition information for the target document image. On the one hand, it can verify the accuracy of the target content recognition model's recognition; on the other hand, it can guide the target content recognition model to continuously optimize the content recognition output, thereby improving the content recognition accuracy of the target document image.

[0054] In some embodiments, the document image content recognition method may further include constructing a target content recognition model. It should be noted that the target content recognition model in this application is obtained by fine-tuning a large language model. Large language models possess powerful understanding and analytical capabilities. Fine-tuning a large language model into a target content recognition model not only reduces the difficulty of constructing the target content recognition model but also improves the accuracy of content recognition.

[0055] Please see Figure 4 In some embodiments, constructing the target content recognition model may include, but is not limited to, steps S401 to S403: Step S401: Obtain the training dataset; wherein, the training dataset includes: training document images, recognition task categories of the training document images, expert content recognition information, and reference content recognition information, wherein the expert content recognition information is obtained by the content recognition expert model based on the training document images for content recognition; Step S402: Perform content recognition on the training document image using a preset large language model and recognition task category to obtain model content recognition information; Step S403: Fine-tune the large language model based on the recognition task category, model content recognition information, expert content recognition information, and reference content recognition information to obtain the target content recognition model.

[0056] In step S401 of some embodiments, the training dataset can come from an open-source image set or from a business image set corresponding to the application scenario of the target content recognition model. The business image set is constructed by collecting images related to the application scenario of the target content recognition model in advance. For example, if the target content recognition model is applied to the content recognition of financial document images, a large number of financial document images need to be collected to construct the training image set. If the target content recognition model is applied to the content recognition of stock transaction chart icons, then stock transaction document images are collected to construct the training image set. It should be noted that the training image set can come entirely from the open-source image set or the business image set, or it can be constructed by integrating the open-source image set and the business image set. This embodiment does not limit the source of the training image set.

[0057] Specifically, the training document images contain only one target feature. Therefore, the recognition task category of the training document images is the same as the feature category of the target feature. This serves as a prompt for the large language model to identify the target feature based on the feature category of the training document images, thus enabling more accurate training of the large language model. It should be noted that after acquiring the training document images, reference content recognition information is manually added. This reference content recognition information is the actual recognition information of the training document images, while expert content recognition information is obtained by the content recognition expert model based on the training document images. Both the reference content recognition information and the expert content recognition information can serve as reference information for the recognition accuracy of the large language model, enabling the training of the large language model to achieve accurate content recognition.

[0058] In some embodiments, the training document image can be an augmented version of an initial document image extracted from an open-source image set and / or a business image set, and the augmentation operation on the initial document image is the same as that on the original document image, which will not be repeated here. Therefore, if the training of a large language model is completed using the augmented training document image, the training effect of the large language model can be improved, thereby improving the accuracy of document image content recognition.

[0059] In step S402 of some embodiments, the task category is used to guide the large language model to recognize target elements in the training document image, and the task category includes formula recognition, table recognition, text recognition, etc. The large language model performs content recognition on each training document image according to each task category to output the model content recognition information corresponding to the task category.

[0060] In step S403 of some embodiments, the training of the large language model is completed by jointly identifying the task category, model content recognition information, expert content recognition information and reference content recognition information. This can achieve collaborative verification, realize the complementary advantages of the large language model and the content recognition expert model, effectively suppress the illusion problem of the large language model and improve the accuracy of content recognition.

[0061] In steps S401 to S403 of this embodiment, a large language model is trained by pre-constructing a training dataset, and the training process incorporates a content recognition expert model. This allows for the complementary advantages of the large language model and the content recognition expert model, effectively improving the illusion problem of the large language model and enhancing the accuracy of content recognition.

[0062] Please see Figure 5 In some embodiments, step S403 may include, but is not limited to, steps S501 to S504: Step S501: Determine the task weight based on the task category. Step S502: Evaluate the matching degree of the model content recognition information, expert content recognition information, and reference content recognition information to obtain the recognition matching degree; Step S503: Build a model to optimize data based on task weights and recognition matching degree; Step S504: Fine-tune the large language model based on the model optimization data to obtain the target content recognition model.

[0063] In step S501 of some embodiments, the task weight represents the recognition difficulty of the task category; the higher the recognition difficulty, the greater the task weight. As disclosed above, the recognition task categories include formula recognition, table recognition, and text recognition, etc. For large language models, formula recognition is more difficult than table recognition, and table recognition is more difficult than text recognition. Therefore, task weights are set according to different recognition task categories. When adjusting the large language model based on task weights, the large language model focuses more on target elements with higher recognition difficulty, thereby training a target content recognition model with higher recognition accuracy.

[0064] In step S502 of some embodiments, the recognition matching degree characterizes the recognition accuracy of the large language model. Specifically, the cross-matching degree between model content recognition information, expert content recognition information and reference content recognition information is calculated, and the consistency evaluation data of model content recognition information inference is also calculated. The recognition matching degree is determined based on the cross-matching degree and the consistency evaluation data.

[0065] In step S503 of some embodiments, task weights and recognition matching degrees are integrated to form optimized data. The optimized data is used to indicate the fine-tuning parameters of the large language model in order to adjust the target content recognition model with higher recognition accuracy.

[0066] Specifically, in this embodiment, the optimized data is a reward function. The task weight and the score of the recognition matching degree are calculated through a dynamic weight scoring function, and then the reward function is determined based on the score.

[0067] In a specific example, if the training image set is defined as ,in To identify task categories, including formula recognition, table recognition, and text recognition, Instructions to guide the large language model in recognition. This is the inference chain composed of corresponding model content recognition information, expert content recognition information, and reference content recognition information. ,in Information is used to identify reference content. To identify information for the model content, For expert content identification information, It will iteratively optimize in the inference chain, while simultaneously iteratively optimizing the large language model.

[0068] The dynamic weighted scoring function is designed as follows:

[0069] in, As task weight, The cross-matching degree refers to the cross-matching degree between model content recognition information, reference content recognition information, and expert content recognition information. For inference chain logical consistency assessment data, This is the adjustment coefficient.

[0070] After determining the score, the PPO algorithm is used to optimize the parameters of the large language model, and the reward function is set as follows:

[0071] in, Let be the score value output by the large language model at the t-th iteration. Let be the score value output by the large language model in the (t-1)th iteration. This is an information entropy regularization term used to prevent the policy from converging prematurely.

[0072] In step S504 of some embodiments, the large language model is fine-tuned through a reward function until the reward function converges, thereby completing the training of the large language model and determining the target content recognition model.

[0073] In steps S501 to S504 of this embodiment, task weights are set by identifying task categories, and the cross-matching degree between model content recognition information, expert content recognition information and reference content recognition information is calculated as the recognition matching degree. Then, optimization data is constructed based on task weights and recognition matching degree. Finally, the large language model is fine-tuned based on the optimization data to adjust the target content recognition model with higher recognition accuracy.

[0074] After fine-tuning the large language model, a target content recognition model is obtained. Then, the enhanced target document image is input into the fine-tuned target content recognition model for content recognition.

[0075] Please see Figure 6 In some embodiments, step S106 includes, but is not limited to, steps S601 to S603: Step S601: Based on the target content recognition model and element category, perform content recognition on the target elements in the target document image to obtain preliminary content recognition information; Step S602: Perform a similarity assessment based on the verified content recognition information and the preliminary content recognition information to obtain the content similarity. Step S603: Optimize the output of the target content recognition model based on content similarity and verification content recognition information to obtain target content recognition information.

[0076] In step S601 of some embodiments, the element category can assist the target content recognition model in quickly locating target elements on the target document image, and then specifically identifying the target elements in the target document image to output preliminary content recognition information. It should be noted that since the target content recognition model is obtained after fine-tuning a large language model, the output preliminary content recognition information also has a certain degree of accuracy. However, in order to output more accurate target content recognition information, it is still necessary to add a content recognition expert model to assist the target content recognition model in reasoning out more accurate target content recognition information.

[0077] In step S602 of some embodiments, content similarity is used to characterize the similarity between the verification content recognition information and the preliminary content recognition information. It should be noted that at least one verification content recognition information is provided, and it originates from at least one content recognition expert model to assist in the output of correct content recognition information.

[0078] Specifically, if at least one verification content identification information is defined as Let n be the number of content recognition expert models invoked, and the invocation of these expert models is primarily determined by the feature category of the target element, with n ≤ 3. Content similarity is calculated as follows: SScore=

[0079] SScore represents content similarity. This provides initial content identification information.

[0080] In step S603 of some embodiments, the output optimization of the target content recognition model mainly involves first constructing a content recognition inference chain from the preliminary content recognition information output by the target content recognition model, and then determining the preliminary content recognition information in the content recognition inference chain as the target content recognition information when the content similarity is the highest, so as to improve the accuracy of content recognition.

[0081] In steps S601 to S603 of this embodiment, during the application of the target content recognition model, not only is content recognition performed based on the enhanced target document image, but also, during the recognition process, verification content recognition information output by the content recognition expert model is added as a reasoning reference for the recognized content, gradually inferring target content recognition information with higher accuracy, thereby improving the accuracy and coherence of content recognition of the document image.

[0082] Please see Figure 7 In some embodiments, step S603 may include, but is not limited to, steps S701 to S702: Step S701: If the content similarity is greater than or equal to the preset second similarity, the preliminary content recognition information output by the target content recognition model is used as the target content recognition information. Step S702: If the content similarity is less than the preset second similarity, perform difference localization on the preliminary content recognition information based on the verification content recognition information to obtain the difference content, and correct the output of the target content recognition model based on the difference content to obtain the target content recognition information.

[0083] In step S701 of some embodiments, the second similarity is used to measure whether the result output by the target content recognition model meets the accuracy requirement. Therefore, when the content similarity is greater than or equal to the second similarity, it indicates that the preliminary content recognition information output by the target content recognition model can be used as the target content recognition information. Conversely, if the content similarity is less than the second similarity, the output of the target content recognition model needs to be optimized, and the preliminary content recognition information is updated step by step until the updated content similarity is greater than or equal to the second similarity. At this point, the output optimization stops, and the preliminary content recognition information output at the last time is used as the target content recognition information.

[0084] In step S702 of some embodiments, if the content similarity is less than the second similarity, the reflection mechanism of the target content recognition model is directly triggered. Specifically, the reflection mechanism of the target content recognition model involves locating the differences between the preliminary content recognition information and the verified content recognition information, generating correction instructions based on the differences, and correcting the output of the target content recognition model according to the correction instructions. Specifically, the target content recognition model will consider and infer new preliminary content recognition information based on the correction instructions, iteratively completing the reflection mechanism until the content similarity is greater than or equal to the second similarity, that is, the final output preliminary content recognition information is used as the target content recognition information.

[0085] Please refer to Figure 8 The embodiments of this application consist of a content recognition platform and at least one expert user terminal. The overall process is divided into a large language model fine-tuning into a target content recognition model, then inputting the image-enhanced target document image into the target content recognition model, and the target content recognition model jointly completing the content recognition inference with the content recognition expert model from the expert user terminal during the content recognition process, so as to output target content recognition information with higher accuracy.

[0086] Specifically, during the fine-tuning of the large language model, initial document images are pre-collected from open-source image sets and / or business image libraries. Semantic segmentation, noise region removal, layer filtering, missing region imputation, and super-resolution reconstruction are used to enhance these initial document images, resulting in training document images with higher resolution and more complete content. Simultaneously, a score is determined by combining the reference content recognition information from the training document images, expert content recognition information, and the model content recognition information output by the large language model based on the training document images. A reward function is then determined based on this score, and the large language model is fine-tuned using this reward function to obtain the target content recognition model.

[0087] When performing document image content recognition, the original document image is first subjected to semantic segmentation, noise region removal, layer filtering, missing region filling, and super-resolution reconstruction to obtain the target document image. This target document image is then input into a content recognition expert model to determine verification content recognition information. The target content recognition model identifies the target elements of the target document image based on element categories. For example, if the target element is text, the text in the target document image is recognized to obtain preliminary content recognition information. After each output of preliminary content recognition information, the target content recognition model calculates a similarity score with the verification content recognition information. If the content similarity score is less than a second similarity score, the differences between the preliminary and verification content recognition information are identified. Based on these differences, correction instructions are determined for the target content recognition model. The output of the target content recognition model is adjusted according to these correction instructions to update the preliminary content recognition information until the updated content similarity score is greater than or equal to the second similarity score. The updated preliminary content recognition information is then used as the target content recognition information, achieving accurate document image recognition.

[0088] In summary, during the original document image enhancement process, a layer cleansing method based on semantic segmentation is employed, and interfering elements are removed through layer association graphs to construct a higher-quality target document image. Simultaneously, during the fine-tuning of the large language model into a target content recognition model, a content recognition expert model is introduced, constructing a reinforcement learning framework that includes logical consistency evaluation to improve the large language model's recognition performance in specific domains. Furthermore, in the content recognition process, a collaborative verification process is implemented between the content recognition expert model and the target content recognition model, achieving complementary advantages between the two, effectively suppressing the illusion problem of the large language model, and improving the accuracy of content recognition.

[0089] Please see Figure 9 This application also provides a document image content recognition device that can implement the above-described document image content recognition method. The device includes: Module 901 is used to acquire the original document image; The classification module 902 is used to classify candidate elements in the original document image to obtain the element category of each candidate element. Image enhancement module 903 is used to perform image enhancement processing on the original document image to obtain the target document image; The expert recognition module 904 is used to input the target document image into a preset content recognition expert model for content recognition and obtain verification content recognition information. The model recognition module 905 is used to perform content recognition on target elements in a target document image by using a preset target content recognition model, element category and verification content recognition information to obtain target content recognition information; wherein, candidate elements include target elements.

[0090] The specific implementation of the document image content recognition device is basically the same as the specific implementation of the document image content recognition method described above, and will not be repeated here.

[0091] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described document image content recognition method. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0092] Please see Figure 10 , Figure 10 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the document image content recognition method of the embodiments of this application. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0093] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described document image content recognition method.

[0094] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0096] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0099] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar elements and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0100] It should be understood that in this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the relationship between related elements, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related elements are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0102] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0105] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method of content recognition of a document image, characterized by, The method comprises: obtaining an original document image; classifying candidate elements in the original document image to obtain an element category of each candidate element; performing image enhancement processing on the original document image to obtain a target document image; inputting the target document image into a preset content recognition expert model to perform content recognition, and obtaining verification content recognition information; performing content recognition on a target element in the target document image through a preset target content recognition model, the element category, and the verification content recognition information to obtain target content recognition information; wherein the candidate element includes the target element.

2. The method of claim 1, wherein, The image enhancement processing on the original document image to obtain a target document image comprises: performing mask processing on the original document image to obtain original image mask data; performing noise region removal on the original document image according to the original image mask data to obtain a preliminary document image; performing feature extraction on each layer of the preliminary document image to obtain layer features; deleting layers different from the target element in the preliminary document image according to the layer features to obtain a candidate document image; performing super-resolution reconstruction on the candidate document image to obtain the target document image.

3. The method of claim 2, wherein, The deleting layers different from the target element in the preliminary document image according to the layer features to obtain a candidate document image comprises: performing graph construction according to the layer features to obtain a layer association graph; wherein the layer association graph includes candidate nodes and target nodes, the candidate nodes represent candidate layers, and the target nodes represent layers of the target element; performing similarity calculation on the candidate nodes and the target nodes according to the layer association graph to obtain node similarity; deleting the candidate layers with a node similarity lower than a preset first similarity as selected layers; deleting the selected layers in the preliminary document image to obtain the candidate document image.

4. The method according to any one of claims 1 to 3, characterized in that, Before the content recognition on the target element in the target document image through the preset target content recognition model, the element category, and the verification content recognition information to obtain target content recognition information, the method further comprises: constructing the target content recognition model, specifically comprising: obtaining a training data set; wherein the training data set includes: a training document image, an identification task category of the training document image, expert content recognition information, and reference content recognition information, the expert content recognition information being obtained by the content recognition expert model according to content recognition of the training document image; performing content recognition on the training document image through a preset large language model and the identification task category to obtain model content recognition information; performing model fine-tuning on the large language model according to the identification task category, the model content recognition information, the expert content recognition information, and the reference content recognition information to obtain the target content recognition model.

5. The method of claim 4, wherein, The model fine-tuning of the large language model according to the identification task category, the model content identification information, the expert content identification information, and the reference content identification information obtains the target content identification model, and the model fine-tuning comprises the following steps: determining a task weight according to the identification task category; evaluating a matching degree of the model content identification information, the expert content identification information, and the reference content identification information to obtain an identification matching degree; constructing model optimization data according to the task weight and the identification matching degree; fine-tuning the large language model according to the model optimization data to obtain the target content identification model.

6. The method according to any one of claims 1 to 3, characterized in that, The content identification of the target element in the target document image by the preset target content identification model, the element category, and the verification content identification information obtains target content identification information, and the content identification comprises the following steps: identifying the content of the target element in the target document image according to the target content identification model and the element category to obtain preliminary content identification information; evaluating a similarity degree according to the verification content identification information and the preliminary content identification information to obtain a content similarity degree; optimizing the output of the target content identification model according to the content similarity degree and the verification content identification information to obtain the target content identification information.

7. The method of claim 6, wherein, The output optimization of the target content identification model according to the content similarity degree and the verification content identification information to obtain the target content identification information comprises the following steps: if the content similarity degree is greater than or equal to a preset second similarity degree, the preliminary content identification information output by the target content identification model is taken as the target content identification information; if the content similarity degree is less than the preset second similarity degree, the preliminary content identification information is positioned according to the difference between the verification content identification information to obtain difference content, and the target content identification model is corrected according to the difference content to obtain the target content identification information.

8. An apparatus for recognizing the contents of a document image, characterized by comprising: The device comprises: an acquisition module configured to acquire an original document image; a classification module configured to perform classification processing on candidate elements in the original document image to obtain an element category of each candidate element; an image enhancement module configured to perform image enhancement processing on the original document image to obtain a target document image; an expert identification module configured to input the target document image into a preset content identification expert model to perform content identification and obtain verification content identification information; a model identification module configured to identify the content of a target element in the target document image by a preset target content identification model, an element category, and the verification content identification information to obtain target content identification information; wherein the candidate elements comprise the target element.

9. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the document image content identification method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the document image content identification method of any one of claims 1 to 7.

Images