A text recognition method, device and computer readable recording medium

Abstract

The application discloses a text recognition method and device and a recording medium, wherein the text recognition method comprises: batch acquiring sample images; performing multi-source labeling and screening processing on each sample image to generate corresponding structured labeled text, so as to construct a training data set; fine-tuning a pre-trained multi-modal large model containing a visual encoder and a text decoder by using the training data set to obtain a target text recognition model; inputting a target image to be recognized into the target text recognition model to output corresponding structured text; wherein the multi-source labeling and screening processing comprises: calling at least two pre-trained text recognition models with different architectures to obtain initial text recognition results; comparing and screening based on a consistency evaluation index; and converting candidate text into structured labeled text conforming to a preset business format by using a format correction model. The application realizes high-precision and end-to-end structured text recognition, and significantly reduces data labeling and system maintenance costs.

Description

Technical Field

[0001] This invention relates to the field of text image recognition technology, and specifically provides a text recognition method, apparatus, and computer-readable recording medium. Background Technology

[0002] Optical Character Recognition (OCR) technology, as an important branch of computer vision, aims to convert text information in images into editable and searchable text data, and is widely used in various scenarios such as document digitization, invoice processing, and industrial quality inspection. With the development of deep learning technology, whole-image text recognition technology has mainly evolved into the following two technical paths.

[0003] The first type of technical approach is a two-stage pipeline method based on "detection-recognition". This method first uses a text detection model (such as DBNet, PSENet, etc.) to scan the entire document image and locate the positions of all text lines (i.e., detection boxes). Then, the image blocks within each detection box are segmented and fed into independent text recognition models (such as CRNN, SVTR, etc.) for character sequence recognition. Finally, a complex post-processing strategy combines all the identified fragmented text lines into a complete text with a logical structure. While this type of method is flexible and easy to adjust for specific scenarios, its fundamental flaw lies in the fragmentation of the model and the rigidity of the post-processing strategy. Specifically: (1) The strategy is complex and difficult to maintain: The post-processing strategy usually contains a large number of rules based on human experience to handle different layouts and splicing scenarios. When business requirements change or new document layouts appear, professional technicians need to manually adjust and update them, resulting in high maintenance costs and long iteration cycles; (2) Poor generalization ability: Complex rule sets are often "overfitted" to specific scenarios. For documents with slightly different layouts, the splicing effect will be greatly reduced, resulting in misaligned lines, missing paragraphs, or structural chaos; (3) Lack of global semantic understanding: Since detection and recognition are carried out in isolation, the entire process lacks an understanding of the global layout and semantic information of the document, making it difficult to handle scenarios with cross-column, cross-page, or complex text-image relationships.

[0004] The second type of technical approach is the end-to-end recognition method based on multimodal large models. In recent years, with the breakthroughs made by large language models in natural language understanding and generation, multimodal large models with image understanding capabilities (such as GPT-4V, Qwen-VL, etc.) have brought a new paradigm to whole-image document recognition. These models can directly take the entire image as input and, by utilizing their powerful image-text understanding and contextual reasoning capabilities, output complete recognition results in one go. However, this end-to-end approach has also exposed new problems in practical applications. First, when general multimodal large models are directly used for OCR tasks, the results are not ideal. The reason is that these general models are designed for broad image understanding and are not specifically optimized for high-precision character-level recognition. Therefore, they perform poorly in text recognition accuracy, with common errors and omissions, failing to meet the stringent accuracy requirements of commercial applications. Second, research shows that multimodal large models suffer from a "text center bias" problem, that is, they over-rely on prior language knowledge in the multimodal understanding process and fail to fully utilize visual information, resulting in insufficient perception of fine-grained visual features. Furthermore, most existing large models are trained on general PDF scenario data, which results in poor performance in specific vertical fields (such as medical reports, industrial manuals, legal documents, etc.), with "understanding gaps" in domain terminology, and the output format often does not match the needs of downstream business systems, still requiring complex post-processing steps.

[0005] In summary, current technologies lack a comprehensive image-to-text recognition solution that can simultaneously achieve high-precision recognition, scene adaptation capabilities, and structured output. How to generate high-quality labeled data for specific business scenarios at low cost and high efficiency, and how to effectively fine-tune large multimodal models to accurately adapt them to the needs of vertical industries, are technical problems that urgently need to be solved by those skilled in the art.

[0006] In view of this, this invention patent is hereby proposed. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a text recognition method, apparatus, and computer-readable recording medium, specifically employing the following technical solutions: In a first aspect, the present invention provides a text recognition method, comprising: Batch acquisition of sample images; Multi-source annotation and filtering are performed on each of the sample images to generate structured annotation text corresponding to the sample images, so as to construct a training dataset containing image-text pairs; Using the training dataset, the pre-trained multimodal large model containing a visual encoder and a text decoder is fine-tuned to obtain a target text recognition model. The target image to be identified is input into the target text recognition model, which generates and outputs structured text corresponding to the target image. The multi-source annotation and filtering process includes: Call at least two pre-trained multi-source text recognition models with different architectures to recognize the same sample image and obtain at least two initial text recognition results; Based on the preset consistency evaluation index, the at least two initial text recognition results are compared pairwise to select texts that meet the consistency conditions as candidate texts. The candidate text is converted into structured annotated text that conforms to a preset business format using a trained format correction model.

[0008] As an optional embodiment of the present invention, in one text recognition method of the present invention, the step of comparing the at least two initial text recognition results pairwise based on a preset consistency evaluation index to select texts that meet the consistency conditions as candidate texts includes: Calculate the edit distance d between the first initial text recognition result and the second initial text recognition result; Based on the edit distance d and the text length L of the first initial text recognition result or the second initial text recognition result, calculate the normalized consistency score s = 1 - d / L; Determine whether the consistency score s is greater than a preset similarity threshold T, where the value of T ranges from 0.8 to 0.95. If s>T, then the confidence scores of the first initial text recognition result and the second initial text recognition result are further compared, and the one with the higher confidence score is determined as the candidate text; If s≤T, then discard both recognition results, or re-perform recognition on the current sample image.

[0009] As an optional embodiment of the present invention, in one text recognition method of the present invention, the pre-trained multimodal large model including a visual encoder and a text decoder is fine-tuned using the training dataset to obtain a target text recognition model, including: A two-stage training strategy is used to fine-tune the pre-trained multimodal large model: Phase 1: Using publicly available large-scale OCR datasets, perform full fine-tuning on the pre-trained multimodal large model to obtain an intermediate model with general text recognition capabilities; The second stage involves using the training dataset to perform full fine-tuning on the intermediate model to obtain a target text recognition model adapted to specific business scenarios.

[0010] As an optional embodiment of the present invention, in one text recognition method of the present invention, when performing full fine-tuning of the intermediate model in the second stage, the method includes: The sample images from the image-text pairs are input into the visual encoder to extract image feature vectors. ; The preset structured prompts are input into the text decoder to extract text feature vectors. ; The image feature vector With the text feature vector Cross-attention fusion is performed at the feature fusion layer to obtain multimodal fused features. ; The multimodal fusion features The text is input into the text decoder to generate predicted text corresponding to the structured cue word format; Based on the predicted text and the structured labeled text in the image-text pair, a loss function is calculated, and the parameters of the intermediate model are updated according to the loss function.

[0011] As an optional embodiment of the present invention, in one text recognition method of the present invention, the loss function L_total is a weighted sum of multiple loss functions, expressed as: ; in, For text generation loss based on cross-entropy, , Let T be the t-th predicted character, and T be the sequence length; For sequence alignment loss based on edit distance, , To predict the edit distance between the sequence and the true sequence, The maximum length of the sequence; The contrast learning loss is used to compare visual features with textual features. sim(·) is the cosine similarity function, τ is the temperature coefficient, and N is the number of samples in the batch; , , The corresponding balance weight coefficients satisfy... + + =1 and > > .

[0012] As an optional embodiment of the present invention, in one text recognition method of the present invention, the at least two pre-trained multi-source text recognition models with different architectures include: A small text recognition model based on a detection-recognition framework is used to acquire pixel-level fine-grained text content. The small text recognition model includes a DBNet detection module and a CRNN recognition module; and An open-source multimodal large language model based on the Transformer architecture, used to obtain global text content based on semantic understanding.

[0013] As an optional embodiment of the present invention, in one text recognition method of the present invention, the format correction model is a large language model fine-tuned by instructions, and the step of using the trained format correction model to convert the candidate text into structured labeled text that conforms to a preset business format includes: The candidate text is concatenated with a preset format conversion prompt to construct a format correction instruction, which is then input into the format correction model. Obtain the structured annotated text that conforms to the preset business format output by the format correction model; The preset business format includes one of JSON, XML, or Markdown formats, and the structured annotation text contains text content and its corresponding layout attribute information.

[0014] As an optional embodiment of the present invention, a text recognition method of the present invention further includes a preprocessing step for the sample image: The batch of acquired sample images are screened for quality, and sample images with blur levels higher than the preset blur threshold or with image damage are removed. Image enhancement processing is performed on the sample images that have passed the quality screening. The image enhancement processing includes at least one of resolution normalization, contrast adjustment and tilt correction.

[0015] In a second aspect, the present invention provides a text recognition device, comprising: The sample acquisition module is used to acquire sample images in batches. The data generation module is used to perform multi-source annotation and filtering processing on each of the sample images to generate structured annotation text corresponding to the sample images, so as to construct a training dataset containing image-text pairs. The training module is used to fine-tune a pre-trained multimodal large model containing a visual encoder and a text decoder using the training dataset to obtain a target text recognition model. The recognition module is used to input the target image to be recognized into the target text recognition model, generate and output structured text corresponding to the target image; The multi-source annotation and filtering process includes: Call at least two pre-trained multi-source text recognition models with different architectures to recognize the same sample image and obtain at least two initial text recognition results; Based on the preset consistency evaluation index, the at least two initial text recognition results are compared pairwise to select texts that meet the consistency conditions as candidate texts. The candidate text is converted into structured annotated text that conforms to a preset business format using a trained format correction model.

[0016] In a third aspect, the present invention provides a computer-readable recording medium storing a computer program that, when executed by a processor, implements the aforementioned text recognition method.

[0017] The present invention achieves the following beneficial technical effects through the above technical solution: First, at the data level, this invention achieves automated and low-cost construction of high-quality training data through multi-source annotation and filtering.

[0018] To address the technical pain points of scarce and costly labeled data in specific business scenarios, this invention designs an innovative multi-source labeling strategy. By calling at least two pre-trained multi-source text recognition models with different architectures (such as a detection-recognition small model and a multimodal large language model) to recognize the same sample image, multiple initial text recognition results are obtained by leveraging the complementary advantages of different model architectures. Based on this, a consistency evaluation metric based on edit distance is introduced to perform pairwise comparisons and consistency filtering on the multiple recognition results, effectively eliminating noisy data and erroneous recognition results, and selecting high-confidence candidate texts. Finally, a format correction model finely tuned by instructions is used to convert the candidate texts into structured labeled text that fully conforms to the preset business format. The entire process requires no manual intervention and can automatically generate high-quality "image-structured text" training pairs for massive sample images, significantly reducing data construction costs while ensuring the accuracy and format standardization of the training data.

[0019] Second, at the model training level, this invention adopts a two-stage training strategy, enabling the target text recognition model to have both general recognition capabilities and scene adaptability.

[0020] The first stage utilizes publicly available large-scale OCR datasets to perform full fine-tuning of the pre-trained multimodal model, enabling it to achieve stable and robust general text recognition capabilities. This addresses the issue of insufficient accuracy when directly applying general-purpose models to OCR tasks. The second stage uses the scenario-specific training datasets generated in the first stage for full fine-tuning, allowing the model to deeply adapt to the layout features, terminology, and output format requirements of specific business scenarios. This "general-first, then specific" training strategy avoids the massive data and computational overhead required to train a large model from scratch, while ensuring high-precision performance in vertical domains, achieving an optimal balance between recognition performance and resource consumption.

[0021] Third, at the model inference level, this invention significantly improves the accuracy of text recognition and structural alignment capability through multimodal feature fusion and multi-task joint optimization.

[0022] During model fine-tuning, this invention introduces a cross-attention mechanism to deeply fuse image and text features, enabling the model to fully capture fine-grained visual information in images. This overcomes the "text center bias" problem common in large multimodal models and improves robustness in recognizing complex layouts, slanted text, and handwritten characters. Simultaneously, a multi-objective loss function, including cross-entropy loss, edit distance loss, and contrastive learning loss, is designed to jointly optimize the model from three dimensions: character-level accuracy, sequence-level alignment, and image-text semantic consistency. Edit distance loss directly optimizes sequence-level similarity, making the model output more structurally aligned with the real text; contrastive learning loss strengthens the semantic association between visual features and corresponding text features, further enhancing the model's accurate understanding of image content. Experiments show that the optimized target text recognition model achieves over 15% higher text recognition accuracy in vertical domains compared to general large multimodal models, and the output text fully meets the structured format requirements of downstream business systems.

[0023] Fourth, at the system application level, this invention achieves end-to-end structured text recognition, significantly reducing the maintenance cost of business systems.

[0024] Compared to traditional pipelined "detection-recognition-post-processing" methods, the target text recognition model trained in this invention can directly take a whole image as input and output structured text that meets business requirements end-to-end, eliminating the need for writing and maintaining complex post-processing rules. When business scenarios or layouts change, only the training dataset needs to be updated and the model incrementally fine-tuned, without the need for manual adjustment of rule strategies, greatly improving the system's adaptability and maintenance efficiency. In addition, the model is fine-tuned based on a lightweight pre-trained large model (such as Qwen-VL 2B), maintaining a small model size while ensuring high accuracy. It has the advantages of fast inference speed and flexible deployment, and can meet the real-time and resource consumption requirements of actual business scenarios.

[0025] In summary, this invention, through its innovative multi-source labeled data generation strategy, two-stage model training method, and multi-task joint optimization mechanism, successfully solves the technical challenges of high cost of labeled data acquisition, weak adaptability of general model scenarios, and complex maintenance of post-processing strategies in existing technologies. It achieves high-precision, high-efficiency, and low-cost whole-image text recognition, demonstrating significant technological progress and broad industrial application value. Attached Figure Description

[0026] Figure 1 A flowchart of a text recognition method according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of the electronic device according to an embodiment of the present invention; Figure 3 A schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0028] Therefore, the following detailed description of embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely illustrates some embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0029] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0031] In the description of this invention, it should be noted that the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. These terms are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] See Figure 1 As shown, this embodiment provides a text recognition method that achieves high-precision, end-to-end structured text recognition through an innovative multi-source labeled data generation strategy, a two-stage model training method, and a multi-task joint optimization mechanism.

[0033] Specifically, a text recognition method according to this embodiment includes the following steps: Step 1: Batch acquire sample images.

[0034] First, batch acquire sample images to be labeled from the business system. These sample images can be document images from specific business scenarios, such as medical reports, industrial equipment manuals, legal documents, and educational question banks. To ensure the diversity and representativeness of subsequent training data, the acquisition of sample images should cover various layout types, lighting conditions, shooting angles, and quality levels within the business scenario.

[0035] Preferably, after acquiring the sample image, a preprocessing step for the sample image is also included: Quality screening: Calculate the blur index (such as Laplacian variance, Tenengrad gradient, etc.) for each sample image and discard images with blur levels exceeding a preset blur threshold; simultaneously, detect image corruption (such as corrupted file headers, decoding failures, etc.) and discard corrupted images. This quality screening ensures that the sample images entering subsequent processing have basic usability, preventing low-quality images from contaminating the training data.

[0036] Image enhancement: Image enhancement processing is performed on sample images that have passed quality screening, including: normalizing the image resolution to a preset size (e.g., 1024×768); adjusting contrast using histogram equalization or adaptive gamma correction; and detecting image tilt angles and performing rotation correction based on Hough transform or deep learning detection methods. Image enhancement processing can improve image quality and enhance the robustness of subsequent recognition models.

[0037] Step 2: Perform multi-source annotation and filtering on each sample image to generate structured annotation text corresponding to the sample image.

[0038] This step automatically generates high-quality structured labeled text through multi-source annotation and filtering to construct a training dataset containing image-text pairs. Specifically, it includes the following sub-steps: Step 2.1: Call at least two pre-trained multi-source text recognition models with different architectures to recognize the same sample image, and obtain at least two initial text recognition results. To fully leverage the complementary advantages of different architectural models, this embodiment utilizes two pre-trained text recognition models with different architectures: This small text recognition model is based on a detection-recognition framework. Specifically, it uses DBNet as the text detection module to locate text line regions in an image, and CRNN as the text recognition module to perform character sequence recognition on the detected text line regions. This type of model excels at acquiring fine-grained text content at the pixel level, is highly sensitive to character shape and font variations, and can accurately recognize every character in an image.

[0039] Open-source multimodal large language models based on the Transformer architecture: specifically employing models such as Qwen-VL and GPT-4V, directly taking the entire image as input, and leveraging their powerful image-text understanding and contextual reasoning capabilities to output the text content of the entire image at once. These models excel at acquiring global text content based on semantic understanding, and can reasonably infer ambiguous or incomplete characters based on context, and understand the overall layout and structure of a document.

[0040] For each sample image, the two types of models mentioned above are called to perform recognition, and two initial text recognition results are obtained, denoted as R_small (small model result) and R_large (large model result).

[0041] Step 2.2: Based on the preset consistency evaluation index, compare at least two initial text recognition results pairwise and select texts that meet the consistency conditions as candidate texts.

[0042] To filter out high-confidence text from multiple initial results, this embodiment introduces a consistency evaluation metric based on edit distance. The specific process is as follows: Calculate the Levenshtein Distance (d) between the first initial text recognition result R_small and the second initial text recognition result R_large. The edit distance measures the minimum number of editing operations (including insertion, deletion, and replacement) required to transform one string into another, and can effectively reflect the similarity between two text sequences.

[0043] Based on the edit distance d and the text length L (which can be either Rsmall or Rlarge, or the average of both), calculate the normalized consistency score s = 1 - d / L. The consistency score s ranges from [0,1], with values ​​closer to 1 indicating greater consistency between the two recognition results.

[0044] The system determines whether the consistency score *s* is greater than a preset similarity threshold *T*. In this embodiment, the threshold *T* ranges from 0.8 to 0.95, and the specific value can be adjusted according to the data accuracy requirements of the business scenario. For example, in an educational question bank scenario, where the accuracy of text is required to be extremely high, *T* can be set to 0.95; in a general document archiving scenario, *T* can be set to 0.85.

[0045] If s > T, it indicates that the recognition results of the two models are highly consistent, and the text has a high confidence level. In this case, the confidence scores of R_small and R_large are further compared (models typically include the confidence score for each character or the entire sequence when outputting text), and the one with the higher confidence score is selected as the candidate text. This optimization mechanism can select the better quality from two highly consistent results, further improving data quality.

[0046] If s ≤ T, it indicates that there is a significant difference in the recognition results of the two models, and at least one of the models may have made a mistake. In this case, discard both recognition results, or re-perform the recognition on the current sample image (e.g., change the model parameters or re-acquire the image).

[0047] Step 2.3: Use the trained format correction model to convert the candidate text into structured labeled text that conforms to the preset business format.

[0048] Even if the candidate text is accurate in content, its format may still not match the requirements of downstream business systems. Therefore, this step introduces a format correction model: The format correction model is a large language model (LLM) fine-tuned by instructions, such as ChatGLM or LLaMA. This model is specially trained to understand and execute format conversion instructions.

[0049] The candidate text is concatenated with a preset format conversion prompt to construct a format correction instruction, which is then input into the format correction model. For example, if the business system requires output in JSON format, the prompt could be: "Please convert the following text to JSON format, containing three fields: 'Title', 'Author', and 'Body':\n{Candidate Text}".

[0050] Retrieves structured annotated text output by the format correction model that conforms to a preset business format. The preset business format includes, but is not limited to, JSON, XML, or Markdown formats. The structured annotated text contains not only the text content but also its corresponding layout attributes, such as heading levels, paragraph divisions, table structures, and list identifiers.

[0051] Through the multi-source annotation and filtering processes described above, this embodiment can automatically generate high-quality "image-structured labeled text" training pairs for massive sample images, thus constructing a training dataset. This process eliminates the need for manual annotation, significantly reducing data construction costs while ensuring the accuracy and format standardization of the training data.

[0052] Step 3: Using the training dataset, fine-tune the pre-trained multimodal large model containing the visual encoder and text decoder to obtain the target text recognition model.

[0053] This step employs a two-stage training strategy to fine-tune the model, enabling it to possess both general recognition capabilities and scene adaptability.

[0054] Step 3.1: First stage – Fine-tuning of general capabilities.

[0055] We utilize publicly available large-scale OCR datasets (such as SynthText, MJSynth, and COCO-Text) to perform full fine-tuning on a pre-trained multimodal large model. The pre-trained multimodal large model specifically refers to a model containing a visual encoder (such as ViT) and a text decoder (such as a Transformer decoder), for example, Qwen-VL 2B. In the first stage of training, the model learns general character recognition capabilities, language modeling capabilities, and image-text alignment capabilities through a large amount of diverse OCR data, obtaining an intermediate model with stable general text recognition capabilities.

[0056] Step 3.2: Second stage - fine-tuning for scene adaptation.

[0057] Using the training dataset (containing image-text pairs) constructed in step 2, the intermediate model obtained in the first stage is fully fine-tuned to obtain a target text recognition model adapted to specific business scenarios. The fine-tuning process in the second stage is further refined into the following steps: Step 3.2.1: Feature extraction.

[0058] The sample images from the image-text pair are input into the visual encoder to extract the image feature vectors. Visual encoders typically consist of multiple layers of Transformer blocks, and their output is a global feature representation of an image.

[0059] Input the pre-set structured prompts into the text decoder to extract text feature vectors. Structured prompts are used to guide the model to output in a specific format, such as: "Please output the text content in the image in JSON format, including the title and body."

[0060] Step 3.2.2: Multimodal feature fusion.

[0061] Image feature vectors With text feature vectors Cross-attention fusion is performed at the feature fusion layer to obtain multimodal fused features. The cross-attention mechanism allows for sufficient information exchange between image features and text features, enabling the model to focus on relevant regions in the image based on textual cues, while adjusting the semantics of the generated text according to the image content, effectively overcoming the "text center bias" problem common in large multimodal models.

[0062] Step 3.2.3: Text generation.

[0063] Multimodal fusion features The input is fed into a text decoder, which generates predicted text corresponding to the structured cue word format. The text decoder generates output character by character in an autoregressive manner, making predictions at each step based on the generated characters and multimodal fusion features.

[0064] Step 3.2.4: Loss calculation and parameter update.

[0065] Based on the predicted text and the structured labeled text in the image-text pairs, a loss function is calculated, and the parameters of the intermediate model are updated according to the loss function. This embodiment employs a loss function design based on multi-task joint optimization: Cross-entropy loss : ,in Let be the t-th predicted character, and T be the sequence length. This loss is used to generate the correct character sequence from the character-level supervised model.

[0066] Edit distance loss : ,in To predict the edit distance between the sequence and the true sequence, The maximum length of the sequence is denoted by . This loss function directly optimizes sequence-level similarity, making the model output more aligned with the overall structure of the real text, and is particularly suitable for scenarios with long texts and complex structures.

[0067] Comparative learning loss : , where sim(·) is the cosine similarity function, τ is the temperature coefficient, and N is the number of samples in the batch. This loss reduces the distance between image features and their corresponding text features, while increasing the distance between them and the text features of other samples, thus strengthening the semantic connection between visual and textual features.

[0068] Total loss: The balance weight coefficients satisfy + + =1 and > > In other words, character-level precision is prioritized, followed by sequence alignment, and finally, semantic consistency between text and images. Specific values ​​can be adjusted according to task requirements; for example, λ1=0.6, λ2=0.3, and λ3=0.1 can be set.

[0069] Through the above multi-task joint optimization, the model is supervised from three dimensions simultaneously. While ensuring character-level recognition accuracy, it improves sequence alignment ability and image-text semantic understanding ability. The final target text recognition model shows higher recognition accuracy and structural regularity in vertical scenarios.

[0070] Step 4: Input the target image to be recognized into the target text recognition model to generate and output the structured text corresponding to the target image.

[0071] During the model application phase, the target image to be recognized, collected from the business scenario, is directly input into the trained target text recognition model. The model outputs end-to-end structured text corresponding to the target image. This text is organized according to a preset business format and can be directly used by downstream systems without additional post-processing.

[0072] For example, in the context of educational question banks, when inputting an image containing test questions, the model can directly output structured text in JSON format, including fields such as question number, question type, question stem, options, and answer, which greatly improves the efficiency of question bank digitization.

[0073] Through the above technical solution, the text recognition method of this embodiment achieves the following technical effects: At the data level: Multi-source annotation and filtering processes leverage the complementary advantages of different architecture models, combined with consistency filtering and format correction, to automatically generate high-quality, formatted, and structured annotation data, solving the technical challenges of scarce and costly annotation data in specific scenarios.

[0074] At the model level: The two-stage training strategy enables the model to first acquire general recognition capabilities and then adapt to specific scenarios, achieving an optimal balance between recognition performance and resource consumption. Multi-task joint optimization supervises model learning from three dimensions: character-level, sequence-level, and semantic-level, significantly improving recognition accuracy and structural alignment capabilities.

[0075] At the application level: the end-to-end structured output eliminates complex post-processing steps and reduces system maintenance costs; the model is based on lightweight pre-trained large model fine-tuning, with fast inference speed and flexible deployment, which can meet the real-time requirements of actual business scenarios.

[0076] Meanwhile, this embodiment provides a text recognition device for implementing the method described in Embodiment 1. The device includes the following modules: The sample acquisition module is used to acquire sample images in batches. This module can connect to the image acquisition devices or database of the business system and automatically retrieve the sample images to be labeled.

[0077] The data generation module performs multi-source annotation and filtering on each sample image to generate structured labeled text corresponding to the sample images, thus constructing a training dataset containing image-text pairs. This module integrates multi-source text recognition model calling interfaces, consistency evaluation units, and format correction model calling interfaces, automating the data generation process.

[0078] The training module is used to fine-tune a pre-trained multimodal model containing a visual encoder and a text decoder using the training dataset to obtain a target text recognition model. This module supports a two-stage training strategy and multi-task loss function optimization, and the training parameters can be configured according to actual needs.

[0079] The recognition module takes the target image to be recognized and inputs it into the target text recognition model, generating and outputting the structured text corresponding to the target image. This module encapsulates the trained model and provides an API interface for business systems to call.

[0080] The specific implementation details of each module correspond to the method steps in Implementation Example 1, and will not be repeated here.

[0081] Figure 2This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. The electronic device includes a processor and a memory. The memory is used to store a computer-executable program. When the computer program is executed by the processor, the processor executes a text recognition method according to the embodiment.

[0082] like Figure 2 As shown, the electronic device is embodied in the form of a general-purpose computing device. There can be one or more processors working collaboratively. This invention also does not preclude distributed processing, meaning that processors can be distributed across different physical devices. The electronic device of this invention is not limited to a single entity, but can also be the sum of multiple physical devices.

[0083] The memory stores a computer-executable program, typically machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps of the method.

[0084] The memory includes volatile memory, such as random access memory (RAM) and / or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).

[0085] Optionally, in this embodiment, the electronic device further includes an I / O interface for exchanging data with external devices. The I / O interface can represent one or more of several bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0086] It should be understood that Figure 2 The electronic device shown is merely one example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as displays, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. Any electronic device capable of executing a computer-readable program in memory to implement the method of the present invention or at least some steps of the method can be considered as an electronic device covered by the present invention.

[0087] Figure 3 This is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. Figure 3As shown, a computer-readable recording medium stores a computer-executable program, which, when executed, implements a text recognition method according to an embodiment of the present invention. The computer-readable recording medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable recording medium may also be any readable medium other than a readable recording medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable recording medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0088] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0089] From the above description of the embodiments, those skilled in the art will readily understand that the present invention can be implemented by hardware capable of executing specific computer programs, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. included in the system. The present invention can also be implemented by computer software executing the methods of the present invention, for example, by control software executed by a microprocessor, electronic control unit, client, server, etc. However, it should be noted that the computer software executing the methods of the present invention is not limited to execution in one or a specific set of hardware entities; it can also be implemented in a distributed manner by unspecified hardware. For computer software, the software product can be stored on a computer-readable recording medium (such as a CD-ROM, USB flash drive, portable hard disk, etc.) or distributed across a network, as long as it enables electronic devices to execute the methods according to the present invention.

[0090] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described herein. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the specific embodiments described above. Therefore, any modifications or equivalent substitutions to the present invention, as well as all technical solutions and improvements that do not depart from the spirit and scope of the invention, are covered within the scope of the claims of the present invention.

Claims

1. A text recognition method, characterized in that, include: Batch acquisition of sample images; Multi-source annotation and filtering are performed on each of the sample images to generate structured annotation text corresponding to the sample images, so as to construct a training dataset containing image-text pairs; Using the training dataset, the pre-trained multimodal large model containing a visual encoder and a text decoder is fine-tuned to obtain a target text recognition model. The target image to be identified is input into the target text recognition model, which generates and outputs structured text corresponding to the target image. The multi-source annotation and filtering process includes: Call at least two pre-trained multi-source text recognition models with different architectures to recognize the same sample image and obtain at least two initial text recognition results; Based on the preset consistency evaluation index, the at least two initial text recognition results are compared pairwise to select texts that meet the consistency conditions as candidate texts. The candidate text is converted into structured annotated text that conforms to a preset business format using a trained format correction model.

2. The text recognition method according to claim 1, characterized in that, The method, based on a preset consistency evaluation index, compares the pairwise results of the at least two initial text recognition results to select texts that meet the consistency criteria as candidate texts, including: Calculate the edit distance d between the first initial text recognition result and the second initial text recognition result; Based on the edit distance d and the text length L of the first initial text recognition result or the second initial text recognition result, calculate the normalized consistency score s = 1 - d / L; Determine whether the consistency score s is greater than a preset similarity threshold T, where the value of T ranges from 0.8 to 0.

95. If s>T, then the confidence scores of the first initial text recognition result and the second initial text recognition result are further compared, and the one with the higher confidence score is determined as the candidate text; If s≤T, then discard both recognition results, or re-perform recognition on the current sample image.

3. The text recognition method according to claim 1, characterized in that, Using the training dataset, a pre-trained multimodal large model containing a visual encoder and a text decoder is fine-tuned to obtain a target text recognition model, including: A two-stage training strategy is used to fine-tune the pre-trained multimodal large model: Phase 1: Using publicly available large-scale OCR datasets, perform full fine-tuning on the pre-trained multimodal large model to obtain an intermediate model with general text recognition capabilities; The second stage involves using the training dataset to perform full fine-tuning on the intermediate model to obtain a target text recognition model adapted to specific business scenarios.

4. The text recognition method according to claim 3, characterized in that, The second stage of fine-tuning the intermediate model includes: The sample images from the image-text pairs are input into the visual encoder to extract image feature vectors. ; The preset structured prompts are input into the text decoder to extract text feature vectors. ; The image feature vector With the text feature vector Cross-attention fusion is performed at the feature fusion layer to obtain multimodal fused features. ; The multimodal fusion features The text is input into the text decoder to generate predicted text corresponding to the structured cue word format; Based on the predicted text and the structured labeled text in the image-text pair, a loss function is calculated, and the parameters of the intermediate model are updated according to the loss function.

5. The text recognition method according to claim 4, characterized in that, The loss function L_total is a weighted sum of multiple loss functions, expressed as: ； in, For text generation loss based on cross-entropy, , Let T be the t-th predicted character, and T be the sequence length; For sequence alignment loss based on edit distance, , To predict the edit distance between the sequence and the true sequence, The maximum length of the sequence; The contrast learning loss is used to compare visual features with textual features. sim(·) is the cosine similarity function, τ is the temperature coefficient, and N is the number of samples in the batch; , , The corresponding balance weight coefficients satisfy... + + =1 and > > .

6. The text recognition method according to claim 1, characterized in that, The at least two different pre-trained multi-source text recognition models include: A small text recognition model based on a detection-recognition framework is used to acquire pixel-level fine-grained text content. The small text recognition model includes a DBNet detection module and a CRNN recognition module; and An open-source multimodal large language model based on the Transformer architecture, used to obtain global text content based on semantic understanding.

7. The text recognition method according to claim 1, characterized in that, The format correction model is a large-scale language model fine-tuned by instructions. The process of using the trained format correction model to convert the candidate text into structured, annotated text conforming to a preset business format includes: The candidate text is concatenated with a preset format conversion prompt to construct a format correction instruction, which is then input into the format correction model. Obtain the structured annotated text that conforms to the preset business format output by the format correction model; The preset business format includes one of JSON, XML, or Markdown formats, and the structured annotation text contains text content and its corresponding layout attribute information.

8. The text recognition method according to claim 1, characterized in that, It also includes a preprocessing step for the sample images: The batch of acquired sample images are screened for quality, and sample images with blur levels higher than the preset blur threshold or with image damage are removed. Image enhancement processing is performed on the sample images that have passed the quality screening. The image enhancement processing includes at least one of resolution normalization, contrast adjustment and tilt correction.

9. A text recognition device, characterized in that, include: The sample acquisition module is used to acquire sample images in batches. The data generation module is used to perform multi-source annotation and filtering processing on each of the sample images to generate structured annotation text corresponding to the sample images, so as to construct a training dataset containing image-text pairs. The training module is used to fine-tune a pre-trained multimodal large model containing a visual encoder and a text decoder using the training dataset to obtain a target text recognition model. The recognition module is used to input the target image to be recognized into the target text recognition model, generate and output structured text corresponding to the target image; The multi-source annotation and filtering process includes: Call at least two pre-trained multi-source text recognition models with different architectures to recognize the same sample image and obtain at least two initial text recognition results; Based on the preset consistency evaluation index, the at least two initial text recognition results are compared pairwise to select texts that meet the consistency conditions as candidate texts. The candidate text is converted into structured annotated text that conforms to a preset business format using a trained format correction model.

10. A computer-readable recording medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements a text recognition method as described in any one of claims 1 to 8.

Images