Character recognition method and system based on large model and OCR technology

Abstract

The invention discloses a character recognition method and system based on a large model and an OCR technology, and relates to the technical field of character recognition, and the method comprises the steps: extracting picture information, carrying out the unified preprocessing, generating a text detection box of a character region through a DBNet lightweight text detection model, and obtaining a text position; quickly identifying characters in the textbox by using a lightweight OCR model to obtain text content, and generating an identification information group; carrying out average value calculation on the confidence coefficient of the lightweight OCR model result, and analyzing the overall confidence coefficient; and for the result with low confidence coefficient, inputting the corresponding identification information group into the multi-modal large model, and carrying out secondary identification. According to the method, the text position set and the text character string sequence are combined into the identification information group, so that the effect of structured storage of detection and identification results is achieved, subsequent information retrieval and multi-modal fusion analysis are facilitated, and the effect of optimizing the subsequent processing efficiency and precision is achieved through the combination of the confidence coefficient screening step.

Description

Technical Field

[0001] This invention relates to the field of text recognition technology, and in particular to a text recognition method and system based on large model and OCR technology. Background Technology

[0002] Traditional OCR methods mainly rely on image processing and machine learning techniques to detect and recognize text through steps such as character segmentation, feature extraction, and template matching. However, with the increasing complexity of application scenarios, the performance of these traditional techniques has been severely limited in unstructured text scenarios, low-resolution images, and complex text scenarios. The rise of deep learning technology has driven significant progress in text recognition methods, especially lightweight OCR models, such as CRNN models that combine CTC decoding or lightweight Transformer recognition models based on autoregressive decoding. These models can achieve efficient text detection and recognition in various scenarios with relatively low computational cost. However, the shortcomings of existing text recognition technologies are mainly reflected in the following aspects: On the one hand, traditional OCR methods are prone to inaccurate detection and recognition when faced with complex backgrounds or irregularly formatted text. On the other hand, existing OCR methods rely more on the surface information of characters, words or sentences and fail to combine contextual semantic information with global image information. Therefore, they cannot fully understand the logical semantic relationship of text in the scene when processing semantically confused text and complex scene images. Summary of the Invention

[0003] In view of the aforementioned existing problems, the present invention is proposed.

[0004] Therefore, this invention provides a text recognition method and system based on large model and OCR technology. The shortcomings of existing text recognition technologies are mainly reflected in the following aspects: On the one hand, traditional OCR methods are prone to inaccurate detection and recognition when faced with complex backgrounds or irregularly formatted text. On the other hand, existing OCR methods mostly rely on surface information of characters, words or sentences, and fail to combine contextual semantic information and global image information. Therefore, they cannot fully understand the logical semantic relationship of text in the scene when dealing with semantically confused text and complex scene images.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a text recognition method based on large model and OCR technology, comprising: Image information is extracted and preprocessed uniformly. Lightweight text detection models such as DBNet are used to generate text detection boxes for text regions to obtain text positions. Lightweight OCR models are then used to quickly recognize the text in the text boxes to obtain text content and generate recognition information groups. The confidence scores of the lightweight OCR model results are averaged to analyze the overall confidence score. The text location and text content output by OCR are combined with the embedding vector of the original image to form a multimodal feature dataset, and the global feature vector of the image is extracted to form a multimodal input matrix; A multimodal large model is used to perform semantic error correction on the OCR model recognition results, generate the corrected text content, and store the data securely.

[0006] As a preferred embodiment of the text recognition method based on large model and OCR technology described in this invention, the step of extracting image information and performing unified preprocessing includes: Use the OpenCV image processing library to read image data and obtain the original height H and width W; Perform color space conversion on the image to unify the color space of the image to RGB format; DBNet's input size is usually fixed, requiring the original image size to be scaled proportionally to the corresponding fixed size; For image data, select appropriate methods for noise removal based on different situations; The pixel values ​​of the scaled image are normalized to ensure that the input feature scale is consistent and to avoid detection bias caused by differences in pixel range. Load the pre-trained lightweight text detection model DBNet and input the image data in batches; Preliminary feature extraction of images is performed through multi-layer convolutional networks, and features from different levels are fused from details to the whole. The model first identifies the definite text locations, treats them as the core regions of the text, and marks them on the image. For text edges, the model automatically calculates a threshold to distinguish between text edges and background; By combining the core region and boundary judgment, a binarized image is generated, clearly separating the regions where all text is located; Extract continuous text regions from the binarized image, filter out noise with too small an area, and fit the most suitable box according to the shape of each text region to ensure that the text detection box can tightly surround the text, which is convenient for subsequent OCR recognition. Map the position of the text detection box from the image size after model processing back to the actual coordinates of the original image; For the coordinate bounding box of the text region output by DBNet, each text region is cropped out from the original image according to its coordinates; For text that is tilted or rotated, affine transformation is used to correct its orientation. Based on the input size requirements of CRNN for images, the text image is scaled or filled to the corresponding size proportionally; Since the CRNN model receives grayscale images as input, a grayscale conversion method is needed to convert the input image into a single-channel grayscale image. The text sequence recognition model CRNN is loaded, and image data is input into the convolutional neural network part of CRNN. ​​After multiple layers of convolution and pooling operations, the visual features of the text image are extracted and a feature map is generated. At the same time, the spatial size of the feature map gradually decreases and the number of channels increases to obtain higher-level semantic features. The feature maps output by the convolutional layer are treated as sequential data, segmented along the width direction, and each column of features is used as the input for a time step and fed into the recurrent layer; The output of the recurrent layer is mapped to the character probability space through a fully connected layer, generating a probability distribution covering all candidate characters and whitespace characters at each time step; The CTC decoding algorithm converts the predicted sequence containing whitespace and repeated characters into an actual text string. A single character is defined as Ti, which is output as the final recognition result.

[0007] As a preferred embodiment of the text recognition method based on large model and OCR technology described in this invention, the step of calculating the average confidence level of the lightweight OCR model results and analyzing the overall confidence level includes: Calculate the average confidence score of each character based on the identified text content. The overall confidence level; For each confidence value, the mean and standard deviation are calculated. The mean and standard deviation of the confidence values ​​are dynamically weighted and summed to serve as the judgment threshold. If the calculated actual confidence value is greater than the judgment threshold, the corresponding identification information group is used as the confidence screening output. If the calculated actual confidence value is less than or equal to the judgment threshold, then the judgment confidence is low.

[0008] As a preferred embodiment of the text recognition method based on large model and OCR technology described in this invention, wherein: the extracted global feature vectors of the image form a multimodal input matrix, including, For identification information groups with low confidence, the text location information is normalized, and the text string is used to generate embedding vectors using the pre-trained language model BERT. The embedded vectors and normalized text location information are combined to form a multimodal feature dataset; Using a pre-trained DBNet model with convolutional neural networks, global feature vectors of the image are extracted through multiple convolutions and combined with a multimodal feature dataset and task-oriented prompts to form a multimodal input matrix.

[0009] As a preferred embodiment of the text recognition method based on large model and OCR technology described in this invention, the step of employing a multimodal large model to perform semantic error correction on the OCR model recognition results includes: Multimodal information processing is performed using a pre-trained multimodal large model, and the multimodal input matrix is ​​used as the model input. The model's input interface is divided into image feature branch, text feature branch, and prompt branch. The model interface completes the encoding of data to generate a combined feature map of joint visual and textual features.

[0010] As a preferred embodiment of the text recognition method based on large model and OCR technology described in this invention, wherein: the generated error-corrected text content includes, Text content identified for each low confidence level Based on comprehensive features, error-correcting text content is generated and composed into a sequence of error-correcting text strings.

[0011] As a preferred embodiment of the text recognition method based on large model and OCR technology described in this invention, the step of secure data storage includes: The error-correcting text string sequence is encrypted using the TLS protocol for data transmission, and the transmitted data is symmetrically encrypted using AES-256. The data is then backed up and stored on a cloud server.

[0012] Secondly, this invention provides a text recognition system based on large model and OCR technology, including, Image preprocessing module: performs unified RGB channel conversion, size scaling, noise removal, pixel value normalization, generates binary segmentation map of text and extracts text detection box position information, and performs local cropping, orientation correction, size scaling, conversion to single-channel grayscale image and other operations to provide high-quality image fragments for subsequent OCR processing; Text recognition module: Uses a lightweight OCR model to quickly recognize text information in a cropped image and obtain the text recognition result; Confidence analysis module: Calculates the average confidence level of the OCR model recognition results to evaluate the overall recognition quality; Multimodal feature construction module: Extracts global feature vectors from the image, combines the output text location information and text content, and constructs a multimodal input matrix for large model processing; Semantic error correction module: Uses a multimodal large model to perform semantic error correction on OCR-recognized text and generate corrected text content.

[0013] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the text recognition method based on large model and OCR technology as described in the first aspect of the present invention.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the text recognition method based on large model and OCR technology as described in the first aspect of the present invention.

[0015] The beneficial effects of this invention are as follows: By combining the text location set with the text string sequence into a recognition information group, the structured storage of detection and recognition results is achieved, which facilitates subsequent information retrieval and multimodal fusion analysis. The combination of confidence screening steps optimizes the efficiency and accuracy of subsequent processing. The construction of a multimodal feature dataset enables multi-angle judgment of text information with low confidence, which can effectively compensate for inference biases that may be caused by single semantic or positional features, and improve the accuracy and reliability of semantic error correction of text recognition results. By constructing a multimodal input matrix and combining global features, local features, and task prompts, an overall synergistic optimization effect is achieved, enhancing the model's comprehensive understanding of complex semantics, contextual relationships, and visual spatial information, thereby achieving non-obvious performance improvements in text recognition, error correction, and analysis tasks. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the text recognition method based on large model and OCR technology in Example 1.

[0018] Figure 2 This is a schematic diagram of the technical route of the text recognition method based on large model and OCR technology in Example 1.

[0019] Figure 3 This is a schematic diagram of the text recognition system based on large model and OCR technology in Example 1. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1, referring to Figures 1 to 3 This is the first embodiment of the present invention, which provides a text recognition method based on large model and OCR technology, including the following steps: S1. Extract image information and perform unified preprocessing to generate text detection boxes for each text region and obtain the position of the text detection boxes. Based on the text detection boxes, perform local cropping on the detected text regions and use a lightweight OCR model to quickly perform text recognition on the image. In summary, obtain the text position and text content in the image and generate a recognition information group. Firstly, extract image information for unified preprocessing, including: Extract image data and scale it to a uniform size. Set a maximum side length threshold. For each image, calculate the scaling ratio and the height and width of the scaled image, as follows:

[0024]

[0025]

[0026] in, Indicates the scaling ratio. Indicates the maximum side length. This represents the maximum raw height H and width W of the image data; Height after scaling the image and the width of the image after scaling The new dimensions of the image after scaling are calculated. Since the DBNet network structure has 5 downsampling operations, the new width and height of the input image are kept in the original aspect ratio, while being adjusted to a multiple of 25 = 32, and bilinear interpolation scaling is performed. For a single-channel grayscale image, the R, G, and B channel values ​​can be made the same by copying the grayscale channel, thus converting the grayscale image into a three-channel image. For images with four RGBA channels (including transparency), the goal of unifying image channels can be achieved by removing the alpha channel (taking the first 3 channels directly) or by blending the alpha value with the background. Load the pre-trained lightweight text detection algorithm DBNet to extract features from the input image, thereby generating a binarized image that separates all regions containing text. Using the bounding box algorithm, for n independent text regions, text detection boxes are fitted according to their shapes. The positions of these text detection boxes are then mapped to the original image to obtain their positional information on the image, forming a text position set, represented as:

[0027] Where B represents the text location set. This indicates the position information of the nth text detection box; For each detected text region, it is treated as an image patch. ; Each image block Input a lightweight text recognition model, perform prediction, and output the text content. , represented as:

[0028] in For the identified text content, The text recognition neural network selects a lightweight recognition architecture based on the features of the text region. For printed text, a CRNN recognition model based on CTC decoding can be used, while for handwritten and irregular text, a Transformer recognition model based on autoregressive decoding can be used. All recognition results are stored sequentially as a sequence of text strings. The text location set and the text string sequence form a complete recognition information group of preliminary detection and recognition information, represented as:

[0029] in This represents the nth text string, where n is the total number of characters in the sequence. This indicates a group of identification information.

[0030] By scaling the input data to a uniform size, the robustness of subsequent algorithms to varying resolutions is improved. Effective noise reduction was achieved by using grayscale conversion and Gaussian filtering, reducing the influence of irrelevant interference in the image, providing a cleaner input for the text detection model, and improving detection accuracy. The lightweight text detection algorithm DBNet was used to segment grayscale images, achieving accurate extraction of text regions while maintaining high speed and resource utilization efficiency, making it suitable for real-time text detection needs in various scenarios. By generating text location sets using bounding box algorithms, the complex full image is transformed into multiple independent text regions, which facilitates subsequent block-by-block cropping and individual processing. This block-based method reduces redundant calculations while improving the detection and recognition accuracy of local text. By cropping the ROI of the detected text region and generating separate image patches, the pixel content of the target region is accurately extracted, avoiding interference from redundant information and providing clearer input for the lightweight OCR model. By inputting image patches into lightweight text recognition models (such as CRNN recognition models based on CTC decoding or Transformer recognition models based on autoregressive decoding), efficient and accurate text content recognition is achieved. At the same time, the lightweight design of the model meets the application requirements in resource-constrained environments. By combining text location sets with text string sequences into recognition information groups, the detection and recognition results are stored in a structured manner, which facilitates subsequent information retrieval and multimodal fusion analysis, and fully preserves the correspondence between text location information and content information. By combining the above technologies, an end-to-end optimization effect is achieved, from image noise reduction and efficient text region extraction to accurate text content recognition, significantly improving the overall performance and applicability of the system in text detection and recognition tasks.

[0031] S2, calculate the average confidence level of the lightweight OCR model results and analyze the overall confidence level; Firstly, the confidence scores of the lightweight OCR model results are averaged to analyze the overall confidence score, including... Calculate the average confidence score of each character based on the identified text content. The overall confidence level is expressed as:

[0032] in Indicates the confidence level value. The length of the string representing the recognized text content. This represents the probability of matching the j-th character in the string; For each confidence value, the mean and standard deviation are calculated. The mean and standard deviation of the confidence values ​​are dynamically weighted and summed, and this sum is used as the judgment threshold. This is because the standard deviation fluctuates more drastically when there are few samples of text. When the sample size is small, the weight of the standard deviation can be dynamically reduced to ensure the stability of the judgment threshold. If the calculated actual confidence value is greater than the judgment threshold, the corresponding recognition information group is used as the confidence screening output. If the calculated actual confidence value is less than or equal to the judgment threshold, then the judgment confidence is low.

[0033] By calculating character confidence, a fine-grained evaluation of character-by-character recognition quality is achieved, which can comprehensively determine the distribution of high and low confidence in a string and avoid the excessive impact of single-character errors on the overall confidence. By combining the mean and standard deviation, the judgment threshold is dynamically generated, which can comprehensively consider the overall confidence level and data volatility, making the screening process more robust and more sensitive to anomalies. By using thresholds to filter the identification information groups, low-quality identification results were effectively filtered out, while high-quality information groups were retained, reducing the adverse interference of low-confidence results for subsequent data processing and correction. By combining confidence screening steps, the efficiency and accuracy of subsequent processing are optimized, and limited computing resources are concentrated on reliable data, thereby improving the overall reliability of the system and the performance of task execution.

[0034] S3 combines the text location and text content output by OCR with the embedding vector of the original image to form a multimodal feature dataset, and extracts the global feature vector of the image to form a multimodal input matrix; Firstly, global feature vectors of the image are extracted to form a multimodal input matrix, including, For identification information groups with low confidence, the text location information is normalized, and the text string is used to generate embedding vectors using the pre-trained language model BERT. The embedded vectors and normalized text location information are combined to form a multimodal feature dataset; Using a pre-trained DBNet model with a convolutional neural network (such as ResNet50), global feature vectors of the image are extracted through multiple convolutions. These vectors are then combined with a multimodal feature dataset and task-oriented prompts to form a multimodal input matrix, represented as follows:

[0035] in Represents the multimodal input matrix. Represents a multimodal feature dataset. Represents the global feature vector. Task-oriented prompts.

[0036] By normalizing the text location information, a unified representation of text regions at different scales and locations is achieved, thereby eliminating the inconsistency of location information in terms of coordinate scale and range, making subsequent multimodal feature fusion more coordinated. By using a pre-trained BERT model to generate text embedding vectors, high-quality semantic representation was achieved. This not only captured the semantic information of the text content but also enhanced the ability to understand the context of low-confidence text, laying the foundation for subsequent analysis and error correction. By combining normalized location information with BERT embedding vectors into a multimodal feature dataset, the effect of cross-modal information complementarity is achieved. Location features provide spatial constraints while semantic embedding captures language retrieval capabilities. The combination of the two enhances the system's ability to jointly model text content and spatial relationships. By constructing a multimodal feature dataset, we have achieved the effect of judging text information with low confidence from multiple perspectives. This can effectively compensate for the inference bias that may be caused by single semantic or positional features, and improve the accuracy and reliability of semantic error correction of text recognition results.

[0037] By using convolutional neural networks (such as ResNet50) in a pre-trained DBNet model to extract global feature vectors, the overall information of the image is captured, enabling the model to understand the global context of the image and supplement the visual background information that may be missed by local text and location features. By combining global feature vectors with multimodal feature datasets, the model achieves the effect of jointly modeling the overall semantics and local details. Global information provides background context, while local information provides accurate text and location information. The two complement each other and enhance the recognition depth of the model. By introducing task-oriented prompts to construct the input matrix, the multimodal features are processed in a task-specific manner, making the model more focused on specific task requirements, thereby improving the inference efficiency and accuracy on the target task. By constructing a multimodal input matrix that combines global features, local features, and task prompts, an overall synergistic optimization effect is achieved, enhancing the model's comprehensive understanding of complex semantics, contextual relationships, and visual spatial information, thereby achieving non-obvious performance improvements in text recognition, error correction, and analysis tasks.

[0038] S4 employs a multimodal large model to perform semantic error correction on the OCR model recognition results, generate the corrected text content, and store the data securely. Prioritize the use of a multimodal large model to perform semantic error correction on the OCR model recognition results, including: Multimodal information processing is performed using a pre-trained multimodal large model (such as BLIP2, CLIP, or LLaVA), and the multimodal input matrix is ​​used as the model input. The model's input interface is divided into image feature branch, text feature branch, and cue branch. The image feature branch is used to understand the visual information of the global context, the text feature branch is used to parse the information of each text region and its content, and the cue branch is used to constrain the optimization direction of the model for a specific task. The model interface completes the encoding of data to generate a combined feature map of joint visual and textual features.

[0039] By using pre-trained multimodal large models (such as BLIP2, CLIP, or LLaVA), the effect of large-scale visual-language knowledge transfer is achieved, enabling the system to enhance its understanding and processing capabilities of multimodal information by utilizing rich pre-training data; By inputting global visual information through image feature branches, the model achieves the effect of capturing the overall context of the scene, enabling it to analyze the global context of the image and help locate the relationship between visual cues and text content. By inputting text regions and content information through text feature branches, the system achieves accurate parsing of local text semantics, ensuring that the text content matches the scene information and laying the foundation for the fusion of multimodal features. By adding task-oriented information through prompt branches, the model's inference direction is constrained, enabling the model to focus on optimizing the current target task and thus more efficiently achieve specific task requirements, such as semantic error correction or text reconstruction. The multimodal input matrix is ​​encoded through the model interface to generate a comprehensive feature of the joint visual-text feature map, achieving the effect of deep fusion of cross-modal information. The model can use the bidirectional complementarity of visual and textual information to jointly model the semantics and distribution of text in complex scenes. By comprehensively generating multimodal features, the model's inference depth and detail accuracy in multimodal tasks are improved. At the same time, the problem of information fragmentation that may be caused by independent processing of vision and text is solved, effectively enhancing the model's global contextual understanding and task efficiency.

[0040] Furthermore, generate error-correcting text content, including, Text content identified for each low confidence level Based on comprehensive features, error-correcting text content is generated and composed into a sequence of error-correcting text strings.

[0041] By correcting text content with low confidence based on comprehensive features, the effect of using multimodal semantic information to correct errors is achieved, which can make up for single-modal errors caused by ambiguous characters or recognition noise. By combining the global visual context of integrated features with the common reference of text content, the effect of contextual constraints on low-confidence text content is achieved, enabling the model to select more reasonable alternative text based on logical consistency in the scene. By generating error-correcting text string sequences, a systematic repair effect on low-confidence text is achieved, while preserving the judgment criteria for multimodal information, thus improving the completeness and accuracy of the final output results. Through the above processing flow, the overall effect of optimizing text content consistency and semantic fluency is achieved, effectively solving the problem of erroneous text fragments due to low confidence, and realizing the full mining and combination of context and visual cues, improving the system's ability to process complex text information in real-world scenarios.

[0042] Furthermore, secure data storage includes, The error-correcting text string sequence is encrypted using the TLS protocol for data transmission, and the transmitted data is symmetrically encrypted using AES-256 and backed up and stored on a cloud server.

[0043] This embodiment also provides a text recognition system based on large model and OCR technology, including, Image preprocessing module: performs unified RGB channel conversion, size scaling, noise removal, pixel value normalization, generates binary segmentation map of text and extracts text detection box position information, and performs local cropping, orientation correction, size scaling, conversion to single-channel grayscale image and other operations to provide high-quality image fragments for subsequent OCR processing; Text recognition module: Uses a lightweight OCR model to quickly recognize text information in a cropped image and obtain the text recognition result; Confidence analysis module: Calculates the average confidence level of the OCR model recognition results to evaluate the overall recognition quality; Multimodal feature construction module: Extracts global feature vectors from the image, combines the output text location information and text content, and constructs a multimodal input matrix for large model processing; Semantic error correction module: Uses a multimodal large model to perform semantic error correction on OCR-recognized text and generate corrected text content.

[0044] This embodiment also provides a computer device applicable to text recognition methods based on large models and OCR technology, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the text recognition method based on large models and OCR technology proposed in the above embodiment.

[0045] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0046] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the text recognition method based on large model and OCR technology as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0047] In summary, this invention achieves structured storage of detection and recognition results by combining text location sets with text string sequences into recognition information groups, thus facilitating subsequent information retrieval and multimodal fusion analysis. The combination of confidence screening steps optimizes subsequent processing efficiency and accuracy. The construction of a multimodal feature dataset enables multi-angle judgment of text information with low confidence, effectively compensating for inference biases that may be caused by single semantic or positional features, and improving the accuracy and reliability of semantic error correction in text recognition results. Furthermore, the construction of a multimodal input matrix combined with global features, local features, and task prompts achieves overall synergistic optimization, enhancing the model's comprehensive understanding of complex semantics, contextual relationships, and visual spatial information, thereby achieving non-obvious performance improvements in text recognition, error correction, and analysis tasks.

[0048] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A text recognition method based on large model and OCR technology, characterized in that, include: Image information is extracted and preprocessed uniformly. The DBNet lightweight text detection model is used to generate text detection boxes for text regions to obtain text positions. The lightweight OCR model is then used to quickly recognize the text in the text boxes to obtain the text content and generate recognition information groups. The confidence scores of the lightweight OCR model results are averaged to analyze the overall confidence score. The text location and text content output by OCR are combined with the embedding vector of the original image to form a multimodal feature dataset, and the global feature vector of the image is extracted to form a multimodal input matrix; A multimodal large model is used to perform semantic error correction on the OCR model recognition results, generate the corrected text content, and store the data securely.

2. The text recognition method based on large model and OCR technology as described in claim 1, characterized in that: The extracted image information undergoes unified preprocessing, including: Use the OpenCV image processing library to read image data and obtain the original height H and width W; Perform color space conversion on the image to unify the color space of the image to RGB format; DBNet's input size is usually fixed, requiring the original image size to be scaled proportionally to the corresponding fixed size; For image data, select appropriate methods for noise removal based on different situations; The pixel values ​​of the scaled image are normalized to ensure that the input feature scale is consistent and to avoid detection bias caused by differences in pixel range. Load the pre-trained lightweight text detection model DBNet and input the image data in batches; Preliminary feature extraction of images is performed through multi-layer convolutional networks, and features from different levels are fused from details to the whole. The model first identifies the definite text locations, treats them as the core regions of the text, and marks them on the image. For text edges, the model automatically calculates a threshold to distinguish between text edges and background; By combining the core region and boundary judgment, a binarized image is generated, clearly separating the regions where all text is located; Extract continuous text regions from the binarized image, filter out noise with too small an area, and fit the most suitable box according to the shape of each text region to ensure that the text detection box can tightly surround the text, which is convenient for subsequent OCR recognition. Map the position of the text detection box from the image size after model processing back to the actual coordinates of the original image; For the coordinate bounding box of the text region output by DBNet, each text region is cropped out from the original image according to its coordinates; For text that is tilted or rotated, affine transformation is used to correct its orientation. Based on the input size requirements of CRNN for images, the text image is scaled or filled to the corresponding size proportionally; Since the CRNN model receives grayscale images as input, a grayscale conversion method is needed to convert the input image into a single-channel grayscale image. The text sequence recognition model CRNN is loaded, and image data is input into the convolutional neural network part of CRNN. ​​After multiple layers of convolution and pooling operations, the visual features of the text image are extracted and a feature map is generated. At the same time, the spatial size of the feature map gradually decreases and the number of channels increases to obtain higher-level semantic features. The feature maps output by the convolutional layer are treated as sequential data, segmented along the width direction, and each column of features is used as the input for a time step and fed into the recurrent layer; The output of the recurrent layer is mapped to the character probability space through a fully connected layer, generating a probability distribution covering all candidate characters and whitespace characters at each time step; The CTC decoding algorithm converts the predicted sequence containing whitespace and repeated characters into an actual text string. A single character is defined as Ti, which is output as the final recognition result.

3. The text recognition method based on large model and OCR technology as described in claim 2, characterized in that: The average confidence level of the lightweight OCR model results is calculated, and the overall confidence level is analyzed, including: Calculate the average confidence score of each character based on the identified text content. The overall confidence level; For each confidence value, the mean and standard deviation are calculated. The mean and standard deviation of the confidence values ​​are dynamically weighted and summed to serve as the judgment threshold. If the calculated actual confidence value is greater than the judgment threshold, the corresponding identification information group is used as the confidence screening output. If the calculated actual confidence value is less than or equal to the judgment threshold, then the judgment confidence is low.

4. The text recognition method based on large model and OCR technology as described in claim 3, characterized in that: The extracted global feature vectors of the image form a multimodal input matrix, including: For identification information groups with low confidence, the text location information is normalized, and the text string is used to generate embedding vectors using the pre-trained language model BERT. The embedded vectors and normalized text location information are combined to form a multimodal feature dataset; Using a pre-trained DBNet model with convolutional neural networks, global feature vectors of the image are extracted through multiple convolutions and combined with a multimodal feature dataset and task-oriented prompts to form a multimodal input matrix.

5. The text recognition method based on large model and OCR technology as described in claim 4, characterized in that: The method employs a multimodal large model to perform semantic error correction on the OCR model recognition results, including: Multimodal information processing is performed using a pre-trained multimodal large model, and the multimodal input matrix is ​​used as the model input. The model's input interface is divided into image feature branch, text feature branch, and prompt branch. The model interface completes the encoding of data to generate a combined feature map of joint visual and textual features.

6. The text recognition method based on large model and OCR technology as described in claim 5, characterized in that: The text content for generating error correction includes, Text content identified for each low confidence level Based on comprehensive features, error-correcting text content is generated and composed into a sequence of error-correcting text strings.

7. The text recognition method based on large model and OCR technology as described in claim 6, characterized in that: The aforementioned data secure storage includes, The error-correcting text string sequence is encrypted using the TLS protocol for data transmission, and the transmitted data is symmetrically encrypted using AES-256. The data is then backed up and stored on a cloud server.

8. A text recognition system based on large model and OCR technology, based on the text recognition method based on large model and OCR technology as described in any one of claims 1 to 7, characterized in that: include, Image preprocessing module: performs unified RGB channel conversion, size scaling, noise removal, pixel value normalization, generates binary segmentation map of text and extracts text detection box position information, and performs local cropping, orientation correction, size scaling, conversion to single-channel grayscale image and other operations to provide high-quality image fragments for subsequent OCR processing; Text recognition module: Uses a lightweight OCR model to quickly recognize text information in a cropped image and obtain the text recognition result; Confidence analysis module: Calculates the average confidence level of the OCR model recognition results to evaluate the overall recognition quality; Multimodal feature construction module: Extracts global feature vectors from the image, combines the output text location information and text content, and constructs a multimodal input matrix for large model processing; Semantic error correction module: Uses a multimodal large model to perform semantic error correction on OCR-recognized text and generate corrected text content.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the text recognition method based on large model and OCR technology as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the text recognition method based on large model and OCR technology as described in any one of claims 1 to 7.

Images