OCR recognition method, system and storage medium based on large model
Through the OCR recognition method based on large models, using multimodal large models and large language models to achieve high-precision text recognition in complex scenarios, solving the problems of low recognition accuracy and insufficient adaptability in the existing technology, and improving user experience and recognition effects.
Patent Information
- Application Number
- CN202410804213.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-21
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-06-21
AI Technical Summary
The existing OCR recognition technology has low recognition accuracy in complex scenarios, and cannot effectively understand the text context. It has poor recognition effect when adapting to lighting changes, different shooting angles and text forms, and its recognition ability decreases when facing deformation or occlusion, which affects practicality and User experience.
Using the OCR recognition method based on the big model, feature extraction of the recognized image and text prompt information through the multimodal big model, the text positioning map is determined and the position code is embedded, and the conversion and fusion process is performed in combination with the preset dialogue information, and the large language model is input. To achieve high-precision identification.
Implement high-precision text recognition in complex scenarios, improve recognition accuracy and adaptability, optimize user experience, can effectively capture image features and text features, accurately identify text locations, and adapt to complex scenarios for high-precision recognition.
Smart Images

Figure CN118379742B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a large model-based OCR recognition method, system and storage medium. Background Art
[0002] Optical Character Recognition (OCR) is a technology that uses the principle of optical character recognition to extract and recognize text information from images. OCR has a wide range of applications, such as express delivery bill recognition, container / carton recognition, and meter reading recognition.
[0003] However, the current OCR recognition technology has the following problems: 1. It only relies on the visual features of the image for text recognition, lacking a deep understanding of the contextual meaning of the text. This leads to low recognition accuracy in complex scenarios (such as when the text is closely integrated into the background, or when the background information is crucial to understanding the text); 2. Due to the limitations of training data and algorithms, the existing technology is not adaptable enough when facing changes in lighting, different shooting angles or text forms (such as tilt, adhesion), which seriously affects the recognition effect; 3. When faced with deformed, blurred or partially occluded text, the recognition ability of the existing OCR system is greatly reduced, and the text content cannot be effectively recognized, affecting the practicality of the system and user experience.
[0004] In view of the above problems, effective technical solutions are urgently needed. Summary of the invention
[0005] The purpose of this application is to provide an OCR recognition method, system and storage medium based on a large model, which can solve the problems of low accuracy and poor recognition effect of OCR recognition in complex scenes, effectively capture the image features of the image to be recognized and the text features of the text prompt information, accurately identify the text position in the image, achieve the purpose of OCR positioning, adapt to complex scenes for high-precision recognition, have high practicality, and effectively optimize the user experience.
[0006] In a first aspect, the present application provides an OCR recognition method based on a large model, the method comprising: obtaining an image to be recognized and text prompt information;
[0007] Based on the set multimodal large model, feature extraction is performed on the image to be recognized and the text prompt information to obtain global image features, first local image features and text features;
[0008] Based on the global image features and text features, a text location map corresponding to the image to be identified is determined, where the text location map stores the probability that each pixel in the image to be identified is text;
[0009] Performing position encoding and embedding on the first local image feature based on the text localization map to obtain a second local image feature;
[0010] The second local image feature and the preset dialogue information are converted and fused to obtain the instruction input information;
[0011] The command input information is input into the set large language model to obtain the OCR recognition result.
[0012] Optionally, the multimodal large model includes a global image encoder, a local image encoder, and a text encoder;
[0013] Accordingly, based on the set multimodal large model, feature extraction is performed on the image to be recognized and the text prompt information to obtain global image features, first local image features and text features, including:
[0014] Inputting the image to be recognized into the global image encoder in the set multimodal large model to obtain the global image features;
[0015] Inputting the image to be recognized into the local image encoder in the multimodal large model to obtain a first local image feature;
[0016] The text prompt information is input into the text encoder in the multimodal large model to obtain text features.
[0017] Optionally, determining a text location map corresponding to the image to be recognized based on global image features and text features includes:
[0018] Performing dot product calculation on the global image features and the text features to obtain a calculation result graph corresponding to the image to be identified, the calculation result graph storing the dot product calculation result of each pixel in the image to be identified;
[0019] The calculation result map is converted into a text positioning map through the set activation function.
[0020] Optionally, performing position encoding and embedding on the first local image feature based on the text localization map to obtain the second local image feature includes:
[0021] Based on the text location map, an original position code corresponding to the first local image feature is enhanced to obtain a target position code;
[0022] The target position code is fused with the first local image feature to obtain the second local image feature.
[0023] Optionally, enhancing the original position code corresponding to the first local image feature based on the text location map to obtain the target position code includes:
[0024] The text location map is sequentially downsampled and dimensionally transformed, and the transformed text location map is superimposed on the original position code corresponding to the first local image feature to obtain an enhanced position code;
[0025] The enhanced position encoding is updated by self-attention to obtain the target position encoding.
[0026] Optionally, the calculation formula corresponding to the self-attention update is as follows: ,
[0027] in, Encode the target location, To enhance position encoding, is the weight matrix of query, is the key weight matrix, is the weight matrix of the values, is the dimension of the key vector, T is the transpose operation, is the normalized exponential function.
[0028] Optionally, the preset dialogue information includes system setting information and question text information;
[0029] Accordingly, the second local image feature and the preset dialogue information are converted and fused to obtain the instruction input information, including:
[0030] Inputting the second local image feature into a set feature converter to obtain image feature information;
[0031] The image feature information, system setting information and question text information are sequentially spliced to obtain the command input information.
[0032] Optionally, the instruction structure corresponding to the instruction input information is as follows: ,
[0033] in, Enter information for the command, Set information for the system. is the image feature information, To ask text information, is the starting character of the input image feature information, It is the terminator of the input image feature information.
[0034] In a second aspect, the present application provides an OCR recognition system based on a large model, the OCR recognition system comprising: a memory and a processor, the memory comprising a program of an OCR recognition method, and when the program of the OCR recognition method is executed by the processor, the following steps are implemented:
[0035] Get the image to be recognized and the text prompt information;
[0036] Based on the set multimodal large model, feature extraction is performed on the image to be recognized and the text prompt information to obtain global image features, first local image features and text features;
[0037] Based on the global image features and text features, a text location map corresponding to the image to be identified is determined, where the text location map stores the probability that each pixel in the image to be identified is text;
[0038] Performing position encoding and embedding on the first local image feature based on the text localization map to obtain a second local image feature;
[0039] The second local image feature and the preset dialogue information are converted and fused to obtain the instruction input information;
[0040] The command input information is input into the set large language model to obtain the OCR recognition result.
[0041] In a third aspect, the present application provides a computer-readable storage medium, which includes a program for a circular sample screening method. When the program for the circular sample screening method is executed by a processor, the steps of the circular sample screening method are implemented.
[0042] As can be seen from the above, the OCR recognition method, system and storage medium based on the large model provided by the present application obtain the image to be recognized and the text prompt information; based on the set multimodal large model, feature extraction is performed on the image to be recognized and the text prompt information to obtain global image features, first local image features and text features; based on the global image features and text features, the text location map corresponding to the image to be recognized is determined; based on the text location map, the first local image features are position-encoded and embedded to obtain the second local image features; the second local image features and the preset dialogue information are converted and fused to obtain the instruction input information; the instruction input information is input into the set large language model to obtain the OCR recognition result. In the above scheme, by extracting features from the image to be recognized and the text prompt information based on the set multimodal large model, the image features of the image to be recognized and the text features of the text prompt information can be effectively captured; by determining the text location map corresponding to the image to be recognized and position-encoding and embedding the first local image features, the text position in the image can be accurately identified to achieve the purpose of OCR positioning, so that the large language model can accurately recognize and understand the text content in the image. By converting and fusing the second local image features and the preset dialogue information, they can be converted into text descriptions that are easy for the large language model to understand, ensuring that the model can accurately interpret the details and text attributes in the image. Finally, the command input information is input into the set large language model to obtain the OCR recognition result, which can adapt to complex scenarios for high-precision recognition, has high practicality, and effectively optimizes the user experience.
[0043] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 A flowchart of an OCR recognition method based on a large model provided in an embodiment of the present application;
[0046] Figure 2 A schematic diagram of the process of the large model-based OCR recognition method provided in an embodiment of the present application;
[0047] Figure 3 A schematic diagram of the structure of the large model-based OCR recognition system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0049] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0050] In natural scenes, text may appear in extremely complex backgrounds, such as irregular surfaces, dynamic environments, or conditions with great changes in lighting. This places high demands on the visual recognition capabilities of the OCR system. On the one hand, the diversity of text size, font, color, and arrangement requires the OCR system to have extremely high flexibility and adaptability. In particular, the recognition of rare fonts or artistic characters is a major challenge for traditional OCR systems. Recognizing text content only through image features is often not enough to understand the true meaning of the text, especially in application scenarios that require contextual interpretation or semantic analysis of the text. On the other hand, in many cases, the text color is similar to the background color, or the text is closely intertwined with the background pattern, making it difficult to accurately separate the text from the background.
[0051] Please refer to Figure 1 , is a flowchart of a large model-based OCR recognition method in some embodiments of the present application, such as Figure 1 As shown, the specific steps include:
[0052] S101, obtaining an image to be recognized and text prompt information;
[0053] S102, based on the set multimodal large model, feature extraction is performed on the image to be recognized and the text prompt information to obtain global image features, first local image features and text features;
[0054] S103, determining a text location map corresponding to the image to be recognized based on global image features and text features, wherein the text location map stores a probability that each pixel in the image to be recognized is text;
[0055] S104, performing position encoding and embedding on the first local image feature based on the text positioning map to obtain a second local image feature;
[0056] S105, converting and fusing the second local image feature and the preset dialogue information to obtain instruction input information;
[0057] S106: Input the instruction input information into the set large language model to obtain an OCR recognition result.
[0058] It should be noted that the image to be identified may contain various backgrounds and text types, and the text prompt information may be a suggestive description for providing the text content of the image to be identified, which can guide the multimodal large model to focus on the text area or features in the image to be identified and improve the accuracy of recognition. The multimodal large model may be a CLIP (Contrastive Language-Image Pre-training) model, a ViLBERT (Vision-and-Language Bidirectional Encoder Representations from Transformers, a visual language bidirectional encoder representation based on Transformer) model, a UNITER (Universal Image-Text Representation Learning, a universal image text representation learning) model, and a VisualBERT (Vision Bidirectional Encoder Representations from Transformers, a visual bidirectional encoder representation based on Transformer) model, etc., which are not limited in this application. Among them, the CLIP model can be used as an optimal multimodal large model, which can be pre-trained on a large number of image-text pairs, map images and texts to the encoder of the same embedding space, and can simultaneously process image and text data, establish cross-modal associations between images and texts, and effectively adapt to various cross-modal natural language processing tasks. The large language model can be a Vicuna-v1.5 model, a GPT (Generative Pre-trained Transformer) model, etc. The Vicuna-v1.5 model is a large language model fine-tuned based on the LLaMA (Language Learning with Adaptive Multi-task Architecture) model, which is not limited in this application. Among them, the Vicuna-v1.5 model can be used as the preferred large language model. It can achieve excellent performance in multiple NLP benchmarks, which is conducive to providing more accurate and reliable results when processing various natural language tasks. Based on its improved training strategies and algorithms, it can achieve performance comparable to other large models on less training data, reducing the demand for large-scale data sets. At the same time, it can adapt to various types of natural language tasks, including text classification, sentiment analysis, question-answering systems, text generation, etc. Developer users can customize and expand it according to their needs to meet specific application scenarios.By utilizing the multimodal characteristics of the multimodal large model and the deep language processing capabilities of the large language model, the accuracy and efficiency of text recognition in complex scenes can be improved. In addition, the multimodal large model has excellent context understanding capabilities, which can better understand the context of the text in the image, including the relationship between the text and the background, the logical connection between the text, etc., which helps to improve the accuracy of OCR, especially when dealing with complex scenes. Specifically, by training the multimodal large model with a large amount of training data of different backgrounds and text types, the multimodal large model can learn more diverse text representations, including different fonts, sizes, colors, and texts in various complex backgrounds, so that the large model can maintain a high recognition accuracy when facing text styles or backgrounds that have not been seen before. Better generalization ability. Optionally, by training the multimodal large model with a large number of data variants, the multimodal large model can better handle tilted, curved or otherwise deformed text. In addition, the global image feature can be a characterization of the overall attribute of the image to be recognized, for example, describing the color features, texture features, and shape features of the image to be recognized. The first local image feature may be a local detail characterizing the image to be identified, used to describe the uniqueness and distinguishability of a specific local area, such as edges, corners, spots, textures, etc. The text feature may be a descriptive attribute characterizing the text prompt information, such as content features, structural features, language features, etc. The text location map may be used to identify the location of the text in the image to be identified. The second local image feature may be a local image feature that effectively embeds text location information. The preset dialogue information may be a dialogue template designed in advance based on the understanding characteristics of a large language model, which may improve the ability of the large language model to perform text recognition and interpretation. The instruction input information may be used to instruct the large language model to perform text recognition and understanding based on the second local image feature.
[0059] In one embodiment, the use of a multimodal large model can enhance visual understanding capabilities and more comprehensively understand the scenes in the image, including elements such as objects, people, actions, and environments, thereby providing richer and more accurate information when answering user queries. For example, the system can determine the most likely text content or intent based on the scene content in the image, and further optimize the answer strategy. In addition, artistic fonts or handwriting are often difficult to be accurately recognized by traditional OCR systems due to their unique style and changes. Large models trained with a large amount of data can significantly improve the recognition ability of these complex fonts. The comprehensive use of multimodal large models and large language models not only improves the ability to recognize text from images, but also enables in-depth analysis of the recognized text to screen out specific content related to the question, such as accurately identifying and extracting date information from a series of texts.
[0060] It should be noted that, by obtaining the image to be recognized and the text prompt information; based on the set multimodal large model, feature extraction is performed on the image to be recognized and the text prompt information to obtain global image features, first local image features and text features; based on the global image features and text features, the text location map corresponding to the image to be recognized is determined; based on the text location map, the first local image features are position-encoded and embedded to obtain the second local image features; the second local image features and the preset dialogue information are converted and fused to obtain the instruction input information; the instruction input information is input into the set large language model to obtain the OCR recognition result. In the above scheme, by extracting features from the image to be recognized and the text prompt information based on the set multimodal large model, the image features of the image to be recognized and the text features of the text prompt information can be effectively captured; by determining the text location map corresponding to the image to be recognized and position-encoding and embedding the first local image features, the text position in the image can be accurately identified to achieve the purpose of OCR positioning, so that the large language model can accurately recognize and understand the text content in the image. By converting and fusing the second local image features and the preset dialogue information, they can be converted into text descriptions that are easy for the large language model to understand, ensuring that the model can accurately interpret the details and text attributes in the image. Finally, the command input information is input into the set large language model to obtain the OCR recognition result, which can adapt to complex scenarios for high-precision recognition, has high practicality, and effectively optimizes the user experience.
[0061] According to an embodiment of the present invention, the multimodal large model includes a global image encoder, a local image encoder, and a text encoder;
[0062] Accordingly, based on the set multimodal large model, feature extraction is performed on the image to be recognized and the text prompt information to obtain global image features, first local image features and text features, including:
[0063] Inputting the image to be recognized into the global image encoder in the set multimodal large model to obtain the global image features;
[0064] Inputting the image to be recognized into the local image encoder in the multimodal large model to obtain a first local image feature;
[0065] The text prompt information is input into the text encoder in the multimodal large model to obtain text features.
[0066] It should be noted that, taking the multimodal large model as the CLIP model as an example, the ResNet part in the CLIP model can be used as a global image encoder, which can be used to extract the global features of the image and capture the overall layout and context information of the image. Specifically, the image to be identified can be input into the global image encoder to extract the global image features. The relevant formula is as follows: ,
[0067] in, represents the ResNet50 part of the pre-trained CLIP model, I is the image to be recognized, is the global image feature, the global image feature The size can be , is the height, is the width and C is the number of channels.
[0068] In addition, the Vision Transformer (ViT) part in the CLIP model can also be used as a local image encoder, which can be used to extract local features of the image and obtain detailed visual information in the image. Specifically, the image to be identified can be input into the local image encoder to extract local image features. The relevant formula is as follows: ,
[0069] in, represents the ViT part of the pre-trained CLIP model, I is the image to be recognized, is a local image feature. It can be composed of a set of tokens, each token represents a local area of the image.
[0070] In addition, text features can be extracted and encoded based on the text encoder in the CLIP model to perform in-depth language processing. Specifically, the text prompt information is input into the text encoder to extract text features. The relevant formula is as follows: ,
[0071] in, is the text encoder representing the pre-trained CLIP model, For text prompt information, is a text feature.
[0072] Optional, this text prompt message The language prompt text is converted into a word vector through a word embedding model. For example, the language prompt template used by the language prompt text can be , the relevant formula is as follows: ,
[0073] in, is the word embedding model, For language hint text, It is a text prompt information.
[0074] According to an embodiment of the present invention, determining a text location map corresponding to an image to be recognized based on global image features and text features includes:
[0075] Performing dot product calculation on the global image features and the text features to obtain a calculation result graph corresponding to the image to be identified, the calculation result graph storing the dot product calculation result of each pixel in the image to be identified;
[0076] The calculation result map is converted into a text positioning map through the set activation function.
[0077] It should be noted that the relevant formula for calculating the text positioning map is as follows: ,
[0078] in, is the global image feature, is the text feature, To calculate the dot product of global image features and text features, P is the text positioning map, is an activation function, for example, a sigmoid function, which is used to convert the dot product calculation result into a probability distribution and determine the score of each pixel. The score of each pixel indicates the probability that the position is a text. The size of the text location map P is , is the height, is the width.
[0079] According to an embodiment of the present invention, position encoding and embedding the first local image feature based on the text positioning map to obtain the second local image feature includes:
[0080] Based on the text location map, an original position code corresponding to the first local image feature is enhanced to obtain a target position code;
[0081] The target position code is fused with the first local image feature to obtain the second local image feature.
[0082] It should be noted that effectively embedding the text location map into the local image feature can help the large language model to accurately recognize and understand the text content in the image. The relevant formula for calculating the second local image feature is as follows: ,
[0083] in, is the local image feature, Encode the target location, is the second local image feature.
[0084] According to an embodiment of the present invention, enhancing the original position code corresponding to the first local image feature based on the text positioning map to obtain the target position code includes:
[0085] The text location map is sequentially downsampled and dimensionally transformed, and the transformed text location map is superimposed on the original position code corresponding to the first local image feature to obtain an enhanced position code;
[0086] It should be noted that the calculation formula for calculating the enhanced position encoding is as follows: ,
[0087] in, is the original position code, P is the text positioning map, is the downsampling operation, is the dimension conversion operation, To enhance the position encoding. The downsampling operation The number of tokens that can be used to match the ViT part of the CLIP model. This dimension conversion operation Can be used to match the tokens dimension of the ViT part of the CLIP model.
[0088] The enhanced position encoding is updated by self-attention to obtain the target position encoding.
[0089] It should be noted that the calculation formula corresponding to the self-attention update is as follows: ,
[0090] in, Encode the target location, To enhance position encoding, is the weight matrix of query, is the key weight matrix, is the weight matrix of the values, is the dimension of the key vector, T is the transpose operation, is the normalized exponential function.
[0091] According to an embodiment of the present invention, the preset dialogue information includes system setting information and question text information;
[0092] Accordingly, the second local image feature and the preset dialogue information are converted and fused to obtain the instruction input information, including:
[0093] Inputting the second local image feature into a set feature converter to obtain image feature information;
[0094] It should be noted that the feature converter can be used to convert the second local image feature into a large language model space for subsequent text recognition and understanding tasks. Specifically, the feature converter can be a multi-layer perceptron (MLP) adapter, and the relevant formula is as follows: ,
[0095] in, as well as are the network parameters of the feature converter, is the activation function, It is the image feature information. It can convert local image features into text descriptions that are easy for large language models to understand, ensuring that the model can accurately interpret the details and text attributes in the image.
[0096] The image feature information, system setting information and question text information are sequentially spliced to obtain the command input information.
[0097] It should be noted that the instruction structure corresponding to the instruction input information is as follows: ,
[0098] in, Enter information for the command, Set information for the system. is the image feature information, To ask text information, is the starting character of the input image feature information, It is the terminator of the input image feature information. The use of preset fixed sentence patterns aims to set the tone of the conversation environment to encourage the large language model to generate targeted, informative and polite responses. The optimized sentence patterns are more concise and clear, which can improve the naturalness and fluency of the interaction.
[0099] Specifically, the system sets the information It is obtained by converting the set prompt text into a word vector through a word embedding model, for example, = "The conversation focuses on interpreting text from images, and AI provides insightful and polite responses", and the relevant formula is as follows: ,
[0100] in, is the word embedding model, To set the prompt text, It is a text prompt information.
[0101] Specifically, the question text information Developers can build flexible and changeable question sentences based on the specific attributes of the text in the image (such as text type, color, orientation, etc.). The optimized question template enhances flexibility and richness of expression and can cover a wider range of query needs. It is obtained by converting the question text into a word vector through a word embedding model, for example, "Recognize the text {} in the image." For example, "Find and describe the {} text located in the image {}." The relevant formula is as follows: ,
[0102] in, is the word embedding model, For the question text, To ask text information.
[0103] Optionally, the question text can contain explicit attribute information, which provides more intuitive and specific guidance for the attribute information in the question sentence, optimizes the attribute description method, and ensures that users can clearly construct targeted queries. For example: “Recognize the text “red, large, English” described in the image.” For example, "Locate and describe the "small, dot-matrix" date in the "lower right corner" of the image."
[0104] For example, Table 1 is a query comparison table of attribute categories and attribute options, which is used to query options corresponding to different attribute types.
[0105] .
[0106] Optionally, the multimodal large model and the large language model in this embodiment can be fine-tuned, wherein the local image encoder responsible for local image feature extraction in the multimodal large model can be fine-tuned using low-rank adaptation technology, while fixing other network parameters. In addition, the large language model can also be fine-tuned using low-rank adaptation technology, while updating the part of the network that updates the position encoding through self-attention during training, while fixing other network parameters.
[0107] In one embodiment, Figure 2 A schematic diagram of the process of the large model-based OCR recognition method provided in the embodiment of the present application is shown in FIG. Figure 2As shown, the image to be recognized 101 is input into the local image encoder 103 to output the first local image feature 106, the image to be recognized is input into the global image encoder 104 to output the global image feature 107, and the text prompt information 102 is input into the text encoder 105 to output the text feature 108. The global image feature 107 and the text feature 108 are input into the first intermediate processing layer 109 to output the OCR positioning result 110, that is, the text positioning map, and the OCR positioning result 110 and the first local image feature 106 are input into the second intermediate processing layer 111 to output the second local image feature (not shown in the figure) after the position coding embedding is completed. The second local image feature is input into the feature converter 114 to output the image feature information to the large language model 117, the system setting information 112, the image start symbol 113 of the image feature information, the image feature information (not shown in the figure), the image end symbol 115 of the image feature information, and the question text information 116 can be combined and input into the large language model 117, and finally the large language model 117 outputs the OCR recognition result 118.
[0108] The present invention also provides an OCR recognition system based on a large model. Figure 3 The schematic diagram of the structure of the large model-based OCR recognition system provided in the embodiment of the present application is shown. The OCR recognition system includes: a memory 201 and a processor 202. The memory includes a program of the OCR recognition method. When the program of the OCR recognition method is executed by the processor, the following steps are implemented:
[0109] S101, obtaining an image to be recognized and text prompt information;
[0110] S102, based on the set multimodal large model, feature extraction is performed on the image to be recognized and the text prompt information to obtain global image features, first local image features and text features;
[0111] S103, determining a text location map corresponding to the image to be recognized based on global image features and text features;
[0112] S104, performing position encoding and embedding on the first local image feature based on the text positioning map to obtain a second local image feature;
[0113] S105, converting and fusing the second local image feature and the preset dialogue information to obtain instruction input information;
[0114] S106: Input the instruction input information into the set large language model to obtain an OCR recognition result.
[0115] It should be noted that, by obtaining the image to be recognized and the text prompt information; based on the set multimodal large model, feature extraction is performed on the image to be recognized and the text prompt information to obtain global image features, first local image features and text features; based on the global image features and text features, the text location map corresponding to the image to be recognized is determined; based on the text location map, the first local image features are position-encoded and embedded to obtain the second local image features; the second local image features and the preset dialogue information are converted and fused to obtain the instruction input information; the instruction input information is input into the set large language model to obtain the OCR recognition result. In the above scheme, by extracting features from the image to be recognized and the text prompt information based on the set multimodal large model, the image features of the image to be recognized and the text features of the text prompt information can be effectively captured; by determining the text location map corresponding to the image to be recognized and position-encoding and embedding the first local image features, the text position in the image can be accurately identified to achieve the purpose of OCR positioning, so that the large language model can accurately recognize and understand the text content in the image. By converting and fusing the second local image features and the preset dialogue information, they can be converted into text descriptions that are easy for the large language model to understand, ensuring that the model can accurately interpret the details and text attributes in the image. Finally, the command input information is input into the set large language model to obtain the OCR recognition result, which can adapt to complex scenarios for high-precision recognition, has high practicality, and effectively optimizes the user experience.
[0116] The present invention also provides a computer-readable storage medium, which includes a program of the OCR recognition method. When the program of the OCR recognition method is executed by a processor, the steps of the OCR recognition method are implemented.
[0117] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0118] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0119] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0120] Those skilled in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.
[0121] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
Claims
1. A large model-based OCR recognition method, characterized in that: include: Get the image to be recognized and the text prompt information; Based on the set multimodal large model, feature extraction is performed on the image to be identified and the text prompt information to obtain global image features, first local image features and text features; Based on the global image features and the text features, determining a text location map corresponding to the image to be recognized, wherein the text location map stores a probability that each pixel in the image to be recognized is text; Performing position encoding and embedding on the first local image feature based on the text localization map to obtain a second local image feature; Converting and fusing the second local image feature and the preset dialogue information to obtain instruction input information; Inputting the instruction input information into a set large language model to obtain an OCR recognition result; The step of performing position encoding and embedding on the first local image feature based on the text positioning map to obtain a second local image feature includes: Based on the text location map, an original position code corresponding to the first local image feature is enhanced to obtain a target position code; Fusing the target position code with the first local image feature to obtain a second local image feature; The step of enhancing the original position code corresponding to the first local image feature based on the text positioning map to obtain the target position code includes: Downsampling and dimension conversion are sequentially performed on the text location map, and the converted text location map is superimposed on the original position code corresponding to the first local image feature to obtain an enhanced position code; Performing self-attention updating on the enhanced position encoding to obtain a target position encoding; The calculation formula corresponding to the self-attention update is as follows: , in, Encode the target position, For the enhanced position code, is the query weight matrix, is the key weight matrix, is the weight matrix of the values, is the dimension of the key vector, is the transpose operation, is the normalized exponential function; The determining, based on the global image feature and the text feature, a text location map corresponding to the image to be recognized includes: Performing dot product calculation on the global image feature and the text feature to obtain a calculation result graph corresponding to the image to be identified, wherein the calculation result graph stores a dot product calculation result corresponding to each pixel in the image to be identified; The calculation result map is converted into a text positioning map by a set activation function; The preset dialogue information includes system setting information and question text information; Correspondingly, the converting and fusing the second local image feature and the preset dialogue information to obtain the instruction input information includes: Inputting the second local image feature into a set feature converter to obtain image feature information; Sequentially concatenating the image feature information, the system setting information, and the question text information to obtain instruction input information; The instruction structure corresponding to the instruction input information is as follows: , in, Enter information for said instruction, setting information for the system, is the image feature information, is the question text information, is the starting character for inputting the image feature information, is the terminator for inputting the image feature information.
2. The OCR recognition method according to claim 1, characterized in that: The multimodal large model includes a global image encoder, a local image encoder and a text encoder; Correspondingly, the multimodal large model based on the setting respectively extracts features of the image to be recognized and the text prompt information to obtain global image features, first local image features and text features, including: Inputting the image to be identified into a global image encoder in a set multimodal large model to obtain global image features; Inputting the image to be recognized into a local image encoder in the multimodal large model to obtain a first local image feature; The text prompt information is input into a text encoder in the multimodal large model to obtain text features.
3. An OCR recognition system based on a large model, characterized in that: The OCR recognition system comprises: a memory and a processor, wherein the memory comprises a program of the OCR recognition method, and when the program of the OCR recognition method is executed by the processor, the following steps are implemented: Get the image to be recognized and the text prompt information; Based on the set multimodal large model, feature extraction is performed on the image to be identified and the text prompt information to obtain global image features, first local image features and text features; Based on the global image features and the text features, determining a text location map corresponding to the image to be recognized, wherein the text location map stores a probability that each pixel in the image to be recognized is text; Performing position encoding and embedding on the first local image feature based on the text localization map to obtain a second local image feature; Converting and fusing the second local image feature and the preset dialogue information to obtain instruction input information; Inputting the instruction input information into a set large language model to obtain an OCR recognition result; The step of performing position encoding and embedding on the first local image feature based on the text positioning map to obtain a second local image feature includes: Based on the text location map, an original position code corresponding to the first local image feature is enhanced to obtain a target position code; Fusing the target position code with the first local image feature to obtain a second local image feature; The step of enhancing the original position code corresponding to the first local image feature based on the text positioning map to obtain the target position code includes: Downsampling and dimension conversion are sequentially performed on the text location map, and the converted text location map is superimposed on the original position code corresponding to the first local image feature to obtain an enhanced position code; Performing self-attention updating on the enhanced position encoding to obtain a target position encoding; The calculation formula corresponding to the self-attention update is as follows: , in, Encode the target position, For the enhanced position code, is the query weight matrix, is the key weight matrix, is the weight matrix of the values, is the dimension of the key vector, is the transpose operation, is the normalized exponential function; The determining, based on the global image feature and the text feature, a text location map corresponding to the image to be recognized includes: Performing dot product calculation on the global image feature and the text feature to obtain a calculation result graph corresponding to the image to be identified, wherein the calculation result graph stores a dot product calculation result corresponding to each pixel in the image to be identified; The calculation result map is converted into a text positioning map by a set activation function; The preset dialogue information includes system setting information and question text information; Correspondingly, the converting and fusing the second local image feature and the preset dialogue information to obtain the instruction input information includes: Inputting the second local image feature into a set feature converter to obtain image feature information; Sequentially concatenating the image feature information, the system setting information, and the question text information to obtain instruction input information; The instruction structure corresponding to the instruction input information is as follows: , in, Enter information for said instruction, setting information for the system, is the image feature information, is the question text information, is the starting character for inputting the image feature information, is the terminator for inputting the image feature information.
4. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a program of the OCR recognition method. When the program of the OCR recognition method is executed by a processor, the steps of the OCR recognition method according to any one of claims 1 to 2 are implemented.
