Text recognition method, model training method, equipment and storage medium

Abstract

The invention discloses a text recognition method, a model training method, equipment and a storage medium, and relates to the technical field of optical character recognition, and the method comprises the steps: obtaining a sample image, and corresponding label textbox information and label text information; inputting the sample image into the single vision model and the residual network model to obtain first sample textbox information and first sample text information; inputting the sample image and the processing requirement into a multi-modal AI large model to obtain second sample textbox information and second sample text information; determining a distillation loss value according to the label textbox information, the first sample textbox information, the second sample textbox information, the label text information, the first sample text information and the second sample text information so as to update parameters of the text recognition model; and inputting the to-be-recognized image into the text recognition model to obtain a text recognition result. According to the method, the generalization ability of text recognition can be improved, and the application scene of text recognition is expanded.

Description

Technical Field

[0001] The present application relates to the technical field of optical character recognition, and in particular to a text recognition method, model training method, device and storage medium. Background Art

[0002] In today's digital age, rapid processing and efficient use of information have become key factors in promoting social development. Optical Character Recognition (OCR) technology plays a vital role as a bridge between paper documents, images, etc. in the real world and text that can be processed by computers.

[0003] Traditional OCR technology is mainly based on feature engineering and machine learning algorithms. In the feature extraction stage, the key features of characters are extracted from the image through edge detection, contour analysis, texture feature extraction and other methods. These features can be information such as the strokes, structure, shape, etc. of the characters. Then, a classifier (such as support vector machine, decision tree, etc.) is used to classify the extracted features to identify the characters.

[0004] Although both traditional methods and deep learning have achieved certain results in the field of OCR, the current optical character recognition technology still has the problem of poor generalization ability. In practical applications, factors such as image quality, font diversity, and background complexity will have a significant impact on the recognition results. For example, in some scanned documents, there may be problems such as blur, noise, and tilt, which makes it difficult for existing OCR systems to accurately recognize characters. In addition, for some special fonts, handwritten text, or text in non-standard formats, the recognition accuracy of the OCR system often drops significantly.

[0005] Therefore, this lack of generalization ability limits the application of OCR technology in a wider range of scenarios, and has also become a key issue that needs to be urgently addressed in the current development of OCR technology. Summary of the invention

[0006] The present application provides a text recognition method, a model training method, a device and a storage medium, which can improve the generalization ability of text recognition and expand the application scenarios of text recognition.

[0007] In order to achieve the above objectives, this application adopts the following technical solutions: In a first aspect, the present application provides a method for text recognition, comprising: Obtain an image to be recognized; Inputting the image to be recognized into a text recognition model to obtain a text recognition result; The text recognition model is trained in the following way: Obtain the sample image and the label text box information and label text information corresponding to the sample image; Input the sample image into a single vision model and a residual network model to obtain the first sample text box information and the first sample text information; Input the sample image and the processing requirements into a multi-modal AI large model to obtain the second sample text box information and the second sample text information; Determine the distillation loss value according to the label text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information, and the second sample text information; Update the parameters of the text recognition model according to the distillation loss value.

[0008] Optionally, the inputting the sample image into a single vision model and a residual network model to obtain the first sample text box information and the first sample text information includes: Extract the color features of the sample image to obtain the first sample feature; use an open-source computer vision library to extract the features of the sample image to obtain the second sample feature; Input the first sample feature into the first single vision model to obtain the first sample text box feature, input the second sample feature into the second single vision model to obtain the second sample text box feature, and fuse the first sample text box feature and the second sample text box feature to obtain the sample text box fusion feature; Obtain the first sample text box information corresponding to the sample text box fusion feature through the first fully connected layer; Input the sample text box fusion feature into the residual network model to obtain the sample text feature; Obtain the first sample text information corresponding to the sample text feature through the second fully connected layer.

[0009] Optionally, the determining the distillation loss value according to the label text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information, and the second sample text information includes: Determine the first loss value according to the label text box information and the first sample text box information, determine the second loss value according to the first text box information and the second sample text box information, determine the third loss value according to the label text information and the first sample text information, and determine the fourth loss value according to the first text information and the second sample text information; Determine the distillation loss value according to the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0010] Optionally, the label text box information corresponding to the sample image is obtained in the following manner: Pre-annotate the text box of the sample image using a multi-modal AI large model, and output the pre-annotation result; Obtain the approved information feedback for the pre-annotation result, where the approved information includes the annotation result after approval; Determine the label text box information corresponding to the sample image according to the approved information.

[0011] Optionally, the label text information corresponding to the sample image is obtained in the following manner: Use at least 3 multi-modal AI large models to identify the sample image, and obtain multiple sample text information and the confidence levels corresponding to the sample text information; If the lengths of multiple sample text information are all the same, or the lengths of multiple sample text information are all different, determine the text information with the highest confidence level among the multiple sample text information as the label text information corresponding to the sample image.

[0012] Optionally, the method further includes: If there is a first type of sample text information set with the same length and a second type of sample text information set with different lengths among the multiple sample text information, determine the text information with the highest confidence level in the first type of sample text information set as the label text information corresponding to the sample image.

[0013] Optionally, the residual network model includes at least a convolutional layer and a residual block group; The convolutional layer includes a first attention mechanism module, which is used to perform parallel processing on the sample image from the channel dimension, height dimension, and width dimension to obtain multi-dimensional fusion features; The residual block group includes a second attention mechanism module, which is used to enhance the features of the similar parts in the multi-dimensional fusion features to obtain sample text features.

[0014] In a second aspect, the present application provides a model training method, and the method includes: Obtain a sample image and the label text box information and label text information corresponding to the sample image; Input the sample image into a single vision model and a residual network model to obtain first sample text box information and first sample text information; Input the sample image and processing requirements into a multi-modal AI large model to obtain second sample text box information and second sample text information; Determine a distillation loss value according to the label text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information, and the second sample text information; Update the parameters of the text recognition model according to the distillation loss value.

[0015] In a third aspect, the present application provides a computing device, including a memory and a processor; Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the computing device is enabled to execute the method according to any one of the first aspect or the second aspect.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the method according to any one of the first aspect or the second aspect.

[0017] It can be seen from the above technical solutions that the present application has at least the following beneficial effects: The present application provides a method for text recognition. This method adopts a distillation training method to train a text recognition model. Specifically, a sample image and label text box information and label text information corresponding to the sample image are obtained, and then the sample image is input into a single vision model and a residual network model to obtain a first sample text box and first sample text information. The sample image and processing requirements are input into a multi-modal AI large model to obtain second sample text box information and second sample text information. Finally, distillation learning is performed using the output result of the multi-modal AI large model, that is, based on the label text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information, and the second sample text information, a distillation loss value is determined, and the parameters of the text recognition model are updated using the distillation loss value, thereby realizing distillation learning. In this solution, the text recognition model is distilled and trained through the output result of the multi-modal AI large model and the label information, which can enable the text recognition model to learn more extensive and general feature representations, so that the text recognition model can better adapt to and process new images, thereby improving the generalization ability of the text recognition model and reducing the risk of overfitting. The text recognition model obtained through this solution is used to recognize the image to be recognized, and the text recognition result obtained is also more accurate.

[0018] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a text recognition method provided in an embodiment of the present application; Figure 2 A schematic diagram of determining label text information provided in an embodiment of the present application; Figure 3 A schematic diagram of a text recognition model provided in an embodiment of the present application; Figure 4 A schematic diagram of a text recognition device provided in an embodiment of the present application; Figure 5 A schematic diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The terms "first", "second", "third", etc. in the specification of this application and the accompanying drawings are used to distinguish different objects rather than to limit a specific order.

[0021] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0022] Traditional optical character recognition methods usually use a series of steps to achieve text recognition, including image preprocessing (grayscale, denoising, and contrast enhancement) to optimize image quality, layout analysis to identify text areas, and character segmentation to separate individual characters. These systems rely on machine learning algorithms, such as support vector machines (SVM) or K-nearest neighbors (k-NN), for character classification.

[0023] However, these methods have some limitations: they are developed based on fixed scenarios and are difficult to migrate to other scenarios, resulting in limited application scope; character recognition relies on manual features and lacks robustness. Especially in the case of font changes or background interference, the recognition effect will significantly decline; in addition, due to the independence of each module, it is difficult to optimize the overall system, and errors are easily transmitted between modules, affecting the overall recognition accuracy. On the other hand, the deep learning-based optical character recognition method reduces the dependence on manual feature design through end-to-end training, can learn complex feature representations, and is suitable for diverse and highly variable texts. This method can improve performance and generalization ability through training with large-scale data. However, deep learning methods also have their drawbacks, including the need for a large amount of labeled data and computing resources, high training and deployment costs, and insufficient generalization ability for small datasets and specific scenarios.

[0024] In view of this, embodiments of the present application provide a model training method, which can be applied to a processing device, which can be a terminal or a server. Terminals include but are not limited to smartphones, tablets, laptops, personal digital assistants, or smart wearable devices, etc. The server can be a cloud server, such as the central server in a central cloud computing cluster or the edge server in an edge cloud computing cluster. Of course, the server can also be a server in a local data center. A local data center refers to a data center directly controlled by a user.

[0025] After the processing device completes model training, the obtained text recognition model can be deployed to a target device, which has a certain computing power. In this way, the small model obtained after training can be used to recognize the text in the image. Further, compared with using a multimodal AI (Artificial Intelligence) large model for text recognition, the multimodal AI large model is usually deployed on the server, and the computing power of the target device is much smaller than that of the server. The multimodal AI large model cannot be deployed to the target device. Therefore, a network connection between the target device and the server is required to call the multimodal AI large model. This method performs distillation training based on the multimodal AI large model to obtain a text recognition model. Even when the target device has no network connection, the text in the image can be recognized through the text recognition model obtained by distillation training, reducing the application conditions and expanding the application scope. Among them, the processing device and the target device can be the same device or different devices. For the sake of easy understanding, the present application takes the processing device and the target device as the same device as an example for introduction.

[0026] To make the technical solutions of the present application clearer and easier to understand, the technical solutions of the present application will be introduced below with reference to the accompanying drawings. As Figure 1As shown in the figure, this is a flowchart of a text recognition method provided by an embodiment of the present application. The method includes: S101. The processing device obtains a sample image, and label text box information and label text information corresponding to the sample image.

[0027] The sample image is an image used for model training, and each sample image corresponds to label text box information and label text information. Among them, the label text box information can be represented by [x0, y0, w0, h0], where x0 represents the abscissa of the center point of the text box, y0 represents the ordinate of the center point of the text box, w0 represents the width of the text box, and h0 represents the height of the text box. The label text information can be represented by text. For example, the label text information can be the text in the label text box.

[0028] In some embodiments, the processing device can directly obtain the pre-labeled sample data, including the sample image, and the label text box information and label text information corresponding to the sample image.

[0029] In other embodiments, in order to improve the labeling efficiency, a multi-modal AI large model can be used for pre-labeling and subsequent manual review, so as to improve the labeling efficiency. Specifically, the processing device uses the multi-modal AI large model to pre-label the text box of the sample image and outputs the pre-labeling result; obtains the review passed information feedback for the pre-labeling result, and the review passed information includes the labeling result after review; determines the label text box information corresponding to the sample image according to the review passed information.

[0030] The multi-modal AI large model refers to a deep learning model with a huge number of parameters, usually including billions or even trillions of parameters. These models are trained through deep learning technology and can process and generate various types of data such as natural language, images, and audio. The pre-labeling result can refer to the text box information in the image to be labeled labeled by the multi-modal AI large model.

[0031] Exemplarily, the user can input "give the center point coordinates, width and height of the text box in the image" through the processing device and upload the image to be labeled. The processing device can input the information input by the user into the multi-modal AI large model. The multi-modal AI large model can identify the image to be labeled, so as to determine the center point coordinates, width and height of the text box in the image, and then obtain the text box information of the image to be labeled, and then display the text box information to the user.

[0032] Next, the user can review the information in the text box. The user can feedback the information of passing the review or not passing the review through the processing device. During the review process, the user can adjust the information in the text box. For example, when the text box is not accurately recognized, the height of the text box can be finely adjusted so that the text box covers all the text. In this way, the user can feedback the information of passing the review for the finely adjusted text box information. The information of passing the review includes the labeled result after passing the review, that is, the text box information adjusted by the user. Then, the processing device determines the text box information of the label corresponding to the image to be labeled for the information of the text box after passing the review. After the image to be labeled is labeled, it becomes a sample image, and then the text box information of the label of the sample image can be obtained.

[0033] The following introduces how to determine the label text information of the sample image.

[0034] In the process of determining the label text information of the sample image, the processing device can use at least 3 multi-modal AI large models to identify the sample image, and then determine the final label text information based on the output results of the multiple multi-modal AI large models. Specifically, using at least 3 multi-modal AI large models to identify the sample image, multiple sample text information and the confidence levels corresponding to the sample text information are obtained; if the lengths of the multiple sample text information are all the same, or the lengths of the multiple sample text information are all different, the text information with the highest confidence level among the multiple sample text information is determined as the label text information corresponding to the sample image. If there is a first type of sample text information set with the same length and a second type of sample text information set with different lengths among the multiple sample text information, the text information with the highest confidence level in the first type of sample text information set is determined as the label text information corresponding to the sample image.

[0035] For ease of understanding, an example is given below.

[0036] Taking the processing device using 3 multi-modal AI large models as an example, the processing device can receive the "Please identify the text in the image and give the confidence level" input by the user and the image to be labeled. The processing device can respectively give the information input by the user to the three multi-modal AI large models, so that the 3 multi-modal AI large models can respectively identify the sample image, and then obtain 3 recognition results, that is, the first recognition result, the second recognition result, and the third recognition result. Among them, the recognition result includes the sample text information in the image to be labeled and the confidence level of the sample text information. Then, the processing device can determine the label text information based on the length of the sample text information, where the length of the sample text information refers to the length of the text represented by the sample text information.

[0037] As Figure 2 shown, this figure is a schematic diagram of determining label text information provided by an embodiment of the present application.

[0038] The processing device can determine the lengths of the sample text information in the first recognition result, the second recognition result, and the third recognition result. If the lengths of these three sample text information are all the same or all different, the sample text information with the highest confidence among these three sample text information is determined as the label text information. If there is a first type of sample text information set with the same length and a second type of sample text information set with different lengths among these three sample text information, the text information with the highest confidence in the first type of sample text information set is determined as the label text information corresponding to the sample image.

[0039] Among them, the first type of sample text information set may include the sample text information corresponding to the first recognition result and the sample text information corresponding to the second recognition result. The lengths of the sample text information corresponding to the first recognition result and the sample text information corresponding to the second recognition result are the same, that is, the text information with the highest confidence is selected from the sample text information corresponding to the first recognition result and the sample text information corresponding to the second recognition result, and this text information is determined as the label text information. The second type of sample text information set may include the sample text information corresponding to the first recognition result and the sample text information corresponding to the third recognition result, and of course, it may also include the sample text information corresponding to the third recognition result.

[0040] In some embodiments, the user can review the label text information corresponding to the sample image and feedback the review passed information, and the review passed information includes the text information passed in the review.

[0041] In the embodiments of the present application, pre-labeling the sample image using the multi-modal AI large model can greatly save time and labor costs. Especially for large-scale image datasets, it can significantly improve the labeling efficiency; when labeling text boxes and text, it can accurately identify text in various fonts, font sizes, colors, and layout styles, reduce labeling errors caused by factors such as manual subjective judgment or visual fatigue, and improve the accuracy of labeling results; and in this solution, after labeling using the multi-modal AI large model, through the methods of manual review and recheck, the accuracy and quality of the labeling results are ensured; the multi-modal AI large model performs labeling according to fixed algorithms and model parameters, which can ensure the consistency of labeling results between different images, avoid the problem of inconsistent labeling standards that may exist among different labelers during manual labeling, make the labeling results more unified and standardized, and is conducive to subsequent data analysis and model distillation training.

[0042] S102. The processing device inputs the sample image into the single vision model and the residual network model to obtain the first sample text box information and the first sample text information.

[0043] Such as Figure 3As shown in the figure, this is a schematic diagram of a text recognition model provided by an embodiment of the present application. The text recognition model includes a feature extraction layer 310, a single vision model 320, a residual network model 330, a first fully connected layer 341, and a second fully connected layer 342. Among them, the feature extraction layer 310 includes an OpenCV (Open Source Computer Vision Library) feature extraction module 312 and a color feature extraction module 311. The single vision model 320 includes a first single vision model 321 and a second single vision model 322. The residual network model 330 includes at least a convolutional layer 331 and a residual block group 332. The convolutional layer 331 introduces a first attention mechanism module 3311. For example, the first attention mechanism module 3311 can be a Triplet Attention (convolutional triple attention) module. The residual block group 332 introduces a second attention mechanism module 3321, and the second attention mechanism module 3321 can be a SimilarityAttention (similarity attention) module.

[0044] In some embodiments, the processing device can input the sample image into the feature extraction layer 310. After being processed by the single vision model 320, the residual network model 330, the first fully connected layer 341, and the second fully connected layer 342, the first sample text box information and the first sample text information are obtained.

[0045] In some embodiments, the processing device can extract color features from the sample image to obtain the first sample feature, and use the Open Source Computer Vision Library to extract features from the sample image to obtain the second sample feature. The processing device can input the sample image into the color feature extraction module 311 and the OpenCV feature extraction module 312 respectively for feature extraction.

[0046] Specifically, the color feature extraction module 311 can call the cv2.imread function to read the data of the sample image, and use the cv2.cvtColor function to convert the channels of the sample image data from BGR to RGB. At this time, the dimension is the original dimension [B1, C1, H1, W1], where B1 represents the batch, C1 represents the channel, H1 represents the height of the image, and W1 represents the width of the image. Here, C1 = 3. Then, normalize the RGB original pixel values of the sample image. For example, if the RGB original pixel values are [230, 100, 20], after normalization, first divide by 255 to get [0.0784, 0.0824, 0.0980]. Then, subtract the mean values [0.485, 0.456, 0.406] in the RGB order respectively, and divide by the standard deviations [0.229, 0.224, 0.225] to finally get [-1.7754, -1.6681, -1.3687]. The purpose of this step is to make all the processed values have a zero mean and unit variance, which can provide faster convergence and more stable training. Use the single vision model 320 (for example, the vision model is separately extracted from the multi-modal AI large model) as the backbone network for the first stage of optical character recognition. Here, make full use of the massive knowledge compression ability of the single vision model 320, and the size of the input first sample feature is not compressed, which can ensure high resolution and greatly improve the subsequent text recognition accuracy. Then, input this first sample feature into the subsequent first single vision model 321 to obtain the first sample text box feature, which can learn rich color information, texture features, illumination and shadow change features. These features are of great value for processing text in complex scenes. Color information helps to distinguish colored text from the background, illumination characteristics can enhance the ability to distinguish real characters and pseudo characters, and multi-channel information fusion improves the comprehensiveness of feature extraction. Especially in color documents, complex backgrounds or natural scenes, RGB images can provide richer visual information than grayscale images, helping the text recognition model achieve more accurate text localization and recognition.

[0047] The OpenCV feature extraction module 312 can obtain enhanced features in the sample image, with a dimension of [B2, C2, H2, W2]. Among them, B2 represents the batch, C2 represents the channel, H2 represents the height of the image, and W2 represents the width of the image. Here, C2 = 1. Since the single vision model 320 takes three-channel data as input, it is necessary to expand [B2, C2, H2, W2] in the 1 dimension. The content can be directly repeated to obtain a new feature [B2, C3, H2, W2], denoted as the second sample feature, where C3 = 3, and this second sample feature is three-channel data. Inputting the second sample feature into the second single vision model 322 can obtain the second sample text box feature. From this clear and concise second sample feature, the shape, contour, geometric structure, and texture pattern of the characters can be learned, and at the same time, the contrast between the characters and the background can be enhanced. This preprocessing effectively reduces background interference and noise effects, highlights the character area, that is, the text box area, making it easier for the text recognition model to extract key information and improving its robustness to complex scenarios and low-quality images, thereby significantly improving the accuracy of character recognition.

[0048] After the processing device obtains the first sample text box feature and the second sample text box feature, the first sample text box feature and the second sample text box feature can be fused to obtain a sample text box fusion feature.

[0049] Then, the sample text box fusion feature is passed through the first fully connected layer to obtain the first sample text box information, for example, [x1, y1, w1, h1, conf1], where x1 represents the abscissa of the center point of the first sample text box, y1 represents the ordinate of the center point of the first sample text box, w1 represents the width of the first sample text box, h1 represents the height of the first sample text box, and conf1 represents the confidence of the first sample text box.

[0050] After the processing device obtains the sample text box fusion feature, the sample text box fusion feature can be input into the residual network model 330 to obtain a sample text feature, and through the second fully connected layer 342, the first sample text information corresponding to the sample text feature is obtained.

[0051] The convolutional layer 331 of the residual network model 330 introduces a first attention mechanism module 3311. The first attention mechanism module 3311 is used to perform parallel processing on the sample image from the channel dimension, height dimension, and width dimension to obtain a multi-dimensional fusion feature; the residual block group 332 introduces a second attention mechanism module 3321. The second attention mechanism module 3321 is used to enhance the features of similar parts in the multi-dimensional fusion feature to obtain a sample text feature.

[0052] The first attention mechanism module 3311 enhances the feature selection ability through channel-spatial joint modeling, specifically including three branches: extracting important channel features through cross-channel interaction; modeling spatial dependencies along the height direction of the image to capture vertical context; and modeling dependencies along the width direction to supplement horizontal context. After fusing the outputs of the three, the model can capture multi-dimensional feature interactions more comprehensively, replacing the local convolution limitation of traditional CNNs.

[0053] The second attention mechanism module 3321 is embedded in the BasicBlock (residual block group) to dynamically enhance the response of feature similarity regions. The specific process is as follows: calculating the correlation between different positions in the feature map through inner product or cosine similarity; normalizing the similarity matrix (Softmax) to generate an attention mask; aggregating features weighted by the mask to highlight local regions related to the task (such as text regions); through this mechanism, the model can adaptively focus on important regions and improve the discrimination ability for text and positions.

[0054] S103. The processing device inputs the sample image and processing requirements into the multi-modal AI large model to obtain the second sample text box information and the second sample text information.

[0055] The processing requirements refer to the requirements input by the user. For example, it can be "Please give the center point coordinates of the text box in the image, the width and height of the text box, and identify the text in the text box".

[0056] The processing device can input the sample image and processing requirements input by the user into the multi-modal AI large model, and then obtain the second sample text box information and the second sample text information output by the multi-modal AI large model. Among them, the second sample text box information can be [x2, y2, w2, h2, conf2], where x2 represents the abscissa of the center point of the first sample text box, y2 represents the ordinate of the center point of the first sample text box, w2 represents the width of the first sample text box, h2 represents the height of the first sample text box, and conf2 represents the confidence of the first sample text box. Among them, conf2 is optional.

[0057] Among them, the execution order of S102 and S103 is not specifically limited in this application. In some embodiments, S102 and S103 can be executed simultaneously, or S103 can be executed first and then S102.

[0058] S104. The processing device determines the distillation loss value according to the sample text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information, and the second sample text information.

[0059] In some embodiments, the processing device determines a first loss value according to the label text box information and the first sample text box information, determines a second loss value according to the first text box information and the second sample text box information, determines a third loss value according to the label text information and the first sample text information, and determines a fourth loss value according to the first text information and the second sample text information; and determines a distillation loss value according to the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0060] The following is an example. The processing device can use the Generalized Intersection over Union (GIoU) method to determine the loss value.

[0061] GIoU is an extension based on IoU (Intersection over Union), which additionally considers the inclusion relationship and relative position information between two bounding boxes. When calculating GIoU, first calculate the smallest enclosing box C that contains the two bounding boxes, then calculate the area of the region in C except for the union of the two bounding boxes, denoted as C - (A ∪ B), where A represents the area of the predicted bounding box (such as the first sample text box), B represents the area of the ground truth bounding box (such as the label text box), and then calculate the first loss value through the following formula: L1 = IoU1 - (C - (A ∪ B)) / C, where IoU1 = (A ∩ B) / (A ∪ B). Here, A ∩ B represents the area of the overlapping part of the two bounding boxes, and A ∪ B represents the total area of the two bounding boxes after merging.

[0062] In some other examples, the label text box information is [25, 30, 55, 20], and the first sample text box information is [26, 29, 56, 22]. The processing device can calculate the first loss value through the following formula: L1 = (|x0 - x1| + |y0 - y1| + |w0 - w1| + |h0 - h1|) / 4.

[0063] Here, L1 represents the first loss value. In the above example, L1 = 1.25.

[0064] The determination methods of the second loss value, the third loss value, and the fourth loss value are similar and will not be elaborated here.

[0065] After obtaining the first loss value, the second loss value, the third loss value, and the fourth loss value, the processing device can sum them up to obtain the distillation loss value, that is LOSS = L1 + L2 + L3 + L4.

[0066] Here, LOSS represents the distillation loss value, L2 represents the second loss value, L3 represents the third loss value, and L4 represents the fourth loss value.

[0067] S105. The processing device updates the parameters of the text recognition model according to the distillation loss value.

[0068] After obtaining the distillation loss value, the processing device can update the parameters of the text recognition model based on the distillation loss value, so as to obtain a text recognition model that meets the requirements.

[0069] In the embodiments of the present application, by minimizing the distillation loss value, the text recognition model can learn richer and more effective features. For example, the teacher model (multi-modal AI large model) has been fully trained on a large amount of data and contains a lot of prior knowledge about text recognition-related features. The student model (text recognition model) adjusts its parameters through the distillation loss and can draw on this knowledge, so as to perform more accurately in the text recognition task and improve the recognition accuracy and recall rate.

[0070] Moreover, when calculating the distillation loss value, not only the sample labels are considered, but also the output results of the teacher model are taken into account. The combination of the two can enable the student model to draw on the generalization ability and feature extraction advantages of the teacher model while closely fitting the true categories of the data, thereby better improving the performance of the text recognition model.

[0071] If only relying on the output of the teacher model, it may cause the student model to overfit the specific pattern of the teacher model and ignore some inherent features of the data. Relying solely on the label information may affect the stability of the model due to problems such as data noise or annotation errors. The combination of the two can balance these two situations to a certain extent, making the student model more stable during the learning process, more tolerant of the noise in the data and the imperfections of the teacher model, and improving the robustness and generalization ability of the model.

[0072] The output of the teacher model provides a soft target for the student model. This soft target contains more information than the hard label, and the label information provides a clear supervision signal. The combination of the two provides a richer learning signal for the student model, enabling the student model to find the optimal parameter space faster, accelerating the convergence speed of the model, reducing the training time and resource consumption. Further, introducing the label information can provide a basic supervision bottom line for model training, ensuring that the student model can at least learn the basic information of the data, avoiding model degradation, and ensuring the effectiveness of model training.

[0073] S106. Obtain the image to be recognized.

[0074] S107. Input the image to be recognized into the text recognition model to obtain the text recognition result.

[0075] After the processing device completes the training of the text recognition model, it can input the image to be recognized into the text recognition model to obtain the text recognition result output by the text recognition model. Among them, the text recognition result can be the text information in the image, or can include the text box information in the image, etc.

[0076] Based on the above description, an embodiment of the present application provides a method for text recognition. This method uses the method of distillation training to train a text recognition model. Specifically, sample images and the corresponding label text box information and label text information are obtained. Then, the sample images are input into a single vision model and a residual network model to obtain the first sample text box and the first sample text information. The sample images and processing requirements are input into a multi-modal AI large model to obtain the second sample text box information and the second sample text information. Finally, distillation learning is performed using the output result of the multi-modal AI large model, that is, based on the label text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information, and the second sample text information, a distillation loss value is determined, and the parameters of the text recognition model are updated using the distillation loss value, thereby realizing distillation learning. In this solution, the text recognition model is distilled and trained through the output result of the multi-modal AI large model, which can enable the text recognition model to learn more extensive and more general feature representations, so that the text recognition model can better adapt to and process new images, thereby improving the generalization ability of the text recognition model and reducing the risk of overfitting. The text recognition result obtained by the text recognition model obtained through this solution for the image to be recognized will also be more accurate.

[0077] An embodiment of the present application also provides a model training method. This model training method is similar to the above S101-S105 and will not be elaborated here.

[0078] As described above in conjunction with Figures 1 - 3 the method for text recognition provided by the embodiment of the present application has been introduced in detail. Next, the device and equipment provided by the embodiment of the present application will be introduced in conjunction with the drawings.

[0079] As Figure 4 shown, this figure is a schematic diagram of a device for text recognition provided by an embodiment of the present application. The device includes: An acquisition module 401, configured to acquire an image to be recognized; A recognition module 402, configured to input the image to be recognized into a text recognition model to obtain a text recognition result; The training module 403 is used to obtain a sample image and the corresponding label text box information and label text information of the sample image; input the sample image into a single vision model and a residual network model to obtain first sample text box information and first sample text information; input the sample image and processing requirements into a multi-modal AI large model to obtain second sample text box information and second sample text information; determine a distillation loss value according to the label text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information and the second sample text information; update the parameters of the text recognition model according to the distillation loss value.

[0080] Optionally, the training module 403 is specifically configured to extract color features of the sample image to obtain first sample features; use an open-source computer vision library to extract features of the sample image to obtain second sample features; input the first sample features into a first single vision model to obtain first sample text box features, input the second sample features into a second single vision model to obtain second sample text box features, and fuse the first sample text box features and the second sample text box features to obtain sample text box fusion features; obtain the first sample text box information corresponding to the sample text box fusion features through a first fully connected layer; input the sample text box fusion features into a residual network model to obtain sample text features; obtain the first sample text information corresponding to the sample text features through a second fully connected layer.

[0081] Optionally, the training module 403 is specifically configured to determine a first loss value according to the label text box information and the first sample text box information, determine a second loss value according to the first text box information and the second sample text box information, determine a third loss value according to the label text information and the first sample text information, and determine a fourth loss value according to the first text information and the second sample text information; determine a distillation loss value according to the first loss value, the second loss value, the third loss value and the fourth loss value.

[0082] Optionally, the training module 403 is specifically configured to pre-annotate the text box of the sample image by using a multi-modal AI large model and output a pre-annotation result; obtain review approval information feedback for the pre-annotation result, where the review approval information includes the annotation result after review approval; determine the label text box information corresponding to the sample image according to the review approval information.

[0083] Optionally, the training module 403 is specifically configured to use at least three multi-modal AI large models to identify the sample image, obtain multiple sample text information and the confidence corresponding to the sample text information; if the lengths of the multiple sample text information are all the same, or the lengths of the multiple sample text information are all different, the text information with the highest confidence in the multiple sample text information is determined as the label text information corresponding to the sample image.

[0084] Optionally, the training module 403 is further configured to, if there is a first type of sample text information set with the same length and a second type of sample text information set with different lengths in the multiple sample text information, determine the text information with the highest confidence in the first type of sample text information set as the label text information corresponding to the sample image.

[0085] Optionally, the residual network model includes at least a convolutional layer and a residual block group; The convolutional layer includes a first attention mechanism module, and the first attention mechanism module is configured to perform parallel processing on the sample image from the channel dimension, height dimension, and width dimension to obtain a multi-dimensional fusion feature; The residual block group includes a second attention mechanism module, and the second attention mechanism module is configured to perform feature enhancement on similar parts in the multi-dimensional fusion feature to obtain sample text features.

[0086] The text recognition device according to the embodiment of the present application can correspond to execute the method described in the embodiment of the present application, and the above other operations and / or functions of each module / unit of the text recognition device are respectively for implementing Figure 1 the corresponding processes of the respective methods in the illustrated embodiments, and for the sake of brevity, will not be described in detail here.

[0087] The embodiment of the present application also provides a computing device. As Figure 5 shown, this figure is a schematic diagram of a computing device provided by the embodiment of the present application. The computing device 500 includes a bus 501, a processor 502, a communication interface 503, and a memory 504. The processor 502, the memory 504, and the communication interface 503 communicate with each other through the bus 501.

[0088] The bus 501 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0089] The processor 502 can be any one or more of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0090] The communication interface 503 is used for external communication.

[0091] The memory 504 may include volatile memory, such as random access memory (RAM). The memory 504 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0092] Executable code is stored in the memory 504, and the processor 502 executes the executable code to perform the foregoing text recognition method.

[0093] Specifically, in the case of implementing Figure 4 the illustrated embodiment, and Figure 4 when each module or unit of the text recognition device described in the embodiment is implemented by software, the software or program code required to execute Figure 4 the functions of each module / unit in can be partially or entirely stored in the memory 504. The processor 502 executes the program code corresponding to each unit stored in the memory 504 to perform the foregoing text recognition method.

[0094] An embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state drive). The computer-readable storage medium includes instructions that direct the computing device to perform the above text recognition method.

[0095] The embodiments of the present application also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, they generate, in whole or in part, the processes or functions described in the embodiments of the present application.

[0096] The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, or data center to another website, computer, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.).

[0097] When the computer program product is executed by a computer, the computer executes any one of the methods for text recognition described above. The computer program product can be a software installation package. In the case where any one of the methods for text recognition described above is needed, the computer program product can be downloaded and executed on the computer.

[0098] The descriptions of the processes or structures corresponding to the above respective drawings have their own focuses. For parts not detailed in a certain process or structure, reference can be made to the relevant descriptions of other processes or structures.

[0099] As described above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application.

Claims

1. A method for text recognition, characterized in that, The method includes: Obtain the image to be recognized; Input the image to be recognized into a text recognition model to obtain a text recognition result; Among them, the text recognition model is trained in the following way: Obtain a sample image and the corresponding label text box information and label text information of the sample image; Input the sample image into a single vision model and a residual network model to obtain first sample text box information and first sample text information; Input the sample image and processing requirements into a multi-modal AI large model to obtain second sample text box information and second sample text information; Determine a distillation loss value according to the label text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information and the second sample text information; Update the parameters of the text recognition model according to the distillation loss value.

2. The method according to claim 1, wherein The step of inputting the sample image into a single vision model and a residual network model to obtain first sample text box information and first sample text information includes: Extract color features from the sample image to obtain first sample features; use an open-source computer vision library to extract features from the sample image to obtain second sample features; Input the first sample features into a first single vision model to obtain first sample text box features, input the second sample features into a second single vision model to obtain second sample text box features, and fuse the first sample text box features and the second sample text box features to obtain sample text box fusion features; Obtain first sample text box information corresponding to the sample text box fusion features through a first fully connected layer; Input the sample text box fusion features into a residual network model to obtain sample text features; Obtain first sample text information corresponding to the sample text features through a second fully connected layer.

3. The method according to claim 2, characterized in that The step of determining the distillation loss value according to the label text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information and the second sample text information includes: Determine a first loss value according to the label text box information and the first sample text box information, determine a second loss value according to the first text box information and the second sample text box information, determine a third loss value according to the label text information and the first sample text information, and determine a fourth loss value according to the first text information and the second sample text information; Determine the distillation loss value according to the first loss value, the second loss value, the third loss value and the fourth loss value.

4. The method according to claim 1, characterized in that The label text box information corresponding to the sample image is obtained in the following way: Use a multi-modal AI large model to pre-annotate the text box of the sample image and output a pre-annotation result; Obtain review passed information feedback for the pre-annotation result, and the review passed information includes the annotation result after review passed; Determine the label text box information corresponding to the sample image according to the review passed information.

5. The method according to claim 1, wherein The label text information corresponding to the sample image is obtained in the following way: The sample image is recognized using at least three multimodal AI large models to obtain multiple sample text information and the confidence levels corresponding to the sample text information; If the lengths of the multiple sample text information are all the same, or the lengths of the multiple sample text information are all different, the text information with the highest confidence level among the multiple sample text information is determined as the label text information corresponding to the sample image.

6. The method according to claim 5, characterized in that, The method further includes: If there is a first type of sample text information set with the same length and a second type of sample text information set with different lengths among the multiple sample text information, the text information with the highest confidence level in the first type of sample text information set is determined as the label text information corresponding to the sample image.

7. The method according to claim 2, characterized in that, The residual network model includes at least a convolutional layer and a residual block group; The convolutional layer includes a first attention mechanism module, and the first attention mechanism module is used to perform parallel processing on the sample image from the channel dimension, height dimension, and width dimension to obtain a multi-dimensional fusion feature; The residual block group includes a second attention mechanism module, and the second attention mechanism module is used to enhance the features of the similar parts in the multi-dimensional fusion feature to obtain sample text features.

8. A model training method, characterized in that, The method includes: Obtain a sample image and the label text box information and label text information corresponding to the sample image; Input the sample image into a single vision model and a residual network model to obtain first sample text box information and first sample text information; Input the sample image and processing requirements into a multimodal AI large model to obtain second sample text box information and second sample text information; Determine a distillation loss value according to the label text box information, the first sample text box information, the second sample text box information, the label text information, the first sample text information, and the second sample text information; Update the parameters of the text recognition model according to the distillation loss value.

9. A computing device, characterized in that, Includes a memory and a processor; Wherein, one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the computing device is enabled to execute the text recognition method according to any one of claims 1 to 7 and the model training method according to claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the text recognition method according to any one of claims 1 to 7 and the model training method according to claim 8.

Images