Image text recognition method and device, computer program product and electronic equipment

Abstract

The invention discloses an image text recognition method and device, a computer program product and electronic equipment. The method comprises the steps of obtaining a to-be-recognized target image; a plurality of pre-trained neural network models are utilized to perform preprocessing operation on the target image, and the preprocessing operation at least comprises textbox detection, direction correction and textbox expansion; and performing text recognition on the preprocessed target image by using a pre-trained text recognition model to obtain corresponding target text content. The technical problem that the text content in the picture cannot be quickly and accurately recognized in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of deep learning, and in particular, to an image text recognition method, an apparatus, a computer program product, and an electronic device. Background Art

[0002] Currently, in the real-name verification business of operators, there are no strict specifications for the photographing devices and writing requirements, which may result in unclear key texts in the pictures and unconventional directions of numbers. The verification operations of operators on relevant pictures usually include two major steps: text detection and text recognition, and both of these two steps have certain requirements for the picture direction. Therefore, when directly analyzing the uploaded original pictures, it may not be possible to accurately identify the text information in the files. For example, if the original number in Picture A is "87654", if the picture rotated 180 degrees is detected and recognized, the final recognition result is likely to be "EhS9L8", thus greatly reducing the accuracy of the verification business.

[0003] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present application provide an image text recognition method, an apparatus, a computer program product, and an electronic device, so as to at least solve the technical problem that the related art cannot quickly and accurately identify the text content in pictures.

[0005] According to one aspect of the embodiments of the present application, an image text recognition method is provided, including: obtaining a target image to be recognized; performing preprocessing operations on the target image by using a plurality of pre-trained neural network models, where the preprocessing operations at least include: text box detection, direction correction, and text box expansion; performing text recognition on the preprocessed target image by using a pre-trained text recognition model to obtain corresponding target text content.

[0006] Optionally, performing preprocessing operations on the target image by using a plurality of pre-trained neural network models includes: performing text detection on the target image by using a text detection model in the plurality of neural network models to determine a first text box corresponding to each text information in the target image; for each first text box in the target image, performing direction analysis on the first text box by using a text direction classification model in the plurality of neural network models to obtain a direction classification result of the first text box, and determining an included angle between the first text box and a preset horizontal direction according to the direction classification result, and performing direction correction on the first text box according to the included angle; determining the target image after direction correction according to the first text boxes after direction correction, and performing text detection on the target image after direction correction again by using the text detection model to determine a second text box corresponding to each of the multiple text information in the target image; performing an expansion process on each second text box.

[0007] Optionally, the training process of the text detection model includes: obtaining multiple groups of first training sample data, where each group of first training sample data includes an image sample and text box samples corresponding to the minimum bounding rectangles of each text information in the image sample; constructing a first neural network model; for each group of first training sample data, inputting the image sample in the first training sample data into the first neural network model to obtain predicted text boxes corresponding to the minimum bounding rectangles of each text information in the image sample output by the first neural network model; constructing a first objective loss function based on each text box sample and the corresponding predicted text box in each group of first training sample data, and adjusting the parameters of the first neural network model based on the first objective loss function until the model converges, obtaining a trained text detection model.

[0008] Optionally, the training process of the text direction classification model includes: obtaining multiple groups of second training sample data, where each group of second training sample data includes a text box sample and a corresponding true direction label, and the true direction label is used to reflect the angle between the text direction in the text box sample and a preset horizontal direction; constructing a second neural network model; for each group of second training sample data, inputting the text box sample in the second training sample data into the second neural network model to obtain a predicted direction label including the text box sample output by the second neural network model; constructing a second objective loss function based on the true direction label and the corresponding predicted direction label in each group of second training sample data, and adjusting the parameters of the second neural network model based on the second objective loss function until the model converges, obtaining a trained text direction classification model.

[0009] Optionally, the process of expanding each second text box includes: traversing each second text box, determining the size information of the second text box and the vertical distance between adjacent lines within the second text box, where the vertical distance between adjacent lines is in pixels; analyzing the size information and the vertical distance between adjacent lines using an object regression model based on the least squares support vector regression algorithm to obtain an expansion coefficient of the second text box, where the object regression model is iteratively trained based on multiple text box samples and corresponding expansion coefficient labels, and the expansion coefficient label corresponding to a text box sample is an expansion coefficient selected from a preset alternative expansion coefficient set corresponding to the text box sample with an identification confidence higher than a preset first threshold and a valid number of recognized characters higher than a preset second threshold; expanding the second text box based on the expansion coefficient.

[0010] Optionally, the preprocessing operation further includes: resolution enhancement. After expanding each second text box, the method further includes: traversing each second text box in the target image, determining the area and perimeter of the second text box, and combining the expansion coefficient of the second text box to determine the number of pixels of the expanded second text box according to the following formula:

[0011]

[0012] In the formula, d expand represents the number of pixels of the expanded second text box, area(rect) represents the area of the second text box, peri(rect) represents the perimeter of the second text box, and out Lssvr represents the expansion coefficient; perform grayscale processing and edge detection processing on the expanded second text box in sequence to determine the grayscale value of each pixel in the processed second text box; based on the number of pixels of the expanded second text box and the grayscale value of each pixel, determine the pixel grayscale variance value of the expanded second text box; when the pixel grayscale variance value is less than a preset third threshold, use a pre-trained text super-resolution model to enhance the resolution of the expanded second text box.

[0013] Optionally, the text super-resolution model at least includes: a structure prior module, a Transformer encoder, and a backbone network, where: the structure prior module is used to map the discrete encodings corresponding to multiple characters in the expanded second text box to a preset structure parameter space to obtain the high-resolution structure prior features of each character; the Transformer encoder is used to extract the category, bounding box, and font style of each character in the expanded second text box; the backbone network is used to extract features of each character in the expanded second text box to obtain the low-resolution features of each character; crop the expanded second text box according to the bounding box of each character detected by the Transformer encoder; fuse the structure prior features of each character output by the structure prior module and the low-resolution features of the character to obtain the super-resolution features of each character; map the super-resolution features of each character back to the original position in the expanded second text box to obtain the expanded and resolution-enhanced second text box.

[0014] Optionally, use a pre-trained text recognition model to perform text recognition on the preprocessed target image to obtain the corresponding target text content, including: for each expanded second text box in the target image, use the text recognition model to perform text recognition on the expanded and resolution-enhanced second text box to obtain the predicted character sequence corresponding to the second text box; based on the predicted character sequences corresponding to each expanded and resolution-enhanced second text box in the target image, determine the target text content corresponding to the preprocessed target image.

[0015] According to another aspect of the embodiments of the present application, an image text recognition device is further provided, including: an acquisition module for acquiring a target image to be recognized; a preprocessing module for performing preprocessing operations on the target image by using a plurality of pre-trained neural network models, where the preprocessing operations at least include: text box detection, orientation correction, and text box expansion; and a recognition module for performing text recognition on the preprocessed target image by using a pre-trained text recognition model to obtain corresponding target text content.

[0016] According to another aspect of the embodiments of the present application, a computer program product is further provided, including: a computer program, where when the computer program is executed by a processor, the above-mentioned image text recognition method is implemented.

[0017] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory and a processor, where a computer program is stored in the memory, and the processor is configured to execute the above-mentioned image text recognition method through the computer program.

[0018] In the embodiments of the present application, by using a plurality of pre-trained neural network models to sequentially perform preprocessing operations on the target image to be recognized, including but not limited to: text box detection, orientation correction, text box expansion, etc.; and then using a pre-trained text recognition model to perform text recognition on the preprocessed target image, the corresponding target text content can be obtained, achieving the technical effect of performing high-precision optical character recognition on the image, and achieving the purpose of improving the recognition accuracy, processing efficiency, and adapting to various text orientations and clarity. Furthermore, the technical problem that the related technology cannot quickly and accurately recognize the text content in the picture is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0020] Figure 1 is a schematic flowchart of an optional image text recognition method according to the embodiments of the present application;

[0021] Figure 2 is a schematic structural diagram of an optional text super-resolution model according to the embodiments of the present application;

[0022] Figure 3 is a schematic structural diagram of an optional image text recognition device according to the embodiments of the present application;

[0023] Figure 4It is a schematic structural diagram of an optional electronic device according to an embodiment of the present application. Detailed implementation manners

[0024] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] In order to better understand the embodiments of the present application, some nouns or terms that appear in the description process of the embodiments of the present application are translated and explained as follows:

[0027] OCR (Optical Character Recognition): It is a technology that converts the text in a picture into editable text. Therefore, it is widely used in fields such as document recognition, document retrieval, and screenshot recognition. Generally, the working process of OCR mainly includes steps such as preprocessing, text detection, feature extraction, and character recognition.

[0028] LSTM (Long Short Term Memory): It is a variant of RNN (Recurrent Neural Network). Due to the problem of gradient disappearance, RNN can only have short-term memory, while LSTM combines short-term memory and long-term memory through gate control, thus solving the problem of gradient disappearance to a certain extent.

[0029] PESNet (Progressive Scale Expansion Network): It is a text detection algorithm based on pixel embedding and shape regression. Its core idea is to generate the bounding boxes of text regions in a progressive expansion manner. This algorithm first segments the input image into different text instances, then generates preliminary text region candidate boxes through PLN (Proposal Localization Network), and further refines and corrects the candidate boxes using PRN (Proposal Refinement Network). Therefore, the key of PESNet lies in its ability to detect text regions of arbitrary shapes and output high-precision text boxes.

[0030] PANNet (Pixel Aggregation Network): It is an optimized version of PESNet, which has been improved in terms of grid structure and post-processing. PANNet is also based on the idea of instance segmentation, but uses a lighter ResNet as the Backbone and performs feature enhancement and fusion through FPEM (Feature Pyramid Enhancement Module) and FFM (Feature Fusion Module). Therefore, when predicting the core text region and text region, PANNet adopts a method of gradually expanding from the core region outward, effectively solving the problem of adhesion of text regions that are close to each other.

[0031] DBNet (Differentiable Binarization Network): It is a segmentation-based text detection algorithm. Its idea is similar to that of object detection algorithms, regarding the text in the image as the object to be detected and the rest as the background. DBNet realizes the automatic training threshold of the network by training the Threshold Map and using an approximate differentiable binarization function. In the test stage, DBNet can calculate an approximate binary map through the probability map and the threshold map, thereby obtaining the position of the text box.

[0032] MobileNet-V3: It is a lightweight neural network model proposed by the Google team in 2019, aiming to optimize deep learning tasks on mobile devices. MobileNet-V3 combines neural architecture search technology and the NetAdapt algorithm, and designs two models suitable for high-resource and low-resource usage scenarios through hardware-aware network architecture search: MobileNetV3-Large and MobileNetV3-Small.

[0033] LSSVR (Least Squares Support Vector Regression) algorithm: It is a variant of the SVR algorithm. Suykens transformed the inequality constraint into an equality constraint, changed the loss function from the sum of errors to the sum of the squares of errors, and the solution algorithm changed from solving a convex quadratic optimization problem to solving a system of linear equations. The number of solution variables decreased from 2N + 1 to N + 1 (N represents the number of training samples). Therefore, the LSSVR algorithm has a lower solution difficulty than the SVR algorithm and a faster training speed.

[0034] Vision Transformer (ViT): Its main idea is to divide the input image into multiple small patches, then convert each small patch into a vector, and finally splice these vectors according to the corresponding position information to form a sequence for subsequent classification.

[0035] AdaIN (Adaptive Instance Normalization): When using a GAN (Generative Adversarial Network) for image style transfer, the network is usually associated with a fixed set of styles and cannot adapt to new styles. Therefore, after passing through the AdaIN layer, it can align the mean and variance of the content features with the mean and variance of the style features.

[0036] CTC (Connectionist Temporal Classification) algorithm: It is an algorithm commonly used in fields such as semantic recognition and text recognition to solve the problem of inconsistent lengths of input and output sequences and inability to align. Among them, in CRNN (Convolutional Recurrent Neural Network), it is the loss function corresponding to the model.

[0037] Embodiment 1

[0038] According to the embodiments of the present application, an image text recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0039] Figure 1 It is a flowchart showing an image text recognition method provided according to the embodiments of the present application. As Figure 1 shown, the method includes the following steps:

[0040] Step S102, obtain the target image to be recognized.

[0041] Step S104, perform preprocessing operations on the target image using multiple pre-trained neural network models. Among them, the above preprocessing operations at least include: text box detection, orientation correction, and text box expansion.

[0042] Among them, the above text box detection is to locate the text area in the target image to obtain the coordinate information of the text box; orientation correction is to determine the angle by which the target image should be rotated by judging the orientation of the target image, so as to ensure that the target image is in the preset horizontal direction; text box expansion is to appropriately expand each text box to avoid missing recognition due to too small a selection range.

[0043] Step S106, perform text recognition on the preprocessed target image using a pre-trained text recognition model to obtain the corresponding target text content.

[0044] Based on the solution defined in the above steps S102 to S106, it can be known that in the embodiment of the present application, by using multiple pre-trained neural network models to perform preprocessing operations on the target image to be recognized in sequence, including but not limited to: text box detection, orientation correction, text box expansion, etc.; and then using a pre-trained text recognition model to perform text recognition on the preprocessed target image, the corresponding target text content can be obtained, achieving the technical effect of performing high-precision optical character recognition on the image, and achieving the purpose of improving the recognition accuracy, processing efficiency, and adapting to various text orientations and clarity.

[0045] The following will describe each step of the image text recognition method in combination with a specific implementation process.

[0046] In the technical solution provided in the above step S102, the target image to be recognized is different according to different application scenarios. For example, in the real-name verification business of operators, the above target pictures can be pictures of various identity documents submitted by users; in addition, the above target image can also be a picture of an application form or contract filled out by the user, and any picture containing text information for verification, such as bank statements, residence certificates, work certificates, etc.

[0047] Since these pictures may be obtained by mobile phone shooting, scanner scanning or other image acquisition methods, and due to different shooting or scanning conditions, the quality and angle of the pictures may vary. Therefore, in order to ensure the accuracy of subsequent text recognition of the target image, the embodiment of the present application can perform preprocessing operations on the target image.

[0048] As an alternative implementation, in the technical solution provided in the above step S104, the target image can be preprocessed as follows, including:

[0049] Step S1041: Use the text detection model in multiple neural network models to detect text in the target image, and determine the first text box corresponding to each text information in the target image.

[0050] Among them, the above first text box is the text box corresponding to the minimum circumscribed rectangle containing each text information in the target image. This step is mainly to initially locate each text information in the target image and provide a basis for subsequent direction correction.

[0051] Step S1042: For each first text box in the target image, use the text direction classification model in multiple neural network models to analyze the direction of the first text box, obtain the direction classification result of the first text box, and determine the angle between the first text box and the preset horizontal direction according to the direction classification result, and correct the direction of the first text box according to the angle.

[0052] Among them, the above steps are mainly to correct the direction of the target image to ensure that the target image is presented in the correct reading direction in subsequent processing.

[0053] Step S1043: Determine the target image after direction correction according to the first text boxes after direction correction, and use the text detection model to detect text in the target image after direction correction again to determine the second text box corresponding to each text information in the target image.

[0054] Among them, each of the above second text boxes is the text box corresponding to the minimum circumscribed rectangle of each text information in the target image. This step is mainly to perform independent and accurate processing on each text information in the target image.

[0055] Step S1044: Perform an expansion process on each second text box.

[0056] This step is to ensure that the text box can fully cover each text information in the target image, while avoiding interference from too much irrelevant information, thereby improving the accuracy and efficiency of text recognition.

[0057] Optionally, the training process of the above text detection model for text detection can include:

[0058] The first step: Obtain multiple groups of first training sample data.

[0059] Among them, each group of first training sample data includes an image sample and a text box sample corresponding to the minimum circumscribed rectangle of each text information in the image sample.

[0060] Step 2: Construct the first neural network model. Among them, the model architecture of the first neural network model can be DBNet (Differentiable Binarization Network), PESNet (Progressive Scale Expansion Network), PANNet (Pixel Aggregation Network), etc. In the embodiments of the present application, DBNet is preferably used.

[0061] Step 3: For each group of first training sample data, input the image samples in the first training sample data into the first neural network model to obtain the predicted text boxes corresponding to the minimum bounding rectangles of each text information in the image samples output by the first neural network model.

[0062] Specifically, if the above first neural network model is DBNet, during the inference process, the front end of the DBNet model will use a convolutional neural network (such as ResNet, VGG or other structures) to extract the features of the image samples to obtain a feature map, where each pixel point corresponds to a feature vector; then, through a series of convolutional layers, upsampling layers and a final fully connected layer in the DBNet model, predict the probability value that each pixel point is text, and obtain a probability prediction map with the same size as the input image sample; then, compare the probability values corresponding to each pixel point in the probability prediction map with a preset threshold to binarize the conversion of the prediction map, that is, mark the pixel points with probability values higher than the threshold as text (usually assigned 1), and mark the pixel points lower than the threshold as non-text (usually assigned 0); perform connected component analysis on the binarized probability prediction map or use contour detection algorithms (such as Canny edge detection, Sobel operator, etc.) for analysis to determine the text contour containing text; finally, for each detected text contour, determine the minimum bounding rectangle with the smallest area and completely containing the text contour, and output the minimum bounding rectangle as the marked predicted text box.

[0063] Step 4: Construct a first target loss function based on each text box sample and the corresponding predicted text box in each group of first training sample data, and adjust the parameters of the first neural network model based on the first target loss function until the model converges to obtain a trained text detection model.

[0064] Through the above steps, the training of the text detection model can be completed, so that the model can be directly used to locate the text area in the target image during subsequent inference, and obtain the first text box corresponding to each text information in the target image.

[0065] Optionally, the training process of the above text direction classification model for text direction classification may include:

[0066] Step 1: Obtain multiple groups of second training sample data. Each group of second training sample data includes text box samples and corresponding true direction labels, and the true direction labels are used to reflect the angle between the text direction in the text box sample and the preset horizontal direction, such as 0 degrees, 90 degrees, 180 degrees, 270 degrees, etc.

[0067] Specifically, the above-mentioned second training sample data can be obtained in the following way: Obtain multiple text image samples. The text image samples can be composed of a single image file, or can be spliced by two or more image samples in the same direction, and the text image samples include texts in different directions; perform preprocessing on the multiple text image samples to obtain corresponding multiple text box samples. The preprocessing operations include but are not limited to: blank area deletion, text box clipping, image enhancement processing; perform text direction annotation on each text box sample obtained by preprocessing. In this process, the actual rotation angle of each text box sample can be annotated manually or using an automated tool to ensure that each sample has an accurate direction label.

[0068] Step 2: Construct a second neural network model. The model architecture of the second neural network model is preferably MobileNet-V3.

[0069] Step 3: For each group of second training sample data, input the text box samples in the second training sample data into the second neural network model to obtain the predicted direction labels of the text box samples output by the second neural network model.

[0070] Step 4: Construct a second objective loss function based on the true direction labels and the corresponding predicted direction labels in each group of second training sample data, and adjust the parameters of the second neural network model based on the second objective loss function until the model converges to obtain a trained text direction classification model.

[0071] Through the above steps, the training of the text direction classification model can be completed, so that in subsequent inference, the model can be directly used to perform direction classification on each first text box in the target image to obtain the direction classification results of each first text box, and determine the rotation angle of the corresponding first text box according to the direction classification results of each first text box, so that the first text box is rotated until the text direction is 0 degrees defined by the text direction classification model, that is, the horizontal direction. For example, if the direction classification result of a certain first text box is 90 degrees, then the first text box can be rotated 90 degrees to make it in the 0 degrees defined by the text direction classification model.

[0072] Since the text direction has a certain impact on the text detection result, after correcting the directions of the first text boxes in the target image, the target image after direction correction is determined based on the first text boxes after direction correction, and the text detection model is used again to perform text detection on the target image after direction correction to determine the second text boxes corresponding to multiple text information in the target image.

[0073] Furthermore, considering that the target image may have a problem of decreased resolution at the edge due to different acquisition methods, resulting in insufficient clarity of the text in the non-central region of the target image. For example, the text content "notice letter" may not detect the right half of the character "letter" due to insufficient resolution or being located at the edge of the image, resulting in the finally recognized text content being "notice commitment".

[0074] Therefore, in order to avoid the above problems and improve the accuracy of text recognition for each subsequent second text box in this application embodiment, the outer expansion process for the second text boxes corresponding to each text information in the target image is eliminated, so as to ensure that the text box after outer expansion can completely cover the text information. It should be noted that the reason for not performing the outer expansion operation after obtaining the first text boxes corresponding to each text information in the target image in the above step S1041 is that the purpose of obtaining the first text box is to find the area where the text information is located in the target image, rather than performing character recognition. Therefore, whether to perform the outer expansion operation will not affect the judgment of the text direction of the corresponding area.

[0075] Optionally, in the technical solution provided in the above step S1044, the outer expansion process for each second text box can be performed in the following manner, including:

[0076] The first step: Traverse each second text box to determine the size information of the second text box and the vertical line spacing within the second text box. Among them, the above size information includes: the width and height of the second text box, and the vertical line spacing is in pixels.

[0077] The second step: Use the target regression model based on the least squares support vector regression algorithm to analyze the size information and the vertical line spacing to obtain the expansion coefficient of the second text box.

[0078] Specifically, the above-mentioned target regression model is iteratively trained based on multiple text box samples and corresponding dilation coefficient labels. Among them, regarding the determination of the dilation coefficient label, first, the dilation coefficient range (i.e., the maximum value and the minimum value) corresponding to each text box sample is determined according to experience, and N equally divided points are determined within this range, forming N + 2 alternative dilation coefficients. These N + 2 alternative dilation coefficients are sequentially applied to the text box sample, obtaining N + 2 text recognition results. One with the highest recognition effect is selected as the dilation coefficient label corresponding to the text box sample according to the following principles: (1) A longer number of valid recognized characters, where valid characters refer to the text obtained by removing the leading and trailing spaces from the recognized string; (2) A higher recognition confidence. Therefore, the dilation coefficient label corresponding to the text box sample can be understood as: a dilation coefficient selected from the set of pre-set alternative dilation coefficients corresponding to the text box sample, with a recognition confidence higher than a pre-set first threshold value and a number of valid recognized characters higher than a pre-set second threshold value.

[0079] Step 3: Perform an outward expansion process on the second text box based on the dilation coefficient.

[0080] Through the above outward expansion process, the dilation coefficient for each second text box can be calculated respectively to achieve the effect of adaptive expansion. Compared with a fixed dilation coefficient, for example, in the open-source PaddleOCR, the method of combining the hyperparameter Vatti Clipping with the hyperparameter unclip_ratio is adopted. Since such hyperparameters are determined in advance according to empirical values, they cannot meet all scenarios. Additionally, if the value of this parameter is too small, it may not be able to well contain all the text pixels to be recognized; conversely, if the value is too large, not only will the area ratio of the text information within the corresponding text box become smaller, but it is very likely to include the pixel information of the upper and lower lines belonging to the text to be recognized, resulting in a decrease in the recognition confidence. Therefore, the above-mentioned outward expansion process provided by the embodiments of the present application can well avoid the problems existing in the existing methods and ensure the accurate recognition of the correct text in the subsequent process.

[0081] In addition, a clear high-resolution image is a prerequisite for ensuring accurate recognition results. However, in application scenarios (such as telecommunications auditing), due to reasons such as slight shaking of the lens when taking the image and low resolution of the imaging device, the collected target image is blurred and unclear, thus affecting the subsequent text recognition accuracy. For this reason, the embodiments of the present application propose to perform resolution detection and enhancement on the second text box after outward expansion to ensure that the text in the image is clear enough and improve the accuracy of the final text recognition result.

[0082] Optionally, the following method can be used to perform resolution enhancement processing on each second text box after outward expansion, including:

[0083] Step 1: Traverse each second text box within the target image, determine the area and perimeter of the second text box, and in combination with the expansion coefficient of the second text box, determine the number of pixels of the second text box after external expansion according to the following formula:

[0084]

[0085] In the formula, d expand represents the number of pixels of the second text box after external expansion, area(rect) represents the area of the second text box, peri(rect) represents the perimeter of the second text box, and out Lssvr represents the expansion coefficient.

[0086] Step 2: Perform grayscale processing and edge detection processing on the second text box after external expansion in sequence to determine the grayscale values of each pixel within the processed second text box.

[0087] Among them, by performing grayscale processing on the second text box after external expansion, it is converted into a grayscale image; then, an edge detection algorithm is applied to perform edge detection on the grayscale image. If the Laplace operator is used to perform edge detection on the grayscale image, a Laplacian image can be obtained, and the pixel grayscale values of the edge part within this Laplacian image will increase significantly, thereby helping to identify the boundaries and details of the text information. Among them, the Laplace operator is a second-order differential operator that can detect sharp changes in grayscale values in an image, that is, edges.

[0088] Step 3: Based on the number of pixels of the second text box after external expansion and the grayscale value of each pixel, determine the pixel grayscale variance value of the second text box after external expansion.

[0089] Among them, the above pixel grayscale variance value can also be called the gradient variance of the pixel points, and this variance reflects the degree of dispersion of the brightness changes of the pixel points within the target image. Among them, a clear image will have more significant pixel gradient changes, while the pixel gradient changes of a blurred or low-resolution image will be smaller.

[0090] Step 4: When the pixel grayscale variance is less than a preset third threshold value, it indicates that the clarity of the second text box after external expansion is insufficient. At this time, the pre-trained text super-resolution model can be used to perform resolution improvement processing on the second text box after external expansion. Among them, the above third threshold value is used to determine whether the second text box after external expansion is clear enough for text recognition, and it can usually be set based on experimental data and business requirements.

[0091] Considering that most of the currently common image super-resolution algorithms execute character recognition in parallel, and then constrain the text image super-resolution task through a loss function or through intermediate features. However, in the face of Chinese characters with complex strokes, this kind of constraint will fail. Therefore, the embodiments of this application mine the structural prior features of the image from the generative network, and use the structural prior features to promote the super-resolution task of the text image.

[0092] Optionally, the above text super-resolution model can also be called AISRNet (Audit Image SuperResolution Net), which at least includes: a structural prior module, a Transformer encoder, and a backbone network, and the flow structure between each part is as Figure 2 shown. The following will combine Figure 2 to elaborate on each component of this model in detail.

[0093] (1) Structural prior module.

[0094] Specifically, the structural prior module includes StyleGAN (Style Generative AdversarialNetworks, style-based generative adversarial network), whose input is the discrete encoding representing different characters, and controls the style of the output image through the w∈W space, that is, maps the initial image style z∈Z. Among them, the discrete encoding of each character can be expressed as and X is the codebook storing all characters. The feature dimension of the discrete encoding of each character is 1*1*512. M represents the number of discrete encodings in the codebook, which covers common Chinese characters, English letters, numbers, etc., and its size is 6623. Therefore, the input-output relationship of the structural prior module can be written as:

[0095] I x = B(x,F(z),Φ G ) = B(x,w,Φ G )

[0096] In the formula, x represents the discrete encoding of a single character, F represents the network that maps the initial image style z to the w space, and Φ G represents the model parameters of StyleGA.

[0097] Therefore, the core of the structural prior module is to map the discrete encodings corresponding to multiple characters in the second text box after expansion to the preset structural parameter space, and obtain the high-resolution structural prior features of each character. These prior structural features are combined in the super-resolution network to help the network understand the structure and contour of each character, so that when restoring blurred text, the details of the characters can be restored more accurately, improving the accuracy of text recognition.

[0098] (2) Transformer Encoder

[0099] Since the input of the structure prior module is the output of the Transformer encoder, in order to ensure that the structure prior module generates the structure of each character, the Transformer encoder can be used to extract the category, bounding box, and font style of each character in the second text box after expansion first.

[0100] It should be noted that Figure 2 the Transformer encoder in [[]] is composed of a Vision Transformer and a CNN (Convolutional Neural Network), and also combines positional encoding to inject positional information. In the same text image, adjacent characters usually have the same font. Therefore, an additional linear layer is used in the Transformer encoder to map the features of all characters into the same prediction.

[0101] (3) Backbone Network

[0102] Specifically, in the backbone network, first, a simple UNet structure is stacked on a lightweight neural network to extract features of each character in the second text box after expansion, obtaining the low-resolution features of each character; the second text box after expansion is cropped according to the bounding box of each character detected by the Transformer encoder, and each character in the second text box after expansion is aligned; the structure prior features of each character output by the structure prior module and the low-resolution features of the character are fused and normalized to obtain the super-resolution features of each character; the super-resolution features of each character are mapped back to the original position in the second text box after expansion, obtaining the second text box after expansion and resolution improvement.

[0103] As an optional implementation manner, in the technical solution provided in step S106 above, the target image can be text-recognized according to the following process to obtain the corresponding target text content, including:

[0104] Step S1061, for each second text box after expansion in the target image, use the text recognition model to perform text recognition on the second text box after expansion and resolution improvement, obtaining the predicted character sequence corresponding to the second text box;

[0105] Step S1062, based on the predicted character sequences corresponding to each second text box after expansion and resolution improvement in the target image, determine the target text content corresponding to the preprocessed target image.

[0106] Specifically, the process of text recognition can be understood as follows: using the convolutional neural network in the text recognition model to perform patch embedding on each second text box after external expansion and resolution improvement, and embedding the position information of each second text box in the target image; for each externally expanded second text box in the target image, this step is cyclically executed: merging the width and height dimensions of the second text box, reducing the whole from three dimensions to two dimensions, then calculating the correlation of different pixel blocks in the spatial position through an attention module (including a global attention module and a local attention module), and then integrating and downsampling the result to restore the three-dimensional structure of the second text box; then, performing global pooling operation and fully connected layer processing on the feature sequence processed by the attention module along the height dimension to generate a compact feature sequence, which contains the feature representation of each character, and then using this feature sequence to generate a transcription sequence, where the same character is transcribed into repeated characters, and the non-text area is transcribed into a blank symbol; finally, using the CTC (Connectionist Temporal Classification) algorithm to analyze the transcription sequence to generate the final target text content.

[0107] In summary, in the image text recognition method provided by the embodiments of the present application, the image is initially detected by the text detection model to roughly determine the first text box where each text information in the image is located, and the file direction classification model is used to classify the directions of each first text box, so as to avoid errors in the model recognition result caused by abnormal local text directions; then, the text detection model is used again to perform text detection on the image after direction correction, accurately locate the second text box where each text information in the image is located, and use the least squares support vector regression algorithm to calculate the best expansion coefficient of each second text box, better for text boxes of different sizes and shapes, ensuring that the expanded second text box can contain sufficient character information, while avoiding unnecessary background interference, optimizing the input of the text, and improving the recognition accuracy; in addition, when the text is not clear enough, the resolution of the image is improved by the text super-resolution model, and the clarity of the blurred text is intelligently restored and enhanced, significantly improving the recognition effect of blurred or low-resolution pictures. The entire automated operation process reduces the need for manual intervention and speeds up the processing speed. Especially when processing a large number of pictures, the recognition efficiency can be significantly improved.

[0108] Embodiment 2

[0109] According to the embodiments of the present application, there is also provided an image text recognition device for implementing the image text recognition method in Embodiment 1, as Figure 3 shown. The image text recognition device at least includes: an acquisition module 32, a preprocessing module 34, and a recognition module 36, where:

[0110] An acquisition module 32, configured to acquire a target image to be recognized;

[0111] A preprocessing module 34, configured to perform preprocessing operations on the target image by using multiple pre-trained neural network models, where the preprocessing operations at least include: text box detection, orientation correction, and text box expansion;

[0112] A recognition module 36, configured to perform text recognition on the preprocessed target image by using a pre-trained text recognition model to obtain corresponding target text content.

[0113] The functions of each module of the image text recognition device are described below in combination with a specific implementation process.

[0114] Optionally, the above preprocessing module 34 can perform preprocessing operations on the target image according to the following steps, including:

[0115] Step S1, use the text detection model in the multiple neural network models to perform text detection on the target image to determine the first text box corresponding to each text information in the target image.

[0116] Wherein, the above first text box is the text box corresponding to the minimum circumscribed rectangle containing each text information in the target image. This step is mainly to initially locate each text information in the target image and provide a basis for subsequent orientation correction.

[0117] Step S2, for each first text box in the target image, use the text orientation classification model in the multiple neural network models to perform orientation analysis on the first text box to obtain the orientation classification result of the first text box, and determine the included angle between the first text box and the preset horizontal direction according to the orientation classification result, and perform orientation correction on the first text box according to the included angle.

[0118] Wherein, the above steps are mainly to perform orientation correction on the target image to ensure that the target image is presented in the correct reading direction in subsequent processing.

[0119] Step S3, determine the target image after orientation correction according to the first text boxes after orientation correction, and use the text detection model to perform text detection on the target image after orientation correction again to determine the second text box corresponding to each text information in the target image.

[0120] Wherein, each of the above second text boxes is the text box corresponding to the minimum circumscribed rectangle of each text information in the target image. This step is mainly to perform independent and precise processing on each text information in the target image.

[0121] Step S4, perform an outer expansion process on each second text box.

[0122] This step is to ensure that the text box can fully cover each text information within the target image, while avoiding interference from excessive irrelevant information, thereby improving the accuracy and efficiency of text recognition.

[0123] In addition, the image text recognition device further includes a model training module, where the model training module is used to train the above-mentioned text detection model and text direction classification model.

[0124] Optionally, the model training module can train the text detection model in the following manner, including:

[0125] The first step: Obtain multiple groups of first training sample data.

[0126] Among them, each group of first training sample data includes an image sample and text box samples corresponding to the minimum bounding rectangles of each text information within the image sample.

[0127] The second step: Construct a first neural network model. Among them, the model architecture of the first neural network model can be DBNet (Differentiable Binarization Network), PESNet (Progressive Scale Expansion Network), PANNet (Pixel Aggregation Network), etc. In the embodiments of the present application, DBNet is preferably used.

[0128] The third step: For each group of first training sample data, input the image sample in the first training sample data into the first neural network model to obtain predicted text boxes corresponding to the minimum bounding rectangles of each text information within the image sample output by the first neural network model.

[0129] The fourth step: Construct a first target loss function based on each text box sample and the corresponding predicted text box in each group of first training sample data, and adjust the parameters of the first neural network model based on the first target loss function until the model converges to obtain a trained text detection model.

[0130] Through the above steps, the training of the text detection model can be completed, and the preprocessing module can then call the text detection model to locate the text area in the target image and obtain the first text boxes corresponding to each text information within the target image.

[0131] Optionally, the model training module can train the text direction classification model in the following manner, including:

[0132] Step 1: Obtain multiple sets of second training sample data. Each set of second training sample data includes text box samples and corresponding true direction labels, and the true direction labels are used to reflect the angle between the text direction in the text box sample and the preset horizontal direction, such as 0 degrees, 90 degrees, 180 degrees, 270 degrees, etc.

[0133] Specifically, the above second training sample data can be obtained in the following way: Obtain multiple text image samples. The text image samples can be composed of a single image file or can be spliced by two or more image samples in the same direction, and the text image samples include text in different directions; perform preprocessing on the multiple text image samples to obtain corresponding multiple text box samples. The preprocessing operations include but are not limited to: blank area deletion, text box clipping, image enhancement processing; perform text direction annotation on each text box sample obtained by preprocessing. In this process, the actual rotation angle of each text box sample can be annotated manually or using an automated tool to ensure that each sample has an accurate direction label.

[0134] Step 2: Construct a second neural network model. The model architecture of the second neural network model is preferably MobileNet-V3.

[0135] Step 3: For each set of second training sample data, input the text box samples in the second training sample data into the second neural network model to obtain the predicted direction labels of the text box samples output by the second neural network model.

[0136] Step 4: Construct a second objective loss function based on the true direction labels and the corresponding predicted direction labels in each set of second training sample data, and adjust the parameters of the second neural network model based on the second objective loss function until the model converges to obtain a trained text direction classification model.

[0137] Through the above steps, the training of the text direction classification model can be completed. The preprocessing module can then call the text direction classification model to classify the directions of each first text box in the target image, obtain the direction classification results of each first text box, and determine the rotation angle of the corresponding first text box according to the direction classification results of each first text box, so that the first text box is rotated until the text direction is 0 degrees defined by the text direction classification model, that is, the horizontal direction.

[0138] In addition, in the technical solution provided in the above step S4, the preprocessing module can perform an expansion process on each second text box in the following way, including:

[0139] Step 1: Traverse each second text box to determine the size information of the second text box and the line spacing between the upper and lower lines within the second text box. Among them, the above size information includes: the width and height of the second text box, and the line spacing between the upper and lower lines is in pixels.

[0140] Step 2: Analyze the size information and the line spacing between the upper and lower lines using the target regression model based on the least squares support vector regression algorithm to obtain the expansion coefficient of the second text box.

[0141] Step 3: Perform an outward expansion process on the second text box based on the expansion coefficient.

[0142] Through the above outward expansion process, the expansion coefficient of each second text box can be calculated respectively to achieve the effect of adaptive expansion.

[0143] In addition, a clear high-resolution image is a prerequisite for ensuring accurate recognition results. However, in application scenarios (such as telecom auditing), due to reasons such as slight shaking of the lens when taking pictures and low resolution of the shooting device, the collected target image is blurred and unclear, thus affecting the accuracy of subsequent text recognition. Therefore, the preprocessing module can also perform resolution detection and enhancement on the outward-expanded second text box to ensure that the text in the image is clear enough and improve the accuracy of the final text recognition result.

[0144] Optionally, the preprocessing module can perform resolution enhancement processing on each outward-expanded second text box through the following method, including:

[0145] Step 1: Traverse each second text box in the target image to determine the area and perimeter of the second text box, and combine the expansion coefficient of the second text box to determine the number of pixels of the outward-expanded second text box according to the following formula:

[0146]

[0147] In the formula, d expand represents the number of pixels of the outward-expanded second text box, area(rect) represents the area of the second text box, peri(rect) represents the perimeter of the second text box, and out Lssvr represents the expansion coefficient.

[0148] Step 2: Perform grayscale processing and edge detection processing on the outward-expanded second text box in sequence to determine the grayscale values of each pixel point within the processed second text box.

[0149] Among them, the second text box after external expansion is grayscale processed to convert it into a grayscale image. Then, an edge detection algorithm is applied to perform edge detection on the grayscale image. If the Laplace operator is used to perform edge detection on the grayscale image, a Laplacian image can be obtained, and the pixel grayscale values of the edge part in the Laplacian image will increase significantly, thereby helping to identify the boundaries and details of the text information. Among them, the Laplace operator is a second-order differential operator that can detect sharp changes in grayscale values in an image, that is, edges.

[0150] Step 3: Based on the number of pixel points of the second text box after external expansion and the grayscale value of each pixel point, determine the pixel grayscale variance value of the second text box after external expansion.

[0151] Among them, the above pixel grayscale variance value can also be called the gradient variance of pixel points, and this variance reflects the degree of dispersion of the brightness changes of pixel points in the target image. Among them, a clear image will have more significant pixel gradient changes, while the pixel gradient changes of a blurred or low-resolution image will be smaller.

[0152] Step 4: When the pixel grayscale variance is less than a preset third threshold value, it indicates that the clarity of the second text box after external expansion is insufficient. At this time, the pre-trained text super-resolution model can be used to perform resolution improvement processing on the second text box after external expansion.

[0153] Specifically, the above text super-resolution model includes at least: a structural prior module, a Transformer encoder, and a backbone network. The following will detail each component of this model.

[0154] (1) Structural prior module.

[0155] Specifically, the structural prior module includes StyleGAN (Style Generative Adversarial Networks, style-based generative adversarial network), whose input is the discrete encoding representing different characters, and controls the style of the output image through the w∈W space, that is, maps the initial image style z∈Z. Among them, the discrete encoding of each character can be expressed as And X is the codebook storing all characters. The feature dimension of the discrete encoding of each character is 1*1*512. M represents the number of discrete encodings in the codebook, and its size covers common Chinese characters, English letters, numbers, etc., with a size of 6623. Therefore, the input-output relationship of the structural prior module can be written as:

[0156] I x = B(x,F(z),Φ G ) = B(x,w,Φ G )

[0157] where x represents the discrete encoding of a single character, F represents the network that maps the initial image style z to the w space, and Φ G represents the model parameters of StyleGA.

[0158] Therefore, the core of the structure prior module is to map the discrete encodings corresponding to multiple characters in the second text box after expansion to a preset structure parameter space, obtaining the high-resolution structure prior features of each character. These prior structure features are combined in the super-resolution network to help the network understand the structure and contour of each character, so that when restoring blurred text, the details of the characters can be restored more accurately, improving the accuracy of text recognition.

[0159] (2) Transformer encoder.

[0160] Since the input of the structure prior module is the output of the Transformer encoder, in order to ensure that the structure prior module generates the structure of each character, the category, bounding box, and font style of each character in the second text box after expansion can be extracted first using the Transformer encoder.

[0161] It should be noted that the Transformer encoder is composed of a vision Transformer and a CNN (Convolutional Neural Network), and also combines position encoding to inject position information. In the same text image, adjacent characters usually have the same font, so an additional linear layer is used in the Transformer encoder to map the features of all characters to the same prediction.

[0162] (3) Backbone network.

[0163] Specifically, in the backbone network, first, a simple UNet structure is stacked on a lightweight neural network to extract features of each character in the second text box after expansion, obtaining the low-resolution features of each character; the second text box after expansion is cropped according to the bounding box of each character detected by the Transformer encoder, and each character in the second text box after expansion is aligned; the structure prior features of each character output by the structure prior module and the low-resolution features of the character are fused and normalized to obtain the super-resolution features of each character; the super-resolution features of each character are mapped back to the original position in the second text box after expansion, obtaining the second text box after expansion and resolution improvement.

[0164] Optionally, the recognition module 36 can perform text recognition on the target image according to the following process to obtain the corresponding target text content, including:

[0165] Step 1: For each expanded second text box in the target image, use the text recognition model to perform text recognition on the expanded and resolution-enhanced second text box to obtain the predicted character sequence corresponding to the second text box.

[0166] Step 2: Based on the predicted character sequences corresponding to the expanded and resolution-enhanced second text boxes in the target image, determine the target text content corresponding to the preprocessed target image.

[0167] It should be noted that each module in the image text recognition device in the embodiments of the present application corresponds one by one to each implementation step of the image text recognition method in Embodiment 1. Since the description in Embodiment 1 is already detailed, some details not shown in this embodiment can be referred to Embodiment 1 and will not be elaborated here.

[0168] Embodiment 3

[0169] According to an embodiment of the present application, there is also provided a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the image text recognition method in Embodiment 1.

[0170] According to an embodiment of the present application, there is also provided a non-volatile storage medium, which includes a stored computer program. The device where the non-volatile storage medium is located executes the image text recognition method in Embodiment 1 by running the computer program.

[0171] According to an embodiment of the present application, there is also provided a processor, which is used to run a computer program. When the computer program runs, it executes the image text recognition method in Embodiment 1.

[0172] According to an embodiment of the present application, there is also provided an electronic device, which includes: a memory and a processor. The memory stores a computer program, and the processor is configured to execute the image text recognition method in Embodiment 1 through the computer program.

[0173] Specifically, when the computer program runs, it executes the following steps: obtain a target image to be recognized; use multiple pre-trained neural network models to perform preprocessing operations on the target image, where the preprocessing operations at least include: text box detection, orientation correction, and text box expansion; use a pre-trained text recognition model to perform text recognition on the preprocessed target image to obtain the corresponding target text content.

[0174] As an optional implementation manner, the above electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 4 Shows a hardware structure block diagram of an electronic device for implementing the image text recognition method. AsFigure 4 As shown, the electronic device 40 may include one or more processors 402 (shown as 402a, 402b, ……, 402n in the figure) (the processor 402 may include, but is not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA), a memory 404 for storing data, and a transmission device 406 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 4 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the electronic device 40 may further include more or fewer components than Figure 4 shown in, or have a different configuration from Figure 4 that shown.

[0175] It should be noted that the above one or more processors 402 and / or other data processing circuits can generally be referred to as "data processing circuits" in this article. The data processing circuit may be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit may be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the electronic device 40. As involved in the embodiments of the present application, the data processing circuit is a kind of processor control (such as the selection of a variable resistance terminal path connected to an interface).

[0176] The memory 404 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image text recognition method in the embodiments of the present application. The processor 402 executes various functional applications and data processing by running the software programs and modules stored in the memory 404, that is, implements the vulnerability detection method of the above application program. The memory 404 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 404 may further include a memory remotely set relative to the processor 402, and these remote memories can be connected to the electronic device 40 through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The transmission device 406 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the electronic device 40. In one example, the transmission device 406 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0178] The display can be, for example, a touch-screen Liquid Crystal Display (LCD), which enables the user to interact with the user interface of the electronic device 40.

[0179] The above-mentioned serial numbers of the embodiments are only for description and do not represent the advantages or disadvantages of the embodiments.

[0180] In the above embodiments of the present application, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0181] In several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0182] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, the functional units in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs.

[0185] The above are only the preferred embodiments of this application. It should be noted that for those of ordinary skill in the art of this technology, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. An image text recognition method, characterized in that, Including: Obtain a target image to be recognized; Use multiple pre-trained neural network models to perform preprocessing operations on the target image, where the preprocessing operations at least include: text box detection, orientation correction, and text box expansion; Use a pre-trained text recognition model to perform text recognition on the preprocessed target image to obtain corresponding target text content.

2. The method according to claim 1, characterized in that Using multiple pre-trained neural network models to perform preprocessing operations on the target image, including: Use the text detection model in the multiple neural network models to perform text detection on the target image, and determine the first text box corresponding to each text information in the target image; For each of the first text boxes in the target image, use the text direction classification model in the multiple neural network models to perform direction analysis on the first text box to obtain the direction classification result of the first text box, and determine the angle between the first text box and the preset horizontal direction according to the direction classification result, and perform orientation correction on the first text box according to the angle; Determine the target image after orientation correction based on each of the first text boxes after orientation correction, and use the text detection model to perform text detection on the target image after orientation correction again to determine the second text box corresponding to each of the multiple text information in the target image; Perform an expansion process on each of the second text boxes.

3. The method according to claim 2, wherein The training process of the text detection model includes: Obtain multiple groups of first training sample data, where each group of the first training sample data includes an image sample and a text box sample corresponding to the minimum bounding rectangle of each text information in the image sample; Construct a first neural network model; For each group of the first training sample data, input the image sample in the first training sample data into the first neural network model to obtain a predicted text box corresponding to the minimum bounding rectangle of each text information in the image sample output by the first neural network model; Construct a first target loss function based on each text box sample and the corresponding predicted text box in each group of the first training sample data, and perform parameter adjustment on the first neural network model based on the first target loss function until the model converges to obtain the text detection model that has completed training.

4. The method according to claim 2, wherein The training process of the text direction classification model includes: Obtain multiple groups of second training sample data, where each group of the second training sample data includes a text box sample and a corresponding true direction label, and the true direction label is used to reflect the angle between the text direction in the text box sample and the preset horizontal direction; Construct a second neural network model; For each group of the second training sample data, input the text box sample in the second training sample data into the second neural network model to obtain a predicted direction label including the text box sample output by the second neural network model; Construct a second objective loss function based on the true direction labels and corresponding predicted direction labels in each group of the second training sample data, and adjust the parameters of the second neural network model based on the second objective loss function until the model converges, obtaining the trained text direction classification model.

5. The method according to claim 2, characterized in that, Perform an expansion process on each of the second text boxes, including: Traverse each of the second text boxes to determine the size information of the second text box and the line spacing between the upper and lower lines within the second text box, where the line spacing between the upper and lower lines is in pixels; Analyze the size information and the line spacing between the upper and lower lines using an objective regression model based on the least squares support vector regression algorithm to obtain the expansion coefficient of the second text box, where the objective regression model is iteratively trained based on multiple text box samples and corresponding expansion coefficient labels, and the expansion coefficient label corresponding to the text box sample is the expansion coefficient selected from the pre-set alternative expansion coefficient set corresponding to the text box sample with an identification confidence higher than a pre-set first threshold value and the number of valid recognized characters higher than a pre-set second threshold value; Perform an expansion process on the second text box based on the expansion coefficient.

6. The method according to claim 2, wherein The preprocessing operation further includes: resolution enhancement, where, after performing the expansion process on each of the second text boxes, the method further includes: Traverse each second text box in the target image, determine the area and perimeter of the second text box, and combine the expansion coefficient of the second text box to determine the number of pixel points of the expanded second text box according to the following formula: where d expand represents the number of pixels of the second text box after external expansion, area(rect) represents the area of the second text box, peri(rect) represents the perimeter of the second text box, out Lssvr represents the expansion coefficient; Perform grayscale processing and edge detection processing on the expanded second text box in sequence to determine the grayscale values of each pixel point within the processed second text box; Determine the pixel grayscale variance value of the expanded second text box based on the number of pixel points of the expanded second text box and the grayscale value of each pixel point; In the case where the pixel grayscale variance value is less than a pre-set third threshold value, use a pre-trained text super-resolution model to enhance the resolution of the expanded second text box.

7. The method according to claim 6, wherein The text super-resolution model at least includes: a structure prior module, a Transformer encoder, and a backbone network, where: The structure prior module is used to map the discrete encodings corresponding to multiple characters within the expanded second text box to a pre-set structure parameter space to obtain high-resolution structure prior features of each character; The Transformer encoder is used to extract the category, bounding box, and font style of each character within the expanded second text box; The backbone network is used to extract features of each character in the expanded second text box, obtaining low-resolution features of each character; clip the expanded second text box according to the bounding box of each character detected by the Transformer encoder; fuse the structural prior features of each character output by the structural prior module and the low-resolution features of the character, obtaining super-resolution features of each character; map the super-resolution features of each character back to the original positions within the expanded second text box, obtaining the expanded and resolution-enhanced second text box.

8. The method according to claim 7, characterized in that, Use a pre-trained text recognition model to perform text recognition on the preprocessed target image, obtaining the corresponding target text content, including: For each expanded second text box in the target image, use the text recognition model to perform text recognition on the expanded and resolution-enhanced second text box, obtaining the predicted character sequence corresponding to the second text box; Based on the predicted character sequences corresponding to the expanded and resolution-enhanced second text boxes in the target image, determine the target text content corresponding to the preprocessed target image.

9. An image text recognition device, characterized in that, Including: An acquisition module, configured to acquire a target image to be recognized; A preprocessing module, configured to perform preprocessing operations on the target image by using a plurality of pre-trained neural network models, where the preprocessing operations at least include: text box detection, orientation correction, and text box expansion; A recognition module, configured to perform text recognition on the preprocessed target image by using a pre-trained text recognition model, obtaining the corresponding target text content.

10. A computer program product, characterized in that, Including: A computer program, where when the computer program is executed by a processor, it implements the image text recognition method according to any one of claims 1 to 8.

11. An electronic device, characterized in that, Including: A memory and a processor, where a computer program is stored in the memory, and the processor is configured to execute the image text recognition method according to any one of claims 1 to 8 through the computer program.

Images