A text recognition model training method and device

Abstract

The application provides a text recognition model training method and device, wherein the text recognition model training method comprises the following steps: determining a sample training set, wherein the sample training set comprises a target sample picture and a corresponding sample label of the target sample picture; inputting the target sample picture into the convolution layer for convolution processing to obtain a convolution vector of the target sample picture; inputting the convolution vector into the self-attention mechanism encoding layer for encoding processing to obtain an encoding vector; inputting the encoding vector and the sample label into the full connection layer to calculate a target loss value, and adjusting parameters of the text recognition model according to the target loss value until a training stop condition is reached.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence in computer technology, and in particular to a text recognition model training method and apparatus, computing device and computer-readable storage medium. Background Technology

[0002] Artificial intelligence (AI) refers to the ability of an engineered (i.e., designed and manufactured) system to perceive its environment, and to acquire, process, apply, and represent knowledge. The development status of key technologies in the field of artificial intelligence includes machine learning, knowledge graphs, natural language processing, computer vision, human-computer interaction, biometrics, and virtual reality / augmented reality.

[0003] In recent years, the most impressive advancements in machine learning have come from the subfield of deep learning (also known as deep neural network learning). Deep learning refers to methods for creating rich hierarchical representations by training neural networks with many hidden layers. It utilizes simple computational units (i.e., "neurons") to form multi-layered networks, where each unit combines a set of input values ​​to produce an output value, which is then passed to other downstream neurons.

[0004] Correspondingly, many natural scenes contain rich textual information, and recognizing this information is crucial for understanding natural scene images. However, scene text recognition suffers from problems such as low text recognition rates and slow processing speeds due to the variety of fonts, backgrounds, image blur or low resolution, and diverse layouts. Summary of the Invention

[0005] In view of this, embodiments of this application provide a text recognition model training method and apparatus, a computing device and a computer-readable storage medium to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this application, a text recognition model training method is provided, comprising:

[0007] The text recognition model includes convolutional layers, self-attention mechanism encoding layers, and fully connected layers;

[0008] A sample training set is determined, wherein the sample training set includes target sample images and sample labels corresponding to the target sample images;

[0009] The target sample image is input into the convolutional layer for convolution processing to obtain the convolution vector of the target sample image;

[0010] The convolution vector of the target sample image is input into the self-attention mechanism encoding layer for encoding processing to obtain the encoding vector;

[0011] The encoding vector and the sample label are input into the fully connected layer to calculate the target loss value, and the parameters of the text recognition model are adjusted according to the target loss value until the training stopping condition is met.

[0012] According to a second aspect of the embodiments of this application, a text recognition model training apparatus is provided, comprising:

[0013] The text recognition model includes a convolutional layer module, a self-attention mechanism encoding layer module, and a fully connected layer module;

[0014] The convolutional layer module is configured to determine a sample training set, wherein the sample training set includes a target sample image and the sample label corresponding to the target sample image, and input the target sample image into the convolutional layer for convolution processing to obtain the convolution vector of the target sample image;

[0015] The self-attention mechanism encoding layer module is configured to input the convolution vector of the target sample image into the self-attention mechanism encoding layer for encoding processing to obtain the encoding vector;

[0016] The fully connected layer module is configured to input the encoding vector and the sample label into the fully connected layer to calculate the target loss value, and adjust the parameters of the text recognition model according to the loss value until the training stopping condition is met.

[0017] According to a third aspect of the embodiments of this application, a text recognition method is provided, comprising:

[0018] Receive an initial image and the corresponding label of the initial image, and input the initial image and the corresponding label of the initial image into the image detection model to obtain the image to be processed;

[0019] The image to be processed is input into a text recognition model to obtain the text information of the image to be processed. The text recognition model is obtained using the above-mentioned text recognition model training method, and the image detection model is the image detection model of the text recognition model training method.

[0020] According to a fourth aspect of the embodiments of this application, a text recognition device is provided, comprising:

[0021] The detection module is configured to receive an initial image and the corresponding label of the initial image, and input the initial image and the corresponding label of the initial image into the image detection model to obtain the image to be processed;

[0022] The recognition module is configured to input the image to be processed into a text recognition model to obtain the text information of the image to be processed. The text recognition model is obtained using the above-mentioned text recognition model training method, and the image detection model is the image detection model of the text recognition model training method.

[0023] According to a fifth aspect of the present application, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the instructions to implement the steps of the text recognition model training method.

[0024] According to a sixth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the text recognition model training method.

[0025] According to a seventh aspect of the present application, a chip is provided that stores computer instructions, which, when executed by the chip, implement the steps of the text recognition model training method.

[0026] In this embodiment, the sample training set is input into a convolutional layer for convolution processing to obtain the convolutional vector of the target sample image. The convolutional vector is then input into a self-attention mechanism encoding layer for encoding to obtain an encoded vector. The target loss value is calculated through a fully connected layer to adjust the parameters of the text recognition model until the training stopping condition is met. In this model, the self-attention mechanism encoding layer not only makes the features proposed by the text recognition model more representative and sequential, but also calculates the target loss value through a fully connected layer to adjust the parameters of the text recognition model, thereby reducing the target loss value and improving the accuracy of text recognition. Attached Figure Description

[0027] Figure 1 This is a structural block diagram of the computing device provided in the embodiments of this application;

[0028] Figure 2 This is a schematic diagram of the text recognition model structure in the text recognition model training method provided in the embodiments of this application;

[0029] Figure 3 This is a flowchart of the text recognition model training method provided in the embodiments of this application;

[0030] Figure 4 This is a schematic diagram of the structure of the fully connected layer in the text recognition model training method provided in this application embodiment;

[0031] Figure 5 This is a flowchart illustrating the specific application process of the text recognition model training method provided in the embodiments of this application;

[0032] Figure 6 This is a flowchart of the text recognition method provided in the embodiments of this application;

[0033] Figure 7 This is a schematic diagram of the structure of the text recognition model training device provided in the embodiments of this application;

[0034] Figure 8 This is a schematic diagram of the structure of the text recognition device provided in the embodiments of this application. Detailed Implementation

[0035] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0036] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "in response to a determination".

[0038] First, the terminology used in one or more embodiments of the present invention will be explained.

[0039] Text detection: Given a text image, automatically detect the text content in the text image.

[0040] Bezier curve: also known as Bezier curve or Bezier curve, is a mathematical curve used in two-dimensional graphics applications. It is a smooth curve drawn based on the coordinates of four points at any location.

[0041] Adaptive Bézier Curve Network (ABCNet): An end-to-end trainable scene text localization framework that uses adaptive Bézier curves.

[0042] Least squares method: a curve fitting method that finds unknown parameters that minimize the total error (distance) between the sample points and the fitted line.

[0043] OCR (Optical Character Recognition): A technology that can scan documents quickly and accurately.

[0044] CTCloss (Connectionist Temporal Classification): A classification loss function based on connection time.

[0045] Centerloss: A loss function designed to minimize intra-class spacing.

[0046] Bezier Align: A feature alignment method based on Bezier curves, similar to ROI.

[0047] transformer: an efficient sequence-to-sequence encoding and decoding structure.

[0048] This application provides a text recognition model training method and apparatus, a computing device and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0049] Figure 1 A structural block diagram of a computing device 100 according to an embodiment of this application is shown. The components of the computing device 100 include, but are not limited to, a memory 110 and a processor 120. The processor 120 is connected to the memory 110 via a bus 130, and a database 150 is used to store data.

[0050] The computing device 100 also includes an access device 140, which enables the computing device 100 to communicate via one or more networks 160. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 140 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0051] In one embodiment of this application, the aforementioned components of the computing device 100 and Figure 1 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 1 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0052] The computing device 100 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 100 can also be a mobile or stationary server.

[0053] The processor 120 can execute the steps in the text recognition model training method described below.

[0054] Scene text detection and recognition have received increasing attention due to their widespread application in computer vision. However, current text detection and recognition methods, due to the diversity of text in terms of size, aspect ratio, font style, perspective distortion, and shape, have largely failed to effectively address the detection and recognition of deformed text. Although the emergence of deep learning has greatly improved the performance of scene text localization tasks, current methods still have significant shortcomings in practical applications, particularly in terms of efficiency.

[0055] In recent years, many end-to-end methods have significantly improved the performance of text localization in scenes of arbitrary shapes. However, these methods either use segmentation-based approaches to maintain complex computations or require a large amount of expensive character-level annotation. Therefore, the text recognition model training method provided in this application proposes a simple and effective end-to-end framework (an end-to-end framework can be understood as a system where the user directly inputs initial data into the model and can directly obtain usable results without having to worry about the structural framework of the intermediate products) for directional or curved scene text localization in images, ensuring improved inference time while also providing better performance. Based on this, this application proposes an end-to-end trainable framework. It can automatically output the text in an image to be recognized by taking an image containing text of arbitrary shapes as input, without considering the intermediate image detection and text recognition calculation processes, thus achieving end-to-end processing. Furthermore, it utilizes an adaptive Bézier curve network (ABCNet) to achieve scene text recognition on images with arbitrary text shapes. It should be noted that the end-to-end framework proposed in this application means that the text localization and text recognition process of the entire image can be determined by a single set of training parameters. That is, first, the text in the image is located, and then the located text is recognized. Originally, these were two stages implemented by two sets of training parameters, but now they can be merged into one stage implemented by a single set of training parameters.

[0056] Furthermore, in the field of text recognition technology, most OCR technologies currently employ a two-stage approach: detection followed by recognition. Moreover, when locating text, most texts are square or straight texts, or text localization is based on segmentation methods. This often results in poor accuracy in recognizing some uncommon similar-looking characters, especially in the highly specialized field of similar-looking character recognition.

[0057] Based on this, the text recognition model training method provided in this application is an end-to-end text recognition method. It uses Bézier curves to fit the label points to make the boundary separation more obvious. It can recognize text of any shape, especially curved text. While improving the prediction accuracy, the computational efficiency is not reduced. Furthermore, the text recognition part uses transformer layers, ctcloss and centerloss to improve the recognition accuracy of similar characters.

[0058] It should be noted that the text recognition model training method provided in this application is applied to a text recognition model in a server. The text recognition model includes convolutional layers, self-attention mechanism encoding layers, and fully connected layers. For details, please refer to [link to relevant documentation]. Figure 2 , Figure 2 A schematic diagram of the text recognition model structure in a text recognition model training method according to an embodiment of this application is shown.

[0059] Figure 2 In the text recognition model, there are convolutional layer a, self-attention mechanism encoding layer b, and fully connected layer c.

[0060] In practical applications, the text recognition model proceeds from bottom to top, sequentially performing algorithmic calculations through a convolutional layer a, a self-attention mechanism encoding layer b, and a fully connected layer c, thereby outputting the text recognition result. The convolutional layer a performs convolution processing on the target image to obtain the convolution vector of the target image, then inputs the convolution vector of the target image into the self-attention mechanism encoding layer b for encoding to obtain the encoded vector, and finally inputs the encoded vector into the fully connected layer c to calculate the loss value, thereby obtaining the text recognition result of the target image.

[0061] Figure 3 A flowchart of a text recognition model training method according to an embodiment of this application is shown, which specifically includes the following steps.

[0062] Step 302: Determine the sample training set, wherein the sample training set includes the target sample image and the sample label corresponding to the target sample image.

[0063] The target sample image can be understood as a sample image containing text content and having text outlines.

[0064] The sample label corresponding to the target sample image can be understood as the feature label in the target sample image that can represent the outline of the text, such as the coordinates of the feature points.

[0065] In practical applications, the server can determine the sample training set through an image detection model. Specifically, determining the sample training set includes:

[0066] The training set of samples is determined using an image detection model.

[0067] The image detection model can be any detection model that recognizes text modules in an image. The image detection model provided in this application can be obtained through pre-training. This application uses ResNet50+FPN and Bézier curves to extract image features as an example for detailed explanation. ResNet50 is a feature extraction network type of ResNet (Residual Network), widely used in object classification and other fields, and as part of the classic neural network backbone for computer vision tasks. FPN (Feature Pyramid Networks) can be understood as a form of feature fusion, constructing a feature pyramid that can be trained end-to-end. This architecture can be flexibly applied to different tasks, including object detection and instance segmentation. ResNet50+FPN can be understood as FPN in this application being implemented on the ResNet50 network structure, where the features of the later layers (high-level features) and the features of the earlier layers (shallow features) are fused together.

[0068] Specifically, the sample training set can be obtained by performing image detection processing on the initial sample images according to the image detection model. The purpose is to use the sample training set to train the text recognition model so that the text recognition model can recognize the text content in the sample training set.

[0069] The text recognition model training method provided in this application uses an image detection model to detect any image containing text, thereby identifying sample images with text regions as a sample training set. This sample training set can then be input into the text recognition model for accurate text recognition.

[0070] To obtain the training set of target sample images, the initial sample images are processed using an image detection model, and Bézier curves are used to fit the label points to accurately determine the text detection boxes, thereby obtaining the images to be processed. Specifically, the training method of the sample image detection model is as follows:

[0071] Receive training data including initial sample images and the labels corresponding to the initial sample images;

[0072] The initial sample image is input into the image feature extraction module for feature extraction to obtain candidate sample images;

[0073] The candidate sample image is input into the curve sequence feature extraction module for feature extraction to obtain the text detection box of the candidate sample image. Based on the text detection box, the sample image to be processed is determined in the initial sample image, and the label corresponding to the sample image to be processed is set.

[0074] The loss value is calculated based on the labels corresponding to the initial sample images and the labels corresponding to the sample images to be processed, and the initial parameters of the image detection model are adjusted according to the loss value until the training stopping condition is met.

[0075] The training data includes initial sample images and their corresponding labels. Initial sample images can be understood as images containing text content, and may also include images of people, landscapes, or other content. The labels corresponding to the initial sample images can be understood as labels representing the text, people, and landscape features in the initial images.

[0076] The image feature extraction module can be understood as a module that extracts features from the initial sample image.

[0077] The curve sequence feature extraction module can be understood as a module that extracts features based on the curve sequence of the text region, such as the shape region composed of the text outline obtained based on Bézier curves.

[0078] The image to be processed can be understood as the image obtained by mapping the shape region composed of text outlines onto the initial sample image.

[0079] Specifically, the sample image detection model can be obtained through continuous iterative training with a large number of initial sample images. The image detection model can receive a large number of initial sample images and the corresponding labels of the initial sample images. The initial sample images are input into the image feature extraction module for feature extraction. For example, the features of people, landscapes or text can be extracted from the initial sample images. After determining the various types of features of the initial sample images, in order to realize the recognition of text in the images, the regions of the images with text areas are determined based on the text features extracted from the initial sample images, and these regions are used as candidate sample images.

[0080] Furthermore, after inputting the candidate sample image into the curve sequence feature extraction module for feature extraction, the text detection box of the text in the candidate sample image can be determined based on the curve sequence features. The text detection box can be the outline of the square text or the outline of the curved text. The image content overlapping with the text detection box in the initial sample image is determined based on the text detection box and used as the sample image to be processed. It should be noted that after the candidate sample image is extracted using the curve sequence feature extraction module, only the control points corresponding to the candidate sample image are obtained (the control points can be understood as the feature points around the text area in the candidate sample image). Therefore, the text detection box determination method provided in this embodiment adopts the calculation method of Bézier curve, that is, the parameter curve c(t), defined as shown in formula (1):

[0081]

[0082] Where n is the degree, bi is the i-th control point (i is a natural number), t is a 0-1 parameter, i.e., a variable parameter used to control the start point to the end point, and Bi,n(t) represents the Bernstein paradoxical polynomial, as shown in formula (2):

[0083]

[0084] Where (i) is the binomial coefficient. In order to fit text of arbitrary shape with Bézier curve, we can comprehensively observe scene text of arbitrary shape from the existing dataset and use a cubic Bézier curve (i.e., n is 3) to fit scene text of different types of arbitrary shape. That is, based on the cubic Bézier curve, scene text of arbitrary shape can be simplified to boundary regression with eight control points, and then the text detection box of candidate sample image can be obtained.

[0085] Furthermore, while obtaining the text detection box, the text region image corresponding to the text detection box can be mapped into the initial sample image based on the text box features. This image with the text region is then used as the sample image to be processed. For example, if the text detection box consists of four text box feature points, A(0,0), B(0,1), C(1,0), and D(1,1), these four text box feature points are mapped into the initial sample image to obtain the corresponding text region image. This text region image is then determined as the sample image to be processed; simultaneously, it also serves as the sample image to be processed. The images are labeled accordingly. Based on the labels of the initial sample images and the labels of the images to be processed, corresponding loss values ​​are calculated. The initial parameters of the image feature extraction module in the image detection model are adjusted based on these loss values ​​until the training stopping condition is met, thus completing the training of the image feature extraction module in the image detection model to accurately extract image features. The loss function used to calculate the loss value can be a 0-1 loss function, an absolute value loss function, a squared difference loss function, a cross-entropy loss function, etc., in practical applications. In this application, the choice of loss function is not limited and is determined by practical application. Here, the squared difference loss function is used as an example for explanation; see Formula 3 below:

[0086] L(Y, f(X)) = (Yf(X)) 2 Formula 3

[0087] Where L(Y, f(X)) represents the loss value, Y represents the region feature value of the label corresponding to the initial sample image (the region feature value of the initial sample image after localization generated during training), and f(X) represents the region feature value of the label corresponding to the sample image to be processed (the feature region feature value of the label generated as a Bézier curve). After calculating the loss value, the image region feature parameters and image feature extraction parameters in the image detection model are adjusted in reverse according to the loss value, and the image detection model is trained again using the next batch of sample images until the loss value is less than the preset threshold and / or the number of training rounds reaches the preset number of rounds.

[0088] In one specific implementation of this application, taking the stopping of image detection model training by the loss value being less than a preset threshold as an example, the preset threshold is 0.4. When the calculated loss value is less than 0.4, it can be determined that the training process of the image detection model has been completed.

[0089] In another specific embodiment provided in this application, taking the stopping of training the image detection model with a preset number of training rounds as an example, the preset number of training rounds is 20 rounds. When the training rounds of the sample images reach 20 rounds, it can be determined that the training of the image detection model has been completed.

[0090] In another specific embodiment provided in this application, two training stopping conditions are set: a preset threshold for the loss value and a preset number of training rounds. The loss value and the number of training rounds are monitored simultaneously. When either the loss value or the number of training rounds meets the preset training stopping condition, it can be determined that the image detection model has completed training.

[0091] In practical applications, the image feature extraction process for the initial sample image can use ResNet50+FPN. In the output text detection box, the original image of the initial sample image is mapped to the coordinate points of the initial sample image, and the obtained polygonal label box is fitted into the form of a Bézier curve. The least squares method can be used to obtain the cubic Bézier curve text detection box. It should be noted that this manual does not impose any restrictions on the method of obtaining the text detection box.

[0092] The text recognition model training method provided in this application training an image detection model to accurately obtain sample images with text content as a sample training set, which is then input into the text recognition model for text content recognition.

[0093] Further, the step of inputting the candidate sample image into the curve sequence feature extraction module for feature extraction to obtain the text detection box of the candidate image includes:

[0094] The candidate image is input into the curve sequence feature extraction module, and features are extracted according to the preset extraction method based on the curve sequence feature extraction conditions to obtain the text detection box of the candidate image.

[0095] The preset extraction method can be understood as any method of extracting curve features, and this application embodiment does not impose too many restrictions on it.

[0096] Specifically, for candidate images that have undergone preliminary feature extraction, it is possible to distinguish between the features of people, scenery, and text. The candidate image is then input into the curve sequence feature extraction module, where the text features of the candidate image can be extracted again based on the curve sequence feature extraction conditions. The curve sequence feature extraction conditions can use the Bézier curve equation. For the text features of the candidate image, the Bézier curve equation can be used to extract features from the candidate image using the least squares method, thus obtaining the text detection box of the candidate image.

[0097] It should be noted that the embodiments of this application are not limited to using the Bézier curve equation to extract features from the text feature parts in the candidate image using the least squares method; other corresponding methods can also be used to extract features from the text feature parts.

[0098] In practical applications, Bézier curves are used because curved text can be recognized based on the Bézier equation. For example, for bow-shaped text, the text box formed by the upper and lower edges of the bow is the bow-shaped text detection box. The upper and lower edges can be calculated and determined using Bézier curves. Furthermore, after obtaining the bow-shaped text detection box, the coordinates of the bow-shaped text detection box can be mapped to the coordinates of the initial sample image, so as to facilitate the subsequent determination of the image to be processed, which consists of multiple coordinates.

[0099] The text recognition model training method provided in this application uses curve sequence feature extraction conditions to extract features from candidate sample images again, and determines the image corresponding to the text detection box. This allows the obtained image to be input into the text recognition model, enabling rapid recognition of the text content, reducing interference from background images other than the text content, and improving the accuracy of text recognition.

[0100] To obtain a target image with minimal background interference, the BezierAlign sampling method is used to select RoI (Region of Interest, which can be understood in machine vision and image processing as the region to be processed delineated from the image being processed using rectangles, circles, ellipses, irregular polygons, etc.) features, effectively outputting a feature map of a fixed size. Specifically, the step of determining the target sample image based on the text detection box in the initial image and setting the label corresponding to the target sample image includes:

[0101] Determine the width, height, and target detection points of the text detection box;

[0102] Based on the target detection point and the width of the detection frame, the target detection point width ratio is determined, and based on the target detection point and the height of the detection frame, the target detection point height ratio is determined.

[0103] Based on the target detection point width ratio, the target detection point parameter position is determined in the text detection box. Based on the target detection point parameter position and the target detection point height ratio, the sample image to be processed is determined, and the label corresponding to the sample image to be processed is set.

[0104] Specifically, to obtain text images with less interference, BezierAlign (a feature alignment method based on Bezier curves) can be used to effectively output a fixed-size feature map. This is achieved by determining the width and height of the text detection box, as well as the target detection point. First, the ratio of the distance from the target detection point to the detection box width to the entire detection box width is calculated. Then, the ratio of the distance from the target detection point to the detection box height is calculated, thus determining the target detection point's width and height ratios. Based on this target detection point, the corresponding target detection point parameter position is determined within the text detection box. Finally, the ratio of this target detection point parameter position to the target detection point height is used to determine the target detection point's position. For example, the sample image to be processed is then determined, and the feature value at that location can be solved by two-dimensional interpolation (two-dimensional interpolation is also called node two-dimensional or interpolation function two-dimensional, in the form of z = f(x,y)), and the label corresponding to the sample image to be processed is set. It should be noted that, in the embodiment of this application, when determining the sample image to be processed, since the detection points of the image in this application are two-dimensional detection points, a two-dimensional interpolation algorithm is used for calculation. Because the calculation of two-dimensional interpolation data must be a rectangular domain, that is, a regular matrix composed of known data points (x,y), or a raster, which can be generated using the meshgid function; however, the calculation method for determining the sample image to be processed in this embodiment of the application is not limited to the two-dimensional interpolation algorithm, and no specific limitation is made here.

[0105] In practical applications, the text detection box can be divided into multiple grids on an equal basis. Each column of any grid is orthogonal to the Bézier curve boundary of the text. The width and height of the sampling points (i.e., target detection points) are equidistant, and bilinear interpolation is performed on the coordinates. The specific operation process includes the following steps: First, for any grid point in the text detection box, calculate the ratio t of the distance from the grid point to the left side of the text detection box to the width of the entire text detection box, and then calculate the ratio of the distance from the grid point to the bottom of the text detection box to the height of the entire text detection box. Second, for the original text detection box, the final corresponding text detection point can be determined according to the two Bézier curve equations, and then the feature value of the text detection point is calculated by two-dimensional interpolation. It should be noted that by calculating each grid point, the region that fits the text can be accurately determined without introducing a large amount of invalid background information into the image to be processed, and the corresponding label is set for the image to be processed.

[0106] The text recognition model training method provided in this application calculates the grid points in the text detection box to determine the region that fits the text, and does not introduce a large amount of invalid background information into the image to be processed. It can not only achieve rapid recognition of the text to be processed, but also improve the accuracy of recognition by filtering out invalid background information.

[0107] Specifically, determining the position of the target detection point parameter in the text detection box based on the target detection point width ratio includes:

[0108] The target parameters corresponding to the text detection box are determined based on the target detection point width ratio, and the target detection point parameter positions are determined in the text detection box based on the target parameters.

[0109] The target detection point parameter position can be understood as the position within the text detection box determined based on the width ratio of the target detection point.

[0110] In practical applications, in order to obtain a text detection box that fits the text area, the target parameters corresponding to the text detection box can be determined based on the width ratio of the target detection point. Then, the parameter positions of the target detection points corresponding to the upper and lower boundaries of the text detection box can be realized. After determining the width ratio and height ratio of the target detection point, the position of the parameter value t corresponding to the Bézier curve parametric equation of the upper and lower edges of the text detection box can be found to obtain the parameter position of the target detection point.

[0111] The text recognition model training method provided in this application can determine the target parameters in the text detection box by the target detection point width ratio, thereby determining the target detection point parameter positions on the upper and lower boundary lines of the text detection box, so as to determine the region that fits the text better, and thus filter out invalid background information in the image content.

[0112] Furthermore, after determining the parameter positions of the upper and lower boundary lines within the text detection box, the distance between the upper and lower boundaries can be determined, thereby determining the feature values ​​of the sample image to be processed; specifically, determining the sample image to be processed based on the ratio of the target detection point parameter position to the target detection point height includes:

[0113] The distance determined by the target detection point parameter position is processed based on the target detection point height ratio, and the feature value of the target detection point is determined based on a preset processing method;

[0114] The sample image to be processed is obtained based on the feature values ​​of the target detection points.

[0115] Specifically, given the height ratio of the target detection point, the distance between the upper and lower boundaries of the text detection box can be determined, this distance can be segmented, and the feature value of the target detection point can be calculated using two-dimensional interpolation.

[0116] In practical applications, the target detection point parameter positions, namely tp (upper boundary parameter point) and bp (lower boundary parameter point), are determined based on the target detection point height ratio t. Then, the line segment from bp to tp is segmented according to the ratio t. The feature value at this point can be solved by two-dimensional interpolation, and finally the sample image to be processed is obtained, which can be understood as the target sample image.

[0117] The text recognition model training method provided in this application determines the distance between the parameter positions of the upper and lower boundaries of the text detection box, thereby obtaining the feature value of the target detection point to obtain the sample image to be processed. This facilitates the subsequent input of the sample image to be processed into the text recognition model for accurate recognition of the content of the sample image.

[0118] Step 304: Input the target sample image into the convolutional layer for convolution processing to obtain the convolution vector of the target sample image.

[0119] Specifically, the obtained target sample image is used as the input to the text recognition model. It is fed into the convolutional layer of the text recognition model for convolution processing to obtain the convolution vector of the target sample image. This convolution vector is then fed into the self-attention mechanism encoding layer for encoding processing.

[0120] In practical applications, the text recognition model training method provided in this manual can be applied to 6 convolutional layers for convolution processing. However, the specific number of convolutional layers is not limited here, and the number of convolutional layers can be set according to the actual application.

[0121] Step 306: Input the convolution vector of the target sample image into the self-attention mechanism encoding layer for encoding processing to obtain the encoding vector.

[0122] Specifically, the target sample image is used as the input to the self-attention mechanism encoding layer for encoding processing to obtain the encoded vector. It should be noted that the self-attention mechanism encoding layer used in this embodiment encodes the convolution vector instead of the original bidirectional LSTM, and the features extracted by the self-attention mechanism encoding layer are more representative and serializable than those extracted by the original bidirectional LSTM.

[0123] Step 308: Input the encoding vector and the sample label into the fully connected layer to calculate the target loss value, and adjust the parameters of the text recognition model according to the target loss value until the training stopping condition is met.

[0124] Specifically, after inputting the encoding vector output by the self-attention mechanism encoding layer and the sample labels into the fully connected layer, the corresponding target loss value can be calculated, and the parameters of the corresponding text recognition model can be adjusted according to the target loss value until the training stopping condition is reached, thus completing the training of the text recognition model.

[0125] Furthermore, the target loss value can be calculated based on two loss values; specifically, the step of inputting the encoding vector into the fully connected layer to calculate the target loss value includes:

[0126] The encoding vector and the sample label are input into the first sub-layer of the fully connected layer to calculate the text arrangement loss value, and the encoding vector and the sample label are input into the second sub-layer of the fully connected layer to calculate the text precision loss value;

[0127] The target loss value is determined based on the text arrangement loss value and the text precision loss value.

[0128] Specifically, to improve the accuracy of text recognition, the order of the recognized text can be determined from the text arrangement, or the accuracy of the recognized text can be determined from the text precision. Alternatively, for the recognition of texts with similar-looking characters, the encoding vector output by the encoding layer based on the self-attention mechanism can be input into the fully connected layer to calculate the loss value.

[0129] For details, see Figure 4 , Figure 4 This paper shows a schematic diagram of the structure of a fully connected layer in a text recognition model training method provided in an embodiment of this application.

[0130] Figure 4 The fully connected layer in the model consists of a first sub-layer and a second sub-layer. The first sub-layer is used to calculate the text order loss value, which can be understood as using the text order loss value to perform string alignment processing on the text recognition part to ensure the string alignment of the output recognized text. The second sub-layer is used to calculate the text accuracy loss value, which can be understood as using the text accuracy loss value to accurately identify similar-looking characters in the recognized text to improve the accuracy of the output text. Then, the final target loss value of the text recognition model is determined based on the text order loss value and the text accuracy loss value.

[0131] In practical applications, the first sub-layer in the fully connected layer can use the CTCLOss loss function to process string alignment, and the second sub-layer can use the Center loss function to process similar-looking characters, so as to improve the accuracy of text recognition for character length and some uncommon similar-looking characters.

[0132] It should be noted that CTCLOSS calculates probabilities for each recognized character using an algorithm. It calculates the optimal probability for each character position within the current path (a path can be understood as multiple possible arrangements of recognized characters; different predicted arrangements represent different paths, meaning the position of each character is different within each path). This probability is calculated by multiplying the accuracy of the predicted positions of all characters preceding the current character by the accuracy of the predicted positions of all characters following the current character. Adding the two products together gives the optimal probability value of the current text's current position in the text arrangement. The position with the optimal probability value is then the output position of the current text. Furthermore, during text recognition, the number of characters can be determined based on the number of labels, ensuring that the number of characters output by the text recognition model matches the number of pre-labeled labels. For example, if the labeled labels contain 10 characters, but the text recognition model only predicts the positions of 9 characters, temporary characters can be interspersed into the predicted text to ensure that the number of characters in the recognized string matches the number of characters in the original labels.

[0133] Center loss is calculated after CTCLoss. It involves identifying and correcting similar-looking characters after the text has been aligned, with the aim of reducing the intra-class distance (for the identified similar-looking characters in the text).

[0134] The text recognition model training method provided in this application can recognize similar-looking characters by adding the calculation of the accurate loss value of characters in the fully connected layer, thereby improving the accuracy of similar-looking character recognition.

[0135] See Figure 5 , Figure 5 The flowchart illustrates the specific application process of the text recognition model training method provided in the embodiments of this application.

[0136] It should be noted that the text recognition model provided in this embodiment uses an image detection model as input to obtain the image to be processed. The image to be processed can be a text image of any shape. In the text recognition model, the image to be processed is first convolved to obtain a convolution vector, and then the convolution vector is input to the encoding layer for encoding. The encoding layer is a two-layer encoding layer with a self-attention mechanism. Therefore, the processing of the convolution vector is more representative and sequential than the features extracted by the original bidirectional LSTM. Finally, the encoded vector is input to the fully connected layer for processing. The fully connected layer uses CTCLoss + Centerloss as the text recognition metric to recognize the text in the image to be processed, thereby training the text recognition model. It should be noted that the addition of the Centerloss loss function to the fully connected layer to process the encoded vector can improve the accuracy of recognizing similar-looking characters in the image to be processed.

[0137] Specifically, in Figure 5 Part A can be an image detection model, which has multiple processing layers to recognize images and obtain images to be processed. First, a large number of initial images and their corresponding labels are input into the image detection model. The initial images are then input into the image feature extraction layer of the image detection model for feature extraction to obtain candidate images. Then, the obtained candidate images are input into the Bézier curve feature extraction layer for feature extraction to obtain text detection boxes for the candidate images. Based on the text detection boxes, the images to be processed are determined in the original initial images. The Bézier curve fits the label points into a curve form, providing a concise and parameterized labeling method that improves accuracy without sacrificing efficiency.

[0138] It should be noted that during the training process of the above image detection model, after identifying the image to be processed, it is also necessary to set corresponding labels for the image to be processed, and calculate the loss value based on the image to be processed and its corresponding labels. The parameters of the image detection model are then adjusted according to the calculated loss value to train the image detection model, thereby achieving accurate detection of images with text features from the initial image, and enabling the localization of directional or curved scene text in the image. This embodiment of the application utilizes a Bézier curve network to adapt to the accurate recognition of scene text of arbitrary shapes.

[0139] In addition, Figure 5Part B can be a text recognition model, which has multiple processing layers to recognize text in the image to be processed, thereby accurately identifying the text portion of the image. First, the image detection model can input the output image to be processed into the text recognition model in real time. The text recognition model inputs the image to be processed into a 6-layer convolutional layer for convolution processing to obtain the convolution vector of the image to be processed. Then, the convolution vector is input into a self-attention mechanism encoding layer for encoding processing to obtain an encoding vector. Finally, the encoding vector is input into a fully connected layer to calculate the target loss value, and the parameters of the text recognition model are adjusted according to the target loss value to achieve training of the text recognition model.

[0140] It should be noted that, in order to improve the accuracy of text recognition, two loss functions are used in the fully connected layer: the CTCLoss loss function and the Centerloss loss function. The CTCLoss loss function is used to adjust the parameters of the text recognition model, which can not only improve the accuracy of recognizing the text arrangement in the image being processed, but also improve the accuracy of recognizing similar characters by adjusting the parameters of the text recognition model using the Centerloss loss function.

[0141] By training the text recognition model as described above, the trained text recognition model can solve the problem of low text recognition rate in images with diverse fonts, diverse backgrounds, blurry images, low image resolution, and diverse layouts. This enables the end-to-end method to improve the performance of text localization in arbitrary shape scenes and also improves the accuracy of recognizing some uncommon similar-looking characters.

[0142] In summary, the text recognition model training method provided in this application involves inputting the sample training set into a convolutional layer for convolution processing to obtain the convolutional vector of the target sample image. This convolutional vector is then input into a self-attention mechanism encoding layer for encoding to obtain an encoded vector. Finally, a fully connected layer is used to calculate the target loss value to adjust the parameters of the text recognition model until the training stopping condition is met. This model utilizes a self-attention mechanism encoding layer, which not only makes the features proposed by the text recognition model more representative and sequential, but also calculates the target loss value through a fully connected layer to adjust the parameters of the text recognition model, thereby reducing the target loss value and improving the accuracy of text recognition.

[0143] Figure 6 A flowchart of a text recognition method according to an embodiment of this application is shown, which specifically includes the following steps.

[0144] It should be noted that this embodiment is an application of the text recognition model training method in the above embodiments, wherein both the image detection model and the text recognition model are trained based on the above text recognition model training method.

[0145] Step 602: Receive the initial image and the corresponding label of the initial image, and input the initial image and the corresponding label of the initial image into the image detection model to obtain the image to be processed.

[0146] It should be noted that the image detection model is the image detection model of the text recognition model training method, and may include an image feature extraction layer and a curve sequence feature extraction layer.

[0147] In practical applications, the image feature extraction layer in the image detection model can be applied to ResNet50+FPN, and the curve series feature extraction layer can be applied to Bézier curves for image feature extraction. Please refer to the description of each application layer in the image detection model in the above embodiments.

[0148] Further, the step of inputting the initial image and its corresponding label into the image detection model to obtain the image to be processed includes:

[0149] The initial image is input into the image feature extraction layer to obtain candidate images;

[0150] The candidate image is input into the curve sequence feature extraction layer for feature extraction to obtain the text detection box of the candidate image;

[0151] Based on the text detection box and the label corresponding to the initial image, a mapping process is performed in the initial image to generate an image to be processed, wherein the label corresponding to the initial image is preset based on the initial image.

[0152] The text recognition method provided in the embodiments of this specification can extract features from candidate images based on curve sequence feature extraction conditions and according to a preset extraction method; specifically, the step of inputting the candidate image into the curve sequence feature extraction layer for feature extraction to obtain the text detection box of the candidate image includes:

[0153] The candidate image is input into the curve sequence feature extraction layer, and features are extracted according to the preset extraction method based on the curve sequence feature extraction conditions to obtain the text detection box of the candidate image.

[0154] It should be noted that the embodiments in this specification can use Bézier curves to extract features from candidate images. The specific feature extraction process can be referred to the process of determining the text detection box of the candidate sample image in the above embodiments, and will not be described in detail here.

[0155] In practical applications, the initial input image can be an image with features such as a blurred background and diverse text shapes. This initial image, along with its corresponding label, is input into an image detection model to obtain the image to be processed. It should be noted that the initial image is first processed by image feature extraction, and then text detection bounding boxes are extracted from the extracted image based on Bézier curves. Then, the labels of the text detection boxes are mapped to the initial image to obtain the corresponding image portion of the text detection box, which serves as the image to be processed. For example, if the text detection box in the initial image is bow-shaped, it will be mapped to the initial image to obtain a bow-shaped image containing text content.

[0156] Step 604: Input the image to be processed into the text recognition model to obtain the text information of the image to be processed.

[0157] It should be noted that the text recognition model includes convolutional layers, self-attention mechanism encoding layers, and fully connected layers.

[0158] Accordingly, the step of inputting the image to be processed into the text recognition model to obtain the text information of the image to be processed includes:

[0159] The image to be processed is input into the convolutional layer for convolution processing to obtain the convolution vector of the image to be processed;

[0160] The convolutional vector is input into the self-attention mechanism encoding layer for encoding processing to obtain the encoded vector;

[0161] The encoded vector is input into the fully connected layer to obtain the text information of the image to be processed.

[0162] The text recognition model is obtained using the text recognition model training method provided in the above embodiments.

[0163] Specifically, the image to be processed output from the image detection model is input into the text recognition model to obtain the text information in the image. In practical applications, the image to be processed can be input into a 6-layer convolutional layer for convolution processing. The specific convolution process will not be elaborated in this embodiment. The convolution vector of the image to be processed is obtained, and the convolution vector is input into the self-attention mechanism encoding layer for encoding to obtain the encoded vector. It should be noted that the features extracted by the self-attention mechanism encoding layer are more representative and serializable. Then, the recognized text content is adjusted through a fully connected layer, and finally the text information of the image to be processed can be output. Using the previous example, the input image to be processed is an arch-shaped image with text content. The text recognition model can output the arch-shaped text in the image to achieve text recognition of the image.

[0164] It should be noted that the fully connected layer in this text recognition model uses text arrangement loss value and text accuracy loss value to adjust the recognized text, which can not only improve the recognition accuracy of text arrangement or order, but also enhance the accuracy of recognizing similar-looking characters.

[0165] In summary, the text recognition method provided in this application uses an image detection model and a text recognition model to recognize images containing text, which can both ensure recognition efficiency and improve the accuracy of text recognition.

[0166] Corresponding to the above method embodiments, this application also provides embodiments of a text recognition model training device. Figure 7 A schematic diagram of the structure of a text recognition model training device according to an embodiment of this application is shown. Figure 7 As shown, the device 700 includes:

[0167] The text recognition model includes a convolutional layer module 702, a self-attention mechanism encoding layer module 704, and a fully connected layer module 706.

[0168] The convolutional layer module 702 is configured to determine a sample training set, which includes target sample images and their corresponding sample labels, and input the target sample images into the convolutional layer for convolution processing to obtain the convolution vector of the target sample images.

[0169] The self-attention mechanism coding layer module 704 is configured to input the convolution vector of the target sample image into the self-attention mechanism coding layer for encoding processing to obtain the encoded vector;

[0170] The fully connected layer module 706 is configured to input the encoding vector and the sample label into the fully connected layer to calculate the target loss value, and adjust the parameters of the text recognition model according to the loss value until the training stopping condition is met.

[0171] Optionally, the convolutional layer module 702 is further configured as follows:

[0172] The determination module is configured to determine the sample training set through an image detection model.

[0173] Optionally, the determining module is further configured to:

[0174] Receive training data including initial sample images and the labels corresponding to the initial sample images;

[0175] The initial sample image is input into the image feature extraction module for feature extraction to obtain candidate sample images;

[0176] The candidate sample image is input into the curve sequence feature extraction module for feature extraction to obtain the text detection box of the candidate image. Based on the text detection box, the sample image to be processed is determined in the initial image, and the label corresponding to the sample image to be processed is set.

[0177] The loss value is calculated based on the labels corresponding to the initial sample images and the labels corresponding to the sample images to be processed, and the parameters of the image detection model are adjusted according to the loss value until the training stopping condition is met.

[0178] Optionally, the fully connected layer module 706 is further configured as follows:

[0179] The encoding vector is input into the first sub-layer of the fully connected layer to calculate the text arrangement loss value, and the encoding vector is input into the second sub-layer of the fully connected layer to calculate the text precision loss value;

[0180] The target loss value is determined based on the text arrangement loss value and the text precision loss value.

[0181] Optionally, the determining module is further configured to:

[0182] The candidate image is input into the curve sequence feature extraction module, and features are extracted according to the preset extraction method based on the curve sequence feature extraction conditions to obtain the text detection box of the candidate image.

[0183] Optionally, the determining module is further configured to:

[0184] Determine the width, height, and target detection points of the text detection box;

[0185] Based on the target detection point and the width of the detection frame, the target detection point width ratio is determined, and based on the target detection point and the height of the detection frame, the target detection point height ratio is determined.

[0186] Based on the target detection point width ratio, the target detection point parameter position is determined in the text detection box. Based on the target detection point parameter position and the target detection point height ratio, the sample image to be processed is determined, and the label corresponding to the sample image to be processed is set.

[0187] Optionally, the determining module is further configured to:

[0188] The target parameters corresponding to the text detection box are determined based on the target detection point width ratio, and the target detection point parameter positions are determined in the text detection box based on the target parameters.

[0189] Optionally, the determining module is further configured to:

[0190] The distance determined by the target detection point parameter position is processed based on the target detection point height ratio, and the feature value of the sample image to be processed is determined based on the preset processing method.

[0191] The sample image to be processed is based on the feature values ​​of the sample image to be processed.

[0192] The above is an illustrative scheme of a text recognition model training device according to this embodiment. It should be noted that the technical solution of this text recognition model training device and the technical solution of the text recognition model training method described above belong to the same concept. For details not described in detail in the technical solution of the text recognition model training device, please refer to the description of the technical solution of the text recognition model training method described above.

[0193] It should be noted that each component in the device claim should be understood as a functional module necessary to implement each step of the program flow or method, and the functional modules are not actual functional divisions or separations. A device claim defined by such a set of functional modules should be understood as a functional module architecture that implements the solution primarily through the computer program described in the specification, and not as a physical device that implements the solution primarily through hardware.

[0194] Corresponding to the above method embodiments, this application also provides embodiments of a text recognition device. Figure 8 A schematic diagram of the structure of a text recognition device according to an embodiment of this application is shown. Figure 8 As shown, the device 800 includes:

[0195] Detection module 802 is configured to receive an initial image and the label corresponding to the initial image, input the initial image and the label corresponding to the initial image into an image detection model to obtain an image to be processed;

[0196] The recognition module 804 is configured to input the image to be processed into a text recognition model to obtain the text information of the image to be processed. The text recognition model is obtained using the above-mentioned text recognition model training method, and the image detection model is the image detection model of the text recognition model training method.

[0197] Optionally, the image detection model includes an image feature extraction layer and a curve sequence feature extraction layer.

[0198] Optionally, the detection module 802 is further configured to:

[0199] The initial image is input into the image feature extraction layer to obtain candidate images;

[0200] The candidate image is input into the curve sequence feature extraction layer for feature extraction to obtain the text detection box of the candidate image;

[0201] Based on the text detection box and the label corresponding to the initial image, a mapping process is performed in the initial image to generate an image to be processed, wherein the label corresponding to the initial image is preset based on the initial image.

[0202] Optionally, the detection module 802 is further configured to:

[0203] The candidate image is input into the curve sequence feature extraction layer, and features are extracted according to the preset extraction method based on the curve sequence feature extraction conditions to obtain the text detection box of the candidate image.

[0204] Optionally, the text recognition model includes convolutional layers, self-attention mechanism encoding layers, and fully connected layers.

[0205] Optionally, the identification module 804 is further configured to:

[0206] The image to be processed is input into the convolutional layer for convolution processing to obtain the convolution vector of the image to be processed;

[0207] The convolutional vector is input into the self-attention mechanism encoding layer for encoding processing to obtain the encoded vector;

[0208] The encoded vector is input into the fully connected layer to obtain the text information of the image to be processed.

[0209] The above is an illustrative scheme of a text recognition device according to this embodiment. It should be noted that the technical solution of this text recognition device and the technical solution of the above-described text recognition method belong to the same concept. For details not described in detail in the technical solution of the text recognition device, please refer to the description of the technical solution of the above-described text recognition method.

[0210] It should be noted that each component in the device claim should be understood as a functional module necessary to implement each step of the program flow or method, and the functional modules are not actual functional divisions or separations. A device claim defined by such a set of functional modules should be understood as a functional module architecture that implements the solution primarily through the computer program described in the specification, and not as a physical device that implements the solution primarily through hardware.

[0211] In one embodiment of this application, a computing device is also provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the instructions to implement the steps of the text recognition method.

[0212] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described text recognition method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described text recognition method.

[0213] An embodiment of this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the text recognition method as described above.

[0214] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the text recognition method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the text recognition method described above.

[0215] This application discloses a chip that stores computer instructions, which, when executed by a processor, implement the steps of the text recognition model training method or text recognition method as described above.

[0216] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0217] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0218] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0219] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0220] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training a text recognition model, characterized in that, The text recognition model includes convolutional layers, self-attention mechanism encoding layers, and fully connected layers; A sample training set is determined, comprising target sample images and their corresponding sample labels. The sample training set is determined using an image detection model, which includes an image feature extraction module and a curve sequence feature extraction module. The image feature extraction module extracts features from the initial sample images to obtain candidate sample images. The curve sequence feature extraction module extracts features from the candidate sample images to obtain text detection boxes for the candidate sample images, thereby determining the target sample image and its corresponding sample label. Determining the target sample image and its corresponding sample label includes: determining the detection box width, detection box height, and target detection point of the text detection box; determining the target detection point width ratio based on the target detection point and the detection box width, and determining the target detection point height ratio based on the target detection point and the detection box height; determining the target detection point parameter position in the text detection box based on the target detection point width ratio; determining the target sample image based on the target detection point parameter position and the target detection point height ratio; and setting the label corresponding to the target sample image. The target sample image is input into the convolutional layer for convolution processing to obtain the convolution vector of the target sample image; The convolutional vector is input into the self-attention mechanism encoding layer for encoding processing to obtain the encoded vector; The encoding vector and the sample label are input into the fully connected layer to calculate the target loss value, and the parameters of the text recognition model are adjusted according to the target loss value until the training stopping condition is met.

2. The text recognition model training method according to claim 1, characterized in that, The training method for the image detection model is as follows: Receive training data including initial sample images and the labels corresponding to the initial sample images; The initial sample image is input into the image feature extraction module for feature extraction to obtain candidate sample images; The candidate sample image is input into the curve sequence feature extraction module for feature extraction to obtain the text detection box of the candidate sample image. Based on the text detection box, the sample image to be processed is determined in the initial sample image, and the label corresponding to the sample image to be processed is set. The loss value is calculated based on the labels corresponding to the initial sample images and the labels corresponding to the sample images to be processed, and the parameters of the image detection model are adjusted according to the loss value until the training stopping condition is met.

3. The text recognition model training method according to claim 1, characterized in that, The step of inputting the encoded vector into the fully connected layer to calculate the target loss value includes: The encoding vector and the sample label are input into the first sub-layer of the fully connected layer to calculate the text arrangement loss value, and the encoding vector and the sample label are input into the second sub-layer of the fully connected layer to calculate the text precision loss value; The target loss value is determined based on the text arrangement loss value and the text precision loss value.

4. The text recognition model training method according to claim 2, wherein inputting the candidate sample image into the curve sequence feature extraction module for feature extraction to obtain the text detection box of the candidate sample image includes: The candidate sample image is input into the curve sequence feature extraction module, and features are extracted according to the preset extraction method based on the curve sequence feature extraction conditions to obtain the text detection box of the candidate sample image.

5. The text recognition model training method according to claim 4, characterized in that, The step of determining the sample image to be processed from the initial sample image based on the text detection box and setting the label corresponding to the sample image to be processed includes: Determine the width, height, and target detection points of the text detection box; Based on the target detection point and the width of the detection frame, the target detection point width ratio is determined, and based on the target detection point and the height of the detection frame, the target detection point height ratio is determined. Based on the target detection point width ratio, the target detection point parameter position is determined in the text detection box. Based on the target detection point parameter position and the target detection point height ratio, the sample image to be processed is determined, and the label corresponding to the sample image to be processed is set.

6. The text recognition model training method according to claim 5, characterized in that, Determining the position of the target detection point parameter in the text detection box based on the target detection point width ratio includes: The target parameters corresponding to the text detection box are determined based on the target detection point width ratio, and the target detection point parameter positions are determined in the text detection box based on the target parameters.

7. The text recognition model training method according to claim 6, characterized in that, The step of determining the sample image to be processed based on the ratio of the target detection point parameter position to the target detection point height includes: The distance determined by the target detection point parameter position is processed based on the target detection point height ratio, and the feature value of the target detection point is determined based on a preset processing method; The sample image to be processed is obtained based on the feature values ​​of the target detection points.

8. A text recognition method, characterized in that, include: The system receives an initial image and its corresponding label, inputs the initial image and its corresponding label into an image detection model to obtain an image to be processed. The image detection model includes an image feature extraction layer and a curve sequence feature extraction layer. The image feature extraction layer is used to extract features from the initial image to obtain candidate images, and the curve sequence feature extraction layer is used to extract features from the candidate images to obtain text detection boxes of the candidate images, thereby determining the image to be processed and its corresponding label. The image to be processed is input into a text recognition model to obtain the text information of the image to be processed. The text recognition model is obtained by the text recognition model training method according to any one of claims 1-7, and the image detection model is the image detection model of the text recognition model training method.

9. The text recognition method according to claim 8, characterized in that, The image detection model includes an image feature extraction layer and a curve sequence feature extraction layer; Accordingly, the step of inputting the initial image and its corresponding label into the image detection model to obtain the image to be processed includes: The initial image is input into the image feature extraction layer to obtain candidate images; The candidate image is input into the curve sequence feature extraction layer for feature extraction to obtain the text detection box of the candidate image; Based on the text detection box and the label corresponding to the initial image, a mapping process is performed in the initial image to generate an image to be processed, wherein the label corresponding to the initial image is preset based on the initial image.

10. The text recognition method according to claim 9, characterized in that, The step of inputting the candidate image into the curve sequence feature extraction layer for feature extraction to obtain the text detection box of the candidate image includes: The candidate image is input into the curve sequence feature extraction layer, and features are extracted according to the preset extraction method based on the curve sequence feature extraction conditions to obtain the text detection box of the candidate image.

11. The text recognition method according to claim 8, characterized in that, The text recognition model includes convolutional layers, self-attention mechanism encoding layers, and fully connected layers. Accordingly, the step of inputting the image to be processed into the text recognition model to obtain the text information of the image to be processed includes: The image to be processed is input into the convolutional layer for convolution processing to obtain the convolution vector of the image to be processed; The convolutional vector is input into the self-attention mechanism encoding layer for encoding processing to obtain the encoded vector; The encoded vector is input into the fully connected layer to obtain the text information of the image to be processed.

12. A text recognition model training device, characterized in that, The text recognition model includes a convolutional layer module, a self-attention mechanism encoding layer module, and a fully connected layer module; The convolutional layer module is configured to determine a sample training set, which includes target sample images and their corresponding sample labels. The target sample images are input into the convolutional layer for convolution processing to obtain the convolution vectors of the target sample images. The sample training set is determined by an image detection model, which includes an image feature extraction module and a curve sequence feature extraction module. The image feature extraction module extracts features from the initial sample images to obtain candidate sample images, and the curve sequence feature extraction module extracts features from the candidate sample images to obtain text detection boxes for the candidate sample images. The process of determining the target sample image and its corresponding sample label includes: determining the detection box width, detection box height, and target detection point of the text detection box; determining the target detection point width ratio based on the target detection point and the detection box width, and determining the target detection point height ratio based on the target detection point and the detection box height; determining the target detection point parameter position in the text detection box based on the target detection point width ratio; determining the target sample image based on the target detection point parameter position and the target detection point height ratio; and setting the label corresponding to the target sample image. The self-attention mechanism encoding layer module is configured to input the convolution vector into the self-attention mechanism encoding layer for encoding processing to obtain an encoded vector; The fully connected layer module is configured to input the encoding vector and the sample label into the fully connected layer to calculate the target loss value, and adjust the parameters of the text recognition model according to the loss value until the training stopping condition is met.

13. A text recognition device, characterized in that, include: The detection module is configured to receive an initial image and the corresponding label of the initial image, input the initial image and the corresponding label of the initial image into an image detection model to obtain an image to be processed. The image detection model includes an image feature extraction layer and a curve sequence feature extraction layer. The image feature extraction layer is used to extract features from the initial image to obtain candidate images. The curve sequence feature extraction layer is used to extract features from the candidate images to obtain text detection boxes of the candidate images, so as to determine the image to be processed and its corresponding label. The recognition module is configured to input the image to be processed into a text recognition model to obtain the text information of the image to be processed, wherein the text recognition model is obtained by the text recognition model training method of any one of claims 1-7, and the image detection model is the image detection model of the text recognition model training method.

14. A computing device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the instructions, it implements the steps of the method according to any one of claims 1-7 or 8-11.

15. A computer-readable storage medium storing computer instructions, characterized in that, When executed by the processor, this instruction implements the steps of the method according to any one of claims 1-7 or 8-11.

16. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-7 or 8-11.

Images