A data enhancement method, system and terminal for scene text recognition

Through adaptive selection of deformation types by data enhancer and optimization of enhancement samples using the loss function feedback from the recognizer, the problem of insufficient diversity and excessive diversity in scene text image data enhancement is solved, and the robustness and recognition performance of the text recognizer are improved. .

CN112149681BActive Publication Date: 2024-11-26PENG CHENG LAB +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011040208.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-28
Publication Date
2024-11-26
Estimated Expiration
2040-09-28

AI Technical Summary

Technical Problem

The existing scene text image data enhancement methods have problems with insufficient diversity and excessive diversity, which affects the performance of text recognizers.

Method used

The data enhancer adaptively selects the deformation type, combines affine deformation and elastic deformation, process the input samples to generate enhancement samples, and optimizes the enhancement samples using the feedback loss function of the scene text identifier to realize the combination of the data enhancer and the identifier Optimize, balance the diversity and affinity of samples.

Benefits of technology

It effectively improves the robustness of the scene text recognizer, and expands the data set by generating appropriate enhancement samples, improving the recognition performance, and solving the problems of insufficient diversity and overdiversity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112149681B_ABST
    Figure CN112149681B_ABST
Patent Text Reader

Abstract

The present invention discloses a data enhancement method, system and terminal for scene text recognition, the method comprising: inputting an input sample into a data enhancer, the data enhancer adaptively selecting a deformation type according to the input sample; the data enhancer deforms the input sample according to the selected deformation type to generate an enhanced sample; the input sample and the enhanced sample are input into a scene text recognizer for recognition, the scene text recognizer outputs a loss function according to the recognition result, and feeds back to the data enhancer; the data enhancer generates an enhanced sample according to the loss function. The present invention realizes the joint optimization of the data enhancer and the scene text recognizer by adaptively generating suitable enhanced samples according to the properties of the input sample and the capabilities of the scene text recognizer through the data enhancer, and balances the diversity and affinity of the samples through the data enhancement method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a data enhancement method, system and terminal for scene text recognition. Background Art

[0002] At present, models based on deep learning generally require a large amount of data for training to achieve better results. However, the limited amount of existing data leads to bottlenecks in model performance. A low-cost solution is data enhancement, which is to perform a series of operations on the original samples to generate new samples.

[0003] Existing data augmentation methods can be divided into two categories: affine deformation and elastic deformation. Affine deformation methods include image rotation, translation, and scaling, such as Figure 1 ( Figure 1 As shown in a), affine deformation can enhance the affinity of samples (affinity refers to the distribution difference between generated samples and original samples). This affine deformation method forces the entire image to perform the same transformation, thereby increasing the global diversity of text images. However, the shape diversity of each character is not taken into account, so there is a problem of insufficient diversity.

[0004] Elastic deformation can improve the diversity of each character, however, this elastic deformation also has some defects for scene text images. For example, due to the similarity of patterns between some characters (such as "O" and "D"), elastic deformation can transform one character into another, such as Figure 1 ( Figure 1 As shown in b), the elastic deformation deforms the “1” on “A133” to “7”. In addition, scene text usually prints each character in a consistent style. Therefore, elastic deformation may be affected by the excessive diversity problem of scene text images, thereby harming the performance of text recognizers.

[0005] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0006] The main purpose of the present invention is to provide a data enhancement method, system and terminal for scene text recognition, aiming to solve the problems of insufficient diversity and excessive diversity in existing scene text image data expansion strategies due to the complexity of text content and shape in the prior art.

[0007] To achieve the above object, the present invention provides a data enhancement method for scene text recognition, the data enhancement method for scene text recognition comprising the following steps:

[0008] Inputting the input sample to a data enhancer, wherein the data enhancer adaptively selects a deformation type according to the input sample;

[0009] The data enhancer performs deformation processing on the input sample according to the selected deformation type to generate an enhanced sample;

[0010] Inputting the input sample and the enhanced sample into a scene text recognizer for recognition, wherein the scene text recognizer outputs a loss function according to the recognition result and feeds the loss function back to the data enhancer;

[0011] The data enhancer generates enhanced samples according to the loss function.

[0012] Optionally, in the data enhancement method for scene text recognition, the deformation types include: affine deformation and elastic deformation.

[0013] Optionally, in the data enhancement method for scene text recognition, the affine deformation is used to perform a linear transformation on the input sample to maintain the affinity of the sample;

[0014] The elastic deformation is used to perform nonlinear transformation on the input sample to improve the local diversity of the sample.

[0015] Optionally, the data enhancement method for scene text recognition, wherein the input sample is input to a data enhancer, and the data enhancer adaptively selects a deformation type according to the input sample, specifically comprises:

[0016] Inputting the input sample into the prediction network, the prediction network extracts features of the input sample and obtains score values ​​corresponding to the two paths of affine deformation and elastic deformation respectively;

[0017] Compare the fractional values ​​corresponding to the affine deformation and elastic deformation paths, and select the one with the larger fractional value as the deformation type.

[0018] Optionally, the data enhancement method for scene text recognition, wherein the step of inputting the input sample and the enhanced sample into a scene text recognizer for recognition, specifically comprises:

[0019] The scene text recognizer inputs the input sample and the picture of the enhanced sample into a neural network;

[0020] The neural network extracts image sequence features, classifies and recognizes the image sequence features, and obtains a recognized character sequence and a loss function.

[0021] Optionally, in the data enhancement method for scene text recognition, the loss function is an adversarial control loss function.

[0022] Optionally, the data enhancement method for scene text recognition further comprises:

[0023] The data enhancer selects a deformation type according to the features of the input sample and the loss function fed back by the scene text recognizer.

[0024] In addition, to achieve the above-mentioned object, the present invention further provides a data enhancement system for scene text recognition, wherein the data enhancement system for scene text recognition comprises:

[0025] Data augmenter and scene text recognizer;

[0026] The data enhancer comprises:

[0027] A gating module, configured to receive an input sample and adaptively select a deformation type according to the input sample;

[0028] An affine deformation module, used for performing a linear transformation on the input sample to generate an enhanced sample so as to maintain the affinity of the sample;

[0029] An elastic deformation module, used for performing nonlinear transformation on the input sample to generate an enhanced sample, so as to improve the local diversity of the sample;

[0030] The scene text recognizer is used to recognize the input sample and the enhanced sample, output a loss function according to the recognition result, and feed it back to the data enhancer;

[0031] The data enhancer is also used to generate enhanced samples according to the loss function.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a data enhancement program for scene text recognition stored in the memory and executable on the processor, wherein the data enhancement program for scene text recognition implements the steps of the data enhancement method for scene text recognition as described above when executed by the processor.

[0033] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium, wherein the storage medium stores a data enhancement program for scene text recognition, and when the data enhancement program for scene text recognition is executed by a processor, the steps of the data enhancement method for scene text recognition as described above are implemented.

[0034] The present invention inputs an input sample into a data enhancer, and the data enhancer adaptively selects a deformation type according to the input sample; the data enhancer deforms the input sample according to the selected deformation type to generate an enhanced sample; the input sample and the enhanced sample are input into a scene text recognizer for recognition, and the scene text recognizer outputs a loss function according to the recognition result and feeds it back to the data enhancer; the data enhancer generates an enhanced sample according to the loss function. The present invention realizes the joint optimization of the data enhancer and the scene text recognizer by adaptively generating suitable enhanced samples according to the properties of the input sample and the capabilities of the scene text recognizer through the data enhancer, effectively improving the robustness of the scene text recognizer, and balancing the diversity and affinity of the samples through the data enhancement method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram comparing the data enhancement effects of two types of deformation enhancement in the prior art and the data enhancement effect of the present invention;

[0036] Figure 2 It is a flow chart of a preferred embodiment of the data enhancement method for scene text recognition of the present invention;

[0037] Figure 3 It is a schematic diagram of the principle of performing data enhancement by a data enhancer and a scene text recognizer in a preferred embodiment of the data enhancement method for scene text recognition of the present invention;

[0038] Figure 4 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] Therefore, in order to overcome the defects of the prior art, the present invention discloses a data enhancement method for scene text recognition as described in a preferred embodiment, such as Figure 2 As shown, the data enhancement method for scene text recognition includes the following steps:

[0041] Step S10: input the input sample to a data enhancer, and the data enhancer adaptively selects a deformation type according to the input sample.

[0042] The data enhancer generates new samples through a series of transformations of the original input samples, which are called enhanced samples.

[0043] Among them, the deformation types include: affine deformation and elastic deformation. The affine deformation is a transformation of the global operation of the image, which performs transformations such as rotation, shift and scaling on the image; the affine deformation is used to perform linear transformation on the input sample to maintain the affinity of the sample; the elastic deformation is to perform local transformations such as stretching and twisting on the sample image; the elastic deformation is used to perform nonlinear transformation on the input sample to improve the local diversity of the sample.

[0044] Specifically, the input sample is input into the prediction network, the prediction network extracts features of the input sample, and obtains score values ​​corresponding to the two paths of affine deformation and elastic deformation respectively; the score values ​​corresponding to the two paths of affine deformation and elastic deformation are compared, and the one with a larger score value is selected as the deformation type.

[0045] Step S20: The data enhancer performs deformation processing on the input sample according to the selected deformation type to generate an enhanced sample.

[0046] Specifically, the data enhancer determines the deformation type, which is affine deformation or elastic deformation, and then the data enhancer deforms the input sample according to the selected deformation type to generate an enhanced sample (the data enhancer generates a new sample, namely the enhanced sample, by subjecting the original input sample to a series of transformations).

[0047] Step S30: input the input sample and the enhanced sample to a scene text recognizer for recognition; the scene text recognizer outputs a loss function according to the recognition result and feeds it back to the data enhancer.

[0048] The scene text recognizer is a model specifically used to recognize text in scene text images. The invariant characteristics of the text can be learned by learning the generated enhanced samples, thereby improving the robustness of the scene text recognizer.

[0049] Specifically, the scene text recognizer inputs the input sample and the enhanced sample images into a neural network; the neural network extracts image sequence features, classifies and recognizes the image sequence features, obtains a recognized character sequence and a loss function, and feeds the loss function back to the data enhancer. The loss function is an adversarial control loss function.

[0050] Step S40: The data enhancer generates enhanced samples according to the loss function.

[0051] Specifically, the loss function obtained by the scene text recognizer (i.e., the loss function loss in deep learning / machine learning, i.e., reflecting the error between the predicted result and the actual result) is fed back to the data enhancer to guide the data enhancer to generate more suitable enhanced samples. According to the loss function obtained by the scene text recognizer, the size of the adversarial control loss function is obtained. Generally speaking, the smaller the adversarial control loss is, the more suitable the enhanced samples generated by the data enhancer are for the scene text recognizer to learn.

[0052] Generally speaking, the smaller the adversarial control loss is, the more suitable the enhanced samples generated by the data enhancer are for the scene text recognizer to learn, because suitable enhanced samples need to ensure diversity and affinity. If the adversarial control loss is small, it means that the diversity and affinity of the enhanced samples are well guaranteed.

[0053] Therefore, the data enhancer adaptively generates enhanced samples according to the properties of the input samples and the capabilities of the scene text recognizer.

[0054] Furthermore, the data enhancer selects a deformation type according to the features of the input sample and a loss function fed back by the scene text recognizer.

[0055] The present invention also discloses a data enhancement system for scene text recognition, wherein the data enhancement system for scene text recognition includes: a data enhancer and a scene text recognizer; the data enhancer includes: a gating module, used to receive input samples and adaptively select deformation types according to the input samples; an affine deformation module, used to perform linear transformation on the input samples to generate enhanced samples to maintain the affinity of the samples; an elastic deformation module, used to perform nonlinear transformation on the input samples to generate enhanced samples to improve the local diversity of the samples; the scene text recognizer is used to identify the input samples and the enhanced samples, output a loss function according to the recognition results, and feed it back to the data enhancer; the data enhancer is also used to generate enhanced samples according to the loss function.

[0056] Specifically, Figure 3 As shown in the figure, the data enhancer consists of three parts: a gating module, an affine deformation module and an elastic deformation module, wherein the gating module can adaptively select the deformation type based on the input sample (the gating module is selected based on the characteristics of the input sample and the loss function feedback of the scene text recognizer); the affine deformation module maintains the affinity of the sample by performing a linear transformation on the sample; and the elastic deformation module improves the local diversity of the sample by performing a nonlinear transformation on the sample. In addition, according to the learning process of the scene text recognizer, the loss function of the data enhancer is set, thereby realizing the joint optimization of the data enhancer and the scene text recognizer.

[0057] Among them, the gating module adaptively selects affine deformation or elastic deformation according to the characteristics of the input sample to generate enhanced samples, and then sends the enhanced samples and input samples to the scene text recognizer for recognition (the scene text recognizer enters the neural network through the input sample picture to extract the picture sequence features, and then classifies and recognizes the sequence features to obtain the recognized character sequence and loss function); the loss function obtained by the scene text recognizer is fed back to the data enhancer to guide the data enhancer to generate more suitable enhanced samples.

[0058] like Figure 1 ( Figure 1 As shown in c), the data enhancement of the present invention can adaptively select the appropriate deformation type according to the sample content. Using the data enhancement method of the present invention, the input sample is first sent to the data enhancer to obtain a suitable enhanced sample, and the input sample and the enhanced sample are sent to the scene text recognizer together, so that the scene text recognizer learns at the same time, so using data enhancement is equivalent to expanding the scale of the data set.

[0059] like Figure 3 As shown in Figure 1, the purpose of the gating module is to predict the deformation type based on the characteristics of the image and the feedback of the scene text recognizer. The prediction network (the prediction network is part of the gating module, which is used to extract features from the input samples and obtain the scores corresponding to the two deformation type paths. The larger the score, the greater the probability of selecting the deformation path) is used to extract features and obtain classification scores α1 and α2, which represent the scores of affine deformation and elastic deformation respectively.

[0060] For the forward propagation (passing the sample through the neural network of the gating module to infer the selection probability X corresponding to the two deformation modules k ) Use the following formula to get the corresponding selection probability X k , where the independent and identically distributed standard Gumbel distribution random variables G k =-log(-log(U k )), U k ~U(0,1) is a variable that follows a uniform distribution, α k These are the α1 and α2 that appeared above. The k subscript can take the value of 1 or 2, representing elastic transformation and affine transformation respectively:

[0061] X k = argmax(α k +G k ); (1)

[0062] When backpropagating (finding the gradient of each parameter of the neural network according to the obtained loss function), the following formula is used to express the selection probability Perform a correction gradient, where τ is the temperature coefficient, set to 0.6:

[0063]

[0064] like Figure 3 As shown in the figure, the main component of the affine deformation module is the space transformer network (hereinafter referred to as STN), which consists of three parts: a positioning network, a grid generator, and a sampler (the sampling rate of the sampler needs to be consistent with the sampling rate of the input samples).

[0065] The input of the localization network is the input sample, and the output is the parameters of the spatial transformation. For the affine deformation module, it is necessary to generate the parameters of the affine deformation matrix, which are used to control the image to rotate, shift, scale, and other operations. Parameter matrix A θ It consists of the following six parameters, θ1 and θ5 are used to control scaling, θ2 and θ4 are used to control rotation, and θ3 and θ6 are used to control translation:

[0066]

[0067] For the grid generator, the input is the parameter matrix and the input sample, and the pixel correspondence between the enhanced sample and the input sample is obtained. Let the subscript i represent the i-th pixel, the superscript t represent the pixel from the enhanced sample, and the superscript s represent the pixel from the input sample. Set as the target coordinates on the augmented sample, and Set as the source coordinates selected on the input sample, the specific generation formula is as follows:

[0068]

[0069] Input the sampling grid (i.e. the correspondence between the input sample and the enhanced sample obtained in the above formula) into the sampler, and use the differentiation method to cut out the corresponding pixels from the input sample (the pixels in the input sample need to be cut out and copied to the enhanced sample). The specific method is bilinear interpolation (the core idea of ​​bilinear interpolation is to perform linear interpolation in two directions respectively, which can ensure end-to-end differentiability). The enhanced sample is represented as I′ and the input sample is represented as I. The sampler generates enhanced samples as follows, where H is the height of the input sample, W is the width of the input sample, h and w are the height and width of the sample respectively, and I hw is the pixel value of the input sample at height h and width w:

[0070]

[0071] like Figure 3As shown in the figure, the overall framework of elastic deformation is still STN, and the specific method it uses is thin plate spline interpolation (hereinafter referred to as TPS). The transformation of TPS is determined by these two sets of control points P′ and P. P′ is the control point on the input sample, called the reference control point. P is the control point on the enhanced sample. Let K represent the number of control points, and R refers to the real number-dimensional space, then the two sets of control points P = [p1, ..., p K ]∈R 2×K , P′=[p′1,…,p′ K ]∈R 2×K .

[0072] The task of the positioning network is to generate basic control points P′, the output vector dimension is 2×K, and then obtain the P′ matrix.

[0073] The grid generator is used to calculate a transformation and apply it to each pixel position on the enhanced sample, generating a sampling grid on the input sample. The transformation matrix β is as follows, where the superscript T indicates the transpose of the matrix:

[0074]

[0075] Among them, Δ P′ is a constant and can be expanded as follows:

[0076]

[0077] Among them, ε is a matrix representing the distance relationship, i and j represent positive integers, and by calculating the basis control point p i and p j The Euclidean distance between them can be obtained by the i-th row and j-th column of ε.

[0078] For the point s′ on the enhanced sample i (x′ i and y′ i Represents the horizontal and vertical coordinates of the point), the corresponding point s on the input sample can be obtained by the following operation i , where d i,k is point s′ i and the kth reference control point p′ k The Euclidean distance of:

[0079]

[0080] Among them, k represents one of the control points, and there are K control points in total, so k takes any value in [1, K], β is the transformation matrix, R (K+3)×1 It is a matrix in the real range with dimension (K+3)×1.

[0081] The final grid S can be obtained by calculating all sampling points in the enhanced sample.

[0082] The sampler will use the input sample I and the sampling grid S to obtain the enhanced sample I', where V represents a differentiable bilinear interpolation:

[0083] I′=V(S,I); (9)

[0084] Furthermore, the loss function L of the scene text recognizer is defined as:

[0085]

[0086] Where T is the length of the symbol sequence, I is the image, y1, y2, ..., y T is the ground truth text represented by a character sequence and p is the corresponding output probability.

[0087] Using the adversarial control loss (ACL) as the loss function of the data enhancer, L(P′) represents the loss of the scene text recognizer for the enhanced sample, and L(P) represents the loss of the scene text recognizer for the input sample:

[0088] l AC =|1.0-exp[L(P′)-αL(P)]|; (11)

[0089] The dynamic α is designed using the following formula:

[0090]

[0091] in, is the classification probability of the enhanced sample for character k, is the classification probability of the true value for character k.

[0092] Furthermore, the training strategy in the data enhancement framework is as follows:

[0093] Input: number of training rounds S and number of iterations N, original training sample set and the corresponding truth value set

[0094] Output: enhancer A and recognizer R;

[0095] (1) For s = 1, 2, ..., S;

[0096] (2) For i=1, 2,...,N:

[0097] (3)P i Enter enhancer A to generate enhanced sample P′ i ;

[0098] (4) Use formula (10) to calculate P i and P′i For the recognition loss L(P i ) and L(P′ i );

[0099] (5) Calculate the loss of enhancer A using formula (11);

[0100] (6) Update the parameters in enhancer A;

[0101] (7)P i Enter the updated enhancer A to generate new enhanced samples P′ i ;

[0102] (8)P i and P′ i The combination enters the recognizer and uses formula (10) to calculate the recognition loss of the recognizer R;

[0103] (9) Update the parameters in the identifier R;

[0104] (10) Update the parameter α using formula (12).

[0105] The above training strategy is used to show how to train the data augmentation and scene text recognition together end-to-end.

[0106] The present invention defines a new module in the data enhancer, namely the gating module, which can select the appropriate deformation type based on the properties of the input samples and the feedback of the scene text recognizer, and defines a new adversarial control loss function to guide the training of the data enhancer.

[0107] The purpose of the present invention is to design a sample-aware data enhancement method for scene text images. Specifically, the data enhancer perceives the characteristics of the sample and adaptively selects elastic deformation or affine deformation. This enhancement method enriches the existing scene text image dataset, thereby solving the problems of insufficient diversity of existing synthetic datasets and high labeling cost of real text image datasets. The robustness of the scene text recognizer is efficiently improved at a relatively low cost. By adding enhanced samples to the training set, the scene text recognizer can significantly improve its performance on benchmark tests.

[0108] Furthermore, if Figure 4 As shown, based on the above-mentioned data enhancement method and system for scene text recognition, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 4 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0109] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a data enhancement program 40 for scene text recognition is stored on the memory 20, and the data enhancement program 40 for scene text recognition can be executed by the processor 10, thereby realizing the data enhancement method for scene text recognition in the present application.

[0110] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the data enhancement method for scene text recognition.

[0111] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0112] In one embodiment, when the processor 10 executes the data enhancement program 40 for scene text recognition in the memory 20, the following steps are implemented:

[0113] Inputting the input sample to a data enhancer, wherein the data enhancer adaptively selects a deformation type according to the input sample;

[0114] The data enhancer performs deformation processing on the input sample according to the selected deformation type to generate an enhanced sample;

[0115] Inputting the input sample and the enhanced sample into a scene text recognizer for recognition, wherein the scene text recognizer outputs a loss function according to the recognition result and feeds the loss function back to the data enhancer;

[0116] The data enhancer generates enhanced samples according to the loss function.

[0117] The deformation types include: affine deformation and elastic deformation.

[0118] Wherein, the affine deformation is used to perform a linear transformation on the input sample to maintain the affinity of the sample;

[0119] The elastic deformation is used to perform nonlinear transformation on the input sample to improve the local diversity of the sample.

[0120] The step of inputting the input sample to the data enhancer, wherein the data enhancer adaptively selects a deformation type according to the input sample, specifically includes:

[0121] Inputting the input sample into the prediction network, the prediction network extracts features of the input sample and obtains score values ​​corresponding to the affine deformation and elastic deformation paths respectively;

[0122] Compare the fractional values ​​corresponding to the affine deformation and elastic deformation paths, and select the one with the larger fractional value as the deformation type.

[0123] The step of inputting the input sample and the enhanced sample into a scene text recognizer for recognition specifically includes:

[0124] The scene text recognizer inputs the input sample and the picture of the enhanced sample into a neural network;

[0125] The neural network extracts features of the image sequence, classifies and recognizes the features of the image sequence, and obtains a recognized character sequence and a loss function.

[0126] Wherein, the loss function is an adversarial control loss function.

[0127] The data enhancement method for scene text recognition further includes:

[0128] The data enhancer selects a deformation type according to the features of the input sample and the loss function fed back by the scene text recognizer.

[0129] The present invention also provides a storage medium, wherein the storage medium stores a data enhancement program for scene text recognition, and when the data enhancement program for scene text recognition is executed by a processor, the steps of the data enhancement method for scene text recognition as described above are implemented.

[0130] In summary, the present invention provides a data enhancement method, system and terminal for scene text recognition, the method comprising: inputting an input sample into a data enhancer, the data enhancer adaptively selecting a deformation type according to the input sample; the data enhancer deforms the input sample according to the selected deformation type to generate an enhanced sample; the input sample and the enhanced sample are input into a scene text recognizer for recognition, the scene text recognizer outputs a loss function according to the recognition result, and feeds it back to the data enhancer; the data enhancer generates an enhanced sample according to the loss function. The present invention realizes the joint optimization of the data enhancer and the scene text recognizer by adaptively generating suitable enhanced samples according to the properties of the input sample and the capabilities of the scene text recognizer through the data enhancer, efficiently improving the robustness of the scene text recognizer, and balancing the diversity and affinity of the samples through the data enhancement method.

[0131] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, an optical disk, etc.

[0132] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A data enhancement method for scene text recognition, It is characterized in that The data enhancement method for scene text recognition includes: Inputting the input sample to a data enhancer, wherein the data enhancer adaptively selects a deformation type according to the input sample; The step of inputting the input sample to the data enhancer, wherein the data enhancer adaptively selects a deformation type according to the input sample, specifically includes: Inputting the input sample into the prediction network, the prediction network extracts features of the input sample and obtains score values ​​corresponding to the two paths of affine deformation and elastic deformation respectively; Compare the scores of the two paths, affine deformation and elastic deformation, and select the one with the larger score as the deformation type; The data enhancer performs deformation processing on the input sample according to the selected deformation type to generate an enhanced sample; Inputting the input sample and the enhanced sample into a scene text recognizer for recognition, wherein the scene text recognizer outputs a loss function according to the recognition result and feeds the loss function back to the data enhancer; The step of inputting the input sample and the enhanced sample into a scene text recognizer for recognition specifically includes: The scene text recognizer inputs the input sample and the picture of the enhanced sample into a neural network; The neural network extracts image sequence features, classifies and recognizes the image sequence features, obtains a recognized character sequence and a loss function, and feeds the loss function back to the data enhancer; The loss function is an adversarial control loss function; The data enhancer generates enhanced samples according to the loss function.

2. The data enhancement method for scene text recognition according to claim 1, It is characterized in that The deformation types include: affine deformation and elastic deformation.

3. The data enhancement method for scene text recognition according to claim 2, It is characterized in that The affine deformation is used to perform a linear transformation on the input sample to maintain the affinity of the sample; The elastic deformation is used to perform nonlinear transformation on the input sample to improve the local diversity of the sample.

4. The data enhancement method for scene text recognition according to claim 1, It is characterized in that The data enhancement method for scene text recognition also includes: The data enhancer selects a deformation type according to the features of the input sample and the loss function fed back by the scene text recognizer.

5. A data enhancement system for scene text recognition, It is characterized in that The data enhancement system for scene text recognition includes: Data augmenter and scene text recognizer; Inputting the input sample into the prediction network, the prediction network extracts features of the input sample and obtains score values ​​corresponding to the two paths of affine deformation and elastic deformation respectively; Compare the scores of the two paths, affine deformation and elastic deformation, and select the one with the larger score as the deformation type; The data enhancer comprises: A gating module, configured to receive an input sample and adaptively select a deformation type according to the input sample; An affine deformation module, used for performing a linear transformation on the input sample to generate an enhanced sample so as to maintain the affinity of the sample; An elastic deformation module, used for performing nonlinear transformation on the input sample to generate an enhanced sample, so as to improve the local diversity of the sample; The scene text recognizer is used to recognize the input sample and the enhanced sample, output a loss function according to the recognition result, and feed it back to the data enhancer; The data enhancer is also used to generate enhanced samples according to the loss function; The scene text recognizer inputs the input sample and the picture of the enhanced sample into a neural network; The neural network extracts image sequence features, classifies and recognizes the image sequence features, obtains a recognized character sequence and a loss function, and feeds the loss function back to the data enhancer; The loss function is an adversarial control loss function.

6. A terminal, It is characterized in that The terminal includes: a memory, a processor, and a data enhancement program for scene text recognition stored in the memory and executable on the processor. When the data enhancement program for scene text recognition is executed by the processor, the steps of the data enhancement method for scene text recognition as described in any one of claims 1 to 4 are implemented.

7. A storage medium, It is characterized in that The storage medium stores a data enhancement program for scene text recognition, and when the data enhancement program for scene text recognition is executed by the processor, the steps of the data enhancement method for scene text recognition according to any one of claims 1 to 4 are implemented.