A general enhancement method for target character recognition in a motion scene and a related device

Abstract

The application discloses a general enhancement method for target character recognition in a motion scene and a related device, uses a network structure construction strategy of a double branch, carries out data fusion and parameter exchange through an implicit text statistical branch and an explicit text recognition branch, can control the model size without increasing the model complexity, and reduces unnecessary parameters of the network. Meanwhile, a basic backbone network is provided for subsequent downstream tasks, and a solution can be provided for text analysis in more complex scenes. The offline learning method used in the application can reduce the cost when iterating and strengthening the model. With the increase of data diversity, the model can dynamically adjust the parameters each time, so that the iteration process can adaptively extract and update the parameters of the data, can improve the convergence speed of the iteration process while ensuring the performance of the model without increasing the network complexity.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of text recognition, and relates to a general enhancement method for target character recognition in a motion scene and a related device. BACKGROUND

[0002] In modern society, text recognition technology has been widely used in various scenarios, such as automatic driving, intelligent security, image recognition, etc. However, traditional text recognition technology often needs to operate under sufficient light conditions, while in complex environments or on the surface of high-speed moving objects, the clarity and stability of the text will be greatly affected, resulting in a significant decrease in recognition accuracy.

[0003] In order to solve this problem, most text recognition technologies in recent years use artificial intelligence mode, which combines advanced technologies in the fields of deep learning and computer vision, and can relatively accurately recognize the text information on the surface of objects under low light and high-speed object motion. Specifically, this technology uses a neural network-based model to extract image features, and performs basic units-operators on hardware to achieve efficient text recognition. In the optimization of operators, this technology uses an end-to-end deep learning framework to automatically generate parameters using model search, thereby achieving automatic tuning and quickly finding the best parameter program configuration. This method abstracts the complex process of manually optimizing network performance into a set of scheduling primitives, thereby greatly reducing the difficulty and complexity of optimization for related program personnel. For this technology, a suitable network plus some data sets can train a model that can be used in practice. If it is applied in complex environments, the traditional technology's requirement for data sets often limits the training of its own model, and due to time and manpower resources, this method is often not feasible.

[0004] Most current text recognition technologies based on convolutional neural networks design a sophisticated recognition paradigm.

[0005] Its core idea is:

[0006] 1. Feature extraction: Convolutional Neural Network (CNN) as a deep learning model can extract feature information in images through multiple convolution operations and pooling operations. In text recognition, CNN can obtain high-dimensional feature representations through multiple convolution and pooling operations on input image data. These feature representations not only contain basic elements in images (such as strokes, lines, edges, etc.), but also include higher-level abstract concepts (such as shapes, textures, etc.), thereby better capturing the features of text.

[0007] 2. Sequence modeling: Text recognition involves modeling and predicting sequence data. To address this challenge, CNNs often add sequence modeling modules such as recurrent neural networks (RNNs) or transformers after the convolutional layers to better handle sequence information. Through these sequence modeling modules, CNNs can map the feature maps output by the convolutional layers to a fixed-length vector representation, providing better input for subsequent classification tasks.

[0008] 3. Classification prediction: CNN-based text recognition techniques usually use fully connected layers or classifiers such as connectionist temporal classification (CTC) for prediction. These classifiers can map the input feature vectors to a fixed number of classes, enabling text recognition. To improve model accuracy and generalization, optimization methods such as data augmentation, transfer learning, and model fusion are also employed.

[0009] During the initial model parameter training, the model is in an initialized state, so the text results predicted based on model search and randomly selected configurations are not significantly different, with high error rates. However, as the model iteratively learns, the parameters gradually become accurate, so that in each iteration, the text prediction network is retrained based on the feedback from the configurations selected according to the proxy model prediction, and the accuracy of the text prediction network also improves.

[0010] Currently, the existing technical solutions are roughly as follows:

[0011] 1. CRNN

[0012] CRNN is the most classic deep learning network applied to text recognition:

[0013] 1) CRNN is an end-to-end trainable neural network for scene text recognition. It proposes an end-to-end trainable neural network model based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs) for scene text recognition. This model can map input image sequences to output text sequences, enabling the automation of the entire text recognition process.

[0014] 2) In the convolutional neural network part of CRNN, the input text image is processed through a series of convolutional layers and pooling layers to extract high-level feature representations. To further improve the feature representation ability, the model uses advanced convolution operations such as deformable convolution and dilated convolution. In the recurrent neural network part, the feature sequence extracted by the convolutional neural network part is input into a recurrent neural network with multiple LSTM units to learn the long-term dependencies between sequences. During training, the network optimizes the model parameters through the backpropagation algorithm to minimize the difference between the output sequence and the true label sequence. In the output layer of the model, a sequence modeling method based on CTC (Connectionist Temporal Classification) is used to map the output feature sequence to the text sequence. The CTC method can handle the alignment problem between the input sequence and the output sequence, thereby avoiding the need for a manual alignment process. This end-to-end design can fully utilize data for model training while avoiding the problems caused by manually designing feature extractors and classifiers.

[0015] 2. SAR

[0016] Compared with CRNN, SAR makes two improvements:

[0017] 1) Faster training and inference speed: The SAR model uses a lightweight convolutional neural network structure and attention mechanism, with fewer parameters, faster training and inference speed. A convolutional neural network (CNN) is used to extract features from the input image, including multiple convolution and pooling layers. The purpose is to extract low-level visual information from the text image, such as strokes, lines, etc. The final output layer is a multi-layer perceptron (MLP) that maps the output of the decoder to a sequence of text characters. This process can be seen as a classification problem, where the output layer of the MLP is a softmax layer that maps the feature vector to a probability distribution over a series of character classes.

[0018] 2) Stronger robustness: SAR uses attention mechanisms in text recognition and achieves good results. The model can more accurately locate and recognize characters when processing text of different lengths and shapes. In the decoding process, an attention mechanism-based decoder is used. The decoder dynamically weights different parts of the input to map the feature representation to a fixed-length vector. This process is performed at each decoding time and for different input image parts, so it can adaptively adapt to different text shapes.

[0019] 3. Mask OCR-L

[0020] Mask OCR-L mainly adds a mask processing method to the model in the CRNN or SAR text recognition scheme in complex scenes. In Mask OCR-L, each object is represented as a binary mask, which can accurately locate the object and identify the sequence text. And it uses data augmentation techniques to expand the data set, and uses a feature pyramid network to extract features of different scales. So that Mask OCR-L can achieve better performance and test accuracy than CRNN, SAR and other technologies in complex scenes. In the training of the model, Mask OCR-L uses a multi-task loss function to train the model. Including the loss function of the text detection and text recognition two tasks, can make the network parameters more suitable for the scheme.

[0021] 1) Mask OCR-L uses a self-supervised learning method to pre-train the encoder and decoder. Estimate the mask patch from the visible patch in the image, and use a CAE decoder composed of four layers, each layer includes a self-attention unit and an FFN unit. Each self-attention module is a 12-head attention. The network model uses the AdamW optimizer for optimization, and uses the linear learning rate scaling rule.

[0022] 2) For some complex environment text recognition, Mask OCR-L uses data preprocessing mechanism, there are two kinds of data preprocessing methods for Chinese and English respectively. For Chinese text line images, due to the large difference in width, Mask OCR-L adjusts the height of the input image to 32 and maintains the aspect ratio, and fills the width of the input image to 400. For English word samples, all input images are directly adjusted to 32x128. For different scenes, Mask OCR-L uses some data enhancement techniques in the training process of downstream recognition, such as rotation, regular distortion and color enhancement schemes to improve the reliability of the data.

[0023] The way Mask OCR-L preprocesses images and optimizes network parameters, although the model is better than previous algorithms in accuracy, can also be targeted for different scenes downstream task processing, but because the model parameters trained by this method are too large, and there is no specific processing scheme for subtasks. Therefore, the speed of this Mask-based text detection and recognition method is between CNN and SAR, and they are also not ideal for high-speed moving target recognition in complex environments.

[0024] In real life, traffic monitoring, industrial automation, security monitoring and other fields, we often encounter the problem of text recognition on the surface of high-speed moving objects in complex environments. For this text recognition technology, its evaluation can be based on three standards: recognition area capture accuracy, low-resolution image restoration, target text segmentation, and digital text recognition accuracy. In practical applications, these three goals often constrain each other, and we need to find a balance between them.

[0025] A commonly used text recognition technology on the surface of high-speed moving objects in a certain scenario is based on a convolutional neural network (CNN) of deep learning. However, convolutional neural networks have three disadvantages in this scenario, namely:

[0026] 1. In each network training upgrade, a large number of fine-tuned data sets are needed to improve the robustness of the network.

[0027] 2. Convolutional neural networks are relatively simple and require a large number of parameters to achieve a certain recognition effect, which also leads to slow model loading and difficulty in meeting real-time recognition requirements in practical applications.

[0028] 3. When processing text recognition on the surface of high-speed moving objects in complex environments, convolutional neural networks have high requirements for raw training data sets and recognition data in applications, and have low accuracy in capturing and recognizing distorted images.

[0029] Considering the need for continuous iteration and updating of the network in applications, the above three characteristics have a large cost in terms of both practice and money, and it is difficult to achieve the speed and accuracy required by the scene in practice. SUMMARY

[0030] The purpose of the present application is to solve the problem of text detection and recognition on the surface of high-speed moving objects in complex environments in the prior art, and to provide a general enhancement method for target text recognition in a moving scenario and related devices. The present application can detect and recognize text on high-speed moving targets in complex environments without increasing the cost of data acquisition and preprocessing to improve the recognition speed and accuracy of the model.

[0031] To achieve the above purpose, the present application adopts the following technical solutions:

[0032] In a first aspect, the present application provides a general enhancement method for target text recognition in a moving scenario, comprising the following steps:

[0033] Cleaning the text image data in the moving scenario to obtain the shallow features of the text image in the moving scenario;

[0034] Using the shallow features, a text prediction model is constructed based on a double-branch network;

[0035] An offline performance prediction model is trained by using the shallow features, and the offline performance prediction model is iteratively optimized;

[0036] The text prediction model is updated by using the optimized offline performance prediction model;

[0037] The motion scene text image is input into the updated text prediction model, and predicted text is output.

[0038] In a second aspect, the present application provides a general enhancement system for target character recognition in a motion scene, comprising:

[0039] A data cleaning module is configured to clean motion scene text image data to obtain shallow features of the motion scene text image;

[0040] An online model construction module is configured to construct a text prediction model based on a double-branch network by using the shallow features;

[0041] An offline model training module is configured to train an offline performance prediction model by using the shallow features, and iteratively optimize the offline performance prediction model;

[0042] A model updating module is configured to update the text prediction model by using the optimized offline performance prediction model;

[0043] A text recognition module is configured to input the motion scene text image into the updated text prediction model, and output predicted text.

[0044] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0045] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the steps of the above method.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] The scene text recognition method designed by the application adopts a more practical and effective data preprocessing strategy. Compared with the current most popular open source data enhancement method ImageAug, Luminance-Retinex not only reduces the human and material resources consumed by data collection, but also significantly improves the recognition accuracy of the model. The application uses a double-branch network structure construction strategy, which is more delicate and powerful than CRNN. That is, through the implicit text statistical branch and the explicit text recognition branch, data fusion and parameter exchange are carried out, which can control the model size and reduce unnecessary network parameters without increasing the complexity of the model. At the same time, it provides a basic backbone network for subsequent downstream tasks, which can provide solutions for text analysis in more complex scenarios. Compared with the SAR technology, the offline learning method used in the application can reduce the cost when iterating and strengthening the model. With the increase of data diversity, the model can dynamically adjust the parameters each time, so that the iteration process can adaptively "extract-update" parameters, which can improve the convergence speed of the iteration process without increasing the network complexity to improve the performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 The flowchart of the method of the application.

[0050] Figure 2 The principle diagram of the system of the application.

[0051] Figure 3 The principle diagram of an embodiment of the application.

[0052] Figure 4 The overall network structure diagram of an embodiment of the application.

[0053] Figure 5 The double-branch sequence complementary feature learning framework diagram of the application.

[0054] Figure 6 The structure diagram of the offline performance model of the application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative work fall within the scope of protection of the present application.

[0057] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0058] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0059] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0060] In the description of the embodiments of the present application, it should also be noted that unless otherwise explicitly specified and limited, if the terms "arrangement", "installation", "connection", "connection" appear, they should be understood in a broad sense, for example, they can be fixedly connected, or detachably connected, or integrally connected; can be mechanically connected, or electrically connected; can be directly connected, or indirectly connected through an intermediate medium, or the communication inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0061] The present application will be described in further detail below with reference to the drawings:

[0062] Referring toFigure 1 The embodiment of the present application discloses a general enhancement method for target character recognition in a motion scene, comprising the following steps:

[0063] S1 cleaning the motion scene text image data to obtain the shallow features of the motion scene text image;

[0064] S2 constructing a text prediction model based on a double-branch network using the shallow features;

[0065] S3 training an offline performance prediction model using the shallow features and iteratively optimizing the offline performance prediction model;

[0066] S4 updating the text prediction model using the optimized offline performance prediction model;

[0067] S5 inputting the motion scene text image into the updated text prediction model to output the predicted text.

[0068] In a feasible implementation of the present application, the cleaning of the motion scene text image data comprises:

[0069] S101 collecting scene text image data to form a basic task set;

[0070] S103 preprocessing the basic task set to obtain the shallow features.

[0071] In a feasible implementation of the present application, the preprocessing of the basic task set comprises:

[0072] S1031 randomly selecting a data set from the basic task set as an initial set;

[0073] S1032 dividing the initial set into a training set, a test set and a validation set by a K-fold cross-validation method;

[0074] S1033 preprocessing the training set according to a Luminance-Retinex method to obtain the shallow features V of the input data through a feature mapping module and output;

[0075] C, H out ,W out ∈V

[0076] wherein C is the output channel, H out is the output height, and W out is the output width.

[0077] In a feasible implementation of the present application, the text prediction model is constructed, comprising:

[0078] ​S201 input the shallow features to the implicit text statistics branch, and output text statistics parameters;

[0079] S202 input the shallow features to the explicit text recognition branch, and output text recognition parameters;

[0080] S203 exchange the text statistics parameters and the text recognition parameters, and perform iteration of the double-branch network; calculate the loss average of all parameter weights of the current iteration double-branch network, evaluate the state of the current double-branch network by using the loss average, and assist in updating the iteration parameters of the next double-branch network iteration;

[0081] S204 input the text statistics parameters, the text recognition parameters and the iteration parameters to the double-branch integration module, input the features integrated by the module to the multilayer perceptron, and obtain deep features;

[0082] S205 input the deep features into the full connection layer for decoding, and obtain the predicted text predicted by the network;

[0083] The double-branch module shares the parameters of the main trunk, and simultaneously provides the optimized parameters to each other.

[0084] In a feasible embodiment of the present application, the implicit text statistics branch is used to predict the number of characters in each target in the sequence data; specifically, given the encoded features from the shared main trunk, the regression prediction based on the mean square error (MSE) loss is implemented; wherein the specific way of loss calculation is as follows:

[0085]

[0086] Wherein, C count represents the length of the predicted sequence text, represents the ground truth of the sequence; the implicit text statistics branch calculates the probability of character occurrence, and uses the parameters to update the loss in network training;

[0087] The explicit text recognition branch is the adjustment of the CTC module in the existing CRNN, and the addition of the shallow variable pressure-based codec framework.

[0088] In a feasible embodiment of the present application, the constructed text prediction model is also evaluated, and the evaluation of the text prediction model is performed by calculating the accuracy, wherein the accuracy Acc is as follows:

[0089]

[0090] Wherein, N accurate represents the number of correct predictions, and N total represents the total number of predicted texts.

[0091] In a feasible implementation of the present application, the offline performance prediction model is trained using the shallow features, and the offline performance prediction model is iteratively optimized, including:

[0092] S301 input the shallow features into the data preprocessing module Backbone, then pass them into the double-branch module, then output the features through the CTC decoder, finally combine the shallow features of the data preprocessing module Backbone and the deep features obtained through the multilayer perceptron MLP, output to the full connection layer to obtain the text result, and obtain the offline performance prediction model;

[0093] S302 take the attribute information, configuration information and loop structure information of the task as input, take the speed and accuracy of text recognition as performance, and constitute a training data, iteratively optimize the offline performance prediction model using the training data; when the total loss is less than the loss change threshold defined by the user, stop iteration.

[0094] As shown in Figure 2 The embodiment of the present application discloses a general enhancement system for target text recognition in a motion scene, which comprises:

[0095] A data cleaning module is configured to clean the motion scene text image data to obtain the shallow features of the motion scene text image.

[0096] An online model construction module is configured to construct a text prediction model based on a double-branch network using the shallow features.

[0097] An offline model training module is configured to train an offline performance prediction model using the shallow features, and iteratively optimize the offline performance prediction model.

[0098] A model updating module is configured to update the text prediction model using the optimized offline performance prediction model.

[0099] A text recognition module is configured to input the motion scene text image into the updated text prediction model, and output the predicted text.

[0100] Embodiment

[0101] As shown in Figure 4 The embodiment of the present application provides a new end-to-end double-branch complementary feature network text recognition method, which comprises the following steps:

[0102] I. Data collection and cleaning

[0103] Step 1, collect data from open source scene text image datasets such as Street View Text, COCO-Text, ICDAR, etc. to form the basic task set. This step is the same as Mask OCR-L, and the dataset of Mask OCR-L can be directly reused.

[0104] Step 2, randomly select a certain number (about 1 / 3 of the number) of data sets from the basic task set as the initial set.

[0105] Step 3, divide the initial set into training set, test set and validation set by K-fold cross-validation method; then preprocess the training set according to the Luminance-Retinex method, obtain the shallow features of the input data through the feature mapping module, and then output the shallow features V, including output channel C, output height H out , output width W out .

[0106] C, H out , W out ∈ V

[0107] The output shallow feature V enters the step 4 implicit text statistical branch and the step 5 explicit text recognition branch respectively.

[0108] II. Constructing online model backbone network

[0109] The effective part of the data will be analyzed by the double branch network, and the encoding module capable of completing the scene text recognition task will be trained.

[0110] Step 4, the shallow feature V enters the implicit text statistical branch, and the text statistical parameters are output.

[0111] The implicit text statistical branch is responsible for predicting the number of characters in each target in the sequence data. Given the encoding features from the shared backbone, it is realized by regression prediction based on mean square error (MSE) loss. The specific way of loss calculation is:

[0112]

[0113] Where C count represents the predicted sequence text length, represents the ground truth of the sequence. The implicit text statistical branch calculates the probability of character occurrence and uses its parameters to update the loss in network training.

[0114] Step 5, the shallow feature V enters the explicit text recognition branch, and the text recognition parameters are output.

[0115] The explicit text recognition branch is adjusted by adjusting the CTC module in the existing CRNN, and a shallow transformer-based codec framework is added, so that the explicit text recognition branch can be more friendly to practical applications.

[0116] Step 6, the text statistical parameters and the text recognition parameters are exchanged, and the iteration of the double-branch network is performed; the loss average avg_loss of all parameter weights of the current iteration double-branch network is calculated, the state of the current model is evaluated by using the avg_loss, and the iteration parameters of the next model iteration are updated.

[0117] Step 7, the text statistical parameters, the text recognition parameters and the iteration parameters are input into the double-branch integration module, and deep features are obtained through the feature integration and the multi-layer perceptron MLP; wherein the double-branch module shares the parameters of the main trunk, and simultaneously provides the optimized parameters to each other.

[0118] Step 8, the deep features are transmitted into the full connection layer for decoding to obtain the predicted text predicted by the network.

[0119] III. Evaluation of online model

[0120] Step 9, for the output predicted text, the gradient of the network is updated by calculating the loss, and the dynamic adjustment of the learning rate and other hyperparameters is performed, the training is evaluated, and the corresponding accuracy Acc is obtained. The calculation method of Acc is as follows:

[0121]

[0122] Wherein, N accurate represents the number of correct predictions, N total represents the number of all predicted texts.

[0123] Step 10, the higher the accuracy score of the network, the better the text recognition effect, and the learning rate of the next training will also be reduced and dynamically adjusted. In this way, the network can obtain the best network model faster than Mask OCR-L.

[0124] IV. Training of offline performance prediction model

[0125] Step 11, the offline performance model is an end-to-end model, the preprocessing part is the LR_Mask method, which is transmitted into the double-branch module after the data preprocessing module Backbone (steps 1-3), then the features are output by the CTC decoder, and finally the shallow features of the data preprocessing module Backbone and the deep features obtained by the MLP are output to the full connection layer to obtain the text result. As shown in Figure 6 , the offline trained performance prediction model will provide support for text recognition in the task environment.

[0126] Step 12, according to (task attribute information, configuration information, loop structure information, performance) to form a training data, the first three are the input of the offline performance prediction model, and the performance is the speed and accuracy of text recognition. Repeat steps 4-6, and stop iteration when the total loss total_loss is less than the user-defined loss change threshold.

[0127] Five, offline network iteration optimization

[0128] In each subsequent iteration of the model, the update strategy adopted by the data model

[0129] Step 13, the developer can replace the MLP layer to adapt to different downstream tasks (such as text replacement, text statistics, etc. Solution). The invention provides a general recognition template, which constitutes the main part of the complex motion scene text recognition, and the remaining downstream module configuration can be adjusted by the developer.

[0130] Step 14, through the data set of a specific scene, the network first cleanses the data, and also performs LR-Mask enhancement on the data. Unlike the processing method of ordinary CRNN network, the invention uses mean supplement technology to perform frame filling operation between blurred frames of moving objects, and through this random data supplement and enhancement, the robustness and recognition accuracy of the network can be significantly improved.

[0131] Step 15, then perform the network iteration process of steps 4-8, maintain the current network optimal configuration measured so far And the best performance For step 9.

[0132] Step 16, add the performance data obtained in step 15 to the measured historical data, and update the offline model.

[0133] Step 17, repeat steps 11-16, the number of repetitions is specified by the user.

[0134] Step 18, output the final optimized model

[0135] By using the invention, accurate text recognition of high-speed moving targets in complex environments can be realized, the model size is only 100M, and the model can be loaded in 1s under the support of Nvidia 2060Ti graphics card, the recognition speed can reach 30fps / s, and the recognition accuracy can reach 97.5%.

[0136] The principle of the invention is:

[0137] 1. Composite offline model building mode, a parameter model is trained on a large open source dataset such as ImageNet by using self-supervised pre-training, which has been proved to be beneficial to downstream tasks by CRNN, Mask OCR-L and other methods. The data set features are learned by using the hidden context automatic coding method. The application is self-adaptively designed for the task by data cleaning. At the same time, a light network structure is used to reduce the parameters of the network, so that the subsequent enhanced data set can learn its features faster. The whole process does not need to update the encoder-decoder module, only the network parameters need to be updated to complete the task of text recognition in new scenarios.

[0138] 2. Enhanced mask framework for complex environment, by using Luminance-Retinex Mask (LR_Mask) technology, during the pre-training process of the encoder, the text image is divided into a set of visible image features and a set of mask image features. The image features are sent to the decoder after Luminance-Retinex enhancement, and the visible mask module is generated. Then, the features of the visible mask module are input into the hidden network regressor with mask query to predict the mask image target. Finally, the mask prediction result and the low-level mask feature are combined and input into the decoder to output the prediction result. Considering that there may be some characters in the application data set that are partially or completely occluded, the application uses a three-channel Mask technology, which processes the target by L-R Gaussian normalization at each layer. The loss function of the encoder pre-training is composed of two parts:

[0139]

[0140] Where, loss s represents the alignment loss function, represents the degree loss of the input image and the preprocessed image, I m represents the parameters of the input image, represents the image parameters after LR_Mask processing, λ represents the weight parameter, loss p represents the prediction loss function, O m represents the real target text, represents the predicted text. Both loss functions are MSE loss, which predicts the target text by the visible part of the target text and other text, and predicts a complete target text by other text. The application adds Mask mask here, which means that the network training process is more robust and reliable.

[0141] 3. Dual-branch sequence complementary feature learning framework, the number of characters in the text is calculated by the hidden framework to verify the explicit text recognition to improve the ability of the network to identify tasks, the overall structure is as follows: Figure 5The scene text recognition task is divided into two sub-tasks of text recognition and character counting by referring to the method in Mask OCR-L, and the shared backbone network is used to extract visual features from the input image after the given data is preprocessed by LR_Mask.

[0142] The double-branch network in the application mainly means that when the network extracts features from the input image, the useful information of the two branches can be transmitted to each other, and more features can be extracted than the single-task network. And the character counting branch can also associate the center of attention of the network with the position of the character level, enhancing the encoding features. Compared with SAR, CTC, Attention and other technologies, although their auxiliary branches are effective, all these technologies lack character-level supervision. The double-branch network of the application allows data to flow in both directions and synchronously enhances the target tasks of both sides to achieve accurate text recognition of moving targets in complex environments.

[0143] An embodiment of the application provides a computer device. The computer device of the embodiment comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. The processor implements the steps in each of the above method embodiments when executing the computer program. Alternatively, the processor implements the functions of each module / unit in each of the above device embodiments when executing the computer program.

[0144] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the application.

[0145] The computer device can be a desktop computer, a notebook, a palm computer, and a cloud server, etc. The computer device can include, but is not limited to, a processor and a memory.

[0146] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0147] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the computer device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory.

[0148] The modules / units integrated in the computer device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can realize the steps of the above-mentioned various method embodiments when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electric carrier signals and telecommunication signals.

[0149] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A general enhancement method for target character recognition in a motion scene, characterized in that, The method comprises the following steps: cleaning the motion scene text image data to obtain shallow features of the motion scene text image; a text prediction model is constructed based on a double-branch network using the shallow features; the text prediction model is constructed, comprising: inputting the shallow features into an implicit text statistical branch to output text statistical parameters; inputting the shallow features into an explicit text recognition branch to output text recognition parameters; performing parameter exchange between the text statistical parameters and the text recognition parameters, and performing iteration of the double-branch network; calculating a loss mean value of all parameter weights of the current iteration double-branch network, evaluating a state of the current double-branch network using the loss mean value, and assisting in updating iteration parameters of the next double-branch network iteration; inputting the text statistical parameters, the text recognition parameters and the iteration parameters into a double-branch integration module, inputting a multilayer perceptron after feature integration, and obtaining deep features; transmitting the deep features into a full connection layer for decoding to obtain predicted text predicted by the network; wherein the double-branch module shares parameters of a main trunk and simultaneously provides optimized parameters to each other; training an offline performance prediction model using the shallow features, and iteratively optimizing the offline performance prediction model; updating the text prediction model using the optimized offline performance prediction model; inputting the motion scene text image into the updated text prediction model to output predicted text.

2. The method of claim 1, wherein the method further comprises: The cleaning of the motion scene text image data comprises: collecting scene text image data to form a basic task set; preprocessing the basic task set to obtain shallow features.

3. The method of claim 2, wherein the method further comprises: The preprocessing of the basic task set comprises: randomly pick from a basic task set as an initial set; dividing an initial set into a training set, a test set and a validation set through K-fold cross-validation; The training set is preprocessed according to the Luminance-Retinex method, and the shallow features of the input data are obtained through a feature mapping module and output; wherein, is an output channel, is an output height, is an output width.

4. The method of claim 1, wherein the method further comprises: The implicit text statistics branch is used to predict the number of characters in each target in the sequence data; specifically: given the encoded features from the shared trunk, a regression prediction is implemented by implementing a regression prediction based on mean squared error loss computation; where the specific way of loss computation is: where, denotes the predicted sequence text length, denotes the ground truth of the sequence; the implicit text statistics branch updates the loss in network training by calculating the probability of character occurrence and using its parameters. the explicit text recognition branch is an adjustment of a CTC module in an existing CRNN, and a coding and decoding framework based on shallow variable pressure is added.

5. The method of claim 1, wherein the method further comprises: Also included is evaluating the constructed text prediction model, specifically evaluating the text prediction model by calculating an accuracy rate as follows: wherein, represents the number of correct predictions, represents the total number of predicted texts.

6. The method of claim 1, wherein the method further comprises: The training of the offline performance prediction model using the shallow features and the iteratively optimizing of the offline performance prediction model comprise: inputting the shallow features into a data preprocessing module Backbone and then into a double-branch module, outputting features through a CTC decoder, and finally outputting text results to a full connection layer to obtain an offline performance prediction model, wherein the text results are obtained from the shallow features of the data preprocessing module Backbone and deep features obtained through a multilayer perceptron MLP; taking attribute information, configuration information and cyclic structure information of a task as input, taking speed and accuracy of text recognition as performance, and constructing a training data, iteratively optimizing the offline performance prediction model using the training data; when a total loss is less than a loss change threshold defined by a user, stopping iteration.

7. A general enhancement system for implementing the method of claim 1 for recognizing target text in a moving scene, characterized by, comprise: a data cleaning module configured to clean motion scene text image data to obtain shallow features of the motion scene text image; an online model construction module configured to construct a text prediction model based on a double-branch network using the shallow features; an offline model training module configured to train an offline performance prediction model using the shallow features, and iteratively optimize the offline performance prediction model; a model updating module configured to update the text prediction model using the optimized offline performance prediction model; and A text recognition module is configured to input the sports scene text image into the updated text prediction model, and output predicted text.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program, when executed by the processor, implements the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program, when executed by the processor, implements the steps of the method according to any one of claims 1-6.

Images