A method and device for scene text recognition

Abstract

The present invention discloses a scene text recognition method and device. The method comprises: acquiring multiple target scene images, wherein the target scene images contain a target object; sequentially inputting each target scene image into an encoder for feature extraction and encoding to generate an image encoding feature sequence; decoding the image encoding feature sequence through a decoder to generate a character recognition result for each target scene image; and inputting the multiple frames of character recognition results into a multi-frame synthesis module to determine a final character recognition result. The method solves the technical problems of low accuracy, low recognition stability, high requirements for acquired images, and poor anti-attack capabilities in the prior art scene text recognition.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a scene text recognition method and device. Background Art

[0002] In recent years, with the development of computer vision technology, text recognition, as an important branch of computer vision technology, has been applied to a wider range of scenarios, such as intelligent traffic monitoring systems, network information security monitoring, insurance claims, etc.

[0003] In traditional optical character recognition technology, traditional image processing (e.g., binarization, morphological transformation, connected domain analysis, and projection analysis) is used to detect and segment text characters one by one. Furthermore, the segmented text areas are input into the recognition module to obtain the text content. The accuracy of text recognition depends on the quality of detection and segmentation, and can achieve ideal results in scenes with a single background and simple application scenarios. However, in more complex and changing application scenarios, traditional methods are difficult to guarantee recognition accuracy. For example, license plate recognition technology is inevitably affected by lighting conditions, image quality, and other factors in actual application scenarios.

[0004] With the development of deep learning technology, many efforts have achieved excellent results in text region detection and recognition in complex scenarios using deep neural networks. Mainstream technical solutions include CRNN and sequence-to-sequence methods, but these methods have the following limitations in practice: poor anti-attack resistance, resulting in incorrect recognition of several characters when recognizing non-text images; low recognition stability, prone to misrecognition in unfavorable scenarios, such as poor shooting angles and complex lighting conditions; and reliable and effective recognition of multi-line or irregular text, requiring the collection of a large number of additional samples and targeted training. Furthermore, existing technologies cover text recognition in single-frame images, but do not address image positioning in complex scenarios or the fusion of text recognition results from multiple frames.

[0005] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0006] The embodiments of the present invention provide a scene text recognition method and device to at least solve the technical problems of low scene text recognition accuracy, low recognition stability, high requirements for collected images, and poor anti-attack ability in the prior art.

[0007] According to one aspect of an embodiment of the present invention, a scene text recognition method and device are provided, including: acquiring multiple target scene images, wherein the target scene images contain target objects; inputting each of the target scene images into an encoder in turn for feature extraction and encoding to generate an image coding feature sequence; decoding the image coding feature sequence through a decoder to generate a character recognition result for each of the target scene images; inputting multiple frames of the character recognition results into a multi-frame synthesis module to determine a final character recognition result.

[0008] Optionally, before inputting each of the above-mentioned target scene images into the encoder in sequence for feature extraction and encoding to generate an image coding feature sequence, the above-mentioned method also includes: inputting the above-mentioned target scene image into a trained target positioning detection network to obtain the region coordinates and region type of the above-mentioned target object; according to the above-mentioned region coordinates and the above-mentioned region type, affine transforming the above-mentioned target scene image to generate a corrected target scene image.

[0009] Optionally, each of the above-mentioned target scene images is input into the encoder in turn for feature extraction and encoding to generate an image coding feature sequence, including: extracting features of the above-mentioned target scene images through the backbone network to obtain a feature sequence; and passing the above-mentioned feature sequence through at least one layer of bidirectional recurrent network to generate the above-mentioned image coding feature sequence.

[0010] Optionally, the target scene image is subjected to feature extraction through a backbone network to obtain a feature sequence, including: processing the target scene image through the backbone network to generate a multi-layer feature map whose length and width are proportional to the length and width of the target scene image; and cutting and separating the multi-layer feature map according to the text arrangement direction in the target scene image to generate the feature sequence.

[0011] Optionally, the backbone network consists of at least one convolutional layer, at least one activation layer, a permutation layer, a cutting layer and a merging layer.

[0012] Optionally, the above-mentioned feature sequence is passed through at least one layer of bidirectional recurrent network to generate the above-mentioned image coding feature sequence, including: inputting the above-mentioned feature sequence into the above-mentioned at least one layer of bidirectional recurrent network from forward and reverse directions respectively to generate two candidate feature sequences; and merging the above-mentioned two candidate feature sequences according to spatial correspondence to obtain the above-mentioned image coding feature sequence.

[0013] Optionally, the decoder is composed of a fully connected layer or a multi-layer perceptron, a connection temporal connection classification model, or the decoder is composed of a fully connected layer or a multi-layer perceptron, an attention mechanism and a recurrent neural network.

[0014] Optionally, when the above-mentioned decoder is composed of a fully connected layer or a multi-layer perceptron and a temporal connection classification model, the above-mentioned method includes: inputting the above-mentioned image encoding feature sequence into the above-mentioned fully connected layer or multi-layer perceptron to obtain a classification probability distribution vector of the same length as the input feature sequence; according to the decoding rule of the above-mentioned target object, decoding the above-mentioned classification probability distribution vector through the above-mentioned connection temporal classification model to generate the above-mentioned character recognition results for each of the above-mentioned target scene images.

[0015] Optionally, when the above-mentioned decoder is composed of a fully connected layer or a multi-layer perceptron, an attention mechanism and a recurrent neural network, it includes: converting each sequence in the above-mentioned image coding feature sequence into a weight vector of each sequence through the above-mentioned fully connected layer or multi-layer perceptron; according to the weight vectors of the above-mentioned sequences, fusing them through the attention mechanism to obtain a feature vector that can represent the current character to be recognized; sending the feature vector of the character to be recognized into the above-mentioned recurrent neural network to obtain the character recognition results of each of the above-mentioned target scene images in turn.

[0016] Optionally, multiple frames of the above-mentioned character recognition results are input into a multi-frame synthesis module to determine the final character recognition result, including: synthesizing the above-mentioned character recognition results, the above-mentioned multi-frame synthesis module determines whether a stable recognition result is obtained, if so, completes the recognition process and outputs the above-mentioned final character recognition result, if not, returns to the above-mentioned encoder and the above-mentioned decoder to repeat the recognition process, wherein the above-mentioned final character recognition result is the character recognition result corresponding to the most repetitions and greater than the first threshold in the voting results of the above-mentioned character recognition results.

[0017] Optionally, before inputting the character recognition results of the multiple frames into the multi-frame integration module and determining the final character recognition result, the method includes: inputting the target scene image into a quality evaluation unit to generate an image quality rating result.

[0018] Optionally, before inputting the above-mentioned character recognition results of multiple frames into a multi-frame integration module and determining the final character recognition result, the above-mentioned method also includes: integrating the above-mentioned character recognition results of each frame, the above-mentioned image quality score results and the encoding rules of the target recognition type, and performing a rationality check on the above-mentioned character recognition results of multiple frames. If yes, input them into the above-mentioned multi-frame integration module; if not, do not input them into the above-mentioned multi-frame integration module.

[0019] Optionally, the target scene image is input into a quality evaluation unit to generate a picture quality score result, including: obtaining a degraded image by performing low-pass filtering on the target scene image, measuring the image quality of the target scene image and the degraded image respectively to generate two sets of vectors, and taking the mean of the sum of the squares of the differences between the elements in the two sets of elements as the picture quality score result.

[0020] Optionally, measuring the image quality of the target scene image and the degraded image to generate two sets of vectors includes: calculating the absolute value of the grayscale difference between each pixel in the target scene image and the degraded image and its adjacent pixels and averaging the difference, dividing the average value by the grayscale mean in a neighborhood around the pixel to generate the two sets of vectors.

[0021] Optionally, the target positioning detection network includes: a second backbone network for processing the target scene image and generating a second feature map whose length and width are proportional to the length and width of the target scene image; a region positioning branch for determining the region description vector of the target object and outputting the region coordinates based on the second feature map; and a classification network branch for predicting the region type based on the second feature map.

[0022] Optionally, according to the above-mentioned area coordinates and the above-mentioned area type, the above-mentioned target scene image is affine transformed to generate a corrected target scene image, including: determining a transformation matrix in combination with the above-mentioned area coordinates and a preset target object template, wherein the coordinates of the above-mentioned preset target object template correspond one-to-one to the above-mentioned area coordinates, and the above-mentioned transformation matrix includes: an affine transformation matrix and / or a perspective transformation matrix; according to the above-mentioned transformation matrix, the above-mentioned target scene image is affine transformed to generate a corrected target scene image.

[0023] Optionally, the above method includes: creating an initial target positioning detection network; training the above initial target positioning detection network using sample data to obtain the above trained target positioning detection network, wherein the above sample data includes first sample data and second sample data generated by applying random perspective changes to the above first sample data.

[0024] Optionally, applying a random perspective change to the first sample data to generate the second sample data includes: applying a random offset within a first range to the coordinates of the first annotation area of ​​the first sample data to generate the second annotation area; determining a training transformation matrix based on the one-to-one correspondence between the training target object template area and the second annotation area; and transforming the first sample data according to the training transformation matrix to generate the second sample data.

[0025] Optionally, before decoding the above-mentioned image coding feature sequence, the above-mentioned method also includes: classifying and locating the above-mentioned image coding feature sequence through the above-mentioned Chinese character area prediction network, determining the first sequence containing Chinese character areas, the second sequence not containing Chinese character areas, the Chinese character area coordinates and the non-Chinese character area coordinates.

[0026] Optionally, the above-mentioned Chinese character region prediction network includes: a Chinese character region prediction classification branch, used to predict the corresponding unique Chinese character region for each pixel in the above-mentioned image coding feature sequence, and determine the above-mentioned image coding feature sequence contained in different Chinese character regions; a Chinese character region coordinate offset prediction branch, used to determine the Chinese character region coordinates on the above-mentioned image coding feature sequence through neighborhood non-maximum suppression.

[0027] Optionally, after the image coding feature sequence is classified and located by the above-mentioned Chinese character area prediction network, the classified feature sequence is input into the above-mentioned decoder and the Chinese character decoder to generate the character recognition result of each of the above-mentioned target scene images; wherein, the classified feature sequence is input into the above-mentioned decoder and the Chinese character decoder to generate the character recognition result of each of the above-mentioned target scene images, including: the above-mentioned first sequence is decoded by the above-mentioned Chinese character decoder to generate the first character recognition result of the Chinese character area, and the above-mentioned second sequence is decoded by the above-mentioned decoder to generate the second character recognition result of the non-Chinese character area; the above-mentioned Chinese character area coordinates and the above-mentioned non-Chinese character area coordinates are combined, and the above-mentioned first character result and the above-mentioned second character result are spliced ​​to generate the character recognition result of each of the above-mentioned target scene images.

[0028] Optionally, the encoder and the decoder perform data augmentation on the training set during training, and the data augmentation includes at least one of the following: gamma correction, Gaussian blur, and up-down and left-right flipping.

[0029] Optionally, the parameters of the encoder and the decoder are determined by a back propagation algorithm and a stochastic gradient descent method.

[0030] Optionally, the above area type is at least one of the following: single-line text, multi-line text, single-column text, and multi-column text.

[0031] Optionally, when the above-mentioned area type is the above-mentioned multi-line text or multi-column text, after the above-mentioned target scene image is affine transformed to generate a corrected target scene image, the above-mentioned method also includes: dividing the above-mentioned corrected target scene image into multiple sub-images containing single-layer text according to the arrangement method and number of multiple layers of text; scaling and aligning the above-mentioned sub-images to the same specifications, and splicing them in the arrangement order to generate a spliced ​​target scene image.

[0032] Optionally, the method for obtaining the above-mentioned target scene image containing the target object includes: the user uses the acquisition device to continuously capture according to the guide box on the graphical interface of the mobile device, or the acquisition device automatically captures based on target detection and target tracking.

[0033] According to another aspect of an embodiment of the present invention, a scene text recognition device is also provided, including: an image acquisition unit, used to obtain multiple target scene images, wherein the above target scene images contain target objects; an encoding unit, used to input each of the above target scene images into an encoder in sequence for feature extraction and encoding, and generate an image encoding feature sequence; a decoding unit, used to decode the above image encoding feature sequence through a decoder, and generate a character recognition result for each of the above target scene images; a multi-frame integration unit, used to input the above character recognition results of multiple frames into a multi-frame integration module, and determine the final character recognition result.

[0034] Optionally, the above-mentioned device also includes: a positioning unit, used to input the above-mentioned target scene image into a trained target positioning detection network to obtain the region coordinates and region type of the above-mentioned target object; and a correction unit, used to affine transform the above-mentioned target scene image according to the above-mentioned region coordinates and the above-mentioned region type to generate a corrected target scene image.

[0035] Optionally, the target positioning detection network includes: a second backbone network for processing the target scene image and generating a second feature map whose length and width are proportional to the length and width of the target scene image; a region positioning branch for determining the region description vector of the target object and outputting the region coordinates based on the second feature map; and a classification network branch for predicting the region type based on the second feature map.

[0036] Optionally, the above-mentioned device also includes: a Chinese character positioning unit, which is used to classify and locate the above-mentioned image coding feature sequence through the above-mentioned Chinese character area prediction network, and determine the first sequence containing Chinese character areas, the second sequence not containing Chinese character areas, the Chinese character area coordinates and the non-Chinese character area coordinates.

[0037] According to another aspect of an embodiment of the present invention, a storage medium is further provided. The storage medium includes a stored program, wherein a device where the storage medium is located executes any one of the above-mentioned scene text recognition methods.

[0038] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is configured to run a program, wherein the program executes any one of the above-mentioned scene text recognition methods when running.

[0039] In an embodiment of the present invention, the following steps are performed: obtaining multiple target scene images, wherein the target scene images contain target objects; inputting each of the target scene images into an encoder in turn for feature extraction and encoding to generate an image coding feature sequence; decoding the image coding feature sequence through a decoder to generate a character recognition result for each of the target scene images; inputting multiple frames of the character recognition results into a multi-frame synthesis module to determine the final character recognition result, thereby solving the technical problems of low scene text recognition accuracy, low recognition stability, high requirements for collected images and poor anti-attack ability in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0041] Figure 1 is a flowchart of an optional scene text recognition method according to an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of an optional feature sequence extraction according to an embodiment of the present invention;

[0043] Figure 3 is a flowchart of another optional scene text recognition method according to an embodiment of the present invention;

[0044] Figure 4 is a schematic diagram of an optional random perspective transformation according to an embodiment of the present invention;

[0045] Figure 5 is an optional schematic diagram of multi-line scene text recognition according to an embodiment of the present invention;

[0046] Figure 6 is a flowchart of another optional scene text recognition method according to an embodiment of the present invention;

[0047] Figure 7 FIG. 4 is a structural block diagram of a scene text recognition device according to one embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0049] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the order used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0050] The following is a flowchart of an optional scene text recognition method according to an embodiment of the present invention. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than shown here.

[0051] refer to Figure 1 , is a flow chart of an optional scene text recognition method according to an embodiment of the present invention. Figure 1 As shown, the scene text recognition method includes the following:

[0052] S100, acquiring a plurality of target scene images, wherein the target scene images include target objects;

[0053] S102, inputting each target scene image into the encoder in sequence for feature extraction and encoding to generate an image coding feature sequence;

[0054] S104, decoding the image encoding feature sequence through a decoder to generate a character recognition result for each target scene image;

[0055] S106, inputting the multi-frame character recognition results into a multi-frame integration module to determine the final character recognition result.

[0056] Through the above steps, multiple target scene images are acquired, wherein the target scene images contain a target object; each target scene image is sequentially input into an encoder for feature extraction and encoding, generating an image encoding feature sequence; the image encoding feature sequence is decoded by a decoder to generate a character recognition result for each target scene image; and the multi-frame character recognition results are input into a multi-frame synthesis module to determine the final character recognition result. Through the above steps, problems such as low recognition accuracy, low recognition stability, high requirements for acquired images, and poor anti-attack capabilities of related technologies when recognizing text in complex scenes are resolved.

[0057] The following is a detailed description of the above implementation steps.

[0058] S100, acquiring a plurality of target scene images, wherein the target scene images include target objects;

[0059] In this embodiment, target scenarios include, but are not limited to, insurance claims, autonomous driving, smart shopping, network information monitoring, and the big data industry. Correspondingly, target objects within the target scenario include, but are not limited to, serial numbers, license plates, street signs, and advertising content. Specifically, text recognition of target objects within the target scenario requires that the target scene image contain an object with text. In this application, manual or automatic detection can be used to ensure that the target scene image contains the target object.

[0060] In an optional embodiment, the method for obtaining the above-mentioned target scene image containing the target object includes: the user uses the acquisition device to continuously capture according to the guide frame on the graphical interface of the mobile terminal device, or the acquisition device automatically captures based on target detection and target tracking. For example, the user can guide the user to select the guide frame in the screen displayed on the interactive interface to control the acquisition screen of the acquisition device to capture the license plate picture containing the target object during the processing. When the application scenario and demand change, for example, when the manual lag makes it impossible to guarantee the capture speed during high-speed driving, the target object is captured by replacing the manual guide frame with a target tracking method based on machine learning or deep learning. When the number of target objects is multiple, such as in the scenario of autonomous driving, the target objects include license plates and road sign information, then the multiple target objects are tracked and acquired simultaneously. This application does not limit the methods of target detection, target tracking and image acquisition.

[0061] S102, inputting each target scene image into the encoder in sequence for feature extraction and encoding to generate an image coding feature sequence;

[0062] In an optional embodiment, step 102, inputting each target scene image into an encoder in sequence for feature extraction and encoding to generate an image encoding feature sequence, may include the following steps:

[0063] S200 extracts the target scene image through the backbone network feature extraction to obtain a feature sequence;

[0064] S202 passes the feature sequence through at least one layer of bidirectional recurrent network to generate an image coding feature sequence.

[0065] Specifically, multiple target scene images to be detected and recognized are sequentially input into the encoder. The encoder's backbone network automatically extracts feature sequences from the input images, feeds these feature sequences into the encoder's recurrent network for prediction and annotation, and generates image encoding feature sequences. Since text often appears in sequence rather than in isolation during scene text recognition, recognizing such sequence-like objects typically requires the system to predict a series of object labels rather than a single label. Therefore, the recognition of such objects can naturally be formulated as a sequence recognition problem.

[0066] In an optional embodiment, step S200, extracting features from the target scene image through a backbone network to obtain a feature sequence, includes:

[0067] The target scene image is processed by the backbone network to generate a multi-layer feature map whose length and width are proportional to the length and width of the target scene image;

[0068] According to the text arrangement direction in the target scene image, the multi-layer feature map is cut and separated to generate a feature sequence.

[0069] In an optional embodiment, the backbone network consists of at least one convolutional layer, at least one activation layer, a permutation layer, a cutting layer and a merging layer.

[0070] In this embodiment, the target scene image passes through at least one convolutional layer of the backbone network to generate a multi-layer feature map whose length and width are proportional to the length and width of the target scene image, with the unit of the image length and width being pixels. Specifically, the length and width ratios can be the same, or they can differ due to different downsampling ratios, but both ratios must satisfy the requirement of being an integer power of 1 divided by 2, for example, 1 / 8 or 1 / 32. The specific values ​​of these ratios depend on the quality of the input image and the recognition accuracy, and can be adjusted by setting the parameters of the convolutional layer.

[0071] In practical applications, text can be arranged in a variety of directions, from left to right, right to left, top to bottom, or diagonally. The specific text direction is also part of the feature information and will affect the readiness of subsequent text recognition. The multi-layer feature map is cut according to the text direction to generate a feature sequence. Because the internal network structure of the backbone network is executed on local areas, each vector in the feature sequence is associated with a region of the original image.

[0072] Figure 2 FIG is a schematic diagram of an optional feature sequence extraction according to an embodiment of the present invention. Taking the feature sequence of a license plate image as an example, Figure 2As shown, the license plate text is sequentially read from left to right. After obtaining the multi-layer feature map, it is cut from left to right in the row direction to generate a feature sequence. Each feature vector of the feature sequence is generated from left to right in the feature map by column. This means that the i-th feature vector is the connection of the i-th column of the multi-layer feature map. The width of each column of the feature sequence is fixed, and each column of the feature sequence corresponds to an area of ​​the original image and can be considered as a descriptor of the area. This embodiment converts the target scene image containing text into multiple feature sequences, which can process text of arbitrary length and arbitrary arrangement. At the same time, the obtained feature sequence contains complete feature information text arrangement and text content.

[0073] In an optional embodiment, the feature sequence is passed through at least one layer of a bidirectional recurrent network to generate an image coding feature sequence, including:

[0074] Input the feature sequence into at least one layer of bidirectional recurrent network from the forward and reverse directions respectively to generate two candidate feature sequences;

[0075] The two candidate feature sequences are merged according to the spatial correspondence to obtain the image coding feature sequence.

[0076] Each frame of the feature sequence uses at least one layer of bidirectional recurrent network to predict the feature sequence, and predicts the label distribution for each feature vector in the sequence. A unidirectional recurrent network is directional and only connects past sequence information. However, the image-based sequence label needs to connect past information as well as be related to future information, that is, the context information in the two directions is interactive and complementary. In the embodiment of the present application, a forward and a backward recurrent network are combined into a bidirectional recurrent network, which can make more full use of context information. The specific manifestation of the network structure is that the network will memorize past information and apply it to the calculation of the current output, that is, the nodes between the hidden layers of the recurrent network are connected, and its input includes not only the output of the input layer at the current moment, but also the output of the hidden layer at the previous moment. The two candidate feature sequences generated by the forward and reverse inputs each have sufficient feature information in that direction. The candidate feature sequences are merged according to the spatial correspondence to obtain the image coding features, and the generated image coding features have feature information in all directions. In addition, the present application supports stacking multi-layer recurrent neural networks. The deep structure has a higher level of abstraction than the shallow abstraction, and the prediction of the feature label distribution is more accurate, thereby further improving the accuracy of subsequent recognition.

[0077] Bidirectional recurrent networks have the ability to capture contextual information within sequences. Similarly, for scene text recognition based on image sequences, using contextual cues is more stable than processing individual characters independently. For example, wide characters may require several consecutive frames to fully describe them. Observing contextual information also makes it easier to distinguish ambiguous characters, and recognition can be performed on sequences of arbitrary length.

[0078] It should be noted that the present application does not limit the specific number of layers of the bidirectional recurrent network, and it will be appropriately changed according to the actual needs of the user in practical applications. The above-mentioned at least one layer of bidirectional recurrent network can be LSTM, RNN, GRU, etc. As well as the output dimension of the backbone network and the input dimension of the recurrent network are not exactly the same, by constructing a dimension conversion layer as a transition between the two structures, the input of the final feature sequence meets the input requirements of the recurrent network. In the above-mentioned recurrent neural network, all layers share parameters, so the parameters to be learned are reduced during training, which greatly improves the training efficiency. At the same time, the parameters that need to be set can be reduced during use, which comprehensively improves the computational efficiency. In addition, in scenarios where the imaging quality is high and stable and the recognition results need to be obtained immediately, a fully connected layer or a multi-layer perceptron can be used instead of the bidirectional recurrent network, which can also be used as a lightweight encoder form.

[0079] S104, decoding the image encoding feature sequence through a decoder to generate a character recognition result for each target scene image;

[0080] Specifically, the role of the decoder is to decode the image coding feature sequence into the character recognition result corresponding to the input target scene image, select the symbol corresponding to the index with the highest probability in each component of the image coding feature sequence as the recognition result, and finally form a sequence as the final recognition sequence.

[0081] In an optional embodiment, the decoder is composed of a fully connected layer or a multi-layer perceptron, a connection temporal connection classification model, or the decoder is composed of a fully connected layer or a multi-layer perceptron, an attention mechanism and a recurrent neural network. In an exemplary embodiment of the present application, two feasible decoding methods are provided according to decoders of different structures.

[0082] Decoding method 1:

[0083] In an optional embodiment, when the decoder is composed of a fully connected layer or a multi-layer perceptron, and a temporal connection classification model, the above method includes:

[0084] Input the image encoding feature sequence into the fully connected layer or multi-layer perceptron to obtain a classification probability distribution vector of the same length as the input feature sequence;

[0085] According to the decoding rules of the target object, the classification probability distribution vector is decoded by the connection temporal classification model to generate the character recognition result of each target scene image.

[0086] Specifically, during the process of converting the image encoding feature sequence into a recognition result, the image encoding feature sequence passes through a fully connected layer or multi-layer perceptron to generate a classification probability distribution vector of the same length as the input feature sequence. If the lengths of the sequences are unequal, it is impossible to align the output sequence vector with the input sequence vector using prior knowledge. The length of the classification probability distribution vector output by the fully connected layer or multi-layer perceptron is equal to the number of character categories to be recognized plus one. The additional categories are the separating boundary classes in the temporal connection classification model. The output layer of the connectionist temporal series model combines the output values ​​at each moment, defining the probabilities of various possible alignments of the output sequence with the input sequence. Furthermore, the probability value of any label distribution sequence can be obtained by summing the probabilities of all different alignments. The output sequence with the highest probability is then found using an optimal algorithm such as dynamic programming. The decoding rules of the target object will change the parameters of the connectionist temporal series classification model. For example, the parameters of the temporal series classification model are set differently when the target object is a license plate or ID document text. In addition, during the model training process, this probability distribution vector sequence is fed into the connectionist temporal classification (CTC) loss function to optimize the network parameters. During the inference process, this probability distribution vector sequence is converted into an output string according to the CTC decoding rules.

[0087] Decoding method 2:

[0088] In an optional embodiment, when the decoder is composed of a fully connected layer or a multi-layer perceptron, an attention mechanism, and a recurrent neural network, it includes:

[0089] Convert each sequence in the image encoding feature sequence into a weight vector of each sequence through a fully connected layer or a multi-layer perceptron;

[0090] According to the weight vectors of each sequence, the feature vector that can represent the current character to be recognized is obtained through the attention mechanism fusion;

[0091] The feature vector of the character to be recognized is sent to the recurrent neural network to obtain the character recognition results of each of the above target scene images in turn.

[0092] Specifically, the embodiment of the present application can support mapping from one sequence of two unequal lengths to another sequence. The output state of each time slice of the decoder is converted into the weight vector of each time slice of the picture coding sequence through the fully connected layer, and then the feature vector that can represent the current character to be recognized is obtained through the attention mechanism and sent to the recurrent neural network unit to obtain the probability distribution vector of all character categories of the current time slice. The above-mentioned recurrent neural network specifically includes: an input layer, multiple hidden layers and an output layer; the nodes between the multiple hidden layers are interconnected; the input of each hidden layer includes the output of the input layer and the output of the previous hidden layer. It should be noted that the above-mentioned recurrent neural network can be RNN, or it can be LSTM, GRU, etc.

[0093] In the decoder of the above structure, the weight vector of each sequence is calculated in the decoding stage, which indirectly reflects the correlation between the feature sequence and the character to be recognized. The decoder outputs the recognition result of each character in sequence according to the recognition order until the output terminator or the maximum supported output character length is reached. When recognizing each character, all the feature vectors obtained by the encoder are input into the decoder and the weights of each group of feature vectors are determined by the attention mechanism and all the hidden layer features are superimposed and input into the decoder for this cyclic decoding process, rather than just sending the last hidden layer or one of the hidden layer features into the decoder for decoding. In addition, the current cyclic neural network inputs not only the feature vector of the current character but also the output of the cyclic neural network at the previous moment. The embodiment of the present application improves the accuracy of scene text recognition by linking the context feature vector and assigning weights to gradually decode the character recognition results.

[0094] Furthermore, the encoder and decoder in the embodiments of the present application support images in various formats, including color, grayscale, and binary images, such as color images in RGB or YUV space, or grayscale images. Furthermore, the input features include information about grayscale and brightness components, enabling simultaneous operation under various lighting conditions. It should be noted that the detection network model for positioning different types of images needs to be trained based on materials of that image type.

[0095] In an optional embodiment, the encoder and decoder perform data augmentation on the training set during training. This augmentation includes gamma correction, Gaussian blurring, and vertical and horizontal flipping. Specifically, this data augmentation can reduce the quality requirements for the input image while supporting both front- and back-facing text recognition, thus enriching application scenarios. For example, horizontal and vertical flipping augmentation supports text recognition in mirrored scenes.

[0096] In an optional embodiment, the parameters of the encoder and decoder are determined by a back propagation algorithm and a stochastic gradient descent method.

[0097] In the embodiment of the present application, the recognition network including the above two decoders supports end-to-end training and deployment of single-line text images. When training the algorithm network model, only a large number of text images and corresponding text content marking information need to be prepared, which avoids the labeling of the image position information of each character, reduces the labeling workload, and makes large-scale labeling possible.

[0098] S106, inputting the multi-frame character recognition results into a multi-frame integration module to determine the final character recognition result.

[0099] In an optional embodiment, the character recognition results of multiple frames are input into a multi-frame integration module to determine the final character recognition result, including:

[0100] Comprehensive character recognition results, the above-mentioned multi-frame synthesis module determines whether a stable recognition result is obtained. If so, the recognition process is completed and the final character recognition result is output. If not, the encoder and decoder are returned to repeat the recognition process, wherein the final character recognition result is the character recognition result corresponding to the most repetitions and greater than the first threshold in the voting results of the character recognition results.

[0101] The multi-frame character recognition results are sequentially input into the multi-frame synthesis module for judgment. The multi-frame synthesis module internally records and judges all the recognition result strings from the current moment. If a stable recognition result is determined, the entire recognition process is completed and the multi-frame voting result is output. Otherwise, the process in S100-S104 is repeated until a stable recognition result is obtained or the user manually exits the recognition process. A stable result means that the effective recognition results of several consecutive frames remain consistent. The specific number of frames is determined by the recognition accuracy requirements. When a stable recognition result is determined, the multi-frame synthesis module votes on the input multi-frame recognition results and uses the string result with the most repetitions and greater than the first threshold as the final output result.

[0102] Specifically, due to the poor recognition and anti-attack performance of single-frame-based character recognition, there is a certain probability of outputting different results when there is a certain disturbance in the input text image. For non-text images, there is also a certain probability of erroneously outputting non-empty results. There is a demand for license plate recognition in scenarios such as vehicle-mounted DVR equipment, security, monitoring, and checkpoints. At the same time, there is also a demand for scene text recognition in fields such as smart retail and smart meters. In these applications, single-frame recognition often cannot meet the needs of actual applications. In order to achieve more stable and accurate operation, the embodiment of the present application improves the accuracy of the recognition system by introducing a multi-frame synthesis module.

[0103] In practical scenarios, imaging quality will inevitably be affected by lighting conditions, imaging clarity, etc. For the recognition of multiple frames of low-quality images, the recognition system may give unrecognizable output. In order to reduce the probability of misrecognition and improve the stability of the system, the embodiment of the present application also takes into account the impact of image quality on recognition.

[0104] In an optional embodiment, the above method includes: inputting the above target scene image into a quality evaluation unit to generate a picture quality rating result. This application does not limit the order of the picture quality evaluation and multi-frame integrated module recognition steps, and the two can be performed serially or in parallel. For example, the above character recognition results of multiple frames can be input into the multi-frame integrated module, and before determining the final character recognition result, the target scene image can be input into the quality evaluation unit to generate a corresponding picture quality rating result.

[0105] In an optional embodiment, the target scene image is input into a quality evaluation unit to generate an image quality score result, including:

[0106] A degraded image is obtained by low-pass filtering the target scene image. The image quality of the target scene image and the degraded image is measured respectively to generate two sets of vectors. The mean of the sum of the squares of the differences between the elements in the two sets of elements is used as the image quality score result.

[0107] In an optional embodiment, the image quality of the target scene image and the degraded image is measured to generate two sets of vectors, including:

[0108] The absolute value of the grayscale difference between each pixel and its adjacent pixels in the target scene image and the above-mentioned degraded image is calculated and averaged, and the above-mentioned average value is divided by the grayscale mean in the neighborhood around the pixel to generate the above-mentioned two sets of vectors.

[0109] The embodiment of the present application predefines an image quality measurement method without a reference image based on the application scenario. There is no need to additionally determine a reference image and an absolute quality standard. Instead, the severity of pixel changes in a local range is represented by comparing the local factors between the degraded image and the target scene image, thereby effectively reflecting the image quality of the object to be evaluated, effectively simplifying the quality evaluation process while ensuring the accuracy of the calculation.

[0110] Before the multi-frame character recognition results are input into the multi-frame synthesis module, the rationality check of the multi-frame character recognition results is carried out. The rationality check of the multi-frame character recognition results is carried out by combining the character recognition results of each frame and the encoding rules of the target recognition type. If the recognition result is reasonable, it is sent to the multi-frame synthesis module. For example, when the recognition target is a VIN code, the rationality check of the recognition result is first carried out according to the VIN encoding rules and the verification result of the check bit. When the recognition target is a license plate, the rationality check is carried out if it does not meet the corresponding type of license plate encoding rules. The unreasonable recognition result is not sent to the multi-frame voting module.

[0111] When considering image quality scores for multiple-frame character recognition results, in an optional embodiment, the method further includes: inputting the multiple-frame character recognition results into a multi-frame integration module before determining a final character recognition result.

[0112] The above-mentioned character recognition results of each frame, the above-mentioned image quality score results and the encoding rules of the target recognition type are comprehensively considered to perform a rationality check on the above-mentioned character recognition results of multiple frames. If yes, the above-mentioned multi-frame comprehensive module is input; if not, the above-mentioned multi-frame comprehensive module is not input.

[0113] Specifically, a rationality check is performed based on multiple factors. For example, if the license plate recognition result satisfies the preset license plate number rules and the image quality score exceeds a preset threshold, it is considered a reasonable character recognition result and the result is sent to the multi-frame result fusion module. In the embodiment of the present application, for low-quality multi-frame image input, it is more likely to give an unrecognizable output, reducing the probability of misrecognition and improving the stability of the system.

[0114] In complex scenes, due to factors such as the large number of subjects and occlusion, the target object may not be obvious in the scene image. Even if the target object is included in the captured image, the correct recognition result cannot be obtained if the target object is not included in the recognition area. Therefore, the embodiment of the present application captures the target scene image in a complex scene and accurately locates the area of ​​the target object through a target positioning detection network.

[0115] In an optional embodiment, each of the target scene images is sequentially input into an encoder for feature extraction and encoding, and before generating an image coding feature sequence, the following steps are included:

[0116] Input the target scene image into the trained target positioning detection network to obtain the region coordinates and region type of the target object;

[0117] According to the region coordinates and the region type, the target scene image is affine transformed to generate a corrected target scene image.

[0118] Figure 3 is a flowchart of another optional scene text recognition method according to an embodiment of the present invention. Figure 3 As shown, the text recognition method includes the following steps:

[0119] S300, acquiring a plurality of target scene images, wherein the target scene images include target objects;

[0120] S302, inputting the target scene image into a trained target positioning detection network to obtain the region coordinates and region type of the target object;

[0121] S304, performing an affine transformation on the target scene image according to the region coordinates and the region type to generate a corrected target scene image;

[0122] S306, inputting each corrected target scene image into the encoder in sequence for feature extraction and encoding to generate an image coding feature sequence;

[0123] S308, decoding the image encoding feature sequence through a decoder to generate a character recognition result for each target scene image;

[0124] S310, inputting the multi-frame character recognition results into a multi-frame integration module to determine the final character recognition result.

[0125] In addition, the above steps S300, S306, S308, S310 and Figure 1 Steps 100, S102, S104, and S106 are the same as in the previous section. Figure 1 The corresponding description will not be described in detail here. Figure 3 The described embodiments and Figure 1 The difference is that the text recognition method further includes S302 and S304, which accurately locate the target object area and correct the target scene image through the target positioning detection network.

[0126] In an optional embodiment, the target positioning detection network includes:

[0127] a second backbone network, configured to process the target scene image and generate a second feature map having a length and width proportional to the length and width of the target scene image;

[0128] A region positioning branch is used to determine a region description vector of the target object based on the second feature map and output the region coordinates of the target object;

[0129] The classification network branch is used to predict the region type of the target object based on the second feature map.

[0130] The target location detection network model can accurately locate the coordinates of the target object's area. The shape of the area is not limited to a quadrilateral, but can be the shape of the area actually occupied by the target object, such as a circular billboard or a triangular road sign. In addition, the target location detection network model supports images in multiple formats, including color images, grayscale images, and binary images, such as color images in RGB or YUV space, or grayscale images. It should be noted that the detection network model for different types of images needs to be trained based on materials of that image type.

[0131] Specifically, the second backbone network in the target location and detection network generates a second feature map whose length and width are proportional to those of the target scene image. The length and width of the feature map image are expressed in pixels. Specifically, the length and width ratios can be the same, or they can differ due to different downsampling ratios, but both ratios must satisfy the requirement of an integer power of 1 divided by 2. While the feature map ensures rich image feature information, it also controls the network structure by varying its size. The specific value of the ratio depends on the quality of the input image and the recognition accuracy, and can be adjusted through the convolutional layer parameter settings.

[0132] For the second feature map, each pixel classification network branch is responsible for predicting the probability that the area centered on the pixel is the target object. Based on this probability, it determines the target object region description vector and outputs the target object's region coordinates. For license plate recognition, for example, after acquiring the corresponding feature map, the region localization branch predicts the probability that each pixel point is the license plate quadrilateral region. Based on the probability distribution, the coordinates of the license plate quadrilateral center and the four vertices are determined and output. Furthermore, the classification network branch predicts the specific region type of the target object based on the second feature map.

[0133] In an optional embodiment, the above-mentioned area type is at least one of the following: single-line text, multiple-line text, single-column text, and multiple-column text.

[0134] The embodiment of the present application predicts the region type by introducing a classification network branch. On the one hand, the arrangement of the feature sequence in the encoder is corrected and adjusted according to the region type. On the other hand, without adjusting the network structure, the prior knowledge of the region type is used to provide reference information for the subsequent encoder and decoder recognition process, making it possible for a recognition network to recognize single-line and multi-line text to improve the recognition effect.

[0135] In actual scenarios, the acquisition device does not face the target directly when collecting images of the target scene, and the lens's own organizational structure causes deformation and distortion in the imaging. By correcting the image before inputting it into the network, the images taken at different angles can be corrected to help accurately locate the target object area and extract more accurate features, thereby maintaining high-accuracy recognition.

[0136] In an optional embodiment, performing an affine transformation on the target scene image according to the region coordinates to generate a corrected target scene image includes:

[0137] Determine a transformation matrix by combining the region coordinates and a preset target object template, wherein the coordinates of the preset target object template correspond one-to-one to the region coordinates, and the transformation matrix includes: an affine transformation matrix and / or a perspective transformation matrix;

[0138] The target scene image is affine transformed according to the transformation matrix to generate a corrected target scene image.

[0139] Specifically, the detection object of the target positioning detection and subsequent recognition network is used as a preset target object template. The present application does not limit the setting method. It can be set manually or automatically confirmed by the area type of the target object. The target object template standardizes the format of the input image. The area coordinates obtained by detection correspond one-to-one to the area coordinates of the preset target object template. The above-mentioned corresponding coordinate information is combined to determine the transformation matrix. The above-mentioned transformation matrix includes: affine transformation matrix and / or perspective transformation matrix. The present application does not limit the determination method of the transformation matrix. According to the above-mentioned transformation matrix, the above-mentioned target scene image is affine transformed to generate a corrected target scene image, and finally the image deformation is eliminated, thereby improving the detection and recognition efficiency while improving the accuracy.

[0140] The target localization and detection network, after acquiring the coordinates of the deformed target region, corrects the coordinates using a transformation matrix to eliminate the distortion. Therefore, the premise for correcting the deformed target image is to ensure that the target region can still be quickly and accurately located within the deformed target image. During the training phase of the target localization and detection network, random perspective transformations are applied to the input image and annotations, enabling the network to achieve more stable and accurate positioning results for images captured from different angles.

[0141] In an optional embodiment, the above method includes:

[0142] Create an initial target positioning and detection network;

[0143] The initial target positioning detection network is trained using sample data to obtain the trained target positioning detection network, wherein the sample data includes first sample data and second sample data generated by applying a random perspective transformation to the first sample data.

[0144] The purpose of this step is to train the initial target location and detection network so that the network can locate and detect images from multiple angles. While using the commonly used image data and the coordinates of the annotated area as the first sample data, the sample data is enriched by applying a random perspective transformation to the first sample data, so that the sample data includes image information from as many angles as possible.

[0145] In an optional embodiment, applying a random perspective transformation to the first sample data to generate the second sample data includes:

[0146] Applying a random offset within a first range to the coordinates of the first annotation area of ​​the first sample data to generate a second annotation area;

[0147] Determine a training transformation matrix based on the one-to-one correspondence between the training target object template region and the second labeled region;

[0148] The first sample data is transformed according to the training transformation matrix to generate second sample data.

[0149] Typically, the first sample dataset includes the coordinates of a first annotated region of the target object. A random offset within a first range is applied to these first annotated region coordinates to generate a second annotated region. The first range is determined based on the required deformation angle range for the subsequent positioning image. The target object template specifies the positioning region coordinates. A one-to-one correspondence between these region coordinates and the second annotated region determines a training transformation matrix. The first sample dataset is then transformed according to the training transformation matrix to generate the second sample dataset.

[0150] Figure 4 FIG. 1 is a schematic diagram of an optional random perspective transformation according to an embodiment of the present invention, taking a license plate image and annotation content as an example. Figure 4 As shown, the coordinates of the four vertices of the license plate quadrilateral in the image are (x i ,y i ), i∈{1, 2, 3, 4}, apply a random offset within an appropriate range to each vertex coordinate (the triangle symbols represent the vertex coordinates of the four vertices after random perturbation) to obtain the corresponding four vertices (x′ i , y′ i ), i∈{1, 2, 3, 4}. At the same time, since the current target positioning detection network detects license plates, the preset target object template is a quadrilateral license plate, and the width and height of the input preset license plate template are w and h respectively. The corresponding vertex coordinates of the four vertices after random perspective transformation in the clockwise direction are (0, 0), (w, 0), (w, h), (0, h) respectively. The transformation matrix can be determined by substituting the four sets of vertex coordinates before and after perspective transformation into the following conversion formula:

[0151]

[0152] According to the random perspective transformation matrix, the original image and annotation information after perspective transformation are obtained. Figure 4 As shown, for the coordinates of the first annotated area marked in the first sample data, including the license plate area and the license plate Chinese character area, the second sample data generated after random perspective transformation is a license plate image in a preset license plate template, and includes the license plate Chinese character area annotation box. This application enriches the sample data by applying random perturbations to the sample data and perspective transforming it into a preset target object template, so that the subsequent network still has a more stable and accurate positioning effect for target images taken at different angles.

[0153] At the same time, during the training phase of the target positioning and detection network, gamma correction with random parameters within a suitable value range, Gaussian blur, and image amplification by flipping the image 180° up and down are applied to the samples, so that the network can support the positioning of both positive and negative target objects.

[0154] In an optional embodiment, when the region type is the multi-line text or multi-column text, after performing an affine transformation on the target scene image to generate a corrected target scene image, the method further includes:

[0155] Dividing the corrected target scene image into multiple sub-images containing single-layer text according to the arrangement and number of the multi-layer texts;

[0156] The above sub-images are scaled and registered to the same size, and then stitched together in the order of arrangement to generate a stitched target scene image.

[0157] Specifically, multiple layers of text information are provided according to the region type, and the corrected target scene image is divided and spliced. The above-mentioned multiple layers of text information include but are not limited to the arrangement method and number of multiple layers of text. This application does not limit the method of image division and splicing. Prior knowledge is provided by the region type, and the target scene image containing multiple layers of text is adjusted to a single layer of text to adapt to the subsequent recognition network that supports single-line text recognition. This not only makes it possible for a recognition network to recognize single-line and multi-line text, but also greatly reduces the number and type requirements of training samples. Taking the double-line yellow license plate image recognition used for large domestic vehicles and trailers as an example, Figure 5 is an optional schematic diagram of multi-line scene text recognition according to an embodiment of the present invention, such as Figure 5 As shown in the figure, first, the initial image is determined to be two lines of text arranged up and down according to the region type result, and the corrected target scene image is divided into two sub-images containing single-line text. The two sub-images are of different sizes, and then scaled and spliced ​​in turn according to the proportion of the sub-images corresponding to the upper and lower parts to generate the final spliced ​​target scene image.

[0158] Due to the complex stroke structure of Chinese characters, scene recognition scenarios often bring difficulties. Especially when the input image quality is poor, errors are often most likely to occur in the recognition of Chinese character parts. When the goal is to recognize such low-quality images with Chinese characters, in order to improve the recognition success rate, the embodiment of the present application also introduces a Chinese character area prediction network. Figure 6 is a flowchart of another optional scene text recognition method according to an embodiment of the present invention.

[0159] In an optional embodiment, before decoding the above-mentioned image coding feature sequence, the above-mentioned method also includes: classifying and locating the above-mentioned image coding feature sequence through the above-mentioned Chinese character region prediction network to determine a first sequence containing Chinese character regions and a second sequence not containing Chinese character regions.

[0160] In an optional embodiment, after the image coding feature sequence is classified and located by the Chinese character region prediction network, the classified feature sequence is input into the decoder and the Chinese character decoder to generate a character recognition result for each of the target scene images;

[0161] The classified feature sequence is input into the decoder and the Chinese character decoder to generate the character recognition results for each target scene image, including:

[0162] The first sequence is decoded by the Chinese character decoder to generate a first character recognition result for the Chinese character area, and the second sequence is decoded by the decoder to generate a second character recognition result for the non-Chinese character area;

[0163] The Chinese character area coordinates and the non-Chinese character area coordinates are combined, and the first character result and the second character result are spliced ​​to generate a character recognition result for each of the target scene images.

[0164] like Figure 6 As shown, after the image passes through the encoder to generate an image coding feature sequence, the coding feature sequence is classified and located through the Chinese character area prediction network to determine whether it contains Chinese characters, and the first sequence containing the Chinese character area and the second sequence not containing the Chinese character area are determined, and the coordinates of the corresponding areas on the feature map are output at the same time.

[0165] In an optional embodiment, the Chinese character region prediction network includes: a Chinese character region prediction classification branch, used to predict a unique Chinese character region corresponding to each pixel in the image coding feature sequence, and determine the image coding feature sequences contained in different Chinese character regions; a Chinese character region coordinate offset prediction branch, used to determine the coordinates of the Chinese character region on the image coding feature sequence by neighborhood non-maximum suppression;

[0166] In order to achieve Chinese character area prediction, similar to the target positioning detection network, the Chinese character area prediction network includes a Chinese character area prediction classification branch and a Chinese character area coordinate offset prediction branch. For each pixel in the image coding feature sequence, at most one unique corresponding Chinese character area is predicted, and the image coding feature sequence is further classified according to the feature sequences contained in different Chinese character areas. The Chinese character area coordinate offset prediction branch finally obtains the coordinate information of each Chinese character in the image through non-maximum suppression. In addition, based on the Chinese character area coordinate information and the preset Chinese character template, the transformation matrix used to align each Chinese character can be uniquely determined. The transformation matrix includes a perspective transformation or an affine transformation matrix, so that the image after each Chinese character is aligned can be predicted, further improving the accuracy of Chinese character recognition, wherein the preset Chinese character template includes a preset number of width and height pixels that are unified for all Chinese characters. In order to implement this method, when preparing training materials, in addition to the overall text content of the text image, the coordinates of each Chinese character area also need to be marked.

[0167] By classifying the sequence, subsequent decoding can be more targeted, improving the accuracy of recognition. The classified feature sequences are input into the decoder and the Chinese character decoder for decoding. The first sequence is predicted by the Chinese character decoder. In order to recognize printed characters and artistic characters, the corresponding types of samples can be amplified in the training Chinese character decoder. This application does not limit the form of the Chinese character decoder, such as RNN, CRNN network. The second sequence is predicted by the decoder for non-Chinese characters. The decoder is the same as the above. Figure 1 The decoder in is consistent and will not be described in detail. It can remove Chinese character materials in the training decoder. Finally, the Chinese character area coordinates and non-Chinese character area coordinates output by the Chinese character area prediction network, the first character result and the second character result are combined to generate the character recognition result. This application decodes and recognizes the sequence by region and network according to the characteristics of different character types, achieves accurate recognition, and avoids the problem of low training efficiency caused by the redundancy of sample materials in training.

[0168] In the embodiments of the present application, the embodiments of the present application can still obtain stable and accurate recognition results even under complex lighting conditions or other adverse factors that affect the accurate acquisition of recognition results, thereby solving the problems of low scene text recognition accuracy, low recognition stability, high requirements for collected images, and poor anti-attack resistance in the prior art.

[0169] According to another aspect of an embodiment of the present invention, a storage medium is further provided. The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the scene text recognition methods.

[0170] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is configured to run a program, wherein the program executes any one of the scene text recognition methods when running.

[0171] According to another aspect of the present invention, a scene text recognition device is provided. Figure 7 , is a structural block diagram of a scene text recognition device according to one embodiment of the present invention. Figure 7 As shown, the scene text recognition device includes an image acquisition unit 700, an encoding unit 702, a decoding unit 704 and a multi-frame integration unit 706.

[0172] The image acquisition unit 700 is configured to acquire a plurality of target scene images, wherein the target scene images include target objects;

[0173] The encoding unit 702 is configured to sequentially input each target scene image into an encoder for feature extraction and encoding to generate an image encoding feature sequence;

[0174] A decoding unit 704 is configured to decode the image coding feature sequence through a decoder to generate a character recognition result for each target scene image;

[0175] The multi-frame integration unit 706 is used to input the character recognition results of the multiple frames into the multi-frame integration module to determine the final character recognition result.

[0176] In an optional embodiment, the device further includes:

[0177] A positioning unit, configured to input the target scene image into a trained target positioning detection network to obtain the region coordinates and region type of the target object;

[0178] A correction unit is used to perform an affine transformation on the target scene image to generate a corrected target scene image according to the region coordinates and the region type.

[0179] In an optional embodiment, the target positioning detection network includes:

[0180] a second backbone network, configured to process the target scene image and generate a second feature map having a length and width proportional to the length and width of the target scene image;

[0181] A region localization branch, configured to determine a region description vector of the target object based on the second feature map and output the region coordinates;

[0182] The classification network branch is used to predict the region type according to the second feature map.

[0183] In an optional implementation, the device further includes:

[0184] The Chinese character positioning unit is used to classify and locate the image coding feature sequence through the Chinese character area prediction network, and determine the first sequence containing the Chinese character area, the second sequence not containing the Chinese character area, the Chinese character area coordinates and the non-Chinese character area coordinates.

[0185] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the above-mentioned units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0187] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0188] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0189] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0190] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A scene text recognition method, characterized in that: include: Acquire multiple target scene images, wherein the target scene images include target objects; Inputting each target scene image into an encoder in sequence for feature extraction and encoding to generate an image coding feature sequence; Classifying and locating the image coding feature sequence through a Chinese character region prediction network to determine a first sequence containing Chinese character regions, a second sequence not containing Chinese character regions, coordinates of Chinese character regions, and coordinates of non-Chinese character regions; Inputting the classified image coding feature sequence into the first decoder and the Chinese character decoder to generate a character recognition result for each target scene image; Inputting the character recognition results of the multiple frames into a multi-frame integration module to determine the final character recognition result; Inputting the classified image coding feature sequence into the first decoder and the Chinese character decoder to generate a character recognition result for each target scene image includes: The first sequence is decoded by the Chinese character decoder to generate a first character recognition result in a Chinese character area, and the second sequence is decoded by the first decoder to generate a second character recognition result in a non-Chinese character area; The Chinese character area coordinates and the non-Chinese character area coordinates are combined, and the first character recognition result and the second character recognition result are spliced ​​to generate a character recognition result for each of the target scene images.

2. The method according to claim 1, characterized in that Before sequentially inputting each target scene image into an encoder for feature extraction and encoding to generate an image coding feature sequence, the method further includes: Inputting the target scene image into a trained target positioning detection network to obtain the region coordinates and region type of the target object; The target scene image is affine transformed according to the region coordinates and the region type to generate a corrected target scene image.

3. The method according to claim 1, characterized in that Each target scene image is sequentially input into the encoder for feature extraction and encoding to generate an image coding feature sequence, including: Extract features of the target scene image through a backbone network to obtain a feature sequence; The feature sequence is passed through at least one layer of a bidirectional recurrent network to generate the image coding feature sequence.

4. The method according to claim 3, characterized in that Extracting features of the target scene image through a backbone network to obtain a feature sequence includes: Processing the target scene image through the backbone network to generate a multi-layer feature map whose length and width are proportional to the length and width of the target scene image; According to the text arrangement direction in the target scene image, the multi-layer feature map is cut and separated to generate the feature sequence.

5. The method according to claim 3, characterized in that The backbone network consists of at least one convolutional layer, at least one activation layer, a permutation layer, a cutting layer and a merging layer.

6. The method according to claim 3, characterized in that Passing the feature sequence through at least one layer of a bidirectional recurrent network to generate the image coding feature sequence includes: Inputting the feature sequence into the at least one layer of bidirectional recurrent network from forward and reverse directions respectively to generate two candidate feature sequences; The two candidate feature sequences are merged according to a spatial correspondence to obtain the image coding feature sequence.

7. The method according to claim 1, characterized in that The first decoder is composed of a fully connected layer or a multi-layer perceptron, a connection temporal connection classification model, or the first decoder is composed of a fully connected layer or a multi-layer perceptron, an attention mechanism and a recurrent neural network.

8. The method according to claim 7, characterized in that When the first decoder is composed of a fully connected layer or a multi-layer perceptron and a temporal connection classification model, the method includes: Inputting the image encoding feature sequence into the fully connected layer or multi-layer perceptron to obtain a classification probability distribution vector of the same length as the input feature sequence; According to the decoding rule of the target object, the classification probability distribution vector is decoded by the temporal connection classification model to generate a character recognition result for each of the target scene images.

9. The method according to claim 7, characterized in that When the first decoder is composed of a fully connected layer or a multi-layer perceptron, an attention mechanism, and a recurrent neural network, it includes: Convert each sequence in the image coding feature sequence into a weight vector of each sequence through the fully connected layer or multi-layer perceptron; According to the weight vectors of each sequence, a feature vector capable of representing the current character to be recognized is obtained through the attention mechanism fusion; The feature vector of the character to be recognized is sent to the recurrent neural network to obtain the character recognition results of each of the target scene images in turn.

10. The method according to claim 1, characterized in that Inputting the character recognition results of the multiple frames into a multi-frame integration module to determine the final character recognition result, including: Based on the character recognition results, the multi-frame synthesis module determines whether a stable recognition result is obtained. If so, the recognition process is completed and the final character recognition result is output. If not, the encoder and the first decoder are returned to repeat the recognition process, wherein the final character recognition result is the character recognition result corresponding to the most repetitions and greater than the first threshold in the voting results of the character recognition results.

11. The method according to claim 1, wherein Before inputting the character recognition results of multiple frames into a multi-frame integration module to determine the final character recognition result, the method includes: inputting the target scene image into a quality evaluation unit to generate an image quality scoring result.

12. The method according to claim 11, characterized in that Before inputting the character recognition results of the multiple frames into a multi-frame integration module and determining a final character recognition result, the method further includes: The character recognition results of each frame, the image quality score results and the encoding rules of the target recognition type are integrated to perform a rationality check on the character recognition results of multiple frames. If yes, they are input into the multi-frame integration module; otherwise, they are not input into the multi-frame integration module.

13. The method according to claim 11, characterized in that Inputting the target scene image into a quality evaluation unit to generate an image quality score result includes: A degraded image is obtained by performing low-pass filtering on the target scene image, and the image quality of the target scene image and the degraded image is measured respectively to generate two sets of vectors, and the mean of the sum of the squares of the differences between the elements in the two sets of vectors is used as the image quality score result.

14. The method according to claim 13, characterized in that Measuring the image quality of the target scene image and the degraded image to generate two sets of vectors, including: The absolute value of the grayscale difference between each pixel and adjacent pixels in the target scene image and the degraded image is calculated and averaged, and the average value is divided by the grayscale mean in the neighborhood around the pixel to generate the two sets of vectors.

15. The method according to claim 2, characterized in that The target positioning detection network includes: a second backbone network, configured to process the target scene image and generate a second feature map having a length and width proportional to the length and width of the target scene image; A region localization branch, configured to determine a region description vector of the target object based on the second feature map and output the region coordinates; The classification network branch is used to predict the region type according to the second feature map.

16. The method according to claim 2, characterized in that Affine transforming the target scene image to generate a corrected target scene image according to the region coordinates and the region type, comprising: Determine a transformation matrix by combining the region coordinates and a preset target object template, wherein the coordinates of the preset target object template correspond one-to-one to the region coordinates, and the transformation matrix includes: an affine transformation matrix and / or a perspective transformation matrix; The target scene image is affine transformed according to the transformation matrix to generate a corrected target scene image.

17. The method according to claim 2, characterized in that The method comprises: Create an initial target localization and detection network; The initial target positioning detection network is trained using sample data to obtain the trained target positioning detection network, wherein the sample data includes first sample data and second sample data generated by applying random perspective change to the first sample data.

18. The method according to claim 17, characterized in that Applying random perspective change to the first sample data to generate second sample data includes: Applying a random offset within a first range to the coordinates of the first annotation area of ​​the first sample data to generate a second annotation area; Determining a training transformation matrix based on the one-to-one correspondence between the training target object template region and the second labeled region; The first sample data is transformed according to the training transformation matrix to generate the second sample data.

19. The method according to claim 1, wherein The Chinese character region prediction network includes: A Chinese character region prediction classification branch is used to predict a unique Chinese character region corresponding to each pixel in the image coding feature sequence, and determine the image coding feature sequences contained in different Chinese character regions; The Chinese character region coordinate offset prediction branch is used to determine the Chinese character region coordinates on the image coding feature sequence through neighborhood non-maximum suppression.

20. The method according to claim 1, wherein The encoder and the first decoder perform data augmentation on the training set during the training process, where the data augmentation includes at least one of the following: gamma correction, Gaussian blur, and up-down and left-right flipping.

21. The method according to claim 1, wherein The parameters of the encoder and the first decoder are determined by a back propagation algorithm and a stochastic gradient descent method.

22. The method according to claim 2, characterized in that The area type is at least one of the following: single-line text, multi-line text, single-column text, and multi-column text.

23. The method according to claim 22, characterized in that When the region type is the multi-line text or multi-column text, after affine transforming the target scene image to generate a corrected target scene image, the method further includes: Dividing the corrected target scene image into a plurality of sub-images containing single-layer texts according to the arrangement manner and number of the multi-layer texts; The sub-images are scaled and registered to the same size, and are spliced ​​in the order of arrangement to generate a spliced ​​target scene image.

24. The method according to claim 1, wherein The method for obtaining the target scene image containing the target object includes: the user uses the acquisition device to continuously capture according to the guide box on the graphical interface of the mobile device, or the acquisition device automatically captures based on target detection and target tracking.

25. A scene text recognition device, characterized in that: include: An image acquisition unit, configured to acquire a plurality of target scene images, wherein the target scene images include a target object; An encoding unit, configured to sequentially input each target scene image into an encoder for feature extraction and encoding, thereby generating an image coding feature sequence; A Chinese character positioning unit is used to classify and locate the image coding feature sequence through a Chinese character region prediction network, and determine a first sequence containing Chinese character regions, a second sequence not containing Chinese character regions, coordinates of Chinese character regions, and coordinates of non-Chinese character regions; A decoding unit, configured to decode the classified image coding feature sequence through a first decoder and a Chinese character decoder to generate a character recognition result for each target scene image; A multi-frame synthesis unit, configured to input the character recognition results of the multiple frames into a multi-frame synthesis module to determine a final character recognition result; The decoding unit is configured to decode the classified image coding feature sequence through the first decoder and the Chinese character decoder to generate a character recognition result for each target scene image, specifically comprising: The first sequence is decoded by the Chinese character decoder to generate a first character recognition result in a Chinese character area, and the second sequence is decoded by the first decoder to generate a second character recognition result in a non-Chinese character area; The Chinese character area coordinates and the non-Chinese character area coordinates are combined, and the first character recognition result and the second character recognition result are spliced ​​to generate a character recognition result for each of the target scene images.

26. The device according to claim 25, characterized in that The device further comprises: A positioning unit, configured to input the target scene image into a trained target positioning detection network to obtain the region coordinates and region type of the target object; A correction unit is used to perform an affine transformation on the target scene image to generate a corrected target scene image according to the region coordinates and the region type.

27. The device according to claim 26, characterized in that The target positioning detection network includes: a second backbone network, configured to process the target scene image and generate a second feature map having a length and width proportional to the length and width of the target scene image; A region localization branch, configured to determine a region description vector of the target object based on the second feature map and output the region coordinates; The classification network branch is used to predict the region type according to the second feature map.

28. A storage medium, characterized in that The storage medium includes a stored program, wherein the program executes the scene text recognition method according to any one of claims 1 to 24.

29. A processor, characterized in that: The processor is configured to run a program, wherein the program, when running, executes the scene text recognition method according to any one of claims 1 to 24.

Images