Scene text information identification method and device, electronic equipment and medium

Abstract

The embodiment of the invention discloses a scene text information identification method and device, electronic equipment and a medium. A specific embodiment of the method comprises the following steps: preprocessing a scene image to be identified, and carrying out regional image segmentation on the preprocessed scene image to obtain a scene sub-image set; performing visual coding on the scene sub-image set, and performing foreground enhancement processing on a visual feature vector set obtained by coding to obtain a visual enhancement feature vector set; performing text recognition on the visual enhancement feature vector set, and performing text correction processing on the recognized text feature vector set to obtain a corrected text feature vector set; carrying out fusion processing on the corrected text feature vector set and the visual feature vector set to obtain a fused feature vector set; and inputting the fused feature vector set into a prediction layer to obtain text information. According to the embodiment, the text recognition accuracy in a complex scene and the universality and robustness of the model can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, electronic device, and medium for recognizing scene text information. Background Art

[0002] At present, the technology of recognizing text information in scene images is often used in various fields. For example, document recognition, license plate recognition and intelligent perception systems. Scene text information recognition can better understand scene information. For scene text information recognition, the usual method is: use a convolutional neural network model or an attention mechanism model to extract visual features of the scene image to be recognized, and recognize the extracted visual features to obtain text information.

[0003] However, the inventors have found that when the above method is used to identify scene text information, the following technical problems often occur:

[0004] First, since only visual features are extracted from the scene image to be recognized, the recognition accuracy of text information is low when the text information is occluded or blurred; in scenarios where there is no contextual information, convolutional neural network models or attention mechanism models are prone to over-correction problems, resulting in low text information recognition accuracy and low model versatility and robustness.

[0005] Second, since the convolutional neural network model or the attention mechanism model only focuses on text sequence or semantic recognition of visual features, it is easily affected by the semantic information of the text context, especially when there are irregularities such as text information obstruction and distortion in the scene image to be identified. It is easy to cause excessive correction of text information and generate a large amount of redundant information, resulting in low accuracy of text information recognition, large memory usage of the model, and low model versatility and robustness.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose scene text information recognition methods, devices, electronic devices, and media to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a method for identifying scene text information, including: acquiring a scene image to be identified that includes text information; preprocessing the scene image to be identified to obtain a preprocessed scene image; performing regional image segmentation on the preprocessed scene image to obtain a scene sub-image set; visually encoding each scene sub-image in the scene sub-image set through a visual feature encoder to generate a visual feature vector to obtain a visual feature vector set; performing foreground enhancement processing on the visual feature vector set through a foreground enhancement network to obtain a visual enhancement feature vector set; based on the visual enhancement feature vector set, executing the following text information identification steps: performing text recognition on the visual enhancement feature vector set according to a preset position embedding vector set to obtain a text feature vector set; performing text correction processing on the text feature vector set to obtain a corrected text feature vector set; fusing the corrected text feature vector set with the visual feature vector set to obtain a fused feature vector set; determining the number of times the text information identification step has been executed; in response to determining that the number of times the step has been executed exceeds a preset execution threshold, inputting the fused feature vector set into a prediction layer to obtain text information.

[0010] In a second aspect, some embodiments of the present disclosure provide a scene text information recognition device, comprising: an acquisition unit, configured to acquire a scene image to be recognized including text information; a preprocessing unit, configured to preprocess the scene image to be recognized to obtain a preprocessed scene image; a region image segmentation unit, configured to perform region image segmentation on the preprocessed scene image to obtain a scene sub-image set; a visual encoding unit, configured to perform visual encoding on each scene sub-image in the scene sub-image set through a visual feature encoder to generate a visual feature vector to obtain a visual feature vector set; a foreground enhancement processing unit, configured to perform visual encoding on the foreground enhancement network through a foreground enhancement network The visual feature vector set is subjected to foreground enhancement processing to obtain a visual enhancement feature vector set; an execution unit is configured to perform the following text information recognition steps based on the visual enhancement feature vector set: according to a preset position embedding vector set, text recognition is performed on the visual enhancement feature vector set to obtain a text feature vector set; text correction is performed on the text feature vector set to obtain a corrected text feature vector set; the corrected text feature vector set is fused with the visual feature vector set to obtain a fused feature vector set; the number of times the text information recognition step has been executed is determined; in response to determining that the number of times the step has been executed exceeds a preset execution threshold, the fused feature vector set is input into a prediction layer to obtain text information.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0013] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the scene text information recognition method of some embodiments of the present disclosure can improve the text recognition accuracy in complex scenes, as well as the versatility and robustness of the model by fully integrating the visual features and language features of the scene image. Specifically, the reason for the low accuracy of the related text information recognition is that: since only the visual features of the scene image to be recognized are extracted, the recognition accuracy of the text information is low when the text information is blocked or blurred; in the scene without context information association, the convolutional neural network model or the attention mechanism model is prone to over-correction, resulting in low text information recognition accuracy, low model versatility and robustness. Based on this, the scene text information recognition method of some embodiments of the present disclosure can first obtain a scene image to be recognized including text information. Here, the scene image to be recognized is used for subsequent text information recognition. Secondly, the scene image to be recognized is preprocessed to obtain a preprocessed scene image. Here, the preprocessing facilitates the conversion of the image to be recognized into a size that can be received by the subsequent visual feature encoder. Thirdly, the preprocessed scene image is subjected to regional image segmentation to obtain a scene sub-image set. Here, regional image segmentation recognizes the text information in the scene image to be identified at a finer granularity, which can improve the recognition accuracy and reduce the consumption of computer resources. Then, through the visual feature encoder, each scene sub-image in the above scene sub-image set is visually encoded to generate a visual feature vector and obtain a visual feature vector set. Here, visual encoding can extract the visual features of the scene image to be identified. Then, through the foreground enhancement network, the above visual feature vector set is subjected to foreground enhancement processing to obtain a visual enhancement feature vector set. Here, through the foreground enhancement network, the weight of the area that does not include text information in the scene image to be identified can be automatically reduced, and the weight of the area containing text information can be increased, which is convenient for improving the accuracy of identifying text information. Finally, based on the visual enhancement feature vector set, the following text information recognition steps are performed: the first step is to perform text recognition on the visual enhancement feature vector set according to the preset position embedding vector set to obtain a text feature vector set. Here, by identifying text features through visual features, the location information of the text information can be accurately identified, so as to subsequently identify the text information and solve the problem of attention drift. The second step is to perform text correction processing on the text feature vector set to obtain a corrected text feature vector set. Here, the text correction processing can correct the text information corresponding to the recognition result character by character, which can improve the accuracy of text recognition. In the third step, the corrected text feature vector set is fused with the visual feature vector set to obtain a fused feature vector set.Here, by combining the iterative fusion of visual and language information, a text feature vector containing visual information is obtained, which can avoid the bias in text information recognition to a certain extent, and then obtain a higher quality fused feature vector. In context-independent scenes, the problem of overcorrection can be avoided to a certain extent. The fourth step is to determine the number of times the text information recognition step has been executed. Here, the visual features and language features can be fully fused to obtain a higher quality fused feature vector. The fifth step is to input the fused feature vector set into the prediction layer in response to determining that the number of executions exceeds the preset execution threshold to obtain text information. Here, text recognition is performed on the fused feature vector set including visual features and language features, which can improve the accuracy of text recognition. It can be concluded that the scene text information recognition method can improve the accuracy of text recognition in complex scenes, as well as the versatility and robustness of the model by fully integrating the visual features and language features of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flow chart of some embodiments of the scene text information recognition method according to the present disclosure;

[0016] Figure 2 is a schematic diagram of an application scenario of the scene text information recognition method according to the present disclosure;

[0017] Figure 3 is a schematic diagram of the structure of some embodiments of the scene text information recognition device according to the present disclosure;

[0018] Figure 4 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] Figure 1 The process 100 of some embodiments of the scene text information recognition method according to the present disclosure is shown. The scene text information recognition method comprises the following steps:

[0026] Step 101: Acquire a scene image to be recognized including text information.

[0027] In some embodiments, the execution subject (e.g., electronic device) of the scene text information recognition method can obtain the scene image to be recognized including text information through a wired connection or a wireless connection. The text information can be text information located in the scene image to be recognized and waiting to be recognized. For example, the scene image to be recognized can be a license plate image. Here, the acquisition can be the scene image to be recognized including text information crawled from the Internet.

[0028] Step 102: preprocess the scene image to be recognized to obtain a preprocessed scene image.

[0029] In some embodiments, the execution subject may preprocess the scene image to be identified to obtain a preprocessed scene image. The preprocessing may include but is not limited to at least one of the following: converting the scene image to be identified into a scene image to be identified of a fixed size, converting the scene image to be identified into a three-channel image, and normalizing the RGB value of the scene image to be identified. The fixed size may be the size of the scene image to be identified specified by the input layer of the visual feature encoder.

[0030] Step 103: performing regional image segmentation on the preprocessed scene image to obtain a scene sub-image set.

[0031] In some embodiments, the execution entity may perform regional image segmentation on the preprocessed scene image to obtain a scene sub-image set. The scene sub-images in the scene sub-image set may be 3*3 pixel regional images of the preprocessed scene image. In practice, the execution entity may segment the preprocessed scene image into a scene sub-image set of 3*3 pixel regions. It should be noted that when the length and width of the preprocessed scene image are not integer multiples of the length and width of the regional image, the preprocessed scene image is subjected to image edge filling processing to obtain a filled scene image. The length and width of the filled scene image are integer multiples of the length and width of the regional image. The image edge filling may be performed using pixels with R=B=G=0 for edge filling.

[0032] Step 104 , visually encode each scene sub-image in the scene sub-image set by using a visual feature encoder to generate a visual feature vector, thereby obtaining a visual feature vector set.

[0033] In some embodiments, the execution subject may perform visual encoding on each scene sub-image in the scene sub-image set through a visual feature encoder to generate a visual feature vector and obtain a visual feature vector set. The visual feature vector may be a vector representing the visual information of the scene sub-image. The visual feature encoder may be a Transformer-based encoder.

[0034] In some optional implementations of some embodiments, the visual feature encoder includes: a visual feature embedding network, a multi-head attention mechanism layer and a first multi-layer perceptron layer; and the visual feature encoder visually encodes each scene sub-image in the scene sub-image set to generate a visual feature vector to obtain a visual feature vector set, which may include the following steps:

[0035] In the first step, the scene sub-image set is input into the visual feature embedding network to obtain a first visual feature vector set. The first visual feature vector in the first visual feature vector set may be a feature vector representing the visual information of the scene sub-image. The visual feature embedding network may be a convolutional neural network that extracts the visual feature vector of each scene sub-image in the scene sub-image set.

[0036] In the second step, based on the first visual feature vector set, the following visual feature extraction steps are performed:

[0037] In a first sub-step, the first visual feature vector set is input into the multi-head attention mechanism layer to obtain a second visual feature vector set. The multi-head attention mechanism layer may be a multi-head attention mechanism layer including layer normalization and residual connection.

[0038] The second sub-step is to input the second visual feature vector set into the first multi-layer perceptron layer to obtain a third visual feature vector set. The first multi-layer perceptron layer is a multi-layer perceptron layer including layer normalization and residual connection.

[0039] The third sub-step is to determine the number of times the above visual feature extraction step has been executed.

[0040] In a fourth sub-step, in response to determining that the number of extraction executions exceeds a preset extraction execution threshold, the third visual feature vector set is determined as the visual feature vector set. The preset extraction execution threshold may be a pre-set threshold for determining whether repeated extractions of visual features are completed. For example, the preset extraction execution threshold may be 12.

[0041] Optionally, after performing the second step, the method may further include the following steps:

[0042] In response to determining that the number of extracted executions does not exceed the preset extraction execution threshold, the third visual feature vector set is used as the first visual feature vector set, the sum of the number of extracted executions and the preset threshold is determined as the number of extracted executions, and the visual feature extraction step is performed again. Here, the preset threshold may be 1.

[0043] In some optional implementations of some embodiments, the visual feature embedding network includes: a first visual feature extraction network layer and a second visual feature extraction network layer; and the step of inputting the scene sub-image set into the visual feature embedding network to obtain a first visual feature vector set includes:

[0044] The first step is to input the above scene sub-image set into the above first visual feature extraction network layer to obtain a fourth visual feature vector set. The above first visual feature extraction network includes: a convolution layer with a convolution kernel of 3*3 and a step size of 2, a batch normalization layer (Batch Normalization) and a ReLU (Rectified Linear Unit, linear rectifier function) activation function. The dimension of the visual feature vector in the above fourth visual feature vector set is half the dimension of the visual feature vector corresponding to the above scene sub-image.

[0045] The second step is to input the fourth visual feature vector set into the second visual feature extraction network layer to obtain the fifth visual feature vector set as the first visual feature vector set. The second visual feature extraction network includes: a convolution layer with a convolution kernel of 3*3 and a step size of 2, a batch normalization layer and a ReLU activation function. The dimension of the fifth visual feature vector in the fifth visual feature vector set is twice the dimension of the fourth visual feature vector, and is also a visual feature vector with the same dimension as the first visual feature vector.

[0046] Step 105, performing foreground enhancement processing on the visual feature vector set through a foreground enhancement network to obtain a visual enhancement feature vector set.

[0047] In some embodiments, the execution subject may perform foreground enhancement processing on the visual feature vector set through a foreground enhancement network to obtain a visual enhancement feature vector set. The visual enhancement feature vectors in the visual enhancement feature vector set may be feature vectors corresponding to different weight values ​​of the visual feature vectors. The foreground enhancement network may be an attention mechanism neural network.

[0048] In some optional implementations of some embodiments, the foreground enhancement network includes: an average pooling layer (Average Pooling) and a second multi-layer perceptron layer; and the foreground enhancement processing of the visual feature vector set through the foreground enhancement network to obtain the visual enhancement feature vector set may include the following steps:

[0049] In the first step, the visual feature vector set is input into the average pooling layer to obtain a visual target feature vector, wherein the visual target feature vector may be a one-dimensional visual feature vector.

[0050] The second step is to input the visual target feature vector into the second multi-layer perceptron layer to obtain a weight vector for the visual target feature vector, wherein the weight vector may be a vector composed of weight values ​​of each visual feature vector in the visual feature vector set.

[0051] In the third step, for each visual feature vector in the visual feature vector set, weighting is performed on the visual feature vector and the weight value corresponding to the visual feature vector in the weight vector to generate a visual weighted feature vector.

[0052] The fourth step is to determine the obtained visual weighted feature vector set as the visual enhancement feature vector set.

[0053] Step 106, based on the visual enhancement feature vector set, perform the following text information recognition steps:

[0054] Step 1061, performing text recognition on the visual enhancement feature vector set according to the preset position embedding vector set to obtain a text feature vector set.

[0055] In some embodiments, the execution subject may perform text recognition on the visual enhancement feature vector set according to the preset position embedding vector set to obtain a text feature vector set. The text feature vector in the text feature vector set may be a feature vector that characterizes the text information contained in the scene image to be identified. The preset position embedding vector in the preset position embedding vector set may be a vector that characterizes the position information of the text. The preset position embedding vector in the preset position embedding vector set is a position embedding vector having the same dimension as the visual enhancement feature vector.

[0056] In some optional implementations of some embodiments, the above-mentioned embedding vector set according to the preset position, performing text recognition on the visual enhancement feature vector set to obtain the text feature vector set may include the following steps:

[0057] In the first step, for each visual enhancement feature vector in the above visual enhancement feature vector set, the following text recognition steps are performed:

[0058] In a first sub-step, a first linear transformation is performed on the visual enhancement feature vector to obtain a key matrix. The key matrix may be a matrix representing the correlation between the preset position embedding vector and the visual enhancement feature vector. The first linear transformation may be the product of the visual enhancement feature vector and a first preset linear parameter matrix. The first preset linear parameter matrix may be a trainable parameter matrix.

[0059] The second sub-step is to perform a second linear transformation on the visual enhancement feature vector to obtain a value matrix. The value matrix may be a matrix representing feature information of the visual enhancement feature vector. The second linear transformation may be the product of the visual enhancement feature vector and a second preset linear parameter matrix. The second preset linear parameter matrix may be a trainable parameter matrix.

[0060] The third sub-step is to perform a third linear transformation on the preset position embedding vector corresponding to the visual enhancement feature vector in the preset position embedding vector set to obtain a query matrix. The preset position embedding vector is a position embedding vector having the same dimension as the visual enhancement feature vector. The query matrix may be a matrix representing feature information of the preset position embedding vector. The third linear transformation may be the product of the preset position embedding vector and a third preset linear parameter matrix. The third preset linear parameter matrix may be a trainable parameter matrix.

[0061] The fourth sub-step is to perform dot multiplication and scaling on the query matrix and the transposed matrix of the key matrix to obtain an initial set of attention weights. The scaling may be performed by dividing the result of the dot multiplication by the arithmetic square root of the corresponding dimension of the key matrix.

[0062] The fifth sub-step is to normalize the initial attention weight set to obtain the attention weight set. The normalization process may be to input the initial attention weight set into a softmax function.

[0063] The sixth sub-step is to perform weighted summation on the above attention weight set and the above value matrix to obtain a text feature vector.

[0064] Step 1062, performing text correction processing on the text feature vector set to obtain a corrected text feature vector set.

[0065] In some embodiments, the execution subject may perform text correction processing on the text feature vector set to obtain a corrected text feature vector set. The corrected text feature vector in the corrected text feature vector set may be a feature vector obtained by correcting the characters in the text information and representing the corrected text information.

[0066] As an example, the execution subject may perform context modeling on the text feature vector set through a pre-trained language model to obtain a corrected text feature vector set. The pre-trained language model may be a single-layer Transformer neural network. The single-layer Transformer neural network may be a single-layer Transformer with a Mask mechanism or a single-layer Transformer without a Mask.

[0067] In some optional implementations of some embodiments, the above-mentioned text correction processing on the text feature vector set to obtain the corrected text feature vector set may include the following steps:

[0068] The first step is to perform text feature splicing on the text feature vector set to obtain a spliced ​​text feature vector. The splicing length of the spliced ​​text feature vector may be the number of text feature vectors included in the text feature vector set. The text feature splicing may be splicing based on the position information corresponding to the text feature vector set. The position information may be the position information of the text corresponding to the text feature vector in the scene image to be identified.

[0069] In the second step, the above-mentioned spliced ​​text feature vector is input into the first BiGRU network layer included in the correction model to obtain the first text output state feature vector and the first text hidden state feature vector. Among them, the number of word vectors included in the above-mentioned text feature vector set can be the number of characters of the text information. The above-mentioned correction model may include: multiple BiGRU (Bidirectional Gated Recurrent Unit, bidirectional gated recurrent unit) network layers, learning perceptrons, multiple attention networks, fusion network layers and GRU (Gated Recurrent Unit, gated recurrent unit) network layers. The above-mentioned multiple BiGRU network layers may be 2 BiGRU network layers. The above-mentioned multiple attention networks may be 2 attention networks. The above-mentioned first text output state feature vector may be a feature vector obtained by splicing the two text hidden state feature vectors output by the forward GRU and the backward GRU in the BiGRU network layer. The above-mentioned first text hidden state feature vector may be an intermediate state feature vector obtained by the first BiGRU network layer processing the above-mentioned word vector sequence, a feature vector representing the feature information in the above-mentioned word vector sequence.

[0070] The third step is to input the first text output state feature vector into the learning perceptron to obtain a text perception feature vector. The text perception feature vector may be a feature vector obtained by classifying the output state feature vector. The learning perceptron may be an LR learning perceptron.

[0071] The fourth step is to input the above text perception feature vector and the above first text hidden state feature vector into the second BiGRU network layer to obtain the second text output state feature vector and the second text hidden state feature vector.

[0072] Step 5: Based on the second text output state feature vector and the second text hidden state feature vector, perform the following second text correction step:

[0073] In the first sub-step, feature concatenation is performed on the second text output state feature vector and the second text hidden state feature vector to obtain a text concatenation feature vector.

[0074] The second sub-step is to input the text concatenation feature vector into the first attention network to obtain the first text weight feature vector. The first attention network may include: a linear layer, an activation function, an L transformer and a classifier. The L transformer may be a transformer that expands the dimension of the feature vector output from the previous level and reduces the dimension of the feature vector output from the last layer of the attention network to the fusion network. The classifier may be a Softmax classifier. The first text weight feature vector may be a feature vector obtained by multiplying the first weight matrix and the text concatenation feature vector. The first weight matrix may be a weight matrix learned by the first attention network.

[0075] The third sub-step is to input the first text weight feature vector into the second attention network to obtain a second text weight feature vector. The second text weight feature vector may be a feature vector obtained by multiplying the second weight matrix and the first text weight feature vector. The second weight matrix may be a weight matrix learned by the second attention network.

[0076] The fourth sub-step is to input the first text weight feature vector and the second text weight feature vector into the fusion network layer to obtain the fused text weight feature vector.

[0077] The fifth sub-step is to perform vector product processing on the fused text weight feature vector and the second text output state feature vector to obtain a text semantic feature vector. The text semantic feature vector may be a feature vector of a character to be recognized in this cycle of the second text correction step.

[0078] The sixth sub-step is to input the text semantic feature vector into the GRU network layer to obtain the text semantic output state feature vector and the text semantic hidden state feature vector. Among them, the text semantic hidden state feature vector can be a feature vector calculated from the hidden state of the previous time step and the input of the current time step in the GRU network. The text semantic output state feature vector can be a feature vector calculated from the hidden state of the current time step in each time step. The hidden state of the GRU network layer can capture the long-term time dependency in the word vector sequence. The output state of the GRU network layer can predict the word vector. The text semantic output state feature vector and the text semantic hidden state feature vector can be the feature vectors of the characters that need to be recognized in the next cycle. The GRU network layer is a network layer that predicts the next character of the character in this cycle of the second text correction step.

[0079] The seventh sub-step is to determine the number of corrected executions of the second text correction step.

[0080] In an eighth sub-step, in response to determining that the number of correction executions exceeds the number of text feature vectors included in the text feature vector set, the text semantic output state feature vector and the text semantic hidden state feature vector are determined as corrected text feature vectors.

[0081] In step 6, in response to determining that the corrected execution times do not exceed the number of text feature vectors included in the text feature vector set, the text semantic output state feature vector and the text semantic hidden state feature vector are used as the second text output state feature vector and the second text hidden state feature vector, the sum of the corrected execution times and the preset threshold is determined as the corrected execution times, and the second text correction step is performed again. Wherein, when the character at the current recognition position is empty, it is indicated by a null identifier.

[0082] The above-mentioned first to sixth steps and their related contents, as an inventive point of an embodiment of the present disclosure, solve the second technical problem mentioned in the background technology: "Since the convolutional neural network model or the attention mechanism model only focuses on the text sequence or semantic recognition of visual features, it is easily affected by the text context semantic information, especially when there are irregularities such as text information shielding and distortion in the scene image to be identified, it is easy to cause over-correction of text information and generate a large amount of redundant information, resulting in low accuracy of text information recognition, large memory occupied by the model, and low versatility and robustness of the model." The factors that lead to low accuracy of text information recognition, large memory occupied by the model, and low versatility and robustness of the model are often as follows: Since the convolutional neural network model or the attention mechanism model only focuses on the text sequence or semantic recognition of visual features, it is easily affected by the text context semantic information, especially when there are irregularities such as text information shielding and distortion in the scene image to be identified, it is easy to cause over-correction of text information and generate a large amount of redundant information. If the above factors are solved, the effect of improving the accuracy of text information recognition, reducing the memory occupied by the model, and improving the versatility and robustness of the model can be achieved. In order to achieve this effect, the present disclosure splices the text feature vector set and inputs it into the multiple BiGRU network layers included in the correction model, which can improve the storage capacity of the text feature vector, retain the context information of adjacent characters in the text information, and extract the long-dependent temporal features of the text information. The characters in the second text output state feature vector and the second text hidden state feature vector output by the multiple BiGRU network layers are subjected to cyclic character correction processing based on multiple attention networks, fusion network layers and GRU network layers, which can fully extract the context information between adjacent characters, avoid the situation of individual character recognition errors to a certain extent, improve the accuracy of text recognition, avoid the problem of over-correction of the model, reduce the generation of redundant information, and improve the versatility and robustness of the model.

[0083] Step 1063, fusing the corrected text feature vector set with the visual feature vector set to obtain a fused feature vector set.

[0084] In some embodiments, the execution entity may fuse the corrected text feature vector set with the visual feature vector set to obtain a fused feature vector set.

[0085] As an example, the above execution entity can input the corrected text feature vector set and the visual feature vector set into a cross attention mechanism (Cross Attention) layer to obtain a fused feature vector set.

[0086] Step 1064, determining the number of times the text information recognition step has been executed.

[0087] In some embodiments, the execution entity may determine the number of times the text information recognition step has been executed.

[0088] Step 1065 , in response to determining that the number of executions exceeds a preset execution threshold, the fused feature vector set is input into a prediction layer to obtain text information.

[0089] In some embodiments, the execution subject may, in response to determining that the number of executions exceeds a preset execution threshold, input the fused feature vector set into the prediction layer to obtain text information. The preset execution threshold may be a pre-set threshold for determining whether the text information recognition step is trained. For example, the preset execution threshold may be 7. The prediction layer may be a fully connected layer. Here, Figure 2 A schematic diagram showing an application scenario of the scene text information recognition method.

[0090] Optionally, after step 1065, the execution subject may further perform the following steps:

[0091] In response to determining that the number of executions does not exceed the preset execution threshold, the fused feature vector set is used as the visual enhancement feature vector set, the sum of the number of executions and the preset threshold is determined as the number of executions, and the text information recognition step is performed again. Here, the preset threshold may be 1.

[0092] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the scene text information recognition method of some embodiments of the present disclosure can improve the text recognition accuracy in complex scenes, as well as the versatility and robustness of the model by fully integrating the visual features and language features of the scene image. Specifically, the reason for the low accuracy of the related text information recognition is that: since only the visual features of the scene image to be recognized are extracted, the recognition accuracy of the text information is low when the text information is blocked or blurred; in the scene without context information association, the convolutional neural network model or the attention mechanism model is prone to over-correction, resulting in low text information recognition accuracy, low model versatility and robustness. Based on this, the scene text information recognition method of some embodiments of the present disclosure can first obtain a scene image to be recognized including text information. Here, the scene image to be recognized is used for subsequent text information recognition. Secondly, the scene image to be recognized is preprocessed to obtain a preprocessed scene image. Here, the preprocessing facilitates the conversion of the image to be recognized into a size that can be received by the subsequent visual feature encoder. Thirdly, the preprocessed scene image is subjected to regional image segmentation to obtain a scene sub-image set. Here, regional image segmentation recognizes the text information in the scene image to be identified at a finer granularity, which can improve the recognition accuracy and reduce the consumption of computer resources. Then, through the visual feature encoder, each scene sub-image in the above scene sub-image set is visually encoded to generate a visual feature vector and obtain a visual feature vector set. Here, visual encoding can extract the visual features of the scene image to be identified. Then, through the foreground enhancement network, the above visual feature vector set is subjected to foreground enhancement processing to obtain a visual enhancement feature vector set. Here, through the foreground enhancement network, the weight of the area that does not include text information in the scene image to be identified can be automatically reduced, and the weight of the area containing text information can be increased, which is convenient for improving the accuracy of identifying text information. Finally, based on the visual enhancement feature vector set, the following text information recognition steps are performed: the first step is to perform text recognition on the visual enhancement feature vector set according to the preset position embedding vector set to obtain a text feature vector set. Here, by identifying text features through visual features, the location information of the text information can be accurately identified, so as to subsequently identify the text information and solve the problem of attention drift. The second step is to perform text correction processing on the text feature vector set to obtain a corrected text feature vector set. Here, the text correction processing can correct the text information corresponding to the recognition result character by character, which can improve the accuracy of text recognition. In the third step, the corrected text feature vector set is fused with the visual feature vector set to obtain a fused feature vector set.Here, by combining the iterative fusion of visual and language information, a text feature vector containing visual information is obtained, which can avoid the bias in text information recognition to a certain extent, and then obtain a higher quality fused feature vector. In context-independent scenes, the problem of overcorrection can be avoided to a certain extent. The fourth step is to determine the number of times the text information recognition step has been executed. Here, the visual features and language features can be fully fused to obtain a higher quality fused feature vector. The fifth step is to input the fused feature vector set into the prediction layer in response to determining that the number of executions exceeds the preset execution threshold to obtain text information. Here, text recognition is performed on the fused feature vector set including visual features and language features, which can improve the accuracy of text recognition. It can be concluded that the scene text information recognition method can improve the accuracy of text recognition in complex scenes, as well as the versatility and robustness of the model by fully integrating the visual features and language features of the image.

[0093] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a scene text information recognition device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the scene text information recognition device can be specifically applied to various electronic devices.

[0094] like Figure 3As shown, a scene text information recognition device 300 includes: an acquisition unit 301, a preprocessing unit 302, a regional image segmentation unit 303, a visual encoding unit 304, a foreground enhancement processing unit 305 and an execution unit 306. Among them, the acquisition unit 301 is configured to: acquire a scene image to be recognized including text information. The preprocessing unit 302 is configured to: preprocess the above-mentioned scene image to be recognized to obtain a preprocessed scene image. The regional image segmentation unit 303 is configured to: perform regional image segmentation on the above-mentioned preprocessed scene image to obtain a scene sub-image set. The visual encoding unit 304 is configured to: perform visual encoding on each scene sub-image in the above-mentioned scene sub-image set through a visual feature encoder to generate a visual feature vector to obtain a visual feature vector set. The foreground enhancement processing unit 305 is configured to: perform foreground enhancement processing on the above-mentioned visual feature vector set through a foreground enhancement network to obtain a visual enhancement feature vector set. The execution unit 306 is configured to: execute the following text information recognition steps based on the visual enhancement feature vector set: perform text recognition on the visual enhancement feature vector set according to the preset position embedding vector set to obtain a text feature vector set; perform text correction processing on the text feature vector set to obtain a corrected text feature vector set; fuse the corrected text feature vector set with the visual feature vector set to obtain a fused feature vector set; determine the number of times the text information recognition step has been executed; in response to determining that the number of times the step has been executed exceeds a preset execution threshold, input the fused feature vector set into the prediction layer to obtain text information.

[0095] It can be understood that the units recorded in the scene text information recognition device 300 are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the scene text information recognition device 300 and the units included therein, and will not be described in detail here.

[0096] Reference below Figure 4 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0097] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 304.

[0098] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0099] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0100] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0101] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0102] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a scene image to be identified including text information; preprocesses the scene image to be identified to obtain a preprocessed scene image; performs regional image segmentation on the preprocessed scene image to obtain a scene sub-image set; visually encodes each scene sub-image in the scene sub-image set through a visual feature encoder to generate a visual feature vector to obtain a visual feature vector set; performs foreground enhancement processing on the visual feature vector set through a foreground enhancement network to obtain a visual enhancement feature vector set; based on the visual enhancement feature vector set, performs the following text information identification steps: according to a preset position embedding vector set, performs text recognition on the visual enhancement feature vector set to obtain a text feature vector set; performs text correction processing on the text feature vector set to obtain a corrected text feature vector set; fuses the corrected text feature vector set with the visual feature vector set to obtain a fused feature vector set; determines the number of times the text information identification step has been executed; in response to determining that the number of times the step has been executed exceeds a preset execution threshold, inputs the fused feature vector set into a prediction layer to obtain text information.

[0103] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor including an acquisition unit, a preprocessing unit, a regional image segmentation unit, a visual encoding unit, a foreground enhancement processing unit and an execution unit. The names of these units do not constitute a limitation on the units themselves in certain cases, for example, the acquisition unit may also be described as a "unit for acquiring a scene image to be identified including text information".

[0106] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0107] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for recognizing scene text information, comprising: Acquire a scene image to be recognized including text information; Preprocessing the scene image to be identified to obtain a preprocessed scene image; Performing regional image segmentation on the preprocessed scene image to obtain a scene sub-image set; Performing visual encoding on each scene sub-image in the scene sub-image set by means of a visual feature encoder to generate a visual feature vector, thereby obtaining a visual feature vector set; Performing foreground enhancement processing on the visual feature vector set through a foreground enhancement network to obtain a visual enhancement feature vector set; Based on the visual enhancement feature vector set, the following text information recognition steps are performed: According to the preset position embedding vector set, text recognition is performed on the visual enhancement feature vector set to obtain a text feature vector set; Performing text correction processing on the text feature vector set to obtain a corrected text feature vector set; The corrected text feature vector set is fused with the visual feature vector set to obtain a fused feature vector set; Determining the number of times the text information recognition step has been performed; In response to determining that the number of executions exceeds a preset execution threshold, the fused feature vector set is input into a prediction layer to obtain text information.

2. The method according to claim 1, wherein: The method further comprises: In response to determining that the number of executions does not exceed the preset execution threshold, the fused feature vector set is used as the visual enhancement feature vector set, the sum of the number of executions and the preset threshold is determined as the number of executions, and the text information recognition step is performed again.

3. The method according to claim 1, wherein: The visual feature encoder comprises: a visual feature embedding network, a multi-head attention mechanism layer and a first multi-layer perceptron layer; and The visual feature encoder is used to visually encode each scene sub-image in the scene sub-image set to generate a visual feature vector, thereby obtaining a visual feature vector set, including: Inputting the scene sub-image set into the visual feature embedding network to obtain a first visual feature vector set; Based on the first visual feature vector set, the following visual feature extraction steps are performed: Inputting the first visual feature vector set into the multi-head attention mechanism layer to obtain a second visual feature vector set, wherein the multi-head attention mechanism layer is a multi-head attention mechanism layer including layer normalization and residual connection; Inputting the second visual feature vector set into the first multi-layer perceptron layer to obtain a third visual feature vector set, wherein the first multi-layer perceptron layer is a multi-layer perceptron layer including layer normalization and residual connection; Determining the number of times the visual feature extraction step has been performed; In response to determining that the number of extraction executions exceeds a preset extraction execution threshold, the third visual feature vector set is determined as the visual feature vector set.

4. The method according to claim 3, wherein: The method further comprises: In response to determining that the number of extracted executions does not exceed the preset extraction execution threshold, the third visual feature vector set is used as the first visual feature vector set, the sum of the number of extracted executions and the preset threshold is determined as the number of extracted executions, and the visual feature extraction step is performed again.

5. The method according to claim 3, wherein: The visual feature embedding network comprises: a first visual feature extraction network layer and a second visual feature extraction network layer; and The step of inputting the scene sub-image set into the visual feature embedding network to obtain a first visual feature vector set includes: Inputting the scene sub-image set into the first visual feature extraction network layer to obtain a fourth visual feature vector set; The fourth visual feature vector set is input into the second visual feature extraction network layer to obtain a fifth visual feature vector set as the first visual feature vector set.

6. The method according to claim 1, wherein: The foreground enhancement network includes: an average pooling layer and a second multi-layer perceptron layer; and The foreground enhancement network is used to perform foreground enhancement processing on the visual feature vector set to obtain a visual enhancement feature vector set, including: Inputting the visual feature vector set into the average pooling layer to obtain a visual target feature vector, wherein the visual target feature vector is a one-dimensional visual feature vector; Inputting the visual target feature vector into the second multi-layer perceptron layer to obtain a weight vector for the visual target feature vector, wherein the weight vector is a vector composed of weight values ​​of each visual feature vector in the visual feature vector set; For each visual feature vector in the visual feature vector set, weighting the visual feature vector and a weight value corresponding to the visual feature vector in the weight vector to generate a visual weighted feature vector; The obtained visual weighted feature vector set is determined as a visual enhancement feature vector set.

7. The method according to claim 1, wherein: The embedding vector set at the preset position and performing text recognition on the visual enhancement feature vector set to obtain the text feature vector set include: For each visual enhancement feature vector in the visual enhancement feature vector set, the following text recognition steps are performed: Performing a first linear transformation on the visual enhancement feature vector to obtain a key matrix; Performing a second linear transformation on the visual enhancement feature vector to obtain a value matrix; Performing a third linear change on the preset position embedding vector in the preset position embedding vector set corresponding to the visual enhancement feature vector to obtain a query matrix, wherein the preset position embedding vector in the preset position embedding vector set is a position embedding vector having the same dimension as the visual enhancement feature vector; Performing point multiplication and scaling processing on the query matrix and the transposed matrix of the key matrix to obtain an initial attention weight set; Normalizing the initial attention weight set to obtain an attention weight set; The attention weight set and the value matrix are weightedly summed to obtain a text feature vector.

8. A scene text information recognition device, comprising: An acquisition unit, configured to acquire a scene image to be recognized including text information; A preprocessing unit, configured to preprocess the scene image to be identified to obtain a preprocessed scene image; A regional image segmentation unit is configured to perform regional image segmentation on the preprocessed scene image to obtain a scene sub-image set; A visual encoding unit is configured to perform visual encoding on each scene sub-image in the scene sub-image set through a visual feature encoder to generate a visual feature vector to obtain a visual feature vector set; a foreground enhancement processing unit configured to perform foreground enhancement processing on the visual feature vector set through a foreground enhancement network to obtain a visual enhancement feature vector set; The execution unit is configured to perform the following text information recognition steps based on the visual enhancement feature vector set: perform text recognition on the visual enhancement feature vector set according to the preset position embedding vector set to obtain a text feature vector set; perform text correction processing on the text feature vector set to obtain a corrected text feature vector set; perform fusion processing on the corrected text feature vector set and the visual feature vector set to obtain a fused feature vector set; Determine the number of times the text information recognition step has been executed; in response to determining that the number of times the step has been executed exceeds a preset execution threshold, input the fused feature vector set into a prediction layer to obtain text information.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images