A Vietnamese scene text recognition method based on improved DAN

Abstract

The present invention relates to the field of text recognition, and in particular to a Vietnamese scene text recognition method based on an improved DAN, comprising: using ResNet as an encoder to extract an original feature map from an input image; inputting the original feature map into a CAM to obtain an attention map, specifically, inputting the original feature map into an FPN network to obtain a fused feature map; inputting the fused feature map into a VSFM module to obtain visual and sequence fusion features; inputting the visual and sequence fusion features into an FCN network to obtain an attention map; and inputting the original feature map, the attention map and the VSFM intermediate features into an enhanced decoder for decoding to obtain a recognition result of a scene map. The method effectively alleviates attention drift, recognizes diacritical marks more accurately, and can effectively distinguish similar characters, so that the recognition result is better.

Description

Technical Field

[0001] The present invention relates to the field of text recognition, and in particular to a Vietnamese scene text recognition method based on improved DAN. Background Art

[0002] Natural scene text recognition has attracted the attention of many researchers. Its wide applications, such as autonomous driving, visual assistance, sign recognition, etc., further highlight its application value. With the development of deep learning, natural scene text recognition has made great progress, and many corresponding methods have emerged. However, most of the existing methods are based on the research of non-tonal languages ​​such as English or French, and there is little research on scene text recognition of some tonal languages ​​such as Vietnamese.

[0003] Vietnamese is a tonal language. Vietnamese characters are composed of nine accents or diacritics on the basis of Latin letters. Four of them are used to add vowels, and the other five represent the tones of Vietnamese. These five tone symbols determine the meaning of each word. The unique composition of Vietnamese characters makes the recognition of Vietnamese characters in natural scenes have the following problems compared with the existing recognition technology mainly for English:

[0004] 1. The number of Vietnamese characters has increased dramatically, and the differences between characters are small, and some characters only have slight differences in diacritical marks;

[0005] 2. Compared with Latin characters, diacritics are smaller in shape, making it more difficult to accurately recognize them. Misrecognition of diacritics will lead to errors in the entire recognition result.

[0006] 3. The presence of diacritics is more likely to cause attention drift problems, resulting in incorrect recognition results. Summary of the invention

[0007] The purpose of the present invention is to provide a Vietnamese scene text recognition method based on improved DAN, which aims to effectively alleviate attention drift, recognize diacritical marks more accurately, and effectively distinguish similar characters, resulting in better recognition results.

[0008] To achieve the above object, the present invention provides a Vietnamese scene text recognition method based on improved DAN, including using ResNet as an encoder to extract an original feature map from an input image;

[0009] The original feature map is input into the CAM to obtain an attention map, specifically, the original feature map is input into the FPN network to obtain a fused feature map; the fused feature map is input into the VSFM module to obtain visual and sequence fusion features; the visual and sequence fusion features are input into the FCN network to obtain an attention map;

[0010] The original feature map, the attention map and the VSFM intermediate features are input into the enhanced decoder for decoding to obtain the recognition result of the scene graph.

[0011] The specific steps of inputting the fusion feature map into the VSFM module to obtain the visual and sequence fusion features are: inputting the fusion feature map into the Bi-GRU with the width as the time step to perform horizontal sequence feature modeling to obtain horizontal sequence modeling features;

[0012] The fused feature map is input into Bi-GRU with the height as the time step to perform vertical sequence feature modeling, and the vertical sequence modeling features are obtained;

[0013] The visual and sequence fusion features are obtained by adding the elements of the horizontal sequence modeling features, the vertical sequence modeling features and the fusion feature map at corresponding positions.

[0014] The specific method of inputting the fused feature map into Bi-GRU to perform horizontal sequence feature modeling with the width as the time step is: modeling from left to right and from right to left in the horizontal direction.

[0015] The fusion feature map is input into Bi-GRU with the height as the time step to perform vertical sequence feature modeling. The specific method of obtaining the vertical sequence modeling features is: modeling from top to bottom and from bottom to top in the vertical direction.

[0016] The specific steps of inputting the original feature map, the attention map and the VSFM intermediate features into the enhanced decoder for decoding to obtain the recognition result of the scene map are:

[0017] S301 multiplies the original feature map and the attention map at each moment to obtain a context vector, concatenates it with the embedding vector of the decoding result at the previous moment, and then inputs it into the GRU together with the hidden information at the previous moment to obtain the hidden information at the current moment;

[0018] S302 concatenates the hidden information and the VSFM intermediate features in the channel dimension and inputs them into the classifier for classification to obtain the recognition result at the current moment;

[0019] S303 repeats S301 to S302. When the character with the highest probability predicted is the terminator, the prediction is stopped, and the recognition results at each moment are combined to obtain the final recognition result.

[0020] Among them, the VSFM intermediate feature is a vertical sequence modeling feature obtained by performing vertical sequence feature modeling on the fused feature map in Bi-GRU, and is obtained by splicing the fused feature map and the vertical sequence modeling feature in the channel dimension.

[0021] The invention discloses a Vietnamese scene text recognition method based on improved DAN, comprising: using ResNet as an encoder to extract an original feature map from an input image; inputting the original feature map into a CAM to obtain an attention map, specifically, inputting the original feature map into an FPN network to obtain a fused feature map; inputting the fused feature map into a VSFM module to obtain visual and sequence fusion features; inputting the visual and sequence fusion features into an FCN network to obtain an attention map; inputting the original feature map, the attention map and the VSFM intermediate features into an enhanced decoder for decoding to obtain a recognition result of a scene map. The method effectively alleviates attention drift, recognizes diacritical marks more accurately, and can effectively distinguish similar characters, so that the recognition result is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a schematic diagram of the algorithm structure of the Vietnamese scene text recognition method based on the improved DAN provided by the present invention.

[0024] Figure 2 It is a comparison diagram of the attention maps generated by the Vietnamese scene text recognition method based on the improved DAN provided by the present invention and the methods of others.

[0025] Figure 3 This is an example of the comparison results between the Vietnamese scene text recognition method based on the improved DAN provided by the present invention and the recognition results of other methods.

[0026] Figure 4 It is a flow chart of a Vietnamese scene text recognition method based on improved DAN of the present invention.

[0027] Figure 5 It is a flow chart of the present invention for inputting the fusion feature map into the VSFM module to obtain the visual and sequence fusion features.

[0028] Figure 6It is a flowchart of the present invention in which the original feature map, the attention map and the VSFM intermediate features are input into the enhanced decoder for decoding to obtain the recognition result of the scene map. DETAILED DESCRIPTION

[0029] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0030] See also Figure 1 to Figure 6 The present invention provides a Vietnamese scene text recognition method based on improved DAN, comprising:

[0031] S101 uses ResNet as an encoder to extract the original feature map fo from the input image x;

[0032] The shape of the original feature map fo is [n,c,h,w], where n is the number of images to be recognized, c is the number of channels, h is the height of the feature map, and w is the width of the feature map. ResNet is a common network structure, and the calculation method is expressed as formula 1:

[0033] f o =ResNet(x) Equation 1

[0034] S102 inputs the original feature map fo into the CAM to obtain an attention map. The specific method is to input the original feature map fo into the FPN network to obtain a fused feature map f u ; The fusion feature map f u Input VSFM module to obtain visual and sequence fusion feature f vs ; The visual and sequence fusion features f vs Input the FCN network to get the attention map;

[0035] The VSFM module is a module in CAM, the attention map is denoted as att, and its shape is [n, maxT, h, w], where maxT represents the maximum time step of decoding. FPN is a classic feature fusion network, and FCN is a seven-layer fully convolutional neural network.

[0036] The original feature map fo is input into the FPN network to obtain the fused feature map f u The specific steps are:

[0037] S201 will fuse the feature map f uWith width w as the time step, input Bi-GRU to model the horizontal sequence features and obtain the horizontal sequence modeling feature f rnn-h ;

[0038] The specific method is: in the horizontal direction, modeling can be done from left to right, or from right to left, which effectively enhances the correlation between characters and alleviates attention drift.

[0039] The number of GRU units is 256. The specific implementation is as follows: first, f u The shape is transformed from [n,c,h,w] to [n×h,w,c], and the calculation method is expressed as formula 2:

[0040] f u =Reshape(f u ) Formula 2

[0041] Then the transformed f u Input GRU to obtain the horizontal sequence modeling feature f rnn-h , the calculation method is expressed as formula 3:

[0042] f rnn-h =Bi-GRU(f u ) Formula 3

[0043] Then f rnn-h Perform shape transformation to restore the shape to [n,c,h,w]. The calculation method is expressed as formula 4:

[0044] f rnn-h =Reshape(f rnn-h ) Formula 4

[0045] S202 fuses the feature map f u Taking height h as the time step, input Bi-GRU to model the vertical sequence features and obtain the vertical sequence modeling feature f rnn-v ;

[0046] The specific method is: modeling from top to bottom in the vertical direction, and modeling from bottom to top, effectively enhancing the correlation between Latin characters and diacritical marks, paying more attention to diacritical marks, and being more sensitive to the case of characters.

[0047] The number of GRU units is 256. The specific implementation is as follows: first, f u The shape is transformed from [n, c, h, w] to [n×w, h, c], and the calculation method is expressed as formula 5:

[0048] f u =Reshape(f u) Formula 5

[0049] Then the transformed f u Input GRU to obtain the vertical sequence modeling feature f rnn-v , the calculation method is expressed as formula 6:

[0050] f rnn-v =Bi-GRU(f u ) Formula 6

[0051] Then f rnn-v Perform shape transformation to restore the shape to [n, c, h, w]. The calculation method is expressed as formula 7:

[0052] f rnn-v =Reshape(f rnn-v ) Formula 7

[0053] S203: modeling the horizontal sequence features f rnn-h and the vertical sequence modeling feature f rnn-v And the fusion feature map f u After adding the elements at the corresponding positions, the visual and sequence fusion feature f is obtained Vs .

[0054] The specific method is: the horizontal sequence modeling feature f rnn-h And the vertical sequence modeling feature f rnn-v And the fusion feature map f u The elements at the same position in are added to obtain the visual and sequence fusion feature f Vs .

[0055] The calculation method is expressed as:

[0056] f vs (i, j) = f rnn-h (i, j)+f rnn-v (i, j)+f u (i, j) i∈(0, h)j∈(0, w) Formula 8

[0057] Combining the horizontal and vertical directions allows modeling from top to bottom, from right to left, and from bottom to top, and from left to right.

[0058] S103 combines the original feature map fo, the attention map Attention map and the VSFM intermediate feature f temp The enhanced decoder is input together for decoding to obtain the recognition result of the scene graph.

[0059] The specific steps are:

[0060] S301 multiplies the original feature map fo and the attention map Attention map at each time t to obtain a context vector C, concatenates it with the embedding vector E of the decoding result at the previous time t-1, and then inputs it into the GRU together with the hidden information at the previous time t-1 to obtain the hidden information at the current time t;

[0061] The calculation method is expressed as formula 9, and it is concatenated with the decoding result E of the previous time t-1, and then input into GRU together with the hidden information of the previous time t-1 to obtain the hidden information of the current time t. The calculation method is expressed as formula 10, and the initial E and hidden are 0:

[0062] C t =f o ⊙att t Formula 9

[0063] hidden t =GRU([C t , E t-1 ], hidden t-1 ) Formula 10

[0064] S302 takes the hidden information obtained at time t and the VSFM intermediate feature f temp After splicing in the channel dimension, the data is input into the classifier for classification to obtain the recognition result at the current moment;

[0065] The VSFM intermediate feature is the fusion feature map f u The vertical sequence modeling features obtained by performing vertical sequence feature modeling in Bi-GRU are used to fused the feature map f u The sequence modeling features in the vertical direction are obtained by concatenating them in the channel dimension.

[0066] Use Softmax activation, and then select the result with the highest probability as the character result recognized at the current moment.

[0067] The hidden information and VSFM intermediate features f temp , spliced ​​in the channel dimension, the calculation method is expressed as 11, and then input into the classifier for classification, activated by the softmax function, and the probability logit of each character recognized at the current time t is obtained t , the calculation method is expressed as 12, and then the result with the highest current probability is selected as the character result res recognized at the current time t t , the calculation method is expressed as 13:

[0068] f temp (i, j) = frnn-v (i, j)+f u (i, j) Formula 11

[0069] logit t =Softmax(Classifier([hidden t , f temp ])) Formula 12

[0070] res t =afgmax(logit t ) Formula 13

[0071] S303 repeats S301 to S302. When the character with the highest probability predicted is the terminator, the prediction is stopped, and the recognition results at each moment are combined to obtain the final recognition result.

[0072] The effects of the method of the present invention are further described below in conjunction with specific embodiments.

[0073] The Vietnamese scene text recognition method based on improved DAN proposed in this invention has the following test environment and experimental results:

[0074] 1) Test environment:

[0075] System environment: Ubuntu 16.04;

[0076] Hardware environment: Memory 256GB, GPU: Tesla V100, CPU: 1.70GHz Intel(R)Xeon(R)E5-2609, Hard disk: 8TB;

[0077] 2) Experimental data:

[0078] 24757 algorithm-generated Vietnamese scene text pictures and 3261 real Vietnamese scene text pictures were randomly divided into 2608 training sets and 653 test sets. We used these real Vietnamese scene text pictures to create two real data sets, one data set (Data_no_bg) is to cover up the redundant background information (using Labelme annotation), and the other data set (Data_have_bg) is to retain the redundant background information.

[0079] 3) Implementation details:

[0080] The Adadelta optimizer is used, the epochs are set to 200, the learning rate is 1.0, the batch-size is set to 256, the number of categories of the classifier in the decoder is 198 (196 Vietnamese characters + Unknown + End_token), the evaluation indicator is the recognition accuracy of the test set acc, and maxT is set to 32.

[0081] 4) Experimental results

[0082] First, the attention map is visualized, such as Figure 2 As shown in the figure, in the first example, the DAN algorithm has attention drift at t=3 and t=4, so it recognizes one more character "T". After adding VSFM, the attention map is also correct and the correct result is recognized. In addition, at t=2, the DAN algorithm did not pay attention to the diacritic, resulting in its recognition result as "E", and at t=3, it mistakenly paid attention to the diacritic, but there was no character "T" with diacritic in the category, so its recognition result became "T". After adding VSFM, we can clearly see that the diacritic is correctly paid attention to at t=2, so it is recognized as In Example 2, we can see that after adding VSFM, the recognition result is "w", while the result of the DAN algorithm is "W", which shows that VSFM is more sensitive to the strokes and uppercase of the characters. At t=3, the DAN algorithm has attention drift, and after adding VSFM, the correct attention map sequence is generated; in Example 3, at t=2, we can see from the attention map that the attention to the diacritic is weaker than that to the Latin characters (the color is slightly lighter), so its recognition result is "Y". After adding VSFM, we can see from the attention map that the diacritic and the character "Y" have the same attention (similar color), so it recognizes the correct result. Therefore, we can conclude that, after combining with VSFM, the algorithm can effectively alleviate the occurrence of attention drift problem, while also paying greater attention to diacritics and strengthening the association between diacritics and Latin characters, so that diacritics are correctly identified during recognition.

[0083] Figure 3 As examples of recognition results of other methods and this method, we can further find that VSFM is sensitive to character strokes and case, has more accurate recognition of similar characters, and is robust to attention drift, while also recognizing diacritical marks more accurately.

[0084] Table 1 shows the comparison results with other methods. The evaluation index is recognition accuracy. It can be seen from the table that this method shows good performance. Its advantage lies in its effective adaptation to the composition of Vietnamese characters, accurate recognition of diacritical marks, strong classification performance for similar characters, and can alleviate the occurrence of attention drift problem.

[0085] Table 1 Performance comparison with other methods

[0086] Dataset DAN CRNN ASTER SEED This method Data_no_bg 70.4 51.7 62.5 41.5 78.3 Data_have_bg 69.5 50.7 60.8 42.7 76.9

[0087] DAN reference in Table 1: Wang Tianwei, ZhuYuanzhi, Jin Lianwen, et al.Decoupledattention network for text recognition[C] / / Proc of the AAAI Conf onArtificial Intelligence.Menlo Park,CA:AAAI,2020,34(07):12216-12224.

[0088] CRNN reference: Shi Baoguang, Bai Xiang, Yao Cong. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition [J]. IEEE transactions on pattern analysis and machineintelligence, 2016, 39(11): 2298-2304.

[0089] ASTER reference: Shi Baoguang, Yang Mingkun, Wang Xinggang, et al. Aster: Anattentional scene text recognizer with flexible rectification[J]. IEEE transactions on pattern analysis and machine intelligence. Piscataway, NJ: IEEE, 2018, 41(9): 2035-2048.

[0090] SEED reference: Qiao Zhi, Zhou Yu, Yang Dongbao, et al. Seed: Semantics enhanced encoder-decoder framework for scene text recognition [C] / / Proc of the IEEE / CVFConf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 13528-13537.

[0091] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A Vietnamese scene text recognition method based on improved DAN, characterized in that: include: Use ResNet as an encoder to extract raw feature maps from the input image; The original feature map is input into the CAM to obtain an attention map, specifically, the original feature map is input into the FPN network to obtain a fused feature map; the fused feature map is input into the VSFM module to obtain visual and sequence fusion features; the visual and sequence fusion features are input into the FCN network to obtain an attention map; Inputting the original feature map, the attention map and the VSFM intermediate features into the enhanced decoder for decoding to obtain a recognition result of the scene map; The specific steps of inputting the fusion feature map into the VSFM module to obtain the visual and sequence fusion features are: inputting the fusion feature map into the Bi-GRU with the width as the time step to perform horizontal sequence feature modeling to obtain horizontal sequence modeling features; The fused feature map is input into Bi-GRU with the height as the time step to perform vertical sequence feature modeling, and the vertical sequence modeling features are obtained; Adding the elements of the horizontal sequence modeling features, the vertical sequence modeling features and the fusion feature map at corresponding positions to obtain visual and sequence fusion features; The specific steps of inputting the original feature map, the attention map and the VSFM intermediate features into the enhanced decoder for decoding to obtain the recognition result of the scene map are: S301 multiplies the original feature map and the attention map at each moment to obtain a context vector, concatenates it with the embedding vector of the decoding result at the previous moment, and then inputs it into the GRU together with the hidden information at the previous moment to obtain the hidden information at the current moment; S302 concatenates the hidden information and the VSFM intermediate features in the channel dimension and inputs them into the classifier for classification to obtain the recognition result at the current moment; S303 repeats S301 to S302, and when the character with the highest probability predicted is the terminator, stops the prediction, and combines the recognition results at each moment to obtain the final recognition result; The VSFM intermediate feature is a vertical sequence modeling feature obtained by performing vertical sequence feature modeling on the fused feature map in Bi-GRU, and is obtained by splicing the fused feature map and the vertical sequence modeling feature in the channel dimension.

2. The Vietnamese scene text recognition method based on improved DAN as claimed in claim 1, characterized in that: The specific method of inputting the fused feature map into Bi-GRU to perform horizontal sequence feature modeling with the width as the time step is: modeling from left to right and from right to left in the horizontal direction.

3. The Vietnamese scene text recognition method based on improved DAN as claimed in claim 1, characterized in that: The fusion feature map is input into Bi-GRU with the height as the time step to perform vertical sequence feature modeling. The specific method of obtaining the vertical sequence modeling features is: modeling from top to bottom and from bottom to top in the vertical direction.

Images