Arabic text recognition method and system
Through feature extraction and attention feature fusion of Arabic text recognition methods, combined with glyph modeling, the problem of difficulty in identifying Arabic text in the existing technology has been solved, and the recognition accuracy rate has been significantly improved.
Patent Information
- Application Number
- CN202111537042.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-13
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-12-13
AI Technical Summary
Existing OCR technology is difficult to effectively identify languages with fewer people, such as Arabic texts, especially due to their unique writing characteristics and symbol differences, resulting in low recognition accuracy.
Using an Arabic text recognition method, through feature extraction, the fusion of channel attention features and spatial attention features, combined with Arabic glyph modeling, the training data set is constructed and the classification number is sorted to form the model category number, using Resnet module and FPN The module and the LSTM unit perform text recognition.
It greatly reduces the problems of special character miss recognition and multiple recognition, and improves the accuracy of Arabic image text recognition in natural scenes.
Smart Images

Figure CN114898345B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of recognition processing, and in particular to an Arabic text recognition method and system. Background Art
[0002] Recognizing text in images is a relatively mature technology in computer pattern recognition. For some mainstream languages such as Chinese, English, and Spanish, due to their large-scale use, they have accumulated huge data resources and have been deeply studied. Therefore, the OCR technology currently on the market can already recognize most text.
[0003] However, for some minority languages, especially those with fewer speakers and used in economically underdeveloped areas, there are relatively few related graphic and text studies, and their data resources are relatively scarce. In addition, the current OCR technology cannot effectively recognize the symbols and writing habits unique to minority languages.
[0004] Each country has its own language, and the languages are quite rich. The language symbols of each ethnic group are also quite different. Take Arabic as an example. Its text direction is from right to left, each letter has different shapes in different positions in the word, there are dots and pronunciation symbols above and below the letters, the extension lines between the letters are of variable length, vertical or horizontal connection, and different letters are of different sizes (height and width). These characteristics make it almost difficult for the current mature OCR recognition technology to recognize it. Summary of the invention
[0005] The present invention is to solve the technical problems existing in the prior art. The present invention provides an Arabic text recognition method, which is applied to an Arabic text recognition system, comprising the steps of:
[0006] Extract features from the input image to obtain a feature map F with high-level semantics;
[0007] Get the channel attention feature Ac of each time step from the feature map F;
[0008] Get the spatial attention feature As at each time step from the feature map F;
[0009] Fuse the feature map F, channel attention feature Ac, and spatial attention feature As to obtain fused features;
[0010] Identify the fused features and obtain text results.
[0011] Furthermore, the step of identifying the fused features and obtaining the text result also includes the step of Arabic glyph modeling:
[0012] Build a training dataset;
[0013] Adding each glyph of each Arabic character to the total category of recognition units;
[0014] Counting the frequency of occurrence of each connected writing element in the training data set, and adding the connected writing elements with a frequency greater than a preset value to the total category of the recognition unit;
[0015] Count the special symbols in the training data set, and add the special symbols whose occurrence frequency is greater than the preset value to the total category of the recognition unit;
[0016] Arrange the classification numbers to form model category numbers.
[0017] The present invention also provides an Arabic text recognition system, comprising a feature extraction unit, a channel attention network unit, a spatial attention network unit, an encoding unit, and a recognition unit, wherein:
[0018] The feature extraction unit is used to extract features from the input image and obtain a feature map F with high-level semantics;
[0019] The channel attention network unit is used to obtain the channel attention feature Ac of each time step from the feature map F;
[0020] The spatial attention network unit is used to obtain the spatial attention feature As at each time step from the feature map F;
[0021] The encoding unit is used to fuse the feature map F, the channel attention feature Ac, and the spatial attention feature As to obtain the fused feature;
[0022] The recognition unit is used to recognize the fused features and obtain text results.
[0023] Furthermore, it also includes a preprocessing unit for preprocessing the input image before inputting it to the feature extraction unit.
[0024] Furthermore, the feature extraction unit includes a Resnet module, an FPN module and a feature fusion module, wherein:
[0025] The Resnet module consists of four convolutional modules connected in series to extract features from the input image and obtain three semantic layer features C2, C3 and C4 at different depths;
[0026] The FPN module is used to fuse the semantic layer features C2, C3 and C4 to form the corresponding M2, M3 and M4 outputs. Its structure is as follows: the output of the semantic layer feature C4 is connected to the convolution layer to obtain feature M4; the output of the semantic layer feature C3 is connected to the convolution layer to extract features, and then added to the 2 times upsampled features of the semantic layer feature M4 to obtain feature M3; the output of the semantic layer feature C2 is connected to the convolution layer to extract features, and then added to the 2 times upsampled features of the semantic layer feature M3 to obtain feature M2;
[0027] The feature fusion module is used to fuse features M2, M3 and M4 to obtain feature map F. Its structure is: feature M4 is passed through the convolution layer and the upsampling layer to obtain feature M4_1; feature M3 is passed through the convolution layer to obtain feature M3_1, feature M2 is passed through the convolution layer and the downsampling layer to obtain feature M2_1, and features M4_1, M3_1 and M2_1 are added to obtain feature map F.
[0028] Furthermore, the channel attention network unit includes an average pooling layer and a multilinear network, where:
[0029] The average pooling layer obtains features from the feature map F that are equal to the number of channels of the feature map F, with a feature size of (1, 1, c), and inputs them into the multilinear network. Finally, the activation function is used to obtain the channel attention Ac at each time step, with a feature size of (T, 1, 1, c).
[0030] Furthermore, the spatial attention network unit includes a convolutional layer, two downsampling convolutional layers and two upsampling convolutional layers connected in series, and finally the spatial attention As of each time step is obtained through the activation function.
[0031] Furthermore, the calculation method of the encoding unit to obtain the fusion feature is:
[0032] Select the fusion feature F_t at time t, where 1≤t≤T;
[0033] Take the t-th feature Ac_t of the first dimension of channel attention Ac(T, 1, 1, c), whose feature size is (1, 1, c);
[0034] Take the t-th feature As_t of the third dimension of spatial attention As(H / 4, W / 4, T), whose feature size is (H / 4, W / 4), expand the third dimension and expand its size to c, and finally the size of As_t is (H / 4, W / 4, c);
[0035] Calculate the fusion feature F_t=Ac_t×As_t×F.
[0036] Furthermore, the recognition unit is composed of multiple LSTM units in series. At time t, the hidden state S at time t-1 is t-1 , and the fused features of the encoding unit are input into the LSTM unit to obtain the character output at time t, so that the character sequence formed by the output results at each moment is output.
[0037] The present invention also discloses an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to execute any of the Arabic text recognition methods.
[0038] In actual applications, the modules in the method and system disclosed in the present invention can be deployed in multiple modules on one target server, or each module can be independently deployed on different target servers. In particular, in order to provide more powerful computing and processing capabilities, the modules can also be deployed on cluster target servers as needed.
[0039] It can be seen that the technical solution adopted by the present invention can significantly reduce the problems of missed recognition and multiple recognition of special characters, and improve the accuracy of Arabic image text recognition in natural scenes.
[0040] In order to have a clearer and more comprehensive understanding of the present invention, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work.
[0042] Figure 1 A schematic diagram of the structure of an Arabic text recognition system according to an embodiment of the present invention;
[0043] Figure 2 A structural schematic diagram of a feature extraction unit according to an embodiment of the present invention;
[0044] Figure 3 Schematic diagram of the structure of a channel attention network unit and a spatial attention network unit in an embodiment of the present invention;
[0045] Figure 4 This is a schematic diagram of the structure of an identification unit according to an embodiment of the present invention;
[0046] Figure 5 The figure is a schematic diagram of the effect of recognizing Arabic text according to an embodiment of the present invention. DETAILED DESCRIPTION
[0047] In view of the current situation of Arabic and the defects of the prior art, the present invention proposes an Arabic text recognition method, which greatly improves the recognition ability of Arabic. The recognition steps include:
[0048] Extract features from the input image to obtain a feature map F with high-level semantics;
[0049] Get the channel attention feature Ac of each time step from the feature map F;
[0050] Get the spatial attention feature As at each time step from the feature map F;
[0051] Fuse the feature map F, channel attention feature Ac, and spatial attention feature As to obtain fused features;
[0052] Identify the fused features and obtain text results.
[0053] In addition, with respect to the writing style of the Arabic language, the present invention, based on the above-mentioned embodiment, further includes modeling Arabic glyphs in the step of identifying the fusion features and obtaining the text result, and the steps include:
[0054] Build a training dataset;
[0055] Adding each glyph of each Arabic character to the total category of recognition units;
[0056] The frequency of occurrence of each continuous writing element in the training data set is counted, and the continuous writing elements with a frequency greater than a preset value are added to the total category of the recognition unit. For example, the continuous writing elements with a frequency greater than 300 are added to the total category of the recognition unit.
[0057] Count the special symbols in the training data set, and add the special symbols whose occurrence frequency is greater than the preset value to the total category of the recognition unit. For example, add the special symbols whose occurrence frequency is greater than 1 to the total category of the recognition unit.
[0058] The classification numbers are sorted to form model category numbers. In some actual examples, 2,000 model category numbers can be formed.
[0059] In addition to being suitable for Arabic, the present invention also has a good recognition effect on text images with text features similar to Arabic. In addition to Arabic, the application of the solution of the present invention to other languages falls within the protection scope of the present invention. In order to further reflect the technical improvement and progress brought about by the technical solution of the present invention, the present invention takes Arabic as an example and further develops a detailed introduction.
[0060] Currently, Arabic is mainly used in twelve countries in West Asia and some countries in Africa. The main characteristics of Arabic language are: the text direction is from right to left, each letter has different shapes in different positions in the word, there are dots and pronunciation symbols above and below the letters, the extension line between the letters can be of variable length, vertical or horizontal connection, and different letters are of different sizes (height and width), etc.
[0061] See also Figure 1 The present invention also discloses an Arabic text recognition system, comprising a feature extraction unit, a channel attention network unit, a spatial attention network unit, an encoding unit, and a recognition unit, wherein:
[0062] The feature extraction unit is used to extract features from the input image and obtain a feature map F with high-level semantics;
[0063] The channel attention network unit is used to obtain the channel attention feature Ac of each time step from the feature map F;
[0064] The spatial attention network unit is used to obtain the spatial attention feature As at each time step from the feature map F;
[0065] The encoding unit is used to fuse the feature map F, the channel attention feature Ac, and the spatial attention feature As to obtain the fused feature;
[0066] The recognition unit is used to recognize the fused features and obtain text results.
[0067] The technical solution of the present invention is further described below in combination with the above-mentioned Arabic text recognition method and Arabic text recognition system.
[0068] First, the feature extraction unit is used to extract features from the input image to obtain a feature map F with high-level semantics.
[0069] See also Figure 2 As a preferred embodiment, Figure 2 The figure is a schematic diagram of the structure of a feature extraction unit according to an embodiment of the present invention. The feature extraction unit includes a Resnet module, an FPN module and a feature fusion module, wherein: the Resnet module is composed of four convolution modules connected in series, namely, the convolution modules Conv1, Conv2, Conv3, and Conv4 in the figure, which are used to extract features from the input image, and three semantic layer features C2, C3, and C4 of different depths are obtained from the following three convolution modules Conv2, Conv3, and Conv4 respectively;
[0070] The FPN module is used to fuse the semantic layer features C2, C3 and C4 to form corresponding M2, M3 and M4 outputs. Its structure is: the output of the semantic layer feature C4 is connected to the convolution layer to obtain feature M4; the output of the semantic layer feature C3 is connected to the convolution layer to extract features, and then added to the twice upsampled features of the semantic layer feature M4 to obtain feature M3; the output of the semantic layer feature C2 is connected to the convolution layer to extract features, and then added to the twice upsampled features of the semantic layer feature M3 to obtain feature M2. In this embodiment, the convolution layers in the FPN module are all selected as 1×1 convolution layers to implement.
[0071] The feature fusion module is used to fuse features M2, M3 and M4 to obtain feature map F. Feature M4 passes through a convolution layer and an upsampling layer to form feature M4_1, feature M3 passes through a convolution layer to form feature M3_1, and feature M2 passes through a convolution layer and a downsampling layer to form feature M2_1. At this time, M4_1, M3_1 and M2_1 have the same feature size. M4_1, M3_1 and M2_1 are added to form the final fused feature F. The convolution layers used in the feature fusion module are all implemented as 3×3 convolution layers.
[0072] In this embodiment, the Resnet module can use Resnet-50 as the backbone feature extraction network, and the last three convolution modules respectively output output features of three different depth semantic layers, corresponding to features C2, C3 and C4.
[0073] Then, the FPN structure is used to fuse the features of each semantic layer, and the corresponding features M2, M3 and M4 are output; the fusion module is used to fuse the corresponding outputs of these three layers to form the final feature map F, which has a size of (H / 4, W / 4, c), where H and W are the height and width of the input image.
[0074] During the research process, the present invention found that there are dots above or below Arabic characters, which is quite different from other languages. In the feature extraction unit, if the downsampling of the convolution module is directly implemented using the conventional MaxPooling module, the feature will lose important dot information. Therefore, based on the characteristics of Arabic, an improved and optimized method is proposed. The downsampling in the convolution module is implemented by a 3×3 convolution with a step size of 2 and a padding of 1, so that the dot information contained in the text information can be obtained.
[0075] Based on the above feature extraction features, the present invention proposes a more preferred method, that is, a CBAM module is connected after each convolution module Conv1, Conv2, Conv13, and Conv4, so that the text area information of the image text can be focused.
[0076] As a preferred implementation, before the feature extraction unit extracts the input image, the input image may be preprocessed to enhance the robustness of the text recognition system. With respect to the characteristics of the Arabic language, the preprocessing of the present invention includes:
[0077] First, since Arabic tones are similar to Chinese pinyin and do not affect the semantic information of the text, all tonal characters in the annotation file are removed;
[0078] Second, the input image is corrected to reduce the impact of excessive tilt on text recognition;
[0079] Third, unify the size of the rectified input image, for example, scale the height to 64 and the width to 256;
[0080] Fourth, after feature enhancement (adjusting contrast, brightness, grayscale value and Gaussian noise) and image normalization, the text image is input into the feature extraction module.
[0081] Secondly, after obtaining the feature map F, the channel attention Ac of each time step and the spatial attention As of each time step are extracted respectively. The encoding unit then fuses the feature map F (feature size is (H / 4, W / 4, c)), the channel attention feature Ac (feature size is (T, 1, 1, c)) and the spatial attention feature As (feature size is (feature size is (H / 4, W / 4, T)) to obtain the fused feature F_t calculation method:
[0082] Select the fusion feature F_t at time t, where 1≤t≤T;
[0083] Take the t-th feature Ac_t of the first dimension of channel attention Ac(T, 1, 1, c), whose feature size is (1, 1, c);
[0084] Take the t-th feature As_t of the third dimension of spatial attention As(H / 4, W / 4, T), whose feature size is (H / 4, W / 4), expand the third dimension and expand its size to c, and finally the size of As_t is (H / 4, W / 4, c);
[0085] Calculate the fusion feature F_t=Ac_t×As_t×F.
[0086] See also Figure 3 As a preferred embodiment, the channel attention network unit of the present invention includes an average pooling layer and a multilinear network, wherein:
[0087] The average pooling layer obtains features from feature map F with the same number of channels as feature map F. The feature size is (1, 1, c), that is, the width and height are both 1, and the number of channels remains unchanged. It is input into a multi-layer perceptron (MLP) or a multi-linear network, and finally the channel attention Ac of each time step is obtained through an activation function. Its feature size becomes (T, 1, 1, c), where T is the maximum size of the output sequence, that is, T feature maps with a height and width of 1 and a number of channels of c are generated. The activation function can be sigmoid.
[0088] The spatial attention network unit consists of a convolutional layer (Conv_0), two downsampling convolutional layers (Conv_1, Conv_2) and two upsampling convolutional layers (Conv_3, Conv_4) connected in series, and finally the spatial attention As of each time step is obtained through the activation function. The specific implementation process is: the spatial attention network unit first passes through a convolutional layer (Conv_0), two downsampling convolutional layers (Conv_1, Conv_2) reduce the scale of the input features to (H / 16, W / 16, c), and then two convolutional layers (Conv_3, Conv_4) restore the input feature scale to (H / 4, W / 4, T), and finally generate T feature maps with the same height and width as the input feature map F, that is, the spatial attention As of each time step.
[0089] Finally, after obtaining the fused features, the recognition unit is used to recognize the fused features to obtain the text results.
[0090] See also Figure 4 As a preferred implementation, the recognition unit is composed of multiple LSTM units connected in series, and the implementation process is as follows:
[0091] LSTM is a network structure for time series processing. At time t, the hidden state S at time t-1 is converted to t-1 , the output Ac_t of the t-th linear layer in the channel attention network unit, whose scale is (1, 1, c), the feature As_t of the t-th channel of the spatial attention network unit, whose scale is (H / 4, W / 4), and the output F of the feature extraction unit are input into the LSTM unit to obtain the character output at time t. In this way, the output results at each time are connected together to form the final predicted string.
[0092] See also Figure 5 , Figure 5 The figure is a schematic diagram of the effect of the embodiment of the present invention on recognizing Arabic text. It has been verified in practice that the present invention can greatly improve the recognition capability of text.
[0093] An embodiment of the present invention further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform any of the Arabic text recognition methods described above.
[0094] It should be noted that, a person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium can include but is not limited to: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), a disk or an optical disk, etc.
[0095] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An Arabic text recognition method, characterized in that: The application in Arabic text recognition system includes the following steps: Extract features from the input image to obtain a feature map F with high-level semantics; Get the channel attention feature Ac of each time step from the feature map F; The spatial attention feature As of each time step is obtained from the feature map F, which is specifically obtained by a spatial attention network unit consisting of a convolutional layer, two downsampling convolutional layers and two upsampling convolutional layers connected in series through an activation function; The feature map F, channel attention feature Ac and spatial attention feature As are fused to obtain fused features, including: Select the fusion feature F_t at time t, where 1≤t≤T; Take the t-th feature Ac_t of the first dimension of channel attention Ac(T, 1, 1, c), whose feature size is (1, 1, c); Take the t-th feature As_t of the third dimension of spatial attention As(H / 4, W / 4, T), whose feature size is (H / 4, W / 4), expand the third dimension and expand its size to c, and finally the size of As_t is (H / 4, W / 4, c); Calculate the fusion feature F_t = Ac_t × As_t × F; The fused features are identified to obtain text results, which also includes the Arabic glyph modeling step: Build a training dataset; Adding each glyph of each Arabic character to the total category of recognition units; Counting the frequency of occurrence of each connected writing element in the training data set, and adding the connected writing elements with a frequency greater than a preset value to the total category of the recognition unit; Count the special symbols in the training data set, and add the special symbols whose occurrence frequency is greater than the preset value to the total category of the recognition unit; Arrange the classification numbers to form model category numbers.
2. An Arabic text recognition system, characterized in that: It includes feature extraction unit, channel attention network unit, spatial attention network unit, encoding unit and recognition unit, among which: The feature extraction unit is used to extract features from the input image and obtain a feature map F with high-level semantics; The channel attention network unit is used to obtain the channel attention feature Ac of each time step from the feature map F; The spatial attention network unit is used to obtain the spatial attention feature As of each time step from the feature map F. The spatial attention network unit consists of a convolution layer, two downsampling convolution layers and two upsampling convolution layers connected in series. Finally, the spatial attention As of each time step is obtained through the activation function. The encoding unit is used to fuse the feature map F, the channel attention feature Ac, and the spatial attention feature As to obtain the fused features, including: Select the fusion feature F_t at time t, where 1≤t≤T; Take the t-th feature Ac_t of the first dimension of channel attention Ac(T, 1, 1, c), whose feature size is (1, 1, c); Take the t-th feature As_t of the third dimension of spatial attention As(H / 4, W / 4, T), whose feature size is (H / 4, W / 4), expand the third dimension and expand its size to c, and finally the size of As_t is (H / 4, W / 4, c); Calculate the fusion feature F_t = Ac_t × As_t × F; The recognition unit is used to recognize the fused features and obtain the text results, and also includes the Arabic glyph modeling step: Build a training dataset; Adding each glyph of each Arabic character to the total category of recognition units; Counting the frequency of occurrence of each connected writing element in the training data set, and adding the connected writing elements with a frequency greater than a preset value to the total category of the recognition unit; Count the special symbols in the training data set, and add the special symbols whose occurrence frequency is greater than the preset value to the total category of the recognition unit; Arrange the classification numbers to form model category numbers.
3. The Arabic text recognition system as claimed in claim 2, characterized in that: It also includes a preprocessing unit for preprocessing the input image before inputting it to the feature extraction unit.
4. The Arabic text recognition system as claimed in claim 2, characterized in that: The feature extraction unit includes a Resnet module, an FPN module and a feature fusion module, wherein: The Resnet module consists of four convolutional modules connected in series to extract features from the input image and obtain three semantic layer features C2, C3 and C4 at different depths; The FPN module is used to fuse the semantic layer features C2, C3 and C4 to form the corresponding M2, M3 and M4 outputs. Its structure is as follows: the output of the semantic layer feature C4 is connected to the convolution layer to obtain feature M4; the output of the semantic layer feature C3 is connected to the convolution layer to extract features, and then added to the 2 times upsampled features of the semantic layer feature M4 to obtain feature M3; the output of the semantic layer feature C2 is connected to the convolution layer to extract features, and then added to the 2 times upsampled features of the semantic layer feature M3 to obtain feature M2; The feature fusion module is used to fuse features M2, M3 and M4 to obtain feature map F. Its structure is: feature M4 is passed through the convolution layer and the upsampling layer to obtain feature M4_1; feature M3 is passed through the convolution layer to obtain feature M3_1, feature M2 is passed through the convolution layer and the downsampling layer to obtain feature M2_1, and features M4_1, M3_1 and M2_1 are added to obtain feature map F.
5. The Arabic text recognition system as claimed in claim 2, characterized in that: The channel attention network unit consists of an average pooling layer and a multilinear network, where: The average pooling layer obtains features from the feature map F that are equal to the number of channels of the feature map F, with a feature size of (1, 1, c), and inputs them into the multilinear network. Finally, the activation function is used to obtain the channel attention Ac at each time step, with a feature size of (T, 1, 1, c).
6. The Arabic text recognition system as claimed in claim 2, characterized in that: The recognition unit is composed of multiple LSTM units in series. At time t, the hidden state S at time t-1 is t-1 , and the fused features of the encoding unit are input into the LSTM unit to obtain the character output at time t, so that the character sequence formed by the output results at each moment is output.
7. An electronic device comprising: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, wherein when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the Arabic text recognition method as claimed in claim 1.
