A font recognition method and system based on deep learning and a storage medium

Abstract

The application provides a font recognition method and system based on deep learning and a storage medium, and the recognition method comprises the following steps: obtaining a segmented image containing a target font; and performing super-resolution reconstruction on the segmented image containing an attention mechanism to obtain a super-resolution font image. The method can obtain a high-definition font image from a document, especially a natural scene or other documents with noise interference, and obtain an accurate classification and recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision processing, in particular to the technical field of computer vision image processing based on deep learning. BACKGROUND

[0002] In recent years, optical character recognition (OCR) has made considerable progress, and at the same time, font recognition, especially Chinese font recognition (CCRF), has also received more attention. For document image processing, the change of font may affect the accuracy of character recognition (OCR), and font information may also affect the layout analysis and understanding of text. For example, using font information of characters can help the document reconstruction system to recover the document content and character style. However, in actual font recognition, there are many challenges such as multiple font types and multiple font deformations (such as bold, italic, etc.).

[0003] Current font recognition methods mainly focus on English or Arabic script font recognition, such as classifying the font of letters by counting the distribution of black pixels, but this method ignores the morphological differences between different characters of the same font, such as the difference in thickness between different characters of the same font, so the final recognition result is not very good. Or like through wavelet transform or global texture analysis to extract the corresponding features of the font, and then calculate the Euclidean distance to determine the font category, but these methods are developed for English or Arabic font recognition, and are not suitable for Chinese font recognition.

[0004] As we know, Chinese characters have a large number of different forms, and the outlines and gray levels of different characters are irregular and vary greatly, which brings great challenges to Chinese font recognition. Currently, the method of Chinese font recognition mainly uses machine learning, which first extracts the corresponding features, and then classifies through SVM or decision tree. However, the current deep learning-based font recognition method is not friendly to document images in natural scenes, and cannot handle the impact of various problems in natural scenes, such as various noises in natural scene document images. The single font of the text image in the natural scene is small, and the feature extraction process of deep learning often needs multiple downsampling to obtain rich semantic information of the image, so the small picture needs to be enlarged to a certain extent before feature extraction. However, the process of image enlargement will inevitably cause loss of pixel values, which will greatly destroy the texture features of the font. SUMMARY

[0005] The present application aims to overcome the shortcomings in the prior art, and provide a new method for Chinese font recognition based on deep learning, which can obtain high-definition font images from documents, especially in natural scenes or other documents with noise interference, and obtain accurate classification and recognition results.

[0006] The present application also aims to provide an efficient implementation system of the above-mentioned recognition method and a storage medium thereof.

[0007] The present application first provides the following technical solutions:

[0008] A font recognition method based on deep learning, comprising: obtaining a segmentation image containing a target font; performing super-resolution reconstruction with an attention mechanism on the segmentation image to obtain a super-resolution font image.

[0009] The technical solution can fully solve the technical problem that the font recognition is affected by the small font proportion and low resolution of the document image in a natural scene, and the final classification result is affected if the single character cut out is directly classified and recognized, and the noise interference in the document in a natural scene can be fully eliminated by introducing the attention mechanism to obtain a feature significant image to be recognized, and the final classification accuracy is improved.

[0010] Further, the font recognition method further comprises: performing font recognition through the super-resolution font image, and the segmentation image and / or an enlarged image thereof.

[0011] It can be understood that the further technical solution actually comprises: performing font recognition through the super-resolution font image and the segmentation image; performing font recognition through the super-resolution font image and an enlarged image of the segmentation image; and performing font recognition through the super-resolution font image, the segmentation image, and the enlarged image of the segmentation image.

[0012] The further technical solution can improve the final classification and recognition effect compared with the technical means of directly using the segmentation image or the super-resolution font image for recognition.

[0013] Further, the enlarged image is obtained by interpolation on the segmentation image.

[0014] The interpolation can be bilinear interpolation, which can obtain a reconstructed image with better reconstruction effect.

[0015] Further, the attention image used in the attention mechanism is an image obtained by taking the inverse of the normalized segmentation image.

[0016] Further, the normalization processing includes: normalizing the gray value of each pixel in the segmented image to the range of 0 to 1.

[0017] The inventor surprisingly found that, in a document image, the text region is the main concern corresponding, the background region needs less concern, and the region where the text is located is black, and other regions are white, that is, the darker the place, the more attention it needs, and the whiter the place, the less attention it needs. Therefore, in the above two further technical solutions, after normalizing the image or more specifically normalizing it to the range of (0, 1), an image with any point gray value I is obtained, and by subtracting each pixel by 1-I, the attention map can be obtained. The attention map is very suitable for processing of text images, and is easy to obtain and has small amount of calculation.

[0018] Further, the super-resolution reconstruction includes:

[0019] The segmented image and / or the image after the magnification and / or normalization processing of the segmented image is fused with the attention map obtained after the processing to obtain a fused image; one or more times of down-sampling feature extraction is performed on the fused image to obtain one or more feature maps;

[0020] The one or more feature maps are respectively subjected to attention allocation to obtain feature maps respectively containing attention weights;

[0021] One or more times of bilinear up-sampling extraction is performed on the feature maps containing attention weights to obtain one or more up-sampling maps;

[0022] The up-sampling map is multiplied with the attention map to obtain a first fused image;

[0023] The first fused image is additively fused with the segmented image and / or the image after the magnification and / or normalization processing of the segmented image to obtain a super-resolution reconstruction image, that is, the super-resolution font image.

[0024] The further technical solution can fully solve the problems of texture structure damage, detail loss, etc. after the single font image is enlarged by methods such as bilinear interpolation, fully restore the texture result of the text, better focus on the text part rather than the irrelevant background part, and improve the classification accuracy.

[0025] Further, the obtaining of the segmented image includes:

[0026] The original document image is subjected to binaryzation processing for foreground and background separation to obtain a first preprocessed image in which the target region is recognizable;

[0027] The first preprocessed image is subjected to enhancement processing to obtain a second preprocessed image;

[0028] target extraction and target position calibration are performed on the second preprocessed image to obtain a final preprocessed image;

[0029] The final preprocessed image is segmented by a projection segmentation method to obtain the segmented image.

[0030] The further technical scheme can obtain an accurate segmented image.

[0031] The application further provides an implementation system of the detection method, which comprises a super-resolution reconstruction module, and the reconstruction module comprises a reconstruction model as follows: the model comprises one feature fusion unit, five down-sampling units, four up-sampling units, four skip-layer connections with attention mechanisms and one final up-sampling unit, wherein the feature fusion unit comprises two input layers for inputting the segmented image and / or an image obtained by magnifying and / or normalizing the segmented image, and an attention image obtained from the segmented image, and one fusion layer for fusing images of the two input layers; the five down-sampling units each comprise two convolution layers and one ReLU activation layer connected in sequence, and the first four down-sampling units each further comprise one max-pooling layer connected with the ReLU activation layer; the four up-sampling units each comprise one up-sampling layer and two convolution layers connected in sequence; the four skip-layer connections with attention mechanisms are located between the first four down-sampling units and the four up-sampling units, and are connected with the first four down-sampling units, the four up-sampling units and the input layer for inputting the attention image in the feature fusion layer, respectively, and the connection operation comprises: multiplying the attention image after adjusting the size of the feature map obtained from the corresponding down-sampling unit to obtain a feature map with an attention mechanism, and splicing the feature map with the attention mechanism and the corresponding up-sampling layer to obtain a feature fusion image of different layers; the final up-sampling unit comprises one up-sampling layer, two convolution layers, another up-sampling layer and another convolution layer connected in sequence, that is, a total of two up-sampling layers and three convolution layers, wherein the first up-sampling layer is connected with the last up-sampling layer in the four up-sampling units and the two input layers in the feature fusion unit, respectively.

[0032] In the further technical solution, the reconstruction model is constructed based on a Unet neural network, can extract multi-level features from different depth layers, retain rich texture information, and use multi-scale context information to synthesize high-quality images, wherein the lower convolution unit part or the encoder part can use the architecture of VGG16, the upper convolution unit part or the decoder part uses the convolution plus bilinear upsampling method to replace the traditional deconvolution operation, can reduce the chessboard artifact, and introduces the corresponding attention mechanism in the input and skip layer connection, improves the final super-resolution effect.

[0033] Further, the system further comprises a font recognition module, the font recognition module comprises a classifier formed by a mobilenetv3 network, and has an input layer for simultaneously or separately inputting the super-resolution font image, and the segmentation image and / or the enlarged image thereof.

[0034] The further technical solution selects a relatively mature and lightweight network mobilenetv3 as the recognition model of the application, adopts a depth separable convolution, can greatly reduce the parameter calculation amount, simultaneously introduces a lightweight attention model (SE), can improve the classification accuracy, ensures that the model can have very high compactness in the case that the accuracy is very close to that of a classic classifier model resnet, densenet, and enables the model to be directly deployed on a mobile terminal.

[0035] In addition, further, by simultaneously inputting the original segmentation image or the enlarged image thereof obtained by a method such as bilinear interpolation and the reconstructed image obtained by the super-resolution module into the font recognition module, the final classification effect can be improved.

[0036] The application further provides a storage medium which stores an algorithm structure for realizing any detection scheme and / or stores any or all modules of any system. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 It is a specific overall identification flowchart of the application;

[0038] Figure 2 It is a specific super-resolution reconstruction flowchart of the application;

[0039] Figure 3 It is a specific super-resolution model structure schematic diagram of the application;

[0040] Figure 4 It is a specific font recognition flowchart of the application;

[0041] Figure 5A specific font recognition model structure diagram of the application. DETAILED DESCRIPTION

[0042] The technical solutions of the application will be described in further detail below in combination with the drawings and specific embodiments, but the protection scope of the application is not limited to the following.

[0043] According to the technical solutions of the application, a specific recognition method includes the following steps as shown in the flowchart: Figure 1

[0044] S1, preprocessing an original document image to obtain a preprocessed image containing target text;

[0045] More specifically, it can include:

[0046] S11, obtaining a gray image of the original document image, and further obtaining a first preprocessed image recognizable by the target region through a binarization and negation process;

[0047] S12, performing morphological operations such as dilation, erosion, and opening operation on the first preprocessed image and / or performing bilateral filtering on the first preprocessed image to obtain a second preprocessed image after target enhancement;

[0048] S13, extracting the target text from the second preprocessed image by a bounding method such as a positive bounding box and determining the position of the target text, and then cutting a picture of the corresponding position from the original image according to the position to obtain a final preprocessed image;

[0049] S2, obtaining a segmented text image from the preprocessed image, and performing magnification and normalization reconstruction preprocessing to obtain a pre-reconstruction text image;

[0050] More specifically, the segmentation can be obtained by a projection segmentation algorithm;

[0051] The reconstruction preprocessing can include:

[0052] The segmented text image is expanded through bilinear interpolation to obtain a magnified text image;

[0053] The magnified text image is normalized to obtain a picture matrix I whose gray value is normalized to (0, 1);

[0054] S3, performing super-resolution reconstruction on the pre-reconstruction text image to obtain a high-resolution text image;

[0055] More specifically, with reference to Figure 2 , the reconstruction can be realized through the following process:

[0056] ​S31 subtracts 1 from the normalized picture matrix I obtained by S2 to obtain an attention map matrix Attention_I;

[0057] S33 concatenates the picture matrix I and the attention map matrix Attention_I in the channel dimension, and inputs them into the super-resolution model to obtain a feature map matrix Feature_I;

[0058] S34 performs a point multiplication operation on the feature map matrix Feature_I and the attention map matrix Attention_I to obtain a first fusion map matrix Fusion1_I

[0059] S35 performs an addition operation on the first fusion map matrix Fusion1_I and the picture matrix I at corresponding positions to obtain a second fusion map matrix Fusion2_I

[0060] S36 extracts a super-resolution picture from the second fusion map matrix Fusion2_I through a final upsampling of the super-resolution model.

[0061] Further, the above reconstruction process can be implemented through a reconstruction model as follows:

[0062] With reference to Figure 3 The super-resolution reconstruction model comprises an improved Unet network and a final upsampling unit, wherein the improved Unet network comprises one feature fusion unit, five downsampling units, four upsampling units, and four skip connections with attention mechanisms, the feature fusion unit comprises two input layers for inputting the enlarged and normalized image I and the attention image, and one fusion layer for fusing the images of the two input layers, each of the five downsampling units comprises two convolution layers and one ReLU layer, and the first four downsampling units further comprise one max-pooling layer connected to the ReLU layer, each of the four upsampling units comprises one upsampling layer and two convolution layers connected in sequence, the skip connections are located between the first four downsampling units and the four upsampling units, and are connected to the first four downsampling units, the four upsampling units, and the attention image input layer in the feature fusion layer, the connection operation is that the attention image is adjusted to the size of the feature map obtained by the corresponding downsampling unit, multiplied by the feature map to obtain a feature map with an attention mechanism, and the feature map with the attention mechanism is spliced with the corresponding upsampling layer to achieve the effect of fusing features of different layers, and the final upsampling unit comprises one upsampling layer, two convolution layers, another upsampling layer, and another convolution layer in sequence, that is, it comprises two upsampling layers and three convolution layers in total, and the first upsampling layer is connected to the last upsampling layer in the four upsampling units and the two input layers in the feature fusion unit, respectively.

[0063] In some more specific preferred embodiments, the number of convolution kernels of the convolution layers of the five downsampling units is 16, 32, 64, 128, and 256, respectively, the convolution kernels are all 3, and the step sizes are all 1; the upsampling magnification ratio of the upsampling layers of the four upsampling units is 2; the number of convolution kernels of the convolution layers of the four upsampling units is 256, 128, 64, and 32, respectively, the convolution kernels are all 3, and the step sizes are all 1; the upsampling magnification ratio of the final upsampling unit is 2, and the number of convolution kernels of the three convolution layers is 8, 4, and 1, respectively, the size of the convolution kernels is all 3, and the step sizes are all 1.

[0064] S4 classifies and recognizes the obtained high-resolution text image to obtain a recognition result;

[0065] More specifically, the identification can be performed as shown in the following example. Figure 4 The process shown includes: taking the enlarged image and the super-resolution reconstructed image as input at the same time, performing font recognition through the recognition model, and obtaining a recognition result. The enlarged image is an enlarged image obtained by a method such as bilinear interpolation of the segmented text image.

[0066] More specifically, see Figure 5 , wherein the recognition model can select the mobilenetv3 classifier.

[0067] In some more specific implementations, the recognition model may utilize the Adam optimization algorithm, preferably using multivariate cross entropy as the loss function, and preferably having an initial learning rate of lr = 0.003. Furthermore, to improve the generalization ability of the model, it is preferred to optimize the hyperparameter lr using a cosine annealing strategy, and / or train using a 5-fold cross-validation approach, i.e., the dataset is divided into five equal parts, and each training run randomly selects three parts as training sets, with the remaining two parts as validation sets. The model parameters that minimize the loss function on the validation set are selected as the optimal parameters and saved.

[0068] In more specific embodiments, the present invention uses a text document image obtained by scanning with a scanner as the input raw image. After performing the three basic preprocessing steps described above on the raw image, basic noise interference is removed, providing a higher-quality image for subsequent operations. Subsequently, through detection of individual characters, segmentation of the image, and reconstruction operations, font recognition accuracy is improved and adaptability is enhanced. During the recognition process, the combination of bilinear interpolation and a super-resolution model effectively preserves detailed information in the text image, such as subtle strokes and equal widths between horizontal and vertical lines, which can distinguish similar fonts belonging to different font types.

[0069] When compared with the recognition accuracy on the disclosed font data set of other prior arts, the optimal accuracy of LFE, DeepFont and AlexNet in the prior art is about 60.69%, 72.93% and 91.05% respectively, and the present application can achieve a recognition accuracy of about 95% on the existing picture data set, and the recognition accuracy is also higher.

[0070] The above are only preferred embodiments of the present application, and it should be understood that the present application is not limited to the forms disclosed herein, should not be considered as excluding other embodiments, and can be used in various other combinations, modifications and environments, and can be modified within the scope of the present application by the above teachings or related art or knowledge. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application shall be within the protection of the appended claims of the present application.

Claims

1. A font recognition method based on deep learning, characterized in that: It includes: S1 preprocesses the original document image to obtain a preprocessed image containing the target text; S2: obtaining a segmented text image from the pre-processed image, and performing pre-reconstruction processing such as magnification and normalization to obtain a pre-reconstructed text image; S3 performs super-resolution reconstruction including an attention mechanism on the text image before reconstruction to obtain a high-resolution text image; The reconstruction is achieved through the following process: S31 subtracts the normalized image matrix I obtained in S2 from 1 to obtain the attention map matrix Attention_I; S33 concatenates the image matrix I and the attention map matrix Attention_I in the channel dimension and inputs them into the super-resolution model to obtain a feature map matrix Feature_I; S34 performs a dot multiplication operation on the feature map matrix Feature_I and the attention map matrix Attention_I to obtain a first fusion map matrix Fusion1_I; S35 performing an addition operation on the elements at corresponding positions of the first fusion image matrix Fusion1_I and the image matrix I to obtain a second fusion image matrix Fusion2_I; S36 extracts the second fusion image matrix Fusion2_I through final upsampling of the super-resolution model to obtain the high-resolution text image; S4 classifies and recognizes the high-resolution text image to obtain a recognition result.

2. The font recognition method according to claim 1, wherein: It also includes: Font recognition is performed using the high-resolution text image and its magnified image.

3. The font recognition method according to claim 1, wherein: The attention image used in the attention mechanism is an image obtained by normalizing and then negating the segmented text image.

4. The font recognition method according to claim 3, wherein: The normalization process includes normalizing the grayscale value of each pixel in the segmented text image to a range of 0 to 1.

5. The font recognition method according to claim 1, wherein: The acquisition of the segmented text image includes: Performing a binarization process on the original document image to separate the foreground and background, and obtaining a first pre-processed image in which the target area is identifiable; performing enhancement processing on the first preprocessed image to obtain a second preprocessed image; performing target extraction and target position calibration on the second preprocessed image to obtain a final preprocessed image; The final pre-processed image is segmented by a projection segmentation method to obtain the segmented text image.

6. A system for implementing the font recognition method according to any one of claims 1 to 5, characterized in that: It includes a super-resolution reconstruction module, which includes the following reconstruction model: The model includes 1 feature fusion unit, 5 downsampling units, 4 upsampling units, 4 skip-layer connections with attention mechanism and 1 final upsampling unit, wherein the feature fusion unit includes 2 input layers for inputting the segmented text image and / or the image after enlargement and / or normalization of the segmented text image, and the attention image obtained from the segmented text image, and 1 fusion layer for fusing the images of the two input layers; the 5 downsampling units respectively include 2 convolutional layers and 1 ReLU activation layer connected in sequence, and the first 4 downsampling units also include 1 maximum pooling layer connected to the ReLU activation layer; the 4 upsampling units respectively include 1 upsampling layer and 2 convolutional layers connected in sequence; the 4 skip-layer connections with attention mechanism are located in the The first four downsampling units and the four upsampling units are respectively connected to the first four downsampling units, the four upsampling units, and the input layer for attention image input in the feature fusion unit, and the connection operation includes: adjusting the attention map to the size of the feature map obtained by the corresponding downsampling unit and multiplying it with it to obtain a feature map with an attention mechanism, and splicing the feature map with the attention mechanism with the corresponding upsampling layer to obtain feature fusion images of different layers; the final upsampling unit includes one upsampling layer, two convolutional layers, another upsampling layer and another convolutional layer connected in sequence, that is, a total of two upsampling layers and three convolutional layers, wherein the first upsampling layer is respectively connected to the last upsampling unit of the four upsampling units and the two input layers in the feature fusion unit.

7. The system according to claim 6, characterized in that The device further comprises a font recognition module, which comprises a classifier formed by a MobileNetV3 network and has an input layer for simultaneously or separately inputting the high-resolution text image and the enlarged image thereof.

8. A storage medium storing an algorithm structure for implementing the identification method according to any one of claims 1 to 5 and / or storing any or all modules of the system according to claim 6 or 7.

Images