A character recognition method and system based on improved CharNet

Abstract

The application discloses a kind of based on the improved CharNet's character recognition method.Step 1: obtaining the image data set to be detected;Step 2: the image to be detected in step 1 is carried out the feature extraction of backbone network, the Hourglass network of the backbone network is improved using adaptive RoIAlign, while removing part of the pooling layer using the dilated convolution with expansion efficiency rate=3, obtain larger feature map with reduced feature loss;Step 3: the feature map with reduced feature loss in step 2 is respectively input into the two parallel branches of CharNet network, and the text recognition result is obtained;Step 4: while using the improved hardening swish function as the activation function of network deep layer in step 2 and step 3.To solve the problem of poor effect when detecting multi-scale text and arbitrary shape text in text recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field, and particularly relates to a character recognition method based on an improved CharNet and a system thereof. BACKGROUND

[0002] With the continuous development of society, a large amount of data information is accompanied in people's daily life and work, and characters are also in various forms in our life. Using a computer to detect and recognize characters in an image can not only improve our work efficiency, but also provide additional information of a scene, such as image-based search, map location positioning, robot navigation, human-computer interaction and the like.

[0003] End-to-end character detection and recognition based on deep learning refers to that features of a detection network and a recognition network are shared instead of being independently split into a recognition task and a detection task, and the network needs to obtain detection and recognition results in one forward process. Researches show that the detection and recognition network can promote each other in this end-to-end character recognition mode, the semantic features of the obtained characters help to standardize detection, and the integrity of the characters in the test stage is also better guaranteed. However, in the past few years, the end-to-end character detection and recognition mainly faces the following problems: (1) the early end-to-end method usually simply combines the detection and recognition network, so that the structure speed of the end-to-end method has no obvious advantage compared with an independent model, and secondly, in order to pursue performance, a complex candidate frame generation network is also selected to a certain extent, which limits the detection speed. (2) The past end-to-end method mainly relies on a complex training mode, the recognition network is first trained, then the detection network is trained, and finally the detection and recognition network are connected in series for joint training, which leads to the fact that the end-to-end method is not actually end-to-end training. SUMMARY

[0004] The application provides a character recognition method based on an improved CharNet, which is used to solve the problem of poor effect in detecting multi-scale text and arbitrary shape text in text recognition.

[0005] The application provides a character recognition system based on an improved CharNet, which is used to solve the problem of recognizing curved text.

[0006] The application is achieved through the following technical solutions:

[0007] A character recognition method based on an improved CharNet, the character recognition method comprising the following steps:

[0008] Step 1: acquiring an image data set to be detected;

[0009] Step 2: the image to be detected in step 1 is subjected to feature extraction of a backbone network, an Hourglass network of the backbone network is improved by using adaptive RoIAlign, and a dilated convolution with an expansion efficiency of rate=3 is used to remove part of the pooling layer to obtain larger feature maps with reduced feature loss;

[0010] Step 3: the feature maps with reduced feature loss in step 2 are respectively input into two parallel branches of a CharNet network to obtain text recognition results;

[0011] Step 4: step 2 and step 3 are performed simultaneously, and an improved hard swish function is used as an activation function of a deep layer of the network.

[0012] An improved CharNet-based character recognition method, wherein step 2 improves an Hourglass network of the backbone network by using adaptive RoIAlign to obtain feature maps with reduced feature loss;

[0013] The Hourglass network adopts a two-layer structure, the upper layer uses a residual module to collect features at the original scale, the lower layer uses maxpooling to down-sample and then inputs the residual module, and finally uses up-sampling to obtain a feature map with the same size as the upper layer to perform feature synthesis;

[0014] The Hourglass uses a down-sampling method to reduce the dimension of the features.

[0015] An improved CharNet-based character recognition method, wherein step 2 removes part of the pooling layer and uses a dilated convolution with an expansion efficiency of rate=3 to obtain larger feature maps;

[0016] In the adaptive RoIAlign layer, the sampling interval bilinear interpolation bandwidth B is adaptively selected according to the size of the feature grid, and the size of the RoI has a relationship, which is expressed as:

[0017]

[0018] In the formula, width represents the width of the RoI after conv3; width' represents the width of the RoI after RoI Align; [·] represents the integer operator.

[0019] An improved CharNet-based character recognition method, wherein step 4 uses an improved hard swish function as an activation function of a deep layer of the network, and the hard swish function is specifically,

[0020] The expression of Swish is:

[0021] Swish(x)=x*Sigmoid(x) (4)

[0022] The expression of Sigmoid(x) is:

[0023]

[0024] The Swish function is replaced by using a piecewise linear function h-Swish to approximate the Swish function, and the expression of h-Swish is:

[0025]

[0026] An improved CharNet-based character recognition system, the shape-robust text detection system comprises an input module and a CharNet network module;

[0027] The input module inputs the image to be detected into the CharNet network;

[0028] The CharNet network module extracts the backbone network features of the image to be detected, and obtains a larger feature map with reduced feature loss;

[0029] The larger feature map with reduced feature loss is input into two parallel branches of the CharNet network respectively, and a text recognition result is obtained.

[0030] An improved CharNet-based character recognition system, the CharNet network module uses adaptive RoIAlign to improve the Hourglass network of the backbone network, and obtains a feature map with reduced feature loss;

[0031] The Hourglass network adopts a two-layer structure, the upper layer uses a residual module to collect features at the original scale, the lower layer uses maxpooling to downsample and then inputs into a residual module, and finally uses upsampling to obtain a feature map with the same size as the upper layer to synthesize features;

[0032] The Hourglass uses a down-sampling method to reduce the dimension of the feature.

[0033] An improved CharNet-based character recognition system, in the CharNet network module, part of the pooling layer is removed and a dilated convolution with an expansion rate of rate=3 is used to obtain a larger feature map;

[0034] In the adaptive RoIAlign layer, the bandwidth B of the bilinear interpolation sampling interval is adaptively selected according to the size of the feature grid, and the size of the RoI, and the expression is:

[0035]

[0036] In the formula, width represents the width of the RoI after conv3; width' represents the width of the RoI after RoI Align; [·] represents an integral operator.

[0037] A character recognition system based on an improved CharNet, wherein an improved hard swish function is used as an activation function of a deep layer of the CharNet network module.

[0038] The expression of the hard swish function h-Swish is as follows:

[0039]

[0040] An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0041] The memory is used for storing a computer program.

[0042] The processor is used for executing the program stored on the memory, and realizes the method steps.

[0043] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize the method steps.

[0044] The beneficial effects of the present application are:

[0045] The present application has simple structure and can reserve more image features.

[0046] The present application replaces maximum pooling with adaptive RoIAlign for dimension reduction, effectively reduces feature loss and improves the response ability of the network to small targets.

[0047] The present application adopts an improved swish function, which effectively improves the model performance on a deep model BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings are provided. Figure 1 It is the method flow chart of the present application.

[0049] The accompanying drawings are provided. Figure 2 It is the network structure image of Hourglass of the present application.

[0050] The accompanying drawings are provided. Figure 3 It is the activation function image in the neural network.

[0051] The accompanying drawings are provided. Figure 4 It is the Relu function image.

[0052] The accompanying drawings are provided. Figure 5is a function image comparison chart of h-Swish and Swish.

[0053] attached Figure 6 is a detection effect diagram of the present application in a partial picture. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0055] A text recognition method based on an improved CharNet, the text recognition method comprising the following steps:

[0056] Step 1: obtaining an image data set to be detected;

[0057] Step 2: performing feature extraction of the image to be detected in step 1 by a backbone network, improving an Hourglass network of the backbone network by using adaptive RoIAlign, reducing loss of a feature map in a dimension reduction process, thereby improving network performance, and removing part of a pooling layer by using a dilated convolution with a rate = 3 to improve resolution, to obtain a larger feature map with reduced feature loss;

[0058] Step 3: inputting the feature map with reduced feature loss in step 2 into two parallel branches of a CharNet network respectively, the two parallel branches comprising a character branch for single character detection and recognition and a text instance detection branch for predicting a text instance bounding box, to obtain a text recognition result;

[0059] Step 4: using an improved hard swish function as an activation function of a deep layer of the network while steps 2 and 3 are being performed.

[0060] By the text recognition algorithm based on the improved CharNet, the problem of poor effect in detecting multi-scale text and text of arbitrary shape in text recognition is solved, and the algorithm has a simple structure and can retain more image features, thereby quickly solving the recognition problem of curved text.

[0061] A text recognition method based on an improved CharNet, step 2 of which improves an Hourglass network of the backbone network by using adaptive RoIAlign, reduces loss of a feature map in a dimension reduction process, thereby improving network performance, and obtains a feature map with reduced feature loss;

[0062] Hourglass improves the fineness of features by capturing multi-scale feature information and multiple intermediate supervised structures to realize high-order semantic feature extraction. The Hourglass network adopts a two-layer structure, the upper layer uses a residual module to collect features at the original scale, the lower layer uses maxpooling to downsample and then sends it to the residual module, and finally uses upsampling to obtain a feature map with the same size as the upper layer to synthesize features;

[0063] Hourglass uses down-sampling to reduce the dimension of features.

[0064] Hourglass extracts high-order semantic information from the input feature mapping at multiple scales; however, in order to obtain features of different scales, Hourglass uses down-sampling to reduce the dimension of features; the maxpooling layer causes a large amount of feature loss during feature dimension reduction, reducing the network's ability to extract and perceive features.

[0065] An improved CharNet-based character recognition method, step 2 removes part of the pooling layer and uses a dilated convolution with an expansion rate of rate = 3 to improve resolution and obtain larger feature maps;

[0066] Traditional RoIAlign uses nearby grid points on the feature map to calculate interpolation. If the sampling interval is too large, some useful information will be lost, for example, if only 2x2 = 4 feature points are taken from a 5x5 feature grid, some features will be missing; therefore, in the adaptive RoIAlign layer, the sampling interval is adaptively selected according to the size of the feature grid to solve the problem of feature loss caused by a too large sampling interval; generally, the bandwidth B of bilinear interpolation is related to the size of the RoI, and the expression is:

[0067]

[0068] In the formula, width represents the width of the RoI after conv3; width' represents the width of the RoI after RoI Align; [·] represents the integer operator.

[0069] Suppose that a 7x7 feature map is down-sampled to 3x3, then since 7 / 3 = 2.33333, 2x2 = 4 points are taken from each 2.3x2.3 grid; when the feature map becomes 14x14, it is also down-sampled to 3x3, then 14 / 3 = 4.66667, i.e. 4x4 = 16 points are taken from the 4.6x4.6 grid, which significantly reduces the feature loss caused by taking too few points.

[0070] A character recognition method based on improved CharNet, the step 4 uses an improved hardening swish function as the activation function of the deep layer of the network, specifically,

[0071] Since the result of each output of the neural network is a linear result, in order to introduce a nonlinear factor into the neural network so that it can fit any form of curve, the neural network must introduce an activation function. F(z) in the figure is the activation function of the neural network. If the activation function F(z) is missing, the output of the neural network is:

[0072] h=w1x1+w2x2+…w n x n (2)

[0073] That is, the input of each layer of the neural network is the linear output combination of the previous layer. The approximation ability of the neural network is limited, so a nonlinear function must be used as an activation function to increase the expression ability of the neural network. The activation function used by CharNet network is the classic Relu.

[0074] As Figure 4 is the Relu function image. The Relu function is:

[0075] Relu(x)=max(0,x) (3)

[0076] The essence of Relu is to get the larger one of the variable value and 0. This function solves the problem of gradient disappearance and has very fast calculation speed, and is superior to tanh function and Sigmoid function in convergence speed. However, since the negative half-axis of the Relu function is always 0, a part of neurons will not be activated, that is, part of the parameters will not be updated in the training process with a higher learning rate setting.

[0077] In order to solve the problem of Relu function, a new type of function is used as the activation function of the neural network.

[0078] Swish is a newly proposed activation function that has great advantages in improving network accuracy. The expression of Swish is:

[0079] Swish(x)=x*Sigmoid(x) (4)

[0080] The expression of Sigmoid(x) is:

[0081]

[0082] Although the use of Swish function can improve the accuracy of the network, due to the complexity of calculation, the calculation speed of the system is affected, therefore, the piecewise linear function h-Swish is used to approximate the Swish function to realize the replacement of the Swish function, the expression of h-Swish is:

[0083]

[0084] An improved CharNet-based character recognition system, the shape-robust text detection system comprises an input module and a CharNet network module;

[0085] The input module inputs the image to be detected into the CharNet network;

[0086] The CharNet network module extracts the backbone network features of the image to be detected, and obtains larger feature maps with reduced feature loss;

[0087] The Hourglass network of the backbone network is improved by adaptive RoIAlign, the loss of the feature map in the dimension reduction process is reduced, thereby improving the network performance, and part of the pooling layer is removed and the dilated convolution with rate=3 is used to improve the resolution;

[0088] The larger feature maps with reduced feature loss are input into two parallel branches of the CharNet network, the two parallel branches include a character branch for single character detection and recognition and a text instance detection branch for predicting the bounding box of the text instance, and the text recognition result is obtained.

[0089] An improved CharNet-based character recognition system, the CharNet network module, the Hourglass network of the backbone network is improved by adaptive RoIAlign, the loss of the feature map in the dimension reduction process is reduced, thereby improving the network performance, and the feature map with reduced feature loss is obtained;

[0090] Hourglass improves the fineness of features by capturing multi-scale feature information and the structure of multiple intermediate supervision to realize high-order semantic feature extraction. The Hourglass network adopts a two-layer structure, the upper layer uses a residual module to collect features at the original scale, the lower layer uses maxpooling to downsample and then sends it into a residual module, and finally uses upsampling to obtain a feature map with the same size as the upper layer to synthesize features;

[0091] Hourglass uses down-sampling to reduce the dimension of features.

[0092] A text recognition system based on an improved CharNet network module, wherein some pooling layers are removed from the CharNet network module and dilated convolutions with a dilation rate of 3 are used to improve resolution and obtain larger feature maps;

[0093] Traditional RoIAlign uses nearby grid points on the feature map to calculate interpolation. However, if the sampling interval is too large, some useful information will be lost. For example, taking only 2×2=4 feature points on a 5×5 feature grid will inevitably lead to feature loss. Therefore, the adaptive RoIAlign layer adaptively selects the sampling interval based on the feature grid size to solve the feature loss caused by excessively large sampling intervals. Generally speaking, the bandwidth B of bilinear interpolation is related to the size of the RoI, expressed as:

[0094]

[0095] In the formula, width represents the width of the RoI after conv3; width′ represents the width of the RoI after RoI Align; and [·] represents the rounding operator.

[0096] A text recognition system based on an improved CharNet network module employs a modified hardened swish function as the activation function for the deeper layers of the network.

[0097] The expression for the hardened swish function h-Swish is:

[0098]

[0099] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0100] Memory, used to store computer programs;

[0101] When a processor executes a program stored in memory, it implements the above method steps.

[0102] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0103] like Figure 6 The image shows the detection results of this algorithm in some images. It can effectively extract text information in complex scenes.

[0104] In order to verify the performance of the algorithm, multiple public benchmarks including ICDAR-2017, SCUT-CTW1500 are selected to test the accuracy and recognition speed. The test results are shown in Table 1:

[0105] Table 1 Comparison results of algorithm accuracy

[0106]

[0107]

[0108] In the table, E2E represents "End-to-End Recognition using F-measure. R represents Recall, P represents Precision, and F represents F-measure. The comparative algorithm CharNet H-57MS represents a multi-scale inference model using Hourglass57, and CharNet H-88MS represents a multi-scale inference model using Hourglass88. As can be seen from the data in the table, the algorithm in this paper performs well in multiple indicators such as Recall, Precision, F-measure, and achieves the highest E2E score.

Claims

1. An improved CharNet-based method for character recognition, characterized in that, The character recognition method comprises the following steps: Step 1: obtaining an image data set to be detected; Step 2: performing feature extraction on the image to be detected in step 1 by using a backbone network, improving the Hourglass network of the backbone network by using adaptive RoI Align, and removing part of the pooling layer to obtain a larger feature map by using a dilated convolution with an expansion rate of rate=3; Step 3: inputting the feature map with reduced feature loss obtained in step 2 into two parallel branches of a CharNet network respectively to obtain a text recognition result; Step 4: using an improved hard swish function as an activation function of a deep layer of the network while steps 2 and 3 are being performed; In step 2, the Hourglass network of the backbone network is improved by using adaptive RoI Align to obtain a feature map with reduced feature loss; The Hourglass network adopts a two-layer structure, the upper layer uses a residual module to collect features at the original scale, the lower layer uses max pooling to down-sample and then inputs the residual module, and finally uses up-sampling to obtain a feature map with the same size as the upper layer for feature synthesis; The Hourglass network uses a down-sampling method to reduce the dimension of features; In step 2, part of the pooling layer is removed to obtain a larger feature map by using a dilated convolution with an expansion rate of rate=3; In the adaptive RoI Align layer, the sampling interval bandwidth B of the bilinear interpolation is adaptively selected according to the size of the feature grid, and the size of the RoI, expressed as: In the formula, width represents the width of the RoI after conv3; widrh' represents the width of the RoI after RoIAlign; and [·] represents an integer operator. 2.The method of claim 1, wherein, In step 4, an improved hard swish function is used as an activation function of a deep layer of the network, specifically, The expression of Swish is: Swish(x)=x*Sigmoid(x) (4) The expression of Sigmoid(x) is: A segmented linear function h-Swish is used to approximate the Swish function to replace the Swish function, and the expression of h-Swish is:

3. A character recognition system based on improved CharNet, characterized in that, The character recognition system comprises an input module and a CharNet network module; The input module inputs an image to be detected into the CharNet network; The CharNet network module performs feature extraction on the image to be detected by using a backbone network to obtain a larger feature map with reduced feature loss; The larger feature map with reduced feature loss is input into two parallel branches of the CharNet network respectively to obtain a text recognition result; The CharNet network module improves the Hourglass network of the backbone network by using adaptive RoI Align to obtain a feature map with reduced feature loss; The Hourglass network adopts a two-layer structure, the upper layer uses a residual module to collect features at the original scale, the lower layer uses max pooling to down-sample and then inputs the residual module, and finally uses up-sampling to obtain a feature map with the same size as the upper layer for feature synthesis; Hourglass uses down-sampling to reduce the dimension of features; In the CharNet network module, the part of the pooling layer is removed and the dilated convolution with rate = 3 is used to obtain larger feature maps; In the adaptive RoI Align layer, the bandwidth B of the bilinear interpolation of the sampling interval is adaptively selected according to the size of the feature grid, and the size of the RoI is related, and the expression is as follows: In the formula, width represents the width of the RoI after the conv3; Width' represents the width of the RoI after the RoIAlign; [·] represents the integral operator.

4. The improved CharNet-based character recognition system of claim 3, wherein, In the CharNet network module, an improved hardening swish function is used as the activation function of the deep network, and the specific expression is as follows, The expression of the hardening swish function h-Swish is as follows:

5. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps of any one of claims 1-2. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps of any one of claims 1-2. ​ 6. A computer-readable storage medium, characterized in that, ​

Images