Shape-robust text detection method and system based on improved charnet

Abstract

The application provides a shape robustness text detection method based on improved CharNet and a system thereof. Step 1, inputting an image to be detected into a CharNet network; step 2, performing backbone network feature extraction on the image to be detected to form a feature map; step 3, inputting the feature map into two parallel branches of the CharNet network respectively, wherein the two parallel branches include a character branch for single character detection and recognition and a text instance detection branch for predicting a text instance bounding box; step 4, obtaining character recognition information based on the character branch for single character detection and recognition; step 5, obtaining character position information based on the text instance detection branch for predicting a text instance bounding box; and step 6, integrating the character recognition information and the character position information to obtain a text recognition result. The method is used to solve the problems of extremely long text and arbitrary shape text that are difficult to identify in text recognition.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of character recognition, and particularly relates to a shape-robust text detection method based on an improved CharNet and a system thereof. BACKGROUND

[0002] Scene text recognition has attracted much attention due to its applications in automated office work, scene understanding, and autonomous driving. Recently, with the rise of deep learning techniques, general object detection techniques have been developed rapidly. However, there are significant differences between the objects of the text task and general detection objects, and most existing detection algorithms can only achieve good results in an environment with regular shapes and controlled aspect ratios. Natural image text recognition has always been considered as two consecutive but independent problems: text detection and text recognition. Due to the success of deep neural networks, both tasks have made great progress. The purpose of text detection is to predict a text box for each text instance. The current best text detection method is extended from the object detection or segmentation framework. Based on the results of text detection, the goal of text recognition is to recognize a sequence of characters from the text picture that is cut out. Generally, text recognition is converted into a sequence labeling problem, which is usually solved by a recurrent neural network (RNN) based on convolutional neural network (CNN) features. This solution is currently the best. However, text recognition through a two-stage method usually has many limitations. First, learning the two tasks independently is a suboptimal problem because it is difficult to fully utilize the nature of the text itself. For example, joint text detection and recognition can provide rich contextual information and the two tasks can complement each other. Recent work has shown that such joint learning can improve the accuracy of both. Second, the two-stage method usually requires multiple consecutive steps, which makes the entire system complex and makes the recognition result largely dependent on the detection performance.

[0003] Recently, many methods have tried to develop a unified text detection and recognition framework (end-to-end detection) by adding an RNN branch to the text detection model to achieve end-to-end recognition, and good performance has been achieved. However, its essence still cannot escape the limitations of RoI Pooling and RNN in the two-stage detection framework, which inevitably affects the performance of recognition. SUMMARY

[0004] The application provides a shape-robust text detection method based on an improved CharNet, which solves the problems of extremely long text and arbitrary shape text that are difficult to identify in text recognition.

[0005] This invention provides a shape-robust text detection system based on an improved CharNet, which solves the problem of extremely long text and text of arbitrary shape that are difficult to recognize in text recognition.

[0006] This invention provides an electronic device.

[0007] This invention provides a computer-readable storage medium.

[0008] This invention is achieved through the following technical solution:

[0009] A shape-robust text detection method based on an improved CharNet, the method comprising the following steps:

[0010] Step 1: Input the image to be detected into the CharNet network;

[0011] Step 2: Extract the backbone network features from the image to be detected to form a feature map;

[0012] Step 3: Input the feature maps from Step 2 into two parallel branches of the CharNet network, namely, a character branch for single character detection and recognition and a text instance detection branch for predicting the bounding box of text instances;

[0013] Step 4: Based on the single-character detection and recognition in Step 3, obtain character recognition information;

[0014] Step 5: Obtain character position information from the text instance detection branch based on the predicted text instance bounding box from Step 3;

[0015] Step 6: After integrating the character recognition information from Step 4 and the character position information from Step 5, the text recognition result is obtained.

[0016] A shape-robust text detection method based on an improved CharNet, wherein step 2 improves the ResNet and Hourglass networks in the backbone network feature extraction, specifically as follows:

[0017] The improvements to the ResNet network include the following steps:

[0018] Step R2.1: Segment the feature map and decompose and reuse the features within the residual;

[0019] Step R2.2 involves using different convolution kernels to obtain a larger receptive field than a single convolution kernel.

[0020] A shape-robust text detection method based on an improved CharNet divides the resulting feature map into several equal parts after passing it through a 1*1 convolutional kernel of ResNet. Taking a 4-part division as an example...Figure 2 The route shown is used for data fusion. x1 is directly fed into y1, x2 goes through a 3*3 convolution kernel and is then fed into y2, x3 is merged with y2 and then goes through a 3*3 convolution kernel and is then fed into y3, x4 is merged with y3 and then goes through a 3*3 convolution kernel and is then fed into y4. Finally, y1, y2, y3 and y4 are combined to form a new feature map and are then fed into the next round of convolution operation.

[0021] A shape-robust text detection method based on an improved CharNet, assuming x i For input data, K i () represents a 3x3 convolution operation, then y i It can be represented as:

[0022]

[0023] The output of Res2Net modules includes combinations of different receptive field sizes. Res2Net proposes a scale-based approach to network performance, based on depth, width, and cardinality dimensions, to further optimize network performance without increasing computational load.

[0024] A shape-robust text detection method based on an improved CharNet, wherein the improvement of the Hourglass network includes the following steps:

[0025] Step H2.1 uses convolution with a stride of 2 and a size of 1*1 to learn the non-linearity provided by pooling, thereby achieving feature dimensionality reduction;

[0026] Step H2.2, improve the symmetrical structure of the Hourglass trunk to an asymmetrical structure.

[0027] A shape-robust text detection method based on an improved CharNet, wherein step H2.1 specifically involves Hourglass improving the fineness of features by capturing multi-scale feature information and multiple intermediate supervision structures; the network adopts a two-layer structure, the upper layer uses a residual module to collect features at the original scale, and the lower layer uses max pooling to downsample and then feeds the samples into the residual module, and finally uses upsampling to obtain a feature map of the same size as the upper layer for feature synthesis;

[0028] Hourglass extracts high-order semantic information from input feature maps at multiple scales; however, to obtain features at different scales, Hourglass uses downsampling for feature dimensionality reduction. Downsampling causes significant feature loss during feature dimensionality reduction, reducing the network's ability to extract and perceive features. When the number of network layers is shallow, the nonlinear changes provided by pooling can meet the network's needs, but as the number of network layers increases, fixed nonlinear changes become insufficient. Therefore, convolutions with a stride of 2 and a size of 1*1 are used to learn the nonlinearity provided by pooling, and even better nonlinearity can be learned from the training set. Compared to pooling, Hourglass can retain more detailed information.

[0029] A shape-robust text detection method based on an improved CharNet, wherein step H2.2 specifically involves improving the symmetrical structure of the Hourglass trunk to an asymmetrical structure, first abstracting and shrinking the feature map to enhance its expressive power, and then fusing and enlarging it to obtain more accurate coordinate information of the object.

[0030] A shape-robust text detection system based on an improved CharNet, comprising an input module, a CharNet extraction module, and a result fusion module.

[0031] Try the input module and input the image to be detected into the CharNet network;

[0032] The CharNet extraction module extracts backbone network features from the image to be detected to form a feature map.

[0033] The feature maps are input into two parallel branches of the CharNet network. The two parallel branches include a character branch for single character detection and recognition and a text instance detection branch for predicting the bounding box of text instances. The character content is processed, and the text instance detection branch for predicting the bounding box of text instances recognizes the character position information.

[0034] The character branch of single-character detection and recognition yields character recognition information;

[0035] The character position information is obtained from the text instance detection branch of the predicted text instance bounding box;

[0036] The result fusion module integrates character recognition information and character position information to obtain text recognition results.

[0037] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0038] Memory, used to store computer programs;

[0039] The processor, when executing a program stored in memory, implements the above method steps.

[0040] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0041] The beneficial effects of this invention are:

[0042] This invention uses a single-stage CharNet for the joint detection and recognition process, avoiding the impact of RoI and RNN recognition branches.

[0043] This invention improves ResNet by feature grouping, enhancing its performance in multi-scale tasks.

[0044] This invention reduces the loss of features by the max pooling layer through adaptive convolution dimensionality reduction, thereby improving the network's ability to distinguish character models. Attached Figure Description

[0045] Figure 1 This is a flowchart of the method of the present invention.

[0046] Figure 2 This is the network structure diagram of the CharNet of this invention.

[0047] Figure 3 This is a diagram of the residual network structure of the improved ResNet model of this invention.

[0048] Figure 4 This invention relates to the asymmetric Hourglass network structure.

[0049] Figure 5 This is a diagram showing the effect of the algorithm of this invention in recognizing text information in an image. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention proposes a shape-robust text detection method based on an improved CharNet. This method adopts a single-stage end-to-end detection approach with individual characters as units. It uses the feature grouping idea of ​​the improved ResNet to split and reuse the features within the residual, thereby improving the network's ability to extract and perceive detailed features. Convolutional dimensionality reduction is used instead of pooling dimensionality reduction within the improved Hourglass, and the feature loss caused by dimensionality reduction is reduced by adaptively learning parameters.

[0052] This improved CharNet-based shape-robust text detection method effectively solves the problem of recognizing extremely long and arbitrarily shaped texts in text recognition, improves the network's performance in acquiring local features, and strengthens the CharNet network's ability to distinguish single characters.

[0053] like Figure 2 The diagram shows the network structure of CharNet in this invention. The algorithm mainly consists of two parallel branches. One branch is used for single-character detection and recognition, and the other branch is used for predicting the bounding box of the text instance. The two branches seamlessly form an end-to-end training model, realizing the detection and recognition of characters in one forward propagation.

[0054] A shape-robust text detection method based on an improved CharNet, the method comprising the following steps:

[0055] Step 1: Input the image to be detected into the CharNet network;

[0056] Step 2: After extracting the backbone network features from the image to be detected, a feature map is formed. The ResNet network and Hourglass network in the backbone network feature extraction are improved.

[0057] Step 3: Input the feature maps from Step 2 into two parallel branches of the CharNet network, namely, a character branch for single character detection and recognition and a text instance detection branch for predicting the bounding box of text instances;

[0058] Step 4: Based on the single-character detection and recognition in Step 3, obtain character recognition information;

[0059] Step 5: Obtain character position information from the text instance detection branch based on the predicted text instance bounding box from Step 3;

[0060] Step 6: After integrating the character recognition information from Step 4 and the character position information from Step 5, the text recognition result is obtained.

[0061] A shape-robust text detection method based on an improved CharNet, wherein step 2 improves the ResNet and Hourglass networks in the backbone network feature extraction, specifically as follows:

[0062] The improvements to the ResNet network include the following steps:

[0063] Step R2.1: Segment the feature map and decompose and reuse the features within the residual;

[0064] Step R2.2 involves using different convolution kernels to perform calculations, resulting in a larger receptive field than a single convolution kernel.

[0065] The improvement of the backbone network is to address the computational pressure and high latency caused by complex calculations required to achieve higher feature extraction performance.

[0066] A shape-robust text detection method based on an improved CharNet divides the resulting feature map into several equal parts after passing it through a 1*1 convolutional kernel of ResNet. Taking a 4-part division as an example... Figure 2 The route shown is used for data fusion. x1 is directly fed into y1, x2 goes through a 3*3 convolution kernel and is then fed into y2, x3 is merged with y2 and then goes through a 3*3 convolution kernel and is then fed into y3, x4 is merged with y3 and then goes through a 3*3 convolution kernel and is then fed into y4. Finally, y1, y2, y3 and y4 are combined to form a new feature map and are then fed into the next round of convolution operation.

[0067] A shape-robust text detection method based on an improved CharNet, assuming x i For input data, K i () represents a 3x3 convolution operation, then y i It can be represented as:

[0068]

[0069] Combining the structure shown in the figure above and the above formula, it can be seen that due to this split hybrid connection structure, the output of Res2Net modules contains combinations of different receptive field sizes. This structure is beneficial for extracting global and local information. At the same time, convolution operations are avoided when i=1, reducing the overall computational pressure. Meanwhile, some features of x can be retained for feature reuse, improving the influence of global features on the feature map and improving model performance. Based on the dimensions of depth, width, and cardinality, Res2Net proposes a scale-based factor on the network's influence, further optimizing network performance without increasing computational load.

[0070] A shape-robust text detection method based on an improved CharNet, wherein the improvement of the Hourglass network includes the following steps:

[0071] It solves the problem of significant feature loss caused by downsampling during feature dimensionality reduction, effectively preserving more detailed information of the features;

[0072] Step H2.1 uses convolution with a stride of 2 and a size of 1*1 to learn the non-linearity provided by pooling, thereby achieving feature dimensionality reduction;

[0073] Step H2.2, improve the symmetrical structure of the Hourglass trunk to an asymmetrical structure.

[0074] A shape-robust text detection method based on an improved CharNet, wherein step H2.1 specifically involves Hourglass improving the fineness of features by capturing multi-scale feature information and multiple intermediate supervision structures to achieve high-order semantic feature extraction; the network adopts a two-layer structure, the upper layer uses a residual module to collect features at the original scale, and the lower layer uses max pooling to downsample and then feeds the samples into the residual module, and finally uses upsampling to obtain a feature map of the same size as the upper layer for feature synthesis;

[0075] Hourglass extracts high-order semantic information from input feature maps at multiple scales; however, to obtain features at different scales, Hourglass uses downsampling for feature dimensionality reduction. Downsampling causes significant feature loss during feature dimensionality reduction, reducing the network's ability to extract and perceive features. When the number of network layers is shallow, the nonlinear changes provided by pooling can meet the network's needs, but as the number of network layers increases, fixed nonlinear changes become insufficient. Therefore, convolutions with a stride of 2 and a size of 1*1 are used to learn the nonlinearity provided by pooling, and even better nonlinearity can be learned from the training set. Compared to pooling, Hourglass can retain more detailed information.

[0076] A shape-robust text detection method based on an improved CharNet, wherein step H2.2 specifically involves improving the symmetrical structure of the Hourglass trunk to an asymmetrical structure, first abstracting and shrinking the feature map to enhance its expressive power, and then fusing and enlarging it to obtain more accurate coordinate information of the object.

[0077] A shape-robust text detection system based on an improved CharNet, the shape-robust text detection system comprising an input module, a CharNet extraction module, and a result fusion module;

[0078] Try the input module and input the image to be detected into the CharNet network;

[0079] The CharNet extraction module extracts backbone network features from the image to be detected to form a feature map, and improves the ResNet and Hourglass networks in the backbone network feature extraction.

[0080] The feature maps are input into two parallel branches of the CharNet network. The two parallel branches include a character branch for single character detection and recognition and a text instance detection branch for predicting the bounding box of text instances. The character content is processed, and the text instance detection branch for predicting the bounding box of text instances recognizes the character position information.

[0081] The character branch of single-character detection and recognition yields character recognition information;

[0082] The character position information is obtained from the text instance detection branch of the predicted text instance bounding box;

[0083] The result fusion module integrates character recognition information and character position information to obtain text recognition results.

[0084] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0085] Memory, used to store computer programs;

[0086] The processor, when executing a program stored in memory, implements the above method steps.

[0087] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0088] Figure 5 This demonstrates the effectiveness of the algorithm in recognizing text information in images. To prove the effectiveness of this invention, a control experiment was designed to show that the proposed algorithm achieves state-of-the-art (SOTA) performance on public benchmarks. Figure 5 As can be seen, the algorithm proposed in this paper has the ability to recognize text information in images.

[0089] To verify the performance of the algorithm presented in this paper, multiple public benchmarks, including ICDAR-2017 and SCUT-CTW1500, were selected for accuracy and recognition speed testing. The test results are shown in Table 1.

[0090] Table 1 Comparison of Algorithm Accuracy Results

[0091]

[0092] In the table, E2E stands for "End-to-End Recognition using F-measure." R represents Recall, P represents Precision, and F represents F-measure. The comparison algorithms CharNet H-57MS represent the multi-scale inference model using Hourglass57, and CharNet H-88MS represent the multi-scale inference model using Hourglass88. The data in the table shows that the algorithm presented in this paper performs excellently across multiple metrics, including Recall, Precision, and F-measure, and achieves the highest E2E score.

Claims

1. A shape-robust text detection method based on an improved CharNet, characterized in that, The detection method includes the following steps: Step 1: Input the image to be detected into the CharNet network; Step 2: Extract the backbone network features from the image to be detected to form a feature map; Step 3: Input the feature maps from Step 2 into two parallel branches of the CharNet network. The two parallel branches include a character branch for single character detection and recognition and a text instance detection branch for predicting the bounding box of text instances. Step 4: Based on the single-character detection and recognition in Step 3, obtain character recognition information; Step 5: Obtain character position information from the text instance detection branch based on the predicted text instance bounding box from Step 3; Step 6: After integrating the character recognition information from Step 4 and the character position information from Step 5, the text recognition result is obtained; The improvement of the ResNet and Hourglass networks in the backbone network feature extraction in step 2 is as follows: The improvements to the ResNet network include the following steps: Step R2.1: Segment the feature map and decompose and reuse the features within the residual; Step R2.2 involves using different convolution kernels to perform calculations, resulting in a larger receptive field than a single convolution kernel. The improvement to the Hourglass network includes the following steps: Step H2.1 uses convolution with a stride of 2 and a size of 1*1 to learn the non-linearity provided by pooling, thereby achieving feature dimensionality reduction; Specifically, step H2.1 involves Hourglass improving the fineness of features by capturing multi-scale feature information and multiple intermediate supervision structures. The network adopts a two-layer structure: the upper layer uses a residual module to collect features at the original scale, and the lower layer uses max pooling to downsample and then feeds the samples into the residual module. Finally, upsampling is used to obtain a feature map of the same size as the upper layer for feature synthesis. Step H2.2, improve the symmetrical structure of the Hourglass trunk to an asymmetrical structure; Specifically, step H2.2 involves improving the symmetrical structure of the Hourglass trunk to an asymmetrical structure by first abstracting and shrinking the feature map to enhance its expressive power, and then fusing and enlarging it to obtain more accurate coordinate information of the object.

2. The shape-robust text detection method based on an improved CharNet according to claim 1, characterized in that, After passing through a 1x1 convolution kernel in ResNet, the resulting feature map is divided into several parts. x1 is directly fed into y1, x2 goes through a 3x3 convolution kernel and is then fed into y2, x3 is merged with y2 and then fed into y3 through a 3x3 convolution kernel, x4 is fused with y3 and then fed into y4 through a 3x3 convolution kernel, and finally y1, y2, y3, and y4 are combined to form a new feature map, which is then fed into the next round of convolution operations.

3. The shape-robust text detection method based on an improved CharNet according to claim 2, characterized in that, Assume x i For input data, K i ( ) represents a 3x3 convolution operation, then y i It can be represented as: The output of Res2Net modules includes combinations of different receptive field sizes. Res2Net proposes a scale-based approach to network performance, based on depth, width, and cardinality dimensions, to further optimize network performance without increasing computational load.

4. The shape-robust text detection method based on an improved CharNet according to claim 1, characterized in that, Hourglass extracts high-order semantic information from the input feature maps at multiple scales; however, in order to obtain features at different scales, Hourglass uses downsampling to reduce feature dimensionality. Downsampling causes a large amount of feature loss during feature dimensionality reduction, reducing the network's ability to extract and perceive features.

5. A shape-robust text detection system based on an improved CharNet, characterized in that, The shape robust text detection system uses a shape robust text detection method based on an improved CharNet as described in claim 1. The shape robust text detection system includes an input module, a CharNet network module, and a result fusion module. The input module inputs the image to be detected into the CharNet network; The CharNet network module extracts backbone network features from the image to be detected to form a feature map. The feature maps are input into two parallel branches of the CharNet network. The two parallel branches include a character branch for single character detection and recognition and a text instance detection branch for predicting the bounding box of text instances. The character content is processed, and the text instance detection branch for predicting the bounding box of text instances recognizes the character position information. The character branch of single-character detection and recognition yields character recognition information; The character position information is obtained from the text instance detection branch of the predicted text instance bounding box; The result fusion module integrates character recognition information and character position information to obtain text recognition results.

6. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-4.

Images