Arbitrary shape text detection method based on edge feature enhancement

Abstract

The present application belongs to the technical field of image processing, and particularly relates to an arbitrary shape text detection method based on edge feature enhancement, which mainly solves the problem of inaccurate text object detection of the existing deep learning-based text detection method. The present application takes the existing deep learning-based text detection method as a text detection branch, first adds a text edge region detection branch to improve the detection capability of the model for the text edge region, then enables the model to extract more shallow features related to the edge region through a shallow feature enhancement module, and finally, the model highlights the relevant features in the features of the two detection branches through an attention mechanism through a branch feature fusion module, and adjusts the two detection branches after fusion. The present application has the advantage of improving the text detection accuracy by enhancing the extraction of text edge-related features by the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an arbitrary shape text detection method based on edge feature enhancement in an arbitrary scene. BACKGROUND

[0002] Traditional scene text detection methods mainly use the shape, edge and other features of the text region to detect the text region, and compared with the traditional detection method, the detection method based on deep learning can extract more relevant and deeper features of the text object, especially the multi-scale complex features, since the accuracy of scene text detection is closely related to the feature extraction of the text region, the detection method based on deep learning can often obtain higher text detection accuracy.

[0003] The text object in a natural scene usually has different shapes, and accurately detecting the arbitrary shape text is the key to improving the text detection accuracy in the detection process of the text object. Therefore, on the basis of the two text detection methods of regression and segmentation, many research works improve the method for detecting the arbitrary shape text object. In terms of the detection method based on regression, the related research usually improves the text bounding box to be regressed, a common method is to improve the representation of the text bounding box, and another method is to adjust the text box obtained initially using a sub-network according to the text region features; and in terms of the detection method based on semantic segmentation, the accuracy of text region detection is improved mainly by increasing the discrimination between different categories of pixels (between the text instance pixels and between the text instance pixels and the background pixels) when performing segmentation, one improvement method gradually expands the pixels or regions with high discrimination on the basis of the segmented text instance core region to obtain accurate text instances, and some methods embed the pixel features into a new space to increase the cohesiveness of the instance pixels.

[0004] In the deep learning-based model, the extraction of target edge information is usually closely related to the extraction of pixel features, and the attention to the target edge information can help the model to extract the pixel features that are easier to detect the edge region. Meanwhile, the edge detection result of the text region and the pixel features obtained by the backbone network affect the text detection accuracy. Therefore, in the deep learning-based text detection method, more edge information is introduced to better extract the pixel features related to the text boundary from the original image, so as to increase the discrimination between different text instances and between the text instance and the background, and further improve the detection accuracy of the arbitrary text region.

[0005] The present application is an arbitrary shape text detection method based on edge feature enhancement, which can further improve the accuracy of arbitrary shape scene text detection. SUMMARY

[0006] The application provides an arbitrary shape text detection method based on edge feature enhancement, which enhances edge features in a feature extraction process to improve the accuracy of scene text detection.

[0007] The specific technical scheme of the application comprises the following steps:

[0008] Step A: obtain an image set containing text object boundary box labels as a training sample set, and pre-process each image in the training sample set;

[0009] Step B: train the training sample of step A by using a deep learning model based on a residual network to obtain a trained text detection learning model;

[0010] Step C: according to the trained learning model of step B, predict the training set images without labels to obtain a text region segmentation graph and a kernel region segmentation graph, and use a PSE algorithm to obtain the final text region prediction result through gradual expansion.

[0011] The pre-processing of each image in the training sample set in step A is specifically:

[0012] Step A1: use a polygon clipping algorithm to obtain a shrinked text box and an expanded text box of the original text region label;

[0013] Step A2: obtain a complete text region label segmentation graph according to the original text box label, a kernel region label segmentation graph according to the shrinked boundary box, and a text edge region label segmentation graph according to the shrinked text box and the expanded text box;

[0014] Step A3: randomly cut the images in the training sample into images with a fixed size of 640*640 pixels, then perform random flipping in the horizontal direction, random scaling and random selection, and normalization processing;

[0015] The deep learning model based on a residual network (ResNet) in step B is specifically:

[0016] Step B1: use the sample image pre-processed in step A as the input of ResNet to obtain a ResNet feature map F R , and use a feature pyramid (FPN) to preliminarily fuse the features to obtain a feature F f1 ;

[0017] Step B2: the shallow feature enhancement module first uses convolution to process the feature F f1 to obtain a deep feature F h of each pixel in a larger scale, and then uses the deep feature F h to obtain a fused feature Ff1 The feature F is peeled off in a subtractive manner to retain as much shallow information as possible l . Then, the feature F l is spliced and convoluted with the feature F P2 with the highest resolution of the FPN to enhance the shallow features to obtain the feature F le . Finally, the feature F le is added to the feature F h pixel by pixel as the image feature F e for subsequent text detection;

[0018] Step B3, the feature F e is processed by different convolution to construct features for text instances and their edges, to obtain the text region feature F t and the text edge region feature F b . In order to highlight the most representative features and suppress irrelevant features, the attention mechanism module is used to process the features F t and F b respectively to give different weights to different features, and the weights are obtained through the SE (Squeeze and Extract) channel attention module. After the attention mechanism processing, the features F t and F b are fused through splicing and twice convolution to obtain a hybrid feature F m containing information of both;

[0019] Step B4, in the complete text instance detection branch, first, the feature F t is used to obtain the text instance region segmentation result using the method of convolution first and then activation. Then, an adaptive binarization method is used to filter the pixels that are incorrectly segmented in the initial segmentation result. First, the feature F m is used to obtain the threshold T for each pixel, and then the initial segmentation result and the threshold T are fused using the binarization method to obtain the final text instance region segmentation result, and the specific formula is: where B is the text segmentation map, and T is the text threshold;

[0020] Step B5, in the text instance edge detection branch, first, the feature F b is used to obtain the initial edge region segmentation result using the method of convolution first and then activation. Then, the hybrid feature information and part of the initial segmentation result information of the text instance region are fused on the basis of the initial segmentation result. For the hybrid feature information, the method of convolution first and then activation is also used to obtain the probability information of each pixel belonging to the edge region. For the initial segmentation result of the text instance region, the method in RPCNet is used to obtain the edge information from the initial segmentation result of the text instance region by calculating the spatial gradient, and the specific formula is: where σ is an activation function, pool 3*3 is an adaptive average pooling operation with kernel size 3, and M is the Text Map. Finally, the three information are spliced, convolved and activated to obtain the final text edge prediction result;

[0021] Step B6, in order to solve the problem of unbalanced positive and negative samples in the training process, first, the OHEM method is used to mask the segmentation result and the actual label according to the positive and negative sample ratio, and the masked result is S and G, then the Dice loss function is used to calculate the loss, and the specific formula is: where S x，y and G x，y are the pixel values of the segmentation result and the actual label mask.

[0022] The gradual expansion algorithm in step C is:

[0023] Step C1, the connected region of the minimum scale kernel segmentation graph S1 is obtained to obtain the core area of different text instances;

[0024] Step C2, on the basis of each core area of S1, the text pixels in the neighborhood are merged to obtain S'2, wherein the merging manner is that in the edge pixels of the core area, according to the classification result in S2, the four neighborhood pixels belonging to the text pixels are expanded and merged into the corresponding core area of S1. Finally, the same expansion and merging operation is used to obtain S'3... S'N in turn. n and S'N-1 are taken as the final text prediction result. n . BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is the flow of an embodiment of the arbitrary shape text detection method based on edge feature enhancement of the application;

[0026] Figure 2 is a schematic diagram of a complete model of the arbitrary shape text detection method based on edge feature enhancement of the application;

[0027] Figure 3 is a schematic diagram of a shallow feature enhancement module of the arbitrary shape text detection method based on edge feature enhancement of the application;

[0028] Figure 4 is a schematic diagram of a branch feature fusion module of the arbitrary shape text detection method based on edge feature enhancement of the application;

[0029] Figure 5 is a schematic diagram of a complete text region detection branch and a text edge region detection branch of the arbitrary shape text detection method based on edge feature enhancement of the application; Detailed Implementation

[0030] like Figure 1 As shown in this embodiment, an arbitrary shape text detection method based on edge feature enhancement is described. The method includes the following three steps:

[0031] Step A: Obtain the set of images containing the bounding box markers of the text objects as the training sample set, and preprocess each image in the training sample set;

[0032] Step B: Train the training samples described in Step A using a deep learning model based on residual networks to obtain a trained text detection learning model;

[0033] Step C: For the unlabeled training set images, predict them according to the learning model trained in Step B to obtain text region segmentation maps and kernel region segmentation maps, and use the PSE algorithm to obtain the final text region prediction results through progressive expansion.

[0034] like Figure 2 , Figure 3 , Figure 4 , Figure 5 As shown, step A, which involves preprocessing each image in the training sample set, specifically includes:

[0035] Step A1: Use the polygon clipping algorithm to obtain the shrinking and expanding text boxes of the original text region markings;

[0036] Step A2: Obtain the complete text region marking segmentation map based on the original marked text box; obtain the kernel region marking segmentation map based on the shrunken bounding box; and obtain the text edge region marking segmentation map based on the shrunken and expanded text boxes.

[0037] Step A3: Randomly crop the images in the training samples into images of a fixed size of 640×640 pixels, then randomly flip them horizontally, and randomly scale and select them, and then normalize them.

[0038] The deep learning model based on ResNet in step B is specifically as follows:

[0039] Step B1: Use the preprocessed sample image from step A as input to ResNet to obtain the ResNet feature map F. R And use the Feature Pyramid Network (FPN) to perform preliminary feature fusion to obtain feature F. f1 ;

[0040] Step B2: The shallow feature enhancement module first uses convolution to process feature F. oObtain the deep features F of each pixel at a larger scale h , to make deep features F h From the fused feature F f1 The feature F is obtained by subtracting to extract as much shallow information as possible. l Subsequently, feature F l The highest resolution feature F of the same FPN P2 Concatenation and convolution are performed to enhance shallow features to obtain feature F. le Finally, feature F le With feature F h The pixel-by-pixel summation serves as the image feature F for subsequent text detection. e ;

[0041] Step B3, transfer feature F e By employing different convolutional processes to construct features oriented towards text instances and their edges, we obtain the text region features F. t and text edge region features F b To highlight the most representative features and suppress irrelevant features, feature F is analyzed separately. t and F b An attention mechanism module is used to assign different weights to different features, where the weights are obtained through the SE (Squeeze and Extract) channel attention module. The feature F processed by the attention mechanism... t and F b Then, by concatenation and two convolutions, the features are fused to obtain a hybrid feature F that contains information from both. m ;

[0042] Step B4: In the complete text instance detection branch, first, based on feature F... t The text instance region segmentation result is obtained using a convolution followed by activation method; then, an adaptive binarization method is used to filter out incorrectly segmented pixels in the initial segmentation result; where feature F is first used. m The threshold T is adjusted item by item, and then a binarization method is used to fuse the initial segmentation result with the threshold T to obtain the final text instance region segmentation result. The specific formula is as follows: Where B is the text segmentation image and T is the text threshold;

[0043] Step B5: In the text instance edge detection branch, first, based on feature F... bInitial edge region segmentation results are obtained using a convolution-then-activation approach. Subsequently, hybrid feature information and partial initial segmentation results of text instance regions are fused into the initial segmentation results. For the hybrid feature information, a convolution-then-activation approach is also used to obtain the probability information of each pixel belonging to an edge region. For the initial segmentation results of text instance regions, the method in RPCNet is used to obtain edge information from the initial segmentation results of text instance regions by calculating spatial gradients. The specific formula is as follows: Where σ is the activation function, pooling 3*3 An adaptive average pooling operation with a kernel size of 3 is performed, where M is the text map. Finally, the three pieces of information are concatenated, convolved, and activated to obtain the final text edge prediction result.

[0044] Step B6: To address the imbalance between positive and negative samples during training, the OHEM method is first used to mask the segmentation results and the actual labels based on the ratio of positive to negative samples, resulting in the masked results S and G. Then, the Dice loss function is used to calculate the loss, with the specific formula as follows: Where S x，y and G x，y It is divided into segmentation results and pixel values ​​after the actual label mask.

[0045] The progressive expansion algorithm described in step C is as follows:

[0046] Step C1: Find the connected regions of the kernel segmentation graph S1 at the smallest scale to obtain the core regions of different text instances;

[0047] Step C2: Based on the core regions of S1, merge the text pixels in their neighborhoods to obtain S′2. The merging method involves expanding and merging the four neighboring pixels belonging to the text pixels in the edge pixels of the core regions into the corresponding core regions of S1, according to their classification results in S2. Finally, use the same expansion and merging operation to obtain S′3...S′ sequentially. n and S′ n This serves as the final text prediction result.

Claims

1. An arbitrary shape text detection method based on edge feature enhancement, characterized in that, The method comprises the following three steps: Step A: obtaining an image set containing text object bounding box labels as a training sample set, and pre-processing each image in the training sample set; Step B: training the training sample set using a deep learning model based on a residual network ResNet to obtain a trained text detection learning model; The deep learning model based on the residual network described in step B is specifically: a deep learning model based on a residual network Step B1: Use the preprocessed sample image from step A as... The input, get Feature map and using feature pyramid Preliminary feature fusion is performed to obtain features ; Step B2: The shallow feature enhancement module first uses convolution to process the features. Obtain deep features of each pixel at a larger scale Deep features From the characteristics after fusion The subtraction method is used to peel away the surface layer to obtain features that retain as much shallow information as possible. Subsequently, the features same Highest resolution features Perform concatenation and convolution to enhance shallow features and obtain features. Finally, the features With features Pixel-by-pixel summation as image features for subsequent text detection ; Step B3, the features By different convolution processing to construct the text instance and its edge features, the text region features and text edge region features ; In order to highlight the most representative features and suppress irrelevant features, the features and Use attention mechanism module to process different features to give different weights, and the weight is obtained by SE channel attention module; After the attention mechanism processing, the features and Through splicing and twice convolution, the mixed features containing both information are obtained ; Step B4, in the complete text instance detection branch, first according to the feature The initial segmentation result of the text instance region is obtained by using the method of activation after convolution; then an adaptive binarization method is used to filter the erroneously segmented pixels in the initial segmentation result of the text instance region; wherein first the feature The text threshold value is obtained , and then a binarization method is used to fuse the initial segmentation result of the text instance region and the text threshold value to obtain the final text instance region segmentation result, and the specific formula is: , wherein is the initial segmentation result of the text instance region, is the text threshold value. Step B5, in the text instance edge detection branch, first according to the characteristics The initial edge region segmentation result is obtained by using the method of activation after convolution; then the mixed feature information and the initial segmentation result information of part of the text instance region are fused on the basis of the initial edge region segmentation result; for the mixed feature information, the method of activation after convolution is also used to obtain the probability information of each pixel belonging to the edge region; for the initial segmentation result of the text instance region, the method in RPCNet is used to obtain the edge information from the initial segmentation result of the text instance region from the way of calculating the spatial gradient, and the specific formula is: wherein is an activation function, is an adaptive average pooling operation with a kernel size of Finally, the three information is spliced, convolved and activated to obtain the final text edge prediction result; According to text edge region features Obtain a kernel region segmentation map; Step B6, to solve the problem of imbalance between positive and negative samples in the training process, first use the OHEM method to mask the segmentation results and actual labels according to the positive and negative sample ratio, and get the result after masking as and , and then make loss function to calculate the loss, the specific formula is: , wherein and are the pixel values of the segmentation results and the actual label after masking respectively; Step C: predicting the unlabeled training set image according to the trained learning model of step B to obtain a text region segmentation map and a kernel region segmentation map, and using a PSE algorithm to obtain a final text region prediction result through gradual expansion.

2. The edge feature enhancement based arbitrary shape text detection method of claim 1, wherein, The pre-processing of each image in the training sample set in step A is specifically as follows: Step A1, using a polygon clipping algorithm to obtain a shrinked text box and an expanded text box of the text object bounding box label; Step A2, obtaining a complete text region label segmentation map according to the text object bounding box, a kernel region label segmentation map according to the shrinked text box, and a text edge region label segmentation map according to the shrinked text box and the expanded text box; Step A3, randomly crop the images in the training samples into pixel fixed size images, then randomly flip in horizontal direction and randomly scale and randomly select, and normalize. 3.The edge feature enhancement based arbitrary shape text detection method of claim 1, wherein, The gradual expansion using the PSE algorithm in step C is specifically as follows: Step C1, segmenting the kernel region of the minimum scale The core region of different text instances is obtained by finding the connected region. Step C2, in The text pixels in the neighborhood are merged on the basis of each core region to obtain , wherein the merging manner is to extend and merge the four-neighborhood pixels belonging to the text pixels into the corresponding core region according to the classification result of the four-neighborhood pixels in the edge pixels of the core region; finally, the same extension and merging operation is used to obtain … , and is taken as the final text prediction result.​

Images