A text detection method based on mixed attention and feature enhancement

Abstract

The application provides a text detection method based on mixed attention and feature enhancement, S1, collecting a text image in a natural scene and pre-processing the image; S2, sending the feature into the FPN for feature extraction, first, after the convolution operation of the convolution kernel being 1x1, the features of each layer become the features with the channel number being 256 dimensions; the subsequent features sequentially enter the mixed attention fusion module MAF module from top to bottom, the semantic information of the high layer acts on the bottom layer feature, and the context information of multiple features is fused; S3, after the splicing operation of the output multiple feature maps, the adaptive feature enhancement module AFE module makes the attention weight of the network more flexible in the spatial dimension, and captures the feature information. The mixed attention fusion module and the adaptive feature enhancement module in the network can reduce the interference of background noise on text detection and improve the detection ability of small scale text.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision text detection, and particularly relates to a text detection method based on mixed attention and feature enhancement. BACKGROUND

[0002] As a basic computer vision technology, text detection aims to automatically identify and locate text regions in images or videos. It is the basis for many advanced computer vision and natural language processing tasks, and can be applied to text recognition, image retrieval, smart city systems, scene understanding and many other fields closely related to our lives.

[0003] Although the development of deep learning technology has improved the performance of text detection, there are still many challenges and difficulties in practical applications, such as changes in text size and direction, occlusion and complex background, etc. Therefore, the research on text-related detection problems is still a key research direction. Solving these problems not only can improve the performance of text detection, but also can improve the performance of other related tasks, bring better user experience and more extensive application scenarios, and make it easier for humans to handle problems in daily life. This requires more accurate text detection technology, which puts higher requirements on text detection technology.

[0004] The text detection method based on sliding window regression mainly slides a fixed size window on the image, and then classifies the image region in the window as text or non-text. The advantage of this method is simple and easy to understand, easy to implement, and can well adapt to different scales and directions of text. However, the disadvantage of this method is high computational complexity, which requires a large number of sliding window operations on the image, slow processing speed for large images, and problems of false detection and missed detection. In addition, this method also needs more sample labeling, and the calculation amount is huge and the detection efficiency is not high. The detection effect of this method mainly depends on the quality of the feature information it obtains.

[0005] The text detection method based on segmentation can more accurately segment the text region and the non-text region such as background, so that the segmentation method can achieve better detection effect. The segmentation-based detection uses a pixel-level method to improve the effect of text detection, which judges all the pixels contained in the image, and divides the pixels belonging to the same text region into the same group, so as to effectively distinguish different text regions. The TextBPN algorithm uses a new adaptive candidate boundary network for arbitrary shape text detection, which can learn to generate accurate boundaries for arbitrary shape text without any post-processing. The network continuously corrects the text boundary through an iterative method, and finally obtains the accurate text boundary. The CM-Net algorithm proposes a new text construction representation method and a multi-view feature unit. The former can fit the contour of the text of any shape through a concentric mask. The latter promotes the network to learn more features related to the concentric mask from multiple perspectives, thereby improving the network performance.

[0006] Compared with the regression method, the segmentation-based method has improved the detection of text, but still has some problems. For example, the segmentation effect of small text in dense areas is poor, and the segmentation regions may be connected. Complex background may also cause false detection.

[0007] In real life, text images are affected by complex background, light and shooting angle changes in real scenes, and other factors. The existing method is prone to miss small text. The existing method pays little attention to the combination of information between each layer of features in the network, which cannot effectively distinguish the background and the text region, resulting in easy omission of small text regions in the picture and causing omission. SUMMARY

[0008] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide a text detection method based on hybrid attention and feature enhancement, which proposes a hybrid attention fusion module and an adaptive feature enhancement module to reduce the interference of background noise on text detection and improve the detection ability of small scale text.

[0009] To achieve the above purpose, the present application provides the following technical scheme: a text detection method based on hybrid attention and feature enhancement, comprising the following steps:

[0010] S1, collecting text images in natural scenes and pre-processing the images;

[0011] S2, ResNet18 is used as the backbone network, and the second layer to the fifth layer of the ResNet-18 is modified to replace the ordinary convolution with deformable convolution; the features are sent into the FPN for feature extraction. First, the features are subjected to a convolution operation with a convolution kernel of 1x1, and each layer of features becomes a 256-dimensional feature with a channel number of 256; the subsequent features enter the mixed attention fusion module (MAF) from top to bottom in sequence. In the MAF module, the semantic information of the high layer acts on the bottom layer features, and the context information of multiple features is fused.

[0012] S3, after the output multiple feature maps are spliced, the adaptive feature enhancement module (AFE) makes the network's attention weight more flexible in the spatial dimension, capturing the feature information.

[0013] Further, step S2 includes:

[0014] The input of the mixed attention fusion module is P i and Out i+1 , P is the feature of this layer, and Out is the result of the next layer feature; the calculation process is as shown in formula (1):

[0015]

[0016] wherein, represents the addition of feature elements, represents the multiplication of feature elements, Up represents the bilinear interpolation operation; CM and SM represent the attention of the channel dimension and the attention of the spatial dimension;

[0017] The attention of the channel dimension is not changed in the channel dimension, but the spatial dimension is compressed. The module processes the input feature map through two parallel maximum pooling layers and average pooling layers, changes the feature map from CxHxW to Cx1x1, and then passes through the MLP module. In this module, it first compresses the channel number to 1 / r of the original, and then expands it to the original channel number, and obtains two activated results through the ReLU activation function. The two output results are subjected to element-by-element addition operation; then an output result is obtained through a Sigmoid activation function.

[0018] The spatial attention module does not change the spatial dimension, but compresses the channel dimension, and then learns the feature information in the spatial dimension; the module focuses on the position information of the target; that is, the input feature image is obtained through maximum pooling and average pooling operation Two 1xHxW feature maps; then the two feature maps are spliced through connection operation, and then a 7x7 convolution operation is performed to change the feature map with 1 channel, and then a Sigmoid is performed to obtain the spatial attention feature map, and finally the output result is multiplied by the original image to change back to CxHxW size; the feature information is combined with the shallow detail information and the deep semantic information through the operation of the two attentions.

[0019] Further, the step S3 comprises:

[0020] The first input feature is spliced together, as shown in formula (2); then it enters the adaptive feature enhancement module, first through a convolution kernel of 3x3, and then through a series of operations, and then added to S to generate T(S), as shown in formula (3).

[0021] Then, the result generated by T(S) after convolution, ReLu, convolution and Sigmoid operation is added to the original T(S) to generate K(S), as shown in formula (4).

[0022] As shown in formula (5), the final generated OutPut feature map is sent to the post-processing module for operation.

[0023] S=Concat([Out2,Out3,Out4,P5]) (2)

[0024]

[0025]

[0026]

[0027] Wherein, Avg represents the average pooling operation, Conv3 represents the convolution operation with a convolution kernel of 3x3, Conv1 represents the convolution operation with a convolution kernel of 1x1; ReLu represents the ReLU activation function, and sigma represents the Sigmoid activation function, And Respectively represent the operation of adding or multiplying the corresponding elements of two feature maps.

[0028] Compared with the prior art, the beneficial effects of the present application are:

[0029] 1) The application proposes a hybrid attention fusion module, which helps and improves the extraction of feature information in the feature pyramid part. The attention module can effectively obtain the context information of the feature image. By fusing local details and global text information, the interference of background noise on detection can be reduced and the attention of the network to small text that is easy to ignore can be improved.

[0030] 2) The application proposes an adaptive feature enhancement module. The module learns the features after the pyramid output, so that the network can adaptively master the importance of different scale and different position feature information. The features are dynamically aggregated, and the detection ability of the network model for small scale text is improved.

[0031] 3) Ablation experiments are carried out on two public data sets and compared with other excellent methods. The method proposed in the application has obvious improvement on text detection. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The flowchart of the text detection method based on hybrid attention and feature enhancement of the application;

[0033] Figure 2 The overall framework of the application;

[0034] Figure 3 The hybrid attention fusion module;

[0035] Figure 4 The channel dimension attention module;

[0036] Figure 5 The spatial dimension attention module;

[0037] Figure 6 The adaptive feature enhancement module. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the application clearer and more apparent, the application will be further described in detail below in combination with the drawings and examples. The specific examples described herein are only used to explain the technical scheme of the application and are not limited to the application.

[0039] This paper proposes a text detection method based on hybrid attention and feature enhancement. The network incorporates a hybrid attention fusion module (MAF) and an adaptive feature enhancement module (AFE) to reduce background noise interference and improve the detection capability for small-scale text. The attention fusion module assists and enhances feature extraction in the feature pyramid section, effectively acquiring contextual information from the feature image. By fusing local details and global text information, it reduces background noise interference and increases the network's attention to easily overlooked small text. The adaptive feature enhancement module learns from the features output by the pyramid, enabling the network to adaptively grasp the importance of feature information at different scales and locations. Features are dynamically aggregated, further improving the network model's ability to detect small-scale text. Furthermore, a combination of the Dice and BCELoss loss functions is used to address the bias caused by imbalanced positive and negative samples during training. Finally, ablation experiments on two public datasets and comparisons with other methods demonstrate the superior performance of the proposed network structure.

[0040] like Figure 1 As shown, the technical solution of the present invention includes the following steps:

[0041] S1: Collect text images in natural scenes and preprocess the images;

[0042] S2: Construct a text detection network based on hybrid attention and feature enhancement;

[0043] S3: Optimize the loss function;

[0044] S4: A text detection network based on hybrid attention and feature enhancement, built through training;

[0045] S5: Verify model performance.

[0046] S1: For images in the two datasets, adjust the resolution to a corresponding input size of 640×640 based on their respective image sizes. During the training phase, utilize data augmentation techniques such as image rotation, color gamut transformation, and scaling to enrich the amount of sample data and improve the model's generalization performance.

[0047] S2.1 uses ResNet18 as the backbone network to extract features;

[0048] S2.2 The overall framework proposed in this invention is as follows: Figure 2 As shown, it aims to address the problems of background interference in detection and missed detection of small-scale text. The proposed text detection network based on hybrid attention and feature enhancement includes:

[0049] The network uses ResNet-18 and modifies it, replacing the ordinary convolutions in layers 2 through 5 with deformable convolutions. Deformable convolutions, compared to ordinary convolutions, have an additional offset attribute that can be learned and updated by the network. The features are then fed into the Feature Processing Network (FPN) for feature extraction. First, the features undergo 1×1 convolution operations, transforming each layer's features into 256-dimensional features. From bottom to top, these correspond to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolutions of the original image. Subsequent features are then processed in a top-down order by the Hybrid Attention fusion (MAF) module. The MAF module integrates high-level semantic information with low-level features, thus fusing more contextual information from multiple features. Finally, the output feature maps are concatenated, and the Adaptive Feature Enhancement (AFE) module makes the network's attention weights more flexible in spatial dimensions, enabling better capture of feature information. This reduces background noise interference with text and improves the detection capability for small text, enabling the network to achieve better text segmentation results.

[0050] like Figure 3 As shown, the hybrid attention fusion module proposed in this invention includes:

[0051] Due to background noise, text may be falsely detected during detection. Therefore, a hybrid attention fusion module is proposed. This module learns and focuses on the feature information in both spatial and channel dimensions, and combines the contextual information of each layer of features to improve the model's attention to the text in the image, reduce the interference of the background on its detection, and improve the detection accuracy.

[0052] The input of the module is P i and Out i+1 P is the feature of this layer, and Out is the result of the feature of the next layer. The calculation process is shown in formula (1):

[0053]

[0054] in, Adding the representative characteristic elements, Multiplying representative characteristic elements U p represents the bilinear interpolation operation. CM and SM represent attention in the channel dimension and attention in the spatial dimension, respectively.

[0055] like Figure 4As shown, the core idea of ​​channel-dimensional attention is to keep the channel dimension unchanged while compressing the spatial dimension. This module passes the input feature map through two parallel max-pooling and average-pooling layers, reducing the feature map from C×H×W to C×1×1, before passing it through the MLP module. In this module, the number of channels is first compressed to 1 / r of the original number, then expanded to the original number, and then processed through a ReLU activation function to obtain two activated results. These two outputs are then added element-wise. Finally, a Sigmoid activation function is used to obtain the final output.

[0056] like Figure 5 As shown, the spatial attention module maintains the spatial dimension unchanged while compressing the channel dimension before learning spatial features. This module focuses on the target's location information. The input feature image is processed through max pooling and average pooling to obtain two 1×H×W feature maps. These two feature maps are then concatenated, followed by a 7×7 convolution to reduce the number of channels to 1. After multiple trials, the 7×7 convolution yields better results than the 3×3 convolution. A sigmoid function is then applied to obtain the spatial attention feature map. Finally, the output is multiplied by the original image to revert to a C×H×W size. Through these two attention operations, the feature information combines shallow detail and deep semantic information, enhancing the network's ability to acquire information from text regions.

[0057] High-level feature maps have lower resolution and less ability to understand details, but still retain rich semantic information. Low-level feature maps, on the other hand, have higher resolution and contain more location and detail information, but their semantic clarity is lower and they contain noise. To fully utilize features at different scales, semantic segmentation methods typically use feature pyramids. Compared to most semantic segmentation methods that fuse features at different scales through simple concatenation or aggregation, the proposed adaptive feature enhancement module aims to dynamically fuse features at different scales, thereby improving the network's focus on textual features.

[0058] Figure 6The demonstration shows that attention is used in the feature extraction stage to enhance the fusion of feature information from different layers, and then the concatenated feature map is fed into the adaptive feature enhancement module. First, the input features are concatenated together, as shown in formula (2). Then, it enters the adaptive feature enhancement module, first through a 3×3 convolution kernel, then through a series of operations such as average pooling, and then added to S to generate T(S), as shown in formula (3). This structure can obtain the same weight value for the output features and the same number of input features, that is, the weight ratio coefficient of each feature channel, so that the network learns an attention in a channel dimension. This attention can be used to determine which channels are key, so that features can be extracted in these channels and the unimportant parts can be discarded.

[0059] Afterwards, T(S) undergoes a series of operations such as convolution, ReLU, convolution, and Sigmoid to generate the result, which is then added to the original T(S) to generate K(S), as shown in formula (4). This part of the structure utilizes the attention mechanism in the spatial dimension to make the weight of attention more important in the spatial dimension, so as to understand and capture important feature information of the text.

[0060] This dual-attribute attention mechanism allows the network to adaptively capture and understand the feature information of text regions, thereby improving the network's detection performance.

[0061] As shown in the formula (5) below, the final OutPut feature map is sent to the post-processing module for operation.

[0062] S=Concat([Out2,Out3,Out4,P5]) (2)

[0063]

[0064]

[0065]

[0066] Where Avg represents the average pooling operation, Conv3 represents a convolution operation with a 3×3 kernel, and Conv1 represents a convolution operation with a 1×1 kernel; ReLU represents the ReLU activation function, and σ represents the Sigmoid activation function. and These represent the addition or multiplication operations performed on corresponding elements of two feature maps, respectively. The final feature map (Output) is generated and output to the post-processing module.

[0067] S3: Loss Function Optimization

[0068] In the network, supervised information is added for learning through the segmentation map, boundary prediction map, and binarization map described above. The segmentation module and the binarization module use the same supervised information. Therefore, loss functions need to be set for these three parts, so the total loss function is the sum of the loss functions of these three parts. Finally, the total loss function can be expressed by the following formula (6):

[0069] L = L s +γL t +λL b (6)

[0070] Where γ and λ are the weighting coefficients balancing these three loss functions, and in this invention they are taken as 1.0 and 10 respectively, representing the loss L of the segmentation map. s Loss L of boundary graph t Loss L of binary graphs b Because text image detection is a binary classification task, but the proportion of non-text regions in the image is larger than that of text regions, this leads to an imbalance in the weight of positive and negative samples. Furthermore, small-scale text regions also constitute a small proportion of the image, and the module crops the image during feature input, making it even more difficult to capture small text regions. Therefore, using the cross-entropy loss function (BCELoss) causes the network to favor negative sample regions (i.e., non-text regions). To address this issue, this invention employs the Dice loss function to balance positive and negative samples. The Dice loss function is used in the segmentation module, the L1 loss function is used in the boundary prediction module, and the BCELoss loss function is used in the binarization module.

[0071] The formula for the Dice coefficient is shown in formula (7):

[0072]

[0073] |X∩Y| represents the intersection between X and Y, where |X| and |Y| represent the number of elements in X and Y, respectively. The coefficient 2 in the numerator is due to the fact that the denominator contains elements that are common to X and Y but are counted twice.

[0074] The Dice loss function is given by equation (8):

[0075]

[0076] Among them, P (m,n) T is the predicted value for the pixel position (m,n) in the text region. (m,n) T is the truth value for the text at that location. If the network detects that location as a text region, then T... (m,n) The output is 1 if the value is 1, otherwise it is 0.

[0077] The cross-entropy loss function is used in the binarized image, and the Online Hard Instance Mining (OHEM) algorithm is used to ensure that the ratio of positive to negative samples is 1:3. The calculation formula is described as follows:

[0078] L c =∑ m,n∈S (T m,n ·logP m,n +(1-T m,n )·log(1-P m,n (9)

[0079] Where S represents the text region predicted by the network. The L1 loss function is applied to the boundary map, and its calculation method is shown in the formula:

[0080] L t =∑ m,n∈U |T m,n * -P m,n * | (10)

[0081] S4.1: Set the training parameters. The model uses stochastic gradient descent (SGD) with momentum as the optimizer, with a momentum coefficient of 0.9 and a weight decay coefficient of 0.0001. The batch size is set to 8 during training. The initial learning rate is 0.007, and a poly strategy is used to decay the learning rate of the network model.

[0082] S4.2: Input the text images from the training set in the dataset into a text detection network based on hybrid attention and feature enhancement for training;

[0083] S5: Based on the trained text detection network using hybrid attention and feature enhancement, the effectiveness of the proposed method is evaluated using standard evaluation metrics in the field of text detection. Three important metrics in the evaluation are described below. The specific calculation methods for each metric are shown below.

[0084]

[0085]

[0086]

[0087] Where TP represents the number of samples where both the prediction and the actual result are correct, FP represents the number of samples where the actual result is negative and the prediction is positive, TN represents the number of samples where both the actual result and the prediction are negative, FN represents the number of samples where the actual result is positive and the prediction is negative, precision P is the proportion of correctly predicted text regions out of all correctly predicted text regions, recall R is the proportion of correctly predicted text regions out of all actually predicted text regions, and the composite index F is defined as the harmonic mean of recall and precision.

[0088] The experimental platform setup for this invention is as follows: The experimental configuration uses the Ubuntu 16.04 operating system, the network model uses the PyTorch framework, 8GB of video memory, and Python version 3.6.

[0089] The following are the comparative experimental results between this invention and existing technologies. The table shows that this invention achieves the best results in terms of the combined accuracy (P) and accuracy (F) metrics, reaching 83.5%. Compared to other algorithms for curved text detection, such as TextSnake and PSENet, this invention improves accuracy by 5.1% and 2.6%, respectively. While the TextField algorithm uses a two-dimensional vector field and complex post-processing operations to enhance detection performance, this invention, although having a lower recall rate than TextField, outperforms it in both accuracy and overall metrics. Furthermore, its post-processing is simpler and its network speed is faster. Compared to the current best-performing segmentation-based detection algorithm, DBNet, the overall metrics have also improved from 82.8% to 83.5%. In addition, this invention also demonstrates good recall, indicating that the proposed model significantly improves the performance of curved text detection.

[0090] Table 1 compares the performance with existing mainstream techniques on the Total-Text dataset.

[0091]

[0092] Table 2 below shows the experimental results of the HAF-FEN model introduced in this invention on the ICDAR2015 dataset. As can be seen from the table, the HAF-FEN model presented in this invention exhibits strong robustness and adaptability on the ICDAR2015 dataset, with a comprehensive index F and accuracy P reaching 82.9% and 90.4%, respectively. Compared with the traditional classic CTPN, SegLink, and EAST models, the F-value of this model is improved by 22.9%, 7.9%, and 4.7%, respectively. Compared with the TextBoxes algorithm, the model designed in this invention improves the comprehensive index F by 1.2% and the accuracy P by 3.2%. The MCN model uses the Markov clustering algorithm, but its prediction performance is poor for text with low node density. Compared with the MCN model, although the recall rate of this invention is lower, all other indicators are significantly higher. The model proposed in this invention improves both accuracy and comprehensive index. It is a relatively lightweight network and can achieve good detection performance on the ICDAR2015 dataset, which has fuzziness, different orientations, and different scales.

[0093] Table 2 compares the performance with existing mainstream technologies on the ICDAR2015 dataset.

[0094]

[0095] The above description merely illustrates preferred embodiments of the present invention, and while the description is relatively specific and detailed, it should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications, improvements, and substitutions without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A text detection method based on hybrid attention and feature enhancement, characterized in that: Includes the following steps: S1, Collect text images in natural scenes and preprocess the images; S2 uses ResNet18 as the backbone network and modifies it by replacing the ordinary convolutions in layers 2 to 5 of ResNet-18 with deformable convolutions. The features are fed into the FPN for feature extraction. First, the features undergo a 1×1 convolution operation, and each layer's features become 256-dimensional features. The subsequent features are fed into the Hybrid Attention Fusion (MAF) module in a top-down order. In the MAF module, the semantic information of the higher levels acts on the lower-level features, fusing the contextual information of multiple features. S3, after the multiple output feature maps are concatenated, the Adaptive Feature Enhancement (AFE) module makes the network's attention weights more flexible in the spatial dimension, capturing feature information; The input to the hybrid attention fusion module is P. i and Out i+1 P represents the feature of this layer, and Out represents the result of the feature of the next layer. The calculation process is shown below: In the formula, Adding the representative characteristic elements, Up represents the multiplication of feature elements; Up represents the bilinear interpolation operation; CM represents channel-dimensional attention, which means that the channel dimension is not changed, but the spatial dimension is compressed. Channel-dimensional attention is provided with a corresponding channel attention module; SM represents spatial-dimensional attention, which means that the spatial dimension is not changed, but the corresponding compression operation is performed on the channel dimension, and then the feature information in the spatial dimension is learned. Spatial-dimensional attention is provided with a corresponding spatial attention module. The processing procedure of the adaptive feature enhancement module is as follows: S=Concat([Out2,Out3,Out4,P5]) In the formula, Avg represents the average pooling operation, Conv3 represents a convolution operation with a 3×3 kernel, and Conv1 represents a convolution operation with a 1×1 kernel; ReLU represents the ReLU activation function, and σ represents the Sigmoid activation function. and These represent the addition or multiplication operations of corresponding elements in two feature maps, respectively.

2. The text detection method based on hybrid attention and feature enhancement according to claim 1, characterized in that: The channel attention module passes the input feature map through two parallel max pooling layers and average pooling layers, transforming the feature map from C×H×W to a size of C×1×1, and then passes it through the MLP module; In the channel attention module, the number of channels is first compressed to 1 / r times the original number, and then expanded to the original number of channels. After passing through the ReLU activation function, two activated results are obtained. The two outputs are added element by element; then the result is obtained by passing the sigmoid activation function.

3. The text detection method based on hybrid attention and feature enhancement according to claim 1, characterized in that: The spatial attention module focuses on the target's location information; that is, the input feature image is processed by max pooling and average pooling to obtain two 1×H×W feature maps; then the two feature maps are concatenated by a concatenation operation, and then a 7×7 convolution operation is performed to transform it into a feature map with 1 channel, and then a sigmoid operation is performed to obtain the spatial attention feature map. Finally, the output result is multiplied by the original image and transformed back to C×H×W size.

Images