Natural scene text detection method and system based on attention mechanism feature fusion and enhancement
By adopting feature fusion and enhancement methods based on attention mechanism in text detection in natural scenes, combining variable convolution and feature splicing at different scales, the error detection and missed detection problems of text detection in natural scenes are solved, significantly improving detection accuracy Sex and feature extraction effects.
Patent Information
- Application Number
- CN202111393620.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-23
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-11-23
AI Technical Summary
Text detection tasks in natural scenes are affected by background complexity, interference and image quality problems, resulting in high false detection and missed detection rates, and it is difficult for existing deep learning methods to effectively extract features.
Adopt the feature fusion and enhancement method based on attention mechanism, through the decoding and fusion of spatial information and channel information, combining variable convolution and feature stitching at different scales, reduce false detection and missed detection, suppress noise, and improve the accuracy of text detection. .
It significantly improves the accuracy and feature extraction effect of natural scene text detection, reduces false detection and missed detection, enhances the accuracy of target information, and suppresses background noise interference.
Smart Images

Figure CN114255456B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and specifically provides a natural scene text detection method and system based on attention mechanism feature fusion and enhancement. Background Art
[0002] Scene images refer to images captured by image acquisition devices in natural scenes. Compared with other elements in the image, text can convey richer and more accurate information. These text information has extremely high auxiliary value in the research of technologies such as autonomous driving and intelligent translation. Therefore, the detection of text in natural scenes is crucial for image understanding. In order to help computers understand images more accurately, it is of great significance to automatically extract text areas from scene images accurately.
[0003] Compared with standard document text, scene text detection is a more challenging task. Text in natural scenes has the characteristics of complex background interference, diversity and variability, which increases the difficulty of text detection. In addition, the quality of scene images collected by image acquisition equipment is also affected by external conditions such as outdoor light, weather, shooting angle, occlusion, etc., resulting in the collected images often having low contrast, blur, distortion, occlusion, etc. All these make text detection in natural scene images a more difficult problem.
[0004] With the advent of deep learning, the traditional method of manually designing features and classifiers for text detection has been gradually replaced by convolutional neural networks, which autonomously learn feature information in images through convolution operations. Currently, text detection methods based on deep learning are mainly divided into two categories: regression-based and segmentation-based.
[0005] The regression-based method mainly improves the general object detection algorithm Faster RCNN or SSD framework according to the characteristics of the text. This type of method uses an external rectangular frame for positioning, which is good for document text detection, but cannot accurately surround the boundaries of text of any shape, and the excess background noise will seriously interfere with subsequent text recognition.
[0006] The segmentation-based method draws on the ideas of the classic semantic segmentation networks FCN and FPN to build a network framework. Each pixel in the input image is predicted and classified separately, and then the pixels of the same type are clustered to determine the location of the target text. This type of method can adapt to text targets of any shape, but the complex background and diverse scale of scene text make it more difficult to extract effective features, resulting in false detection and missed detection. Summary of the invention
[0007] In view of the above-mentioned defects in the prior art, the present invention adopts a natural scene text detection method and system based on attention mechanism feature fusion and enhancement to effectively segment scene text images, which can reduce false detection and missed detection problems by enhancing the spatial information and channel information of its features.
[0008] In a first aspect, the present invention provides a natural scene text detection method based on attention mechanism feature fusion and enhancement, comprising the following steps:
[0009] S1, obtain natural scene text image;
[0010] S2, extracting features from the natural scene text image, performing n downsampling operations on the whole, and taking the feature map after the [2,n]th downsampling as the first feature map;
[0011] S3, except for the first feature map after the last downsampling, perform spatial information feature extraction on each other first feature map to obtain a spatial information mask; perform channel information feature extraction on the last first feature map to obtain a channel weight vector;
[0012] S4, decoding and fusing the first feature map, the spatial information mask, and the channel weight vector based on the attention mechanism step by step, and obtaining a second feature map with significant features through upsampling, multiplication, and addition operations;
[0013] S5. Use convolution to adjust the number of channels of the fused features, and use upsampling at different rates to unify the original size. Figure 1 / 4 size, and concatenate according to the channel dimension to obtain the third feature map;
[0014] S6, construct different combinations of convolution, pooling, and splicing to further extract, fuse, and enhance features of the third feature map;
[0015] S7. Upsample the features obtained in S6 to the original image size and perform convolution to obtain segmentation masks of the text core area and boundary area in the natural scene text image.
[0016] Furthermore, the process of step S2 is specifically as follows: uniformly scale the natural scene text image to be recognized to a size of A×A; use variable convolution to extract features, and perform 5 downsamplings to obtain the first feature maps in2, in3, in4, and in5 of sizes A / 4, A / 8, A / 16, and A / 32 respectively.
[0017] Furthermore, the process of step S3 is specifically to construct a combination of convolution kernel pooling and splicing, and use the spatial information extraction module SAM to extract the first feature map in kPerform maximum pooling and average pooling by position, concatenate and convolve the features after maximum pooling and average pooling to obtain a spatial information mask; perform maximum pooling and average pooling on the first feature map in5 according to the channel dimension to obtain two information vectors, perform a full connection operation on the two information vectors and add them by position to obtain a channel weight vector.
[0018] Furthermore, the process of step S4 is specifically as follows: the feature fusion module AFFM processes the first feature map in from the shallow encoding end. k (k=2,3,4) and the second feature map from the deep decoding end is out k+1 (k=2,3,4) performs decoding fusion to obtain the second feature map out after fusion of this layer m (m=2,3,4,5). Specifically, the channel weight vector c5 obtained in step S3 is multiplied channel by channel with the first feature map in4 from the encoding end to obtain the channel-weighted encoding feature in4'; the spatial information mask S4 obtained in step S3 is multiplied position by position with the out5 feature after double upsampling from the decoding end to obtain the spatial information weighted decoding feature out5'; then in4' and out5' are added position by position to obtain the second feature map out4 after the first decoding fusion. Repeat the steps in S3 again to extract the spatial and channel information S3 and c4 of in3 and out4, repeat step S4 to obtain the second feature out3 after the second decoding fusion, and so on to obtain a series of second feature maps out2, out3, out4, out5 that have undergone 3, 2, 1, and 0 decodings and have scales of A / 4, A / 8, A / 16, and A / 32 respectively (here the second feature out5 is directly output by in5 without any feature extraction).
[0019] Furthermore, the process of step S5 is specifically to obtain the fused second feature map out m After a layer of 3×3 convolution, the features are further extracted, and the channel dimension is uniformly adjusted to C. Then, the features after the channel adjustment are upsampled by 0, 2, 4, and 8 times to the original Figure 1 / 4 size, we get features P2, P3, P4, and P5, and concatenate them in the channel dimension as the third feature map F, whose dimension is 4C×A / 4×A / 4.
[0020] Furthermore, the process of step S6 is specifically as follows: the feature enhancement module JAM uses the channel information extraction module CAM and the spatial information extraction module SAM to model the dependency relationship between channels and spatial positions of the third feature map F, and then expands the obtained 4C×1×1 channel information weight vector and 1×A / 4×A / 4 spatial information mask to 4C×A / 4×A / 4 dimensions and multiplies them by position to obtain the weighted feature F', and then passes through the sigoid activation function and multiplies the third feature F to obtain the enhanced feature F".
[0021] In a second aspect, the present invention provides a natural scene text detection system based on attention mechanism feature fusion and enhancement, including an acquisition module for acquiring a natural scene text image; a coding extraction module for performing a convolution operation on the natural scene text image to extract features, and performing n downsampling to obtain a first feature map; a spatial information extraction module for extracting spatial detail information of the first feature map to obtain a spatial information mask; a channel information extraction module for extracting semantic information from a second feature map after decoding and fusion from an adjacent deeper layer to obtain a channel weight vector; a feature fusion module for decoding step by step, that is, combining and weighting the first feature map of this layer, the second feature map of an adjacent deeper layer, the spatial information mask, and the channel weight vector, and adding and fusing them to obtain the second feature map of this layer after information enhancement; a feature splicing module for upsampling second feature maps of different scales to the same scale and then splicing them in the channel dimension to obtain a third feature map; a feature enhancement module for modeling the spatial and channel relationships of the third feature map.
[0022] According to a third aspect of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the method described in any one of the above technical solutions when running.
[0023] According to a fourth aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the method described in any one of the above technical solutions.
[0024] The present invention has the following beneficial effects: compared with the existing feature extraction network, the ResNet50 network with variable convolution is used to extract features, the feature extraction information is comprehensive and the effect is good; a feature fusion module based on the attention mechanism is designed, and the effective information of features at different levels is extracted and fused using spatial attention and channel attention, so that the decoded features can contain more and more accurate target information; a feature enhancement module based on attention is designed, and convolution and pooling operations of different dimensions are used to extract more significant features, and play a role in suppressing noise. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0026] Figure 1 This is a flowchart of natural scene text detection based on attention mechanism feature fusion and enhancement according to an embodiment of the present invention.
[0027] Figure 2 for Figure 1 A diagram of the network structure used in the embodiments.
[0028] Figure 3 Schematic diagram of the decoding fusion module AFFM.
[0029] Figure 4 Schematic diagram of the spatial information extraction module SAM structure.
[0030] Figure 5 Schematic diagram of the channel information extraction module CAM structure.
[0031] Figure 6 Schematic diagram of the feature enhancement module JAM structure. DETAILED DESCRIPTION
[0032] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0033] This embodiment is a natural scene text detection method based on attention mechanism feature fusion and enhancement. The overall method flow is as follows: Figure 1 As shown, the network architecture and internal details of the module are as follows Figure 2-Figure 6 As shown, the method comprises the following steps:
[0034] S1. Obtain natural scene text images.
[0035] In this embodiment, the image containing the scene text is obtained by means of street cameras and street photography.
[0036] S2. Modify the ResNet50 network to extract features of the natural scene text image obtained in S1, and perform downsampling n times as a whole. In this embodiment, 5 times is taken as an example, and the features after the 2nd, 3rd, 4th, and 5th downsampling are taken as the first feature map.
[0037] Specifically, Figure 2 As shown in the backbone network on the left, the natural scene text images to be recognized are uniformly scaled to a size of A×A. The ResNet50 series network is modified to use variable convolution to extract features, so that it performs 5 downsamplings to obtain the first feature maps in2, in3, in4, and in5 with sizes of A / 4, A / 8, A / 16, and A / 32 respectively.
[0038] S3. Perform spatial information feature extraction on each acquired first feature map to obtain a spatial information mask; perform semantic information feature extraction on the last first feature map to obtain a channel weight vector.
[0039] Specifically, Figure 4 As shown, the spatial information extraction module SAM is used to extract the first feature map in k Perform maximum pooling and average pooling by position, concatenate the two 1×H×W features, and perform 7×7 convolution to obtain a 1×H×W spatial information mask; Figure 5 As shown, the channel information extraction module CAM module performs maximum pooling and average pooling on the first feature in5 according to the channel dimension to obtain two C×1×1 information vectors, performs a full connection operation on them and adds them by position to obtain a C×1×1 channel weight vector.
[0040] S4. The first feature map, the spatial information mask, and the channel weight vector are decoded and fused based on the attention mechanism step by step, and a second feature map with significant features is obtained through upsampling, multiplication, and addition operations.
[0041] Specifically, Figure 3 As shown, the first feature map from the shallow encoding end of the feature fusion module AFFM is in k (k=2,3,4), the second feature map from the deep decoding end is out k+1 (k=2,3,4), the output second feature map out after feature fusion m(m=2,3,4,5). Specifically, the channel weight vector c5 obtained in step S3 is multiplied channel by channel with the first feature map in4 from the encoder to obtain the channel-weighted encoded feature in4'; the spatial information mask S4 obtained in step S3 is multiplied position by position with the out5 feature after double upsampling from the decoder to obtain the spatial information weighted decoded feature out5'; then in4' and out5' are added position by position to obtain the second feature map out4 after the first decoding fusion. Repeat the steps in S3 again to extract the spatial and channel information S3 and c4 of in3 and out4, repeat step S4 to obtain the second feature out3 after the second decoding fusion, and so on to obtain a series of second feature maps out2, out3, out4, out5 that have undergone 3, 2, 1, and 0 decodings and have scales of A / 4, A / 8, A / 16, and A / 32 respectively (here the second feature out5 is directly output by in5 without any feature extraction). The reason for this construction is that the shallower feature maps in the convolutional neural network contain more spatial detail information, and the deeper feature maps contain more deep semantic information. By extracting effective information at different levels through the attention mechanism and weighting the features of other parts, it can effectively enhance the significant features and suppress background noise while enhancing the target.
[0042] For this embodiment, Figure 2 As shown in the middle part, decoding fusion starts from the smallest feature map in5, that is, first extract the spatial information mask of in4 and multiply it by position with the linear upsampling of 2 times out5 (here out5 = in5), extract the channel information weight vector of out5 and multiply it by channel with in4, and then add the weighted two features by position to obtain the decoded fused feature out4. Similarly, the decoding method of other levels is similar.
[0043] S5. Use convolution to adjust the number of channels of the fused features, and use upsampling at different rates to unify the original size. Figure 1 / 4 size and concatenated according to the channel dimension to obtain the third feature map.
[0044] Specifically, Figure 2 As shown in the middle part, the fused second feature map out m After a layer of 3×3 convolution, the features are further extracted. The channel dimension is uniformly adjusted to C, and then the features out3, out4, and out5 after the adjustment of the channel are upsampled by 2, 4, and 8 times respectively to the original Figure 1 / 4 size, and obtain features P2, P3, P4, and P5, and concatenate them in the channel dimension as the third feature map F after the final decoding feature fusion, whose dimension is 4C×A / 4×A / 4.
[0045] S6. Construct different combinations of convolution, pooling, and splicing to further extract, fuse, and enhance the features of the third feature map.
[0046] Specifically, Figure 6 As shown in the figure, the feature enhancement module JAM uses a parallel approach to process features. This module uses CAM and SAM to model the dependencies between channels and spatial positions of the third feature map F, and then expands the obtained 4C×1×1 channel information weight vector and 1×A / 4×A / 4 spatial information mask to 4C×A / 4×A / 4 dimensions and multiplies them by position to obtain the weighted feature F', which is then multiplied with the input feature F after the Sigmoid activation function to obtain the enhanced feature F". In order to avoid network degradation, residual connections are added in the module to ensure the effectiveness of model training.
[0047] S7. Upsample the final features to the original image size and perform convolution to obtain segmentation masks of the text core area and boundary area in the natural scene text image.
[0048] Another embodiment provided by the present invention is a natural scene text image segmentation system based on an attention mechanism, comprising an acquisition module, a coding extraction module, a spatial information extraction module, a channel information extraction module, a decoding fusion module, a feature splicing module and a feature enhancement module, wherein:
[0049] The acquisition module is used to acquire natural scene text images.
[0050] The encoding extraction module is used to perform convolution operation on the acquired natural scene text image to extract features, and perform downsampling five times to obtain the first feature maps of four different scales.
[0051] The spatial information extraction module SAM is used to extract the spatial detail information of the first feature map and obtain the spatial information mask for the decoding fusion module. In this embodiment, the spatial information extraction module SAM is embedded in the AFFM module and the JAM module. The internal structure of SAM is as follows: Figure 4 As shown, it includes a first pooling unit, a splicing unit, and a convolution unit, wherein
[0052] The first pooling unit performs maximum pooling and average pooling on the first feature map according to position;
[0053] The concatenation unit concatenates the two feature maps obtained by maximum pooling and average pooling according to the channel dimension;
[0054] The convolution unit is used to perform convolution operation on the concatenated features to extract features and obtain a spatial information mask.
[0055] The channel information extraction module CAM is used to extract the semantic information of the second feature map after the previous layer decoding fusion, and obtain the channel weight vector for the next layer decoding fusion module. In this embodiment, it is embedded in the AFFM module and the JAM module. The internal structure of CAM is as follows: Figure 5 As shown, it includes a second pooling unit, a fully connected unit, and an addition unit, wherein
[0056] The second pooling unit performs global maximum pooling and global average pooling on the second feature map according to the channel dimension;
[0057] The fully connected unit uses the fully connected operation to establish the inter-channel dependency relationship between the feature vectors after the maximum and average pooling;
[0058] The addition unit is used to add the two fully connected feature vectors by position to obtain the channel weight vector.
[0059] The decoding fusion module AFFM is used for decoding step by step, that is, combining and weighting the first feature map of the current layer, the second feature map of the previous layer, the spatial information mask, and the channel weight vector, and adding and fusing them to obtain the second feature map of the current layer after information enhancement. In this embodiment, there are a total of 3 decoding fusion modules.
[0060] The feature splicing module is used to upsample the second feature maps of four different scales to the same scale and then splice them in the channel dimension to obtain the third feature map.
[0061] The feature enhancement module JAM is used to model the spatial and channel relationships of the third feature map to enhance the feature representation capability.
[0062] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A natural scene text detection method based on attention mechanism feature fusion and enhancement, characterized in that: The following steps are included: S1, obtain natural scene text image; S2, extracting features from the natural scene text image, performing n downsampling operations on the whole, and taking the feature map after the [2,n]th downsampling as the first feature map; S3, except for the first feature map after the last downsampling, perform spatial information feature extraction on each other first feature map to obtain a spatial information mask; perform channel information feature extraction on the last first feature map to obtain a channel weight vector; S4, decoding and fusing the first feature map, the spatial information mask, and the channel weight vector based on the attention mechanism step by step, and obtaining a second feature map with significant features through upsampling, multiplication, and addition operations; S5. Use convolution to adjust the number of channels of the fused features, and use upsampling of different ratios to unify the size to 1 / 4 of the original image, and splice according to the channel dimension to obtain the third feature map; S6, construct different combinations of convolution, pooling, and splicing to further extract, fuse, and enhance features of the third feature map; S7. Upsample the features obtained in S6 to the original image size and perform convolution to obtain segmentation masks of the text core area and boundary area in the natural scene text image.
2. The natural scene text detection method according to claim 1, characterized in that: The specific process of step S2 is to uniformly scale the natural scene text image to be recognized to a size of A×A; use variable convolution to extract features, and perform 5 downsampling to obtain the first feature maps in2, in3, in4, and in5 with sizes of A / 4, A / 8, A / 16, and A / 32 respectively.
3. The natural scene text detection method according to claim 2, characterized in that: The process of step S3 is specifically as follows: constructing a combination of convolution kernel pooling and splicing, using the spatial information extraction module SAM to perform maximum pooling and average pooling on the first feature map in4 by position, splicing and convolving the features after maximum pooling and average pooling to obtain the spatial information mask S4; performing maximum pooling and average pooling on the first feature map in5 by channel dimension to obtain two information vectors, performing a full connection operation on the two information vectors and adding them by position to obtain the channel weight vector c5.
4. The natural scene text detection method according to claim 3, characterized in that: The process of step S4 is as follows: the feature fusion module AFFM processes the first feature map in from the shallow encoding end. k (k=2,3,4) and the second feature map out from the deep decoding end k+1 (k=2,3,4) performs decoding fusion to obtain the second feature map out after fusion of this layer m (m=2,3,4,5).
5. The natural scene text detection method according to claim 4, characterized in that: The specific process of step S5 is to obtain the fused second feature map out m After a layer of 3×3 convolution, the features are further extracted, and the channel dimension is uniformly adjusted to C. The features after channel adjustment are upsampled by 0, 2, 4, and 8 times respectively and uniformly adjusted to 1 / 4 of the original image size to obtain features P2, P3, P4, and P5. They are then concatenated in the channel dimension as the third feature map F with a dimension of 4C×A / 4×A / 4.
6. The natural scene text detection method according to claim 5, characterized in that: The specific process of step S6 is that the feature enhancement module JAM uses the channel information extraction module CAM and the spatial information extraction module SAM to model the dependency relationship between channels and spatial positions of the third feature map F, and then expands the obtained 4C×1×1 channel information weight vector and 1×A / 4×A / 4 spatial information mask to 4C×A / 4×A / 4 dimensions and multiplies them by position to obtain the weighted feature F', and then passes through the sigoid activation function and multiplies it with the third feature map F to obtain the enhanced feature F".
7. A natural scene text detection system based on attention mechanism, characterized in that: include, An acquisition module, used to acquire natural scene text images; The encoding extraction module is used to perform a convolution operation on the natural scene text image to extract features, and perform n-times downsampling to obtain a first feature map; A spatial information extraction module, used to extract spatial detail information of the first feature map to obtain a spatial information mask; A channel information extraction module is used to extract semantic information from the second feature map of an adjacent deeper layer to obtain a channel weight vector; The feature fusion module is used for decoding step by step, that is, the first feature map of the current layer, the second feature map of the adjacent deeper layer, the spatial information mask, and the channel weight vector are combined and weighted and added to obtain the second feature map of the current layer after information enhancement; A feature concatenation module is used to upsample the second feature maps of different scales to the same scale and then concatenate them in the channel dimension to obtain a third feature map; The feature enhancement module is used to model the spatial and channel relationships of the third feature map.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.
9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.
