A scene text detection method based on multi-scale feature fusion
Patent Information
- Application Number
- CN202410285092.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-13
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2044-03-13
Smart Images
Figure CN118072331B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision target recognition, and in particular to a scene text detection method using multi-scale feature fusion. Background Technology
[0002] Text detection, as an important computer vision task, has broad application prospects and practical value. However, due to the complexity of image and video sources and the diverse scene variations they contain, text detection faces numerous challenges and difficulties, including complex background interference, changes in viewpoint and lighting, and issues related to multi-scale and arbitrary text shapes. To address these challenges, researchers have proposed various effective text detection algorithms, but further research and improvement are still needed to enhance detection performance and applicability.
[0003] Traditional scene text detection methods typically rely on hand-designed features and models, requiring tedious image preprocessing and feature extraction. However, these methods often exhibit limited robustness and accuracy when dealing with challenges such as complex scenes, multi-scale text, and varying lighting conditions.
[0004] With the development of convolutional neural networks, an increasing number of deep learning-based text detection methods have been proposed. These methods can directly learn more discriminative feature representations from raw images, thereby improving the accuracy and robustness of scene text detection. In deep learning-based scene text detection, multi-scale text detection is a key issue, requiring models to effectively handle text at different scales while maintaining high-precision detection results. To address the challenge of multi-scale text, researchers have proposed various methods to achieve multi-scale feature fusion to improve the accuracy and robustness of text detection. However, simply concatenating or chaining multi-scale features is insufficient for effectively fusing information. Summary of the Invention
[0005] This invention addresses the problems existing in the prior art by providing a multi-scale feature fusion method for scene text detection, thereby improving the accuracy of text detection.
[0006] To address the above technical problems, this invention provides the following technical solution: a scene text detection method based on multi-scale feature fusion, comprising the following steps:
[0007] S1. Construct a text detection dataset, including text images and the corresponding recognition results for the text images;
[0008] S2. A feature extraction network is constructed by taking text images as input and multi-scale features of text images as output, and combining the dilated convolutional pooling module DCMP.
[0009] S3. Using the multi-scale features of the text image as input and the multi-scale fused feature map N as output, a multi-scale feature fusion network is constructed by combining the multi-scale feature fusion module MFAF.
[0010] S4. Construct a text detection head network with multi-scale fused feature map N as input and recognition results corresponding to the text image as output;
[0011] S5. Construct a multi-scale feature fusion text detection network based on feature extraction network, multi-scale feature fusion network, and text detection head network. Train the multi-scale feature fusion scene text detection network with text detection dataset to obtain a multi-scale feature fusion text detection model. Then, use the model to detect the collected text images to obtain recognition results.
[0012] Furthermore, the feature extraction network in step S2 mentioned above uses ResNet-50 as the backbone network.
[0013] Furthermore, the feature extraction network in step S2 described above is configured to perform the following actions:
[0014] S201. Extract features from the original text image, extracting five feature maps G1, G2, G3, G4, and G5 with sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image.
[0015] S202. Input feature maps G2, G3, G4, G5 into the feature pyramid network to obtain new feature maps P2, P3, P4, P6.
[0016] S203. Input the feature map G4 into the dilated convolutional pooling module DCMP to obtain a new feature map P5.
[0017] Furthermore, the aforementioned dilated convolutional pooling module DCMP is configured to perform the following actions: performing convolution operations on feature map G4 using four dilated convolutions with different dilation rates to obtain four feature maps Y1, Y2, Y3, and Y4 of the same size; fusing these four feature maps into a single feature map Y'1 using the attention feature fusion mechanism AFF; and finally reducing the channel dimension of feature map Y'1 through a convolutional block to obtain a new feature map P5.
[0018] Furthermore, the multi-scale feature fusion module MFAF in step S3 above is configured to perform the following actions:
[0019] S301. Use spatial attention to obtain the weight information of new feature maps P2, P3, P4, P5, and P6, and multiply them with the corresponding layer principal elements of the feature maps to obtain feature maps C1, C2, C3, C4, and C5 with spatial attention. The calculation formula is as follows:
[0020] Ci-1 =P i ×spatial_Att(P i ),i∈[2,6]#(1)
[0021] Among them, P i Represents a completely new feature map, C i-1 This represents a feature map with spatial attention, where spatial_Att(·) represents spatial attention;
[0022] S302. Use the SEWeight module to extract feature maps with spatial attention. i The channel attention vectors are calculated to obtain channel attention vectors Z1, Z2, Z3, Z4, and Z5 at different scales. Then, the SoftMax function is used to process the obtained channel attention vectors to obtain the corresponding attention weights att. i The calculation formula is as follows:
[0023]
[0024] S303. Simultaneously perform adaptive feature fusion and channel attention acquisition on feature maps C1, C2, C3, C4, and C5 with spatial attention to obtain feature map F and feature map M. The calculation formula is as follows:
[0025] F=Cat(att1⊙C1,att2⊙C2,att3⊙C3,att4⊙C4,att5⊙C5)#(3)
[0026] Among them, att i Representing feature map C i The corresponding spatial attention weights, Cat() indicates concatenation by channel;
[0027] M=Cat(w1×C1,w2×C2,w3×C3,w4×C4,w5×C5)#(4)
[0028] Among them, w i It is based on C i Generated adaptive weights;
[0029] S304. The two feature maps F and M are fused using the attention feature fusion mechanism AFF to obtain the feature map N output by the multi-scale feature fusion network.
[0030] Furthermore, the aforementioned dilated convolution pooling module DCMP performs convolution operations on the feature map G4 using four dilated convolutions with different dilation rates: r = 1, 6, 12, and 18.
[0031] Furthermore, the aforementioned multi-scale feature fusion scene text detection method reduces the channel dimension of feature map Y'1 by passing it through a 1*1 convolutional block to obtain a new feature map P5.
[0032] Compared to existing technologies, the beneficial technical effects of the above technical solution adopted in this invention are as follows: In the feature processing network part, a dilated convolutional pooling module (DCMP) and a multi-scale feature fusion module (MFAF) are proposed. This allows for detailed extraction of feature information at different scales, combining the four-layer feature map output by the original feature extraction network with the feature map output by DCMP. Figure 1 Using this as the fifth layer input to the Multi-Scale Feature Fusion (MFAF) module, cross-level multi-scale feature adaptive fusion effectively alleviates the problems of feature misalignment and insufficient feature extraction during the feature pyramid feature extraction process. This structure can make fuller use of the feature information in the image, especially for the detection of small and blurred text, which is more accurate. Attached Figure Description
[0033] Figure 1 This is an overall flowchart of the present invention;
[0034] Figure 2 The multi-scale feature fusion scene text detection network framework diagram provided by the present invention;
[0035] Figure 3 This is a schematic diagram of the DCMP (Diffuse Convolution Pooling) module of the present invention.
[0036] Figure 4 This is a schematic diagram of the multi-scale feature fusion module (MFAF) of the present invention;
[0037] Figure 5 This is a schematic diagram of the detection effect of the present invention; Detailed Implementation
[0038] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.
[0039] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.
[0040] refer to Figure 1This invention provides a scene text detection method with multi-scale feature fusion, including the following steps: S1, constructing a text detection dataset, including text images and recognition results corresponding to the text images;
[0041] S2. A feature extraction network is constructed by taking text images as input and multi-scale features of text images as output, and combining the dilated convolutional pooling module DCMP.
[0042] S3. Using the multi-scale features of the text image as input and the multi-scale fused feature map N as output, a multi-scale feature fusion network is constructed by combining the multi-scale feature fusion module MFAF.
[0043] S4. Construct a text detection head network with multi-scale fused feature map N as input and recognition results corresponding to the text image as output;
[0044] S5. Construct a multi-scale feature fusion text detection network based on feature extraction network, multi-scale feature fusion network, and text detection head network. Train the multi-scale feature fusion scene text detection network with text detection dataset to obtain a multi-scale feature fusion text detection model. Then, use the model to detect the collected text images to obtain recognition results.
[0045] As a preferred embodiment of the present invention, reference Figure 2 The feature extraction network uses ResNet-50 as the backbone network to extract five feature maps G1, G2, G3, G4, and G5 with sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image. G2, G3, G4, and G5 are then input into the feature pyramid network to obtain new feature maps P2, P3, P4, and P6. Additionally, G4, as a separate feature map, is input into the designed dilated convolutional pooling module DCMP to obtain a new feature map P5.
[0046] like Figure 3 As shown, in step S2, the dilated convolutional pooling module DCMP is configured to perform the following actions: four dilated convolutions with different dilation rates (r = 1, 6, 12, 18) are used to perform convolution operations on feature map G4 to obtain four feature maps Y1, Y2, Y3, Y4 of the same size. These four feature maps are then fused into a single feature map Y'1 through the attention feature fusion mechanism AFF. Finally, feature map Y'1 is passed through a convolutional block to reduce the channel dimension, resulting in a new feature map P5.
[0047] refer to Figure 4 Step S3 uses the multi-scale feature fusion module MFAF to fuse multiple feature maps of different scales. The input is the four-layer feature maps P2, P3, P4, P6 output by the above feature extraction network and the one-layer feature map P5 output by the dilated convolutional pooling module DCMP.
[0048] In a preferred embodiment of the present invention, the multi-scale feature fusion module MFAF is configured to perform the following actions:
[0049] S301. Use spatial attention to obtain the weight information of new feature maps P2, P3, P4, P5, and P6, and multiply them with the corresponding layer principal elements of the feature maps to obtain feature maps C1, C2, C3, C4, and C5 with spatial attention. The calculation formula is as follows:
[0050] C i-1 =P i ×spatial_Att(P i ),i∈[2,6]#(1)
[0051] Among them, P i Represents a completely new feature map, C i-1 This represents a feature map with spatial attention, where spatial_Att(·) represents spatial attention;
[0052] S302. Use the SEWeight module to extract feature maps with spatial attention. i The channel attention vectors are calculated to obtain channel attention vectors Z1, Z2, Z3, Z4, and Z5 at different scales. Then, the SoftMax function is used to process the obtained channel attention vectors to obtain the corresponding attention weights att. i The calculation formula is as follows:
[0053]
[0054] S303. Simultaneously perform adaptive feature fusion and channel attention acquisition on feature maps C1, C2, C3, C4, and C5 with spatial attention to obtain feature map F and feature map M. The calculation formula is as follows:
[0055] F=Cat(att1⊙C1,att2⊙C2,att3⊙C3,att4⊙C4,att5⊙C5)#(3)
[0056] Among them, att i Representing feature map C i The corresponding spatial attention weights, Cat() indicates concatenation by channel;
[0057] M=Cat(w1×C1,w2×C2,w3×C3,w4×C4,w5×C5)#(4)
[0058] Among them, w i It is based on C i Generated adaptive weights;
[0059] S304. The two feature maps F and M are fused using the attention feature fusion mechanism AFF to obtain the feature map N output by the multi-scale feature fusion network.
[0060] Finally, the two feature maps F and M are fused using the attention feature fusion mechanism AFF to obtain the feature map N output by the multi-scale feature fusion network.
[0061] The following table shows the performance tests conducted on the MASRA-TD500 dataset, using accuracy (P), recall (f), and F-score (F) as evaluation metrics.
[0062] Table 1 shows the ablation experiments in the MASRA-TD500 dataset.
[0063] ResNet-50 × × 90.13 79.42 84.23 ResNet-50 √ × 94.84 79.03 86.22 ResNet-50 × √ 93.26 83.16 87.91 ResNet-50 √ √ 93.88 84.36 88.87
[0064] The following table shows the performance tests conducted on the ICDAR-2015 dataset, using accuracy (P), recall (f), and F-score (F) as evaluation metrics.
[0065] Table 2 shows the ablation experiments in the ICDAR-2015 dataset.
[0066] ResNet-50 × × 88.24 85.60 86.90 ResNet-50 √ × 89.73 84.75 87.17 ResNet-50 × √ 90.53 84.35 87.33 ResNet-50 √ √ 91.56 84.75 88.02
[0067] The results of the ablation experiments show that, with a slight increase in model parameters, the dilated convolutional pooling module DCMP and the multi-scale feature fusion module MFAF proposed in this invention both improve the evaluation metrics on the dataset.
[0068] On the ICDAR-2015 dataset, the detection performance of this invention is as follows: Figure 5 As shown.
[0069] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.
Claims
1. A scene text detection method based on multi-scale feature fusion, characterized in that, Includes the following steps: S1. Construct a text detection dataset, including text images and the corresponding recognition results for the text images; S2. Taking text images as input and multi-scale features of the text images as output, a feature extraction network is constructed using the dilated convolutional pooling (DCMP) module. The feature extraction network uses ResNet-50 as the backbone network and is configured to perform the following actions: S201. Perform feature extraction on the original text image, extracting five feature maps with sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image. , , , , ; S202, Feature map , , , The input is fed into the feature pyramid network to obtain a new feature map. , , , ; S203, Regarding feature maps The input is fed into the dilated convolutional pooling module DCMP to obtain a novel feature map. ; The dilated convolutional pooling module DCMP is configured to perform the following actions on feature maps: Four dilated convolutions with different dilation rates are used to obtain four feature maps Y1, Y2, Y3, and Y4 of the same size. These four feature maps are then fused into a single feature map Y using the attention feature fusion mechanism (AFF). ’ 1. Finally, the feature map Y ’ 1. By reducing the channel dimension through convolutional blocks, a new feature map is obtained. ; S3. Using the multi-scale features of the text image as input and the multi-scale fused feature map N as output, a multi-scale feature fusion network is constructed by combining the multi-scale feature fusion module MFAF. S301. Obtain new feature maps using spatial attention. , , , , The weight information is multiplied by the corresponding layer principal element of the feature map to obtain feature maps C1, C2, C3, C4, and C5 with spatial attention. The calculation formula is as follows: ， in, This represents a completely new feature map. This represents a feature map with spatial attention. Represents spatial attention; S302. Use the SEWeight module to extract feature maps with spatial attention. Channel attention is used to obtain channel attention vectors at each different scale. , , , , Then, the SoftMax function is used to process the obtained channel attention vector to obtain the corresponding attention weights. The calculation formula is as follows: ， S303. Simultaneously perform adaptive feature fusion and channel attention acquisition on feature maps C1, C2, C3, C4, and C5 with spatial attention to obtain feature map F and feature map M. The calculation formula is as follows: ， in, Representing feature map C i The corresponding spatial attention weights, Cat() indicates concatenation by channel; ， in, It is based on C i Generated adaptive weights; S304, Combine the two feature maps and The feature map N output by the multi-scale feature fusion network is obtained by using the attention feature fusion mechanism (AFF). S4. Construct a text detection head network with multi-scale fused feature map N as input and recognition results corresponding to the text image as output; S5. Construct a multi-scale feature fusion text detection network based on feature extraction network, multi-scale feature fusion network, and text detection head network. Train the multi-scale feature fusion scene text detection network with text detection dataset to obtain a multi-scale feature fusion text detection model. Then, use the model to detect the collected text images to obtain recognition results.
2. The scene text detection method based on multi-scale feature fusion according to claim 1, characterized in that, The dilated convolutional pooling module DCMP targets feature maps. The convolution operation was performed using four dilated convolutions with different dilation rates: r=1, 6, 12, and 18.
3. The scene text detection method based on multi-scale feature fusion according to claim 1, characterized in that, Feature map Y ’ 1. After reducing the channel dimension using a 1x1 convolutional block, a new feature map is obtained. .