A natural scene text detection method based on adaptive multi-scale feature fusion
This natural scene text detection method, which utilizes an adaptive multi-scale feature fusion approach, employs a ResNet-18 backbone network and SSKM and CAFM modules for feature extraction and fusion. This addresses the issues of poor detection of small text and high computational cost in existing methods, achieving higher accuracy and robustness in text detection.
Patent Information
- Application Number
- CN202311208412.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-19
- Publication Date
- 2026-02-13
- Estimated Expiration
- 2043-09-19
AI Technical Summary
Existing segmentation-based deep learning methods struggle to separate closely spaced characters in text detection, and their post-processing is complex, computationally expensive, and affects detection accuracy.
A natural scene text detection method with adaptive multi-scale feature fusion is adopted, which includes a feature extraction network, an adaptive multi-scale feature fusion network and a text detection head network. Feature extraction and fusion are performed through a ResNet-18 backbone network, a spatially selective kernel module SSKM and a coordinate adaptive fusion module CAFM, and the text instance is reconstructed using the pixel aggregation algorithm PA.
It improves the detection performance of small texts, alleviates inconsistencies within the feature pyramid, reduces the loss of calculating feature fusion weights, and enhances detection accuracy and robustness.
Smart Images
Figure CN117253236B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of computer vision and image processing, and particularly relates to a natural scene text detection method based on adaptive multi-scale feature fusion. BACKGROUND
[0002] Text detection refers to a process of automatically detecting text in an image using computer vision techniques. The purpose is to extract text information from complex scenes for subsequent natural language processing or other applications.
[0003] Deep learning is a common method for solving text detection tasks. Deep learning models can automatically learn feature representations from a large amount of training data, avoiding the problems caused by manually designed features. On the other hand, deep learning models have good scalability and adaptability, and can handle text of different shapes, sizes and directions, and are usually superior to traditional machine learning methods in performance. In addition, with the continuous development of deep learning technology, more and more architectures and algorithms are proposed and improved, making neural networks perform better in text detection tasks.
[0004] Currently, deep learning methods based on segmentation are widely used in text detection tasks. This method regards the text region as a group of pixels, and determines the text region by segmenting and classifying the pixels. Although this method can better detect irregularly shaped text, it is difficult to separate close text. At the same time, the segmentation-based method generally needs complex post-processing to combine pixel-level results into text lines, and the overhead is often large during prediction.
[0005] Text detection, as a preliminary work of text recognition, directly affects the accuracy of later recognition. Compared with traditional character recognition, the background in natural scenes is complex and difficult to detect, so there is still a lot of room for improvement in text detection research. SUMMARY
[0006] In order to solve the above problems, the purpose of the present application is to provide a natural scene text detection method based on adaptive multi-scale feature fusion.
[0007] In order to achieve the above purpose, the natural scene text detection method based on adaptive multi-scale feature fusion provided by the present application includes the following steps in order:
[0008] Step 1: Select a public dataset as a natural scene text detection dataset, then divide it into a training set and a validation set in proportion, and then preprocess the natural scene text images in the training set to obtain input images;
[0009] Step 2: constructing an adaptive multi-scale natural scene text detection network M1, the adaptive multi-scale natural scene text detection network M1 comprising a feature extraction network, an adaptive multi-scale feature fusion network and a text detection head network;
[0010] Step 3: inputting the input image obtained in step 1 into the adaptive multi-scale natural scene text detection network M1 constructed in step 2, calculating a network loss function and using back propagation to iteratively train network parameters to convergence, thereby obtaining a trained adaptive multi-scale natural scene text detection network M2;
[0011] Step 4: inputting the natural scene text image in the test set obtained in step 1 into the trained adaptive multi-scale natural scene text detection network M2 obtained in step 2, using a pixel aggregation algorithm PA to reconstruct each text instance in the natural scene text image, and combining all the text instances to obtain a text detection result.
[0012] In step 1, the public dataset is selected as the natural scene text detection dataset, then the natural scene text images in the training set are preprocessed, and the method for obtaining the input image is:
[0013] The public dataset ICDAR2015, CTW1500 and Total-Text is selected as the natural scene text detection dataset, then the natural scene text images in the training set are preprocessed, and the method is to scale the natural scene text image to a fixed pixel size, then perform random horizontal flipping, random rotation and random cropping to obtain the input image.
[0014] In step 2, the feature extraction network selects ResNet-18 as the backbone network, which is used to extract five layers of feature maps C1, C2, C3, C4 and C5 with sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, and the corresponding channel numbers are 64, 64, 128, 256 and 512, respectively; the feature map C1 is used to generate a new feature map through a designed spatial selective kernel module SSKM;
[0015] The spatial selection kernel module SSKM uses a nonlinear method to aggregate information from multiple different size convolution kernels to achieve adaptive receptive field size, which sequentially performs three operations: branch, fusion and weight selection; wherein the branch operation uses 3x3 and 5x5 convolution kernels to generate paths of different receptive field sizes; the fusion operation is used to combine and aggregate information from each path, reduce the channel dimension by 1x1 convolution operation, to obtain a comprehensive representation of the selective spatial weight; the weight selection operation aggregates the spatial feature maps of different size convolution kernels by convolution and Softmax, as the weight is multiplied by the corresponding path and added to obtain a new feature map C'1.
[0016] In step 2, the adaptive multi-scale feature fusion network structure includes two cascaded feature pyramid enhancement modules FPEM and a coordinate adaptive fusion module CAFM, and the input is five layers of feature maps C'1, C2, C3, C4, C5 output by the feature extraction network; the processing method is as follows:
[0017] First, input the five layers of feature maps C'1, C2, C3, C4, C5 into the feature pyramid enhancement module FPEM to enhance the extracted features; then, input the enhanced features into the coordinate adaptive fusion module CAFM; the coordinate adaptive fusion module CAFM rescales and adaptively fuses the input five layers of feature maps of different scales; for a certain layer of feature map X l , first adjust all five layers of feature maps to the same size by up or down sampling convolution operation, and the channel number is 128; then, the five layers of feature maps adjusted to the same size are spliced in the channel to obtain a feature map F l ; next, the coordinate attention CA method is used to further extract the feature map F l , and then the channel number is reduced to 5 by 1x1 convolution operation, and each channel result is taken as the weight layer σ l of the corresponding layer of feature map; finally, the weight layer σ l is multiplied pixel by pixel with the feature map F l adjusted to the same size and completed channel splicing corresponding to the layer, and summed to obtain the fused feature map Y l of the layer, the calculation formula is as follows:
[0018] Y l =X 1→l *ε(α l )+X 2→l *ε(β l )+X 3→l *ε(γ l )+X 4→l *ε(δ l )+X 5→l *ε(μ l )
[0019] wherein X n→l represents adjusting the n-th feature map to the l-th size, and l , β l , γ l , δ l , μ l represent five weight layers respectively, and 1 represents the coordinate attention;
[0020] Finally, the five feature maps Y 2 , Y 3 , Y 4 , Y 5 of different scales are adjusted to the same size through up or down sampling convolution operation, and then channel splicing is performed to obtain a feature map F output by the adaptive multi-scale feature fusion network.
[0021] In step 2, the input of the text detection head network is the feature map F output by the adaptive multi-scale feature fusion network, and the output includes six channels, which are: one channel for predicting whether the pixel is a text region, one channel for predicting whether the pixel is a text kernel, and four channels for predicting the similarity vector of the pixel and the surrounding four direction pixels.
[0022] In step 3, the expression of the network loss function is as follows:
[0023] L = L tex + αL ker + β(L agg + L dis )
[0024] wherein L tex is the loss of the text region; L ker is the loss of the text kernel; L agg is the aggregation loss of the text pixel to the corresponding text kernel; L dis is the distance loss between different text kernels; and the coefficients α and β are used to balance the importance of the loss L tex of the text region, the loss L ker of the text kernel, the aggregation loss L agg of the text pixel to the corresponding text kernel, and the distance loss L dis between different text kernels.
[0025] The natural scene text detection method based on adaptive multi-scale feature fusion provided by the present application has the following advantages:
[0026] 1. The feature map output by the feature extraction network is changed from four layers to five layers, and a feature map with a size of 1 / 2 of the input image size is added, which can retain more small text information features. At the same time, because the feature map has shallow layer features and more interference information, a spatial selective kernel module SSKM is proposed to further preprocess and screen the bottom layer feature information to obtain a new feature map, which can avoid the introduction of too much invalid feature information and affect the detection accuracy, enrich the feature expression of the feature extraction network, and improve the problem that the existing method has poor detection effect on small text.
[0027] 2. In the adaptive multi-scale feature fusion network part, a coordinate adaptive fusion module CAFM based on adaptive spatial feature fusion ASFF and coordinate attention CA is proposed to adaptively fuse the multi-scale features across layers, which can alleviate the inconsistency within the feature pyramid, and to a certain extent, avoid the loss caused by frequent up or down sampling when calculating the feature fusion weight. This structure makes the feature fusion more sufficient, and can obtain more accurate text detection head network input features. BRIEF DESCRIPTION OF DRAWINGS
[0028] Figure 1 is the natural scene text detection method flow chart based on adaptive multi-scale feature fusion provided by the application.
[0029] Figure 2 is the adaptive multi-scale natural scene text detection network structure schematic diagram provided by the application.
[0030] Figure 3 is the spatial selective kernel module SSKM structure diagram provided by the application.
[0031] Figure 4 is the adaptive multi-scale feature fusion network structure diagram provided by the application.
[0032] Figure 5 is the detection effect diagram of the application. DETAILED DESCRIPTION
[0033] The natural scene text detection method based on adaptive multi-scale feature fusion provided by the application will be described in detail below in combination with the drawings and specific embodiments.
[0034] As shown in Figure 1 , the natural scene text detection method based on adaptive multi-scale feature fusion provided by the application includes the following steps performed in sequence:
[0035] Step 1: Select a public data set as a natural scene text detection data set, then divide it into a training set and a validation set in proportion, then preprocess the natural scene text images in the training set to obtain input images;
[0036] The public data sets of ICDAR2015, CTW1500 and Total-Text are selected as the natural scene text detection data set, and then are divided into a training set and a verification set in a ratio of 8:2. Then, the natural scene text images in the training set are preprocessed by scaling the natural scene text images into a fixed pixel size, and then performing random horizontal flipping, random rotation and random cropping to obtain input images.
[0037] Step 2: constructing an adaptive multi-scale natural scene text detection network M1, the adaptive multi-scale natural scene text detection network M1 comprising a feature extraction network, an adaptive multi-scale feature fusion network and a text detection head network;
[0038] As shown in Figure 2 , an adaptive multi-scale natural scene text detection network M1 comprising a feature extraction network, an adaptive multi-scale feature fusion network and a text detection head network is constructed;
[0039] The feature extraction network selects ResNet-18 as the backbone network, which is used to extract five layers of feature maps C1, C2, C3, C4 and C5 with sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, and the corresponding channel numbers are 64, 64, 128, 256 and 512, respectively. The feature map C1 contains relatively rich small target text information, but it belongs to shallow features, and the extraction of feature information is not sufficient, so the feature map C1 is generated through a designed spatial selective kernel module SSKM to pre-process and screen the bottom layer feature information, so as to avoid introducing too much invalid feature information and affecting the detection accuracy.
[0040] As shown in Figure 3 , the spatial selective kernel module SSKM uses a nonlinear method to aggregate information from multiple different size convolution kernels to achieve an adaptive receptive field size, which performs three operations in order: branch, fusion and weight selection; the branch operation borrows the Inception idea and uses 3x3 and 5x5 convolution kernels to generate paths with different receptive field sizes; the fusion operation is used to combine and aggregate information from each path, and the channel dimension is reduced through 1x1 convolution operation to obtain a comprehensive representation of the selective spatial weight; the weight selection operation aggregates the spatial feature maps of different size convolution kernels through convolution and Softmax, and then adds them after multiplication with the corresponding path to obtain a new feature map C'1.
[0041] As shown in Figure 4 , the adaptive multi-scale feature fusion network structure comprises two cascaded feature pyramid enhancement modules FPEM and a coordinate adaptive fusion module CAFM, and the input is the five layers of feature maps C'1, C2, C3, C4 and C5 output by the feature extraction network.
[0042] The processing method is as follows:
[0043] Firstly, five layers of feature maps C'1, C2, C3, C4 and C5 are input into a feature pyramid enhancement module FPEM to enhance the extracted features. The feature pyramid enhancement module FPEM is a U-shaped module based on separable convolution, which can enhance the multi-scale features extracted by the backbone network with small computational overhead. In addition, the feature pyramid enhancement module FPEM is stackable, and the receptive field of the network will increase as the number of stacked layers increases. Considering the balance between speed and accuracy, two feature pyramid enhancement modules FPEM are cascaded in the present application;
[0044] After that, the enhanced features are input into a coordinate adaptive fusion module CAFM. The coordinate adaptive fusion module CAFM is an improved algorithm based on adaptive spatial feature fusion ASFF, which can adaptively learn the spatial weights of feature map fusion. It rescales and adaptively fuses the input five layers of different scale feature maps. For a certain layer of feature map X l , first, all five layers of feature maps are adjusted to the same size and channel number of 128 through up or down sampling convolution operation; then, the five layers of feature maps adjusted to the same size are channel spliced to obtain a feature map F l ; Next, in order to preserve as much as possible the spatial hierarchical information of each feature map due to frequent up or down sampling, the coordinate attention CA method is used to further extract the feature map F l , and then the channel number is reduced to 5 through 1x1 convolution operation, and each channel result is taken as the weight layer σ l of the corresponding layer feature map; finally, the weight layer σ l is multiplied with the feature map F l adjusted to the same size and completed channel splicing corresponding layer by layer and pixel by pixel, and summed to obtain the fused feature map Y l of the layer, and the calculation formula is as follows:
[0045] Y l =X 1→l *ε(α l )+X 2→l *ε(β l )+X 3→l *ε(γ l )+X 4→l *ε(δ l )+X 5→l *ε(μ l )
[0046] Wherein, X n→l represents the n-th layer feature map adjusted to the l-th layer size, α l , β l, gamma l , delta l , mu l represent five weight layers respectively, and epsilon(·) represents coordinate attention.
[0047] Finally, the feature maps Y 1 , Y 2 , Y 3 , Y 4 , Y 5 of five different scales are adjusted to the same size through up or down sampling convolution operation, and then channel splicing is performed to obtain a feature map F output by the adaptive multi-scale feature fusion network.
[0048] The input of the text detection head network is the feature map F output by the adaptive multi-scale feature fusion network, and the output includes six channels, which are: one channel for predicting whether the pixel is a text region, one channel for predicting whether the pixel is a text kernel, and four channels for predicting the similarity vector of the pixel and the surrounding four direction pixels.
[0049] Step 3: input the input image obtained in step 1 into the adaptive multi-scale natural scene text detection network M1 constructed in step 2, calculate the network loss function, and use back propagation to iteratively train the network parameters to convergence to obtain a trained adaptive multi-scale natural scene text detection network M2.
[0050] The expression of the network loss function is as follows:
[0051] L = L tex + alpha L ker + beta (L agg + L dis )
[0052] Wherein, L tex is the loss of the text region; L ker is the loss of the text kernel; L agg is the aggregation loss of the text pixel to the corresponding text kernel; L dis is the distance loss between different text kernels; the coefficients alpha and beta are used to balance the importance of the loss L tex of the text region, the loss L ker of the text kernel, the aggregation loss L agg of the text pixel to the corresponding text kernel, and the distance loss L dis between different text kernels, which are set to 0.5 and 0.25 respectively in the present application.
[0053] Step 4: input the natural scene text image in the test set obtained in step 1 into the trained adaptive multi-scale natural scene text detection network M2 obtained in step 2, reconstruct each text instance in the natural scene text image by using the pixel aggregation algorithm PA, and combine all the text instances to obtain a text detection result.
[0054] In order to verify the effect of the present application, the present inventors carried out the following experiments.
[0055] All experiments were implemented by using the Pytorch deep learning framework, trained and tested on two Nvidia 2080ti natural scene text images, the training round was 600, the batch size was 16, the Adam algorithm was used for optimization, the initial learning rate was 0.001, and the learning rate was reduced by using the poly method. Then, each text instance in the Nvidia 2080ti natural scene text image was reconstructed by using the pixel aggregation algorithm PA, and all the text instances were combined to obtain a text detection result.
[0056] The following table is the performance test structure of three different networks on the CTW1500 dataset, and the accuracy, recall rate and harmonic mean are used as evaluation indexes.
[0057] Experiment 1: no improvement, only PAN network is used for experiment; experiment 2: CAFM is added to the adaptive feature fusion network; experiment 3: SSKM is added to the feature extraction network and CAFM is added to the adaptive feature fusion network.
[0058] Accuracy Recall Harmonic Mean Model Size Experiment 1 80.70％ 80.02％ 80.36％ 147.2M Experiment 2 83.15％ 79.63％ 81.35％ 172.5M Experiment 3 83.28％ 79.98％ 81.62％ 173.4M
[0059] The comparative experiment results show that under the premise of a small amount of improvement of the model parameters, the coordinate adaptive fusion module CAFM and the spatial selective kernel module SSKM provided by the present application can improve the evaluation indexes on the dataset, and have stronger detection effect and model robustness in the natural scene text detection task.
[0060] The detection effect of the present application on the CTW1500 dataset is shown in the following table. Figure 5
Claims
1. A natural scene text detection method based on adaptive multi-scale feature fusion, the natural scene text detection method based on adaptive multi-scale feature fusion comprising the following steps performed in sequence: Step 1: selecting a public dataset as a natural scene text detection dataset, then dividing it into a training set and a validation set in proportion, and then preprocessing the natural scene text images in the training set to obtain input images; Step 2: constructing an adaptive multi-scale natural scene text detection network M1, the adaptive multi-scale natural scene text detection network M1 comprising a feature extraction network, an adaptive multi-scale feature fusion network, and a text detection head network; Step 3: inputting the input images obtained in Step 1 into the adaptive multi-scale natural scene text detection network M1 constructed in Step 2, calculating a network loss function, and iteratively training network parameters to convergence using back propagation, to obtain a trained adaptive multi-scale natural scene text detection network M2; Step 4: inputting the natural scene text images in the test set obtained in Step 1 into the trained adaptive multi-scale natural scene text detection network M2 obtained in Step 2, and using a pixel aggregation algorithm PA to reconstruct each text instance in the natural scene text image, and combining all the text instances to obtain a text detection result; characterized in that In Step 2, the feature extraction network selects ResNet-18 as a backbone network for extracting five layers of feature maps C1, C2, C3, C4, and C5 with sizes of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, and the corresponding channel numbers are 64, 64, 128, 256, and 512, respectively; and the feature map C1 is used to generate a brand-new feature map through a designed spatially selective kernel module SSKM; The spatially selective kernel module SSKM uses a nonlinear method to aggregate information from multiple different size convolution kernels to achieve an adaptive receptive field size, and performs three operations in sequence: branching, fusion, and weight selection; wherein the branching operation uses 3x3 and 5x5 convolution kernels to generate paths with different receptive field sizes; the fusion operation is used to combine and aggregate information from each path, and reduces the channel dimension through a 1x1 convolution operation to obtain a comprehensive representation of the selective spatial weight; the weight selection operation aggregates the spatial feature mappings of different size convolution kernels through convolution and Softmax, and adds them after being multiplied by the corresponding paths to obtain a brand-new feature map C'1.
2. The natural scene text detection method based on adaptive multi-scale feature fusion according to claim 1, characterized in that: In Step 2, the adaptive multi-scale feature fusion network structure comprises two cascaded feature pyramid enhancement modules FPEM and a coordinate adaptive fusion module CAFM, and the input is the five layers of feature maps C'1, C2, C3, C4, and C5 output by the feature extraction network; the processing method is as follows: First, input the five layers of feature maps C'1, C2, C3, C4, and C5 into the feature pyramid enhancement module FPEM to enhance the extracted features; then, input the enhanced features into the coordinate adaptive fusion module CAFM; The coordinate adaptive fusion module CAFM rescales and adaptively fuses the input five layers of feature maps of different scales; For a layer feature map X l , first adjust all five layers of feature maps to the same size and channel number of 128 through up or down sampling convolution operation; then splice the five layers of feature maps adjusted to the same size in the channel to obtain a feature map F l ; next, further extract the feature map F l using the coordinate attention CA method, and then reduce the channel number to 5 through 1×1 convolution operation, and take each channel result as the weight layer σ l of the corresponding layer feature map; finally, multiply the weight layer σ l with the feature map F l adjusted to the same size and completed channel splicing corresponding to the layer pixel by pixel and summed to obtain the fused layer feature map Y l , and the calculation formula is as follows: Y l = X 1→l * ε(α l ) + X 2→l * ε(β l ) + X 3→l * ε(γ l ) + X 4→l * ε(δ l ) + X 5→l * ε(μ l ) wherein X n→l represents adjusting the n-th feature map to the l-th size, a l , β l , γ l , δ l , μ l represent five weight layers respectively, and ε(·) represents coordinate attention; Finally, the feature maps Y at five different scales are... 1 Y 2 Y 3 Y 4 Y 5 By adjusting the convolutional size to the same size through upsampling or downsampling, and then performing channel concatenation, the feature map F output by the adaptive multi-scale feature fusion network is obtained.