Text Detection Method and System Based on Cross-Level Feature Enhancement and Auxiliary Feature Guidance

Abstract

This invention discloses a text detection method and system based on cross-level feature enhancement and auxiliary feature guidance, belonging to the field of computer vision technology. The method includes the following steps: extracting feature maps of different levels from the image to be detected; for low-level features, dynamically adjusting the receptive field by adjusting the convolutional kernel size to obtain multi-scale information, generating enhanced features for low-level features; for high-level features, using differential convolution to enhance high-frequency information at text edges, generating enhanced features for high-level features; fusing features from different levels using cross-level feature enhancement to generate fused features; extracting semantic information from high-level features to generate auxiliary features; for the generated auxiliary features and fused features, using attention guidance to determine the attribution of text kernel pixels, obtaining attention-guided features; and performing text-to-non-text prediction on pixels to generate text bounding boxes. The scheme in this invention enables the network to more accurately segment text regions.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a text detection method and system based on cross-level feature enhancement and auxiliary feature guidance. Background Technology

[0002] As a crucial medium for human communication and knowledge dissemination, text plays a vital role in daily life, production, and learning. Natural scene images contain rich textual information, which is essential for understanding the surrounding environment. In recent years, with the gradual maturation of artificial intelligence theory and the continuous development of computer vision devices, computer vision technology has received increasing attention. Accurately extracting textual information from images helps in understanding scene images. Text detection, as the first step in obtaining textual information from images, accurately and quickly locating text regions facilitates the efficient execution of subsequent tasks such as text recognition, image layout analysis, and machine translation.

[0003] Currently, text detection technology is widely used in various aspects of life. In the field of autonomous driving: cameras and sensors installed on autonomous vehicles can capture images of the surrounding scene, and then text detection and recognition technologies can be used to extract text information such as signs, billboards, and license plates from the images. This helps the vehicle judge its surrounding environment, thereby improving the safety of autonomous driving. In the field of real-time photo translation: with the continuous improvement of people's living standards, overseas travel has become a popular way to travel. Many translation software programs offer photo translation services. These programs reliably detect and recognize text from smartphone images and translate the text information in real time, which can greatly facilitate people's travel. Therefore, natural scene text detection technology, as one of the fundamental tasks in the field of computer vision technology, has important research value and broad application prospects.

[0004] The main task of text detection is to detect text regions in natural scene images and mark them with rectangular or polygonal bounding boxes. Early traditional text detection methods primarily studied the feature differences between text and background. However, with the continuous development of deep learning technology, deep learning-based text detection methods have gradually gained widespread attention. These methods can be divided into regression-based natural scene text detection methods, contour fitting-based text detection methods, and segmentation-based text detection methods. Regression-based text detection methods generate a predicted score map through a fully convolutional network, and then directly regress the bounding boxes of the text regions based on the score map. In 2017, Zhou et al. proposed an efficient and accurate scene text detector (EAST), which directly predicts the geometric features of text regions through a fully convolutional network without the need for complex anchoring mechanisms, improving detection speed while maintaining accuracy. In 2021, He et al. proposed a multi-oriented scene text detector (MOST), which first generates a coarse detection result, and then dynamically adjusts the receptive field through deformable convolution to generate a more refined result. Although regression-based text detection methods perform well in multi-directional text detection, they are limited by the shape of the bounding box, making it difficult to achieve accurate results when detecting curved text.

[0005] Contour-fit-based text detection methods are similar in model to regression- and segmentation-based methods. Their unique feature lies in directly fitting text contours using a novel approach. In 2018, Long et al. proposed TextSnake (a flexible representation for detecting text of arbitrary shapes), which innovatively uses a series of overlapping disks to cover the text region. The text boundary curve is formed by the tangents of all the disks, transforming the text region detection problem into predicting the center coordinates, radius, and orientation of each disk. The Fourier Contour Embedding for Arbitrary-Shaped Text Detection (FCENet), proposed in 2021, predicts the Fourier feature vectors of text instances and then reconstructs the text contours in the image space domain using inverse Fourier transform. Compared to segmentation-based text detection methods, these methods require predicting more parameters during the testing phase, impacting model performance.

[0006] Segmentation-based text detection methods transform the text detection task into a pixel-level classification problem, providing high-precision text localization. The Progressive Scale Expansion Network (PSENet), proposed in 2019, innovatively employs a scale expansion method. This method generates a text kernel for each text instance and iteratively merges adjacent text pixels, starting from the kernel's pixels, to obtain the final text region. However, this method has a complex post-processing process and low detection efficiency. In 2020, Liao et al. proposed the Differentiable Binarization Network (DBNet), which effectively simplifies the post-processing, significantly improves detection speed, and demonstrates superior performance on a lightweight backbone network. This method mainly introduces a differentiable binarization function, optimizing the binarization process along with the model during training. During testing, the differentiable binarization part can be removed without affecting the model's detection speed.

[0007] While the aforementioned text detection methods have achieved excellent results, detecting text instances in natural scene images presents inherent challenges: large differences in text scale and extreme aspect ratios. To address these challenges, existing methods typically employ feature pyramid networks for feature fusion. However, when fusing multi-scale features directly through upsampling and pixel-level addition, small-scale and large-scale text share the same resolution feature map, neglecting the uniqueness of features within different receptive fields, leading to missed detections of small-scale text. Furthermore, directly fusing multi-level features from top to bottom ignores the feature differences between different levels, resulting in inaccurate text edge segmentation. This can cause the same text instance with extreme aspect ratios to be detected as multiple text regions, impacting the model's detection performance.

[0008] To alleviate these problems, this invention proposes a text detection method and system based on cross-level feature enhancement and auxiliary feature guidance. The proposed text detection method is used to detect text in scenes of arbitrary shapes, addressing the challenge of detecting text with large scale differences and extreme aspect ratios in natural scene images. This method employs unequal processing for features at different levels, dynamically adjusting the receptive field of low-level features to obtain multi-scale information, and introducing differential convolution to enhance high-frequency edge information in high-level features, thereby more accurately locating text at different scales. Furthermore, this method utilizes high-level features to generate auxiliary features to help the network distinguish different text instances, avoiding the problem of a single text with an extreme aspect ratio being detected as multiple text instances. This method provides a research scheme for deep learning methods in natural scene text detection and, to some extent, promotes the development of visual tasks such as image understanding. Summary of the Invention

[0009] The purpose of this invention is to provide a text detection method and system based on cross-level feature enhancement and auxiliary feature guidance, so as to solve the problem that existing methods proposed in the background art have difficulty detecting text with large scale differences and extreme aspect ratios in natural scene images.

[0010] To achieve the above objectives, the present invention employs the following technical solution:

[0011] In its first aspect, this invention proposes a text detection method based on cross-level feature enhancement and auxiliary feature guidance, comprising the following steps:

[0012] S1. Extract feature maps at different levels from the image to be detected;

[0013] S2. For low-level features, the receptive field is dynamically adjusted by adjusting the size of the convolution kernel to obtain multi-scale information and generate enhanced features for low-level features.

[0014] S3. For high-level features, differential convolution is used to enhance high-frequency information at text edges, generating enhanced features for high-level features.

[0015] S4. Use cross-level feature enhancement to fuse features from different levels. The enhanced features of low-level features and the enhanced features of high-level features are fused one-to-one. The fusion is carried out gradually until all features from different levels are fused to generate fused features.

[0016] S5. Extract semantic information from high-level features and generate auxiliary features;

[0017] S6. For the generated auxiliary features and fused features, the attention-guided method is used to determine the ownership of the text kernel pixels, and the attention-guided features are obtained.

[0018] S7. Based on the attention-guided features, perform text-to-non-text prediction on pixels to generate text bounding boxes.

[0019] This text detection method addresses the challenge of detecting text with large scale differences and extreme aspect ratios in natural scene images. It employs unequal processing for features at different levels, dynamically adjusting the receptive field of low-level features to obtain multi-scale information, and introducing differential convolution to enhance high-frequency edge information in high-level features, thereby more accurately locating text at different scales. Furthermore, this method utilizes high-level features to generate auxiliary features to help the network distinguish different text instances, avoiding the problem of detecting a single text with an extreme aspect ratio as multiple text instances.

[0020] Preferably, S1 is specifically as follows:

[0021] The image to be detected is processed by the backbone network to obtain feature maps of four different levels, from stage 2 to stage 5. The number of channels is adjusted to 256 through 1×1 convolutional layers. The adjusted multi-level features are represented as {X2,X3,X4,X5}.

[0022] Preferably, step S2 is as follows:

[0023] The low-level features are first dynamically adjusted in receptive field using convolution operations with different kernel sizes to obtain multi-scale information. Then, feature maps of different receptive field sizes are concatenated along the channel dimension, and the number of channels is adjusted using 1×1 convolutions to obtain multi-scale features. Subsequently, attention scores are learned to determine the channel importance of different receptive fields. Finally, the weights of the corresponding channels are multiplied by the multi-scale features to obtain the enhanced features of the low-level features. This process of generating enhanced features is represented as follows:

[0024] X' L =Conv 1×1 ([Conv 1×1 (X i ),Conv 3×3 (X i ),Conv 5×5 (X i )])

[0025] X L =σ(f FC (ReLU(f FC (f GAP (X' L )))))×X' L

[0026] Among them, X i X represents low-level features. L Represents enhanced features of low-level features, [·] represents features concatenated along the channel dimension, Conv k×k (·) represents a convolution operation with a kernel size of k×k, f GAP (·) indicates global average pooling, f FC (·) represents a fully connected layer, ReLU(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function.

[0027] Preferably, step S3 is as follows:

[0028] High-level features are enhanced using differential convolution. First, sub-pixel convolutions are applied to increase the resolution of the input features. Then, three convolutional layers are used to enhance the features. These layers use ordinary convolutions with 3×3 kernels to aggregate local features, central differential convolutions, and corner differential convolutions to enhance high-frequency edge information. Finally, the enhanced features from the three branches are summed pixel-by-pixel and then activated using the ReLU activation function to generate enhanced features for the high-level features. The functional representation of this enhanced feature generation process is as follows:

[0029] X' H =f sub×N (Conv 3×3 (X j ))

[0030] X H =ReLU(Conv 3×3 (X' H )+Conv CD (X' H )+Conv AD (X' H ))+X' H

[0031] Among them, X j X represents a high-level feature. H Conv represents an enhanced feature representing a higher-level feature. 3×3 (·) represents a convolution operation with a kernel size of 3×3, f sub×N (·) denotes a subpixel convolution operation with an upsampling rate of N, Conv CD (·) denotes central difference convolution, Conv AD (·) represents angular difference convolution, and ReLU(·) represents the ReLU activation function.

[0032] Preferably, step S4 is as follows:

[0033] The features of different levels are combined in a one-to-one correspondence between low-level features and high-level features to obtain several sets of preliminary cross-level features. First, each set of preliminary cross-level features generates its own enhanced features and then they are fused. After fusion, the features are combined again in a one-to-one correspondence to obtain several sets of two-level cross-level features. Then, each set of two-level cross-level features generates its own enhanced features and they are fused, until all features are fused.

[0034] First, the cross-level features X2 and X4, and X3 and X5 are fused using the first step of feature enhancement to generate X2′ and X3′ respectively. Then, the different-level features X2′ and X3′ obtained in the first step are fused using the second step of feature enhancement. By fusing the extracted multi-level features in a stepwise manner, a fused feature X is generated. CFeature enhancement fusion involves adding the enhanced features of low-level features and the enhanced features of high-level features to obtain the fused feature, as shown below:

[0035] X′ i =X H +X L

[0036] Among them, X L X represents the enhancement feature of low-level features. H X represents the enhanced feature of a higher-level feature. i ′ represents the fused feature after enhanced feature fusion.

[0037] Preferably, step S5 is as follows:

[0038] Feature X5 is first enriched with semantic information through an information extraction block, which consists of a 1×1 convolution, a ReLU activation function, and batch normalization. Then, subpixel convolution is used to increase the resolution. Finally, pixel-level subtraction is performed with feature X4 to remove redundant information, resulting in the feature X5. Its function representation is as follows:

[0039] X5'=f BN (Conv 1×1 (ReLU(f BN (Conv 1×1 (X5)))))+X5

[0040]

[0041] Among them, Conv 1×1 (·) represents a 1×1 convolution operation, f BN (·) denotes batch normalization, ReLU(·) denotes the ReLU activation function, f sub×2 (·) indicates a subpixel convolution operation with an upsampling rate of 2;

[0042] Features after noise suppression Further feature enhancement is performed; firstly, texture information is preserved through parallel global average pooling and global max pooling, and then the two pooled features are added at the pixel level before weight generation to obtain the attention map X. W Finally, attention maps and noise-suppressed features are utilized. Auxiliary feature X is obtained by performing multiplication operations and residual connections. A ; indicates the following:

[0043]

[0044] Among them, f GAP (·) indicates global average pooling, fGMP (·) indicates global max pooling, f FC (·) represents a fully connected layer, ReLU(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function. The transformation operation represents the reshaping of a vector into an attention map.

[0045] Preferably, step S6 is as follows:

[0046] For auxiliary features and fused features, the resolution of the auxiliary features is first increased by a factor of 4 by upsampling using bilinear interpolation. Then, the auxiliary features are concatenated with the fused features along the channel dimension. Finally, attention weights X are generated by convolution with a kernel size of 3×3 and activation functions. S Finally, the attention weights are multiplied by the fused features and a residual connection is performed to obtain the guided features X. G ; indicates the following:

[0047] X Cat =[X C ,f up×4 (X A )]

[0048] X S =σ(Conv 3×3 (ReLU(Conv 3×3 (X Cat ))))

[0049] X G =X C ×X S +X C

[0050] Among them, f up×4 (·) indicates a bilinear interpolation upsampling operation with a scaling factor of 4, [·] indicates feature concatenation along the channel dimension, Conv 3×3 (·) represents a 3×3 convolutional layer, ReLU(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function.

[0051] Preferably, step S7 is as follows:

[0052] Feature X G The input is processed by two parallel branches to obtain single-channel threshold maps X. T And single-channel text probability map X P The parallel branch includes upsampling operations and text-to-non-text prediction of pixels; subsequently, a post-processing process is performed to obtain a binary image, from which text bounding boxes are generated. The functions for prediction and generating the binary image are expressed as follows:

[0053] X O=σ(f deconv (f deconv (Conv 3×3 (X G ))))

[0054]

[0055] Among them, Conv 3×3 (·) represents a 3×3 convolutional layer, f deconv (·) indicates that the feature size is doubled through deconvolution, and σ(·) represents the Sigmoid activation function. B This represents the generated binary image, where (i,j) represents the pixel coordinates in the image.

[0056] In a second aspect, this invention proposes a text detection system based on cross-level feature enhancement and auxiliary feature guidance, comprising:

[0057] The backbone network is constructed by replacing the 3×3 convolutional layers in stages 3, 4, and 5 of ResNet-50 with deformable convolutions to obtain feature maps {X2,X3,X4,X5} at four different levels from stage 2 to stage 5. The output features of each stage are then input into the cross-level feature enhancement module.

[0058] The cross-level feature enhancement module dynamically adjusts the receptive field of low-level features to obtain multi-scale information, introduces differential convolution to enhance high-frequency edge information of high-level features, and adds the enhanced low-level and high-level features to output the cross-level enhanced fused feature X. C The method employs cross-level feature enhancement to fuse features from different levels. The cross-level features are first fed into the cross-level feature enhancement module, and then the output features from both levels are fed into the cross-level feature enhancement module to generate the fused feature X. C ;

[0059] The auxiliary weight generation module is used to enhance the semantic information in high-level features and generate auxiliary features X. A The semantic information of high-level feature X5 is enriched by the information extraction block of the auxiliary weight generation module, and then pixel-level subtraction is performed with feature X4 to remove useless noise in the background to obtain the feature. Will Texture information is preserved through global average pooling and global max pooling operations, and then attention map X is obtained through pixel-level addition and weight generation. W Finally, attention maps and noise-suppressed features are utilized. Auxiliary feature X is obtained by performing multiplication operations and residual connections. A ;

[0060] The attention guidance module is used to guide the correct segmentation of text regions and obtain the guided features X.G ;

[0061] The output module uses upsampling operations and text-to-non-text prediction on pixels to generate text bounding boxes.

[0062] Compared with the prior art, the beneficial effects of the present invention are:

[0063] This invention proposes a text detection method based on cross-level feature enhancement and auxiliary feature guidance, aiming to address the challenge of detecting text with large scale differences and extreme aspect ratios in natural scene images. The invention mainly comprises a cross-level feature enhancement module (CFEM), an auxiliary weight generation module (AWGM), and the latter's attention guidance for the former. Specifically, the CFEM enhances features by applying unequal processing to features at different levels, resulting in fused features containing more effective information; and the AWGM enhances the semantic information in high-level features, enabling the network to correctly segment text regions. Extensive experiments on four text detection benchmark datasets demonstrate the effectiveness of the proposed method. This invention provides a research framework for deep learning methods in natural scene text detection, and to some extent promotes the development of visual tasks such as image understanding. Attached Figure Description

[0064] Figure 1 This is a diagram showing the overall network structure of the text detection system based on cross-level feature enhancement and auxiliary feature guidance in this invention.

[0065] Figure 2 This is a structural diagram of the cross-level feature enhancement module in this invention;

[0066] Figure 3 This is a structural diagram of the auxiliary weight generation module in this invention;

[0067] Figure 4 This is a structural diagram of the attention guidance module in this invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example 1:

[0070] This invention proposes a text detection network based on cross-level feature enhancement and auxiliary feature guidance. The overall network structure of this text detection network is as follows: Figure 1 As shown, the text detection network consists of a feature extraction network, a cross-level feature enhancement module (CFEM), an auxiliary weight generation module (AWGM), an attention guidance module, and an output module. The specific text detection method in this invention is as follows:

[0071] Step 1: Construct the backbone network and extract features.

[0072] Feature extraction is performed using ResNet-50 as the backbone network. To enhance the feature extraction capability of the backbone network, this invention replaces the 3×3 convolutional layers in stages 3, 4, and 5 of ResNet-50 with deformable convolutions. The input image passes through the backbone network to obtain feature maps at four different levels, from stage 2 to stage 5.

[0073] First, the number of channels is adjusted to 256 using 1×1 convolutional layers. The adjusted multi-level features are represented as {X2,X3,X4,X5}, with resolutions of {1 / 4,1 / 8,1 / 16,1 / 32} of the input size.

[0074] Step 2: Construct a cross-level feature enhancement module to enhance features by applying unequal processing to features at different levels, thereby obtaining fused features containing more effective information.

[0075] Text scale varies greatly in natural scene images. However, simply using a series of upsampling operations and pixel summation to obtain fused features ignores the differences in features extracted by convolutional networks of different depths, thus affecting the model's ability to recognize text at different scales and reducing the model's detection accuracy. Therefore, this invention employs a Cross-level Feature Enhancement Module (CFEM) to apply different processing methods to features at different levels. The specific structure of the Cross-level Feature Enhancement Module (CFEM) is as follows: Figure 2 As shown, this invention first feeds the feature maps of the four levels into the cross-level feature enhancement module CFEM, and then passes the enhanced features through the cross-level feature enhancement module CFEM to obtain a more informative fused feature X. C .

[0076] (1) Dynamically adjust the receptive field of low-level features to obtain multi-scale information;

[0077] This invention dynamically adjusts the receptive field of the input low-level features by adjusting the convolution kernel size to obtain multi-scale information. Then, feature maps with different receptive field sizes are concatenated along the channel dimension, and the number of channels is adjusted using a 1×1 convolution to ensure that the low-level and high-level features correspond in the channel dimension. After obtaining the multi-scale features, attention scores are learned to determine the channel importance of different receptive fields. For multi-scale features, spatial information is first integrated using global average pooling, followed by feature vector prediction using two fully connected layers and a ReLU activation function. Finally, the weights of the corresponding channels are obtained through a Sigmoid activation function and multiplied with the multi-scale features to obtain the enhanced features of the low-level features. The specific implementation process is as follows:

[0078] X' L =Conv 1×1 ([Conv 1×1 (X i ),Conv 3×3 (X i ),Conv 5×5 (X i )])

[0079] X L =σ(f FC (ReLU(f FC (f GAP (X' L )))))×X' L

[0080] Where [·] represents the concatenation of features along the channel dimension, Conv k×k (·) represents a convolution operation with a kernel size of k×k, f GAP (·) indicates global average pooling, f FC (·) represents a fully connected layer, ReLU(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function.

[0081] (2) Differential convolution is introduced into high-level features to enhance high-frequency edge information;

[0082] In natural scene images, high-frequency information at text edges is crucial for distinguishing text regions from background regions. As feature extraction networks become deeper and edge information becomes smoother, high-frequency information in high-level features is gradually lost. This severely impacts the model's ability to accurately locate text boundaries, especially when detecting small-scale text, where the loss of high-frequency information is even more pronounced. Therefore, this invention introduces differential convolution to enhance the information of the input high-level features. For the input high-level features, the resolution is first increased through sub-pixel convolution, which includes a 3×3 convolutional layer and channel shuffling. Subsequently, the generated high-resolution features are enhanced using differential convolution.

[0083] Specifically, the model first uses three convolutional layers to enhance features. First, a standard 3×3 convolutional layer aggregates local features. Then, center difference convolution and corner difference convolution enhance high-frequency edge information. Next, the enhanced features from the three branches are summed pixel-by-pixel and then activated using the ReLU function to generate enhanced features for higher-level features. Finally, the enhanced features from the low-level and high-level features are summed to output cross-level enhanced features. By addressing the inequality in cross-level features, the model's ability to detect multi-scale text is enhanced. The specific implementation process can be represented as follows:

[0084] X' H =f sub×N (Conv 3×3 (X j ))

[0085] X H =ReLU(Conv 3×3 (X' H )+Conv CD (X' H )+Conv AD (X' H ))+X' H

[0086] X′ i =X H +X L

[0087] Among them, Conv 3×3 (·) represents a convolution operation with a kernel size of 3×3, f sub×N (·) represents a subpixel convolution operation with an upsampling rate of N, such as Figure 1 As shown, in the first step of feature enhancement, N is set to 4, and in the second step of feature enhancement, N is set to 2. Conv CD (·) denotes central difference convolution, Conv AD (·) represents angular difference convolution, and ReLU(·) represents the ReLU activation function.

[0088] Step 3: Construct an auxiliary weight generation module to generate auxiliary features.

[0089] Convolutional neural networks can extract rich feature information from natural scene images. However, low-level feature information contains a lot of redundant noise, which seriously interferes with the model's accurate localization of text instances, causing the model to incorrectly aggregate background pixels into the text kernel or incorrectly segment the text region. High-level features contain semantic information that usually involves the entire image or text instance, making it easier to understand the context of the image and helping the model to correctly segment the text region. Therefore, to address the above problems, this invention generates auxiliary features through the Auxiliary Weight Generation Module (AWGM), and then combines them with the fused features to accurately determine text instances through attention guidance.

[0090] The specific structure of the auxiliary weight generation module AWGM is as follows: Figure 3 As shown, the input features of this module are high-level features X4 and X5 extracted from the backbone network. Input feature X5 is first enriched with semantic information through information extraction blocks, and then the resolution of the feature map is magnified by two times through subpixel convolution. Subsequently, pixel-level subtraction is performed with feature X4 to remove useless noise in the background. The information extraction block mainly consists of 1×1 convolution, ReLU activation function, and batch normalization operation to generate features. The process is as follows:

[0091] X5'=f BN (Conv 1×1 (ReLU(f BN (Conv 1×1 (X5)))))+X5

[0092]

[0093] Among them, Conv 1×1 (·) represents a 1×1 convolution operation, f BN (·) denotes batch normalization, ReLU(·) denotes the ReLU activation function, f sub×2 (·) indicates a subpixel convolution operation with an upsampling rate of 2.

[0094] Subsequently, to more accurately generate auxiliary features to guide the attention of the fused features, this invention further enhances the noise-suppressed features. First, texture information is preserved through parallel global average pooling and global max pooling. Then, the two pooled features are added at the pixel level and weighted to generate the attention map X. W Finally, attention maps and noise-suppressed features are utilized. Multiplication and residual connections are performed to obtain auxiliary features, which help the model effectively distinguish text instances. Auxiliary feature X is generated. A The process is as follows:

[0095]

[0096] Among them, f GAP (·) indicates global average pooling, f GMP (·) indicates global max pooling, f FC (·) represents a fully connected layer, ReLU(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function. The transformation operation reshapes the vector into the shape of the attention map to facilitate subsequent operations.

[0097] Step 4: Provide attention guidance to correctly segment the text region.

[0098] For the generated auxiliary features and fused features, this invention uses attention guidance to help the model determine the ownership of text kernel pixels. The attention guidance process is as follows: Figure 4 As shown. The implementation process of attention guidance is as follows:

[0099] X Cat =[X C ,f up×4 (X A )]

[0100] X S =σ(Conv 3×3 (ReLU(Conv 3×3 (X Cat ))))

[0101] X G =X C ×X S +X C

[0102] Among them, f up×4 (·) indicates a bilinear interpolation upsampling operation with a scaling factor of 4, [·] indicates feature concatenation along the channel dimension, Conv 3×3 (·) represents a 3×3 convolutional layer, ReLU(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function.

[0103] Step 5: Generate text bounding boxes.

[0104] X G The input is fed into two parallel output modules for post-processing, and single-channel threshold maps X are obtained through prediction. T And single-channel text probability map X P The output module mainly includes upsampling operations and text-to-non-text prediction of pixels. After post-processing, a binary image is obtained, from which text bounding boxes are generated. The implementation process is shown in the following formula:

[0105] X O =σ(f deconv (f deconv (Conv 3×3 (X G ))))

[0106]

[0107] Among them, Conv 3×3 (·) represents a 3×3 convolutional layer, f deconv (·) indicates that the feature size is doubled through deconvolution, and σ(·) represents the Sigmoid activation function. B This represents the generated binary image, where (i,j) represents the pixel coordinates in the image.

[0108] Experimental verification:

[0109] (1) Dataset:

[0110] To verify the effectiveness of the proposed method, experiments were conducted on four text detection benchmark datasets. The datasets used specifically include:

[0111] The MSRA-TD500 dataset contains 500 natural images, including 300 training images and 200 test images. It includes both Chinese and English languages. Considering the limited number of training images, an additional 400 images from the HUST-TR400 dataset were added during training.

[0112] The ICDAR2015 dataset was captured in natural street scenes, and the text in the images exhibits blurring and distortion. The training set contains 1000 images, and the test set contains 500 images.

[0113] Total-Text is a classic arbitrary-shape text detection dataset. The dataset contains 1555 images, of which 1255 are used for training and the remaining 300 are used for testing.

[0114] The CTW1500 dataset contains horizontal, slanted, and curved text. It includes 1,000 training images and 500 test images, covering multiple languages, primarily Chinese and English.

[0115] SynthText is a large artificially synthesized dataset containing over 850,000 synthetic text images. The method proposed in this invention is pre-trained only on this dataset.

[0116] (2) Experimental setup:

[0117] The method proposed in this invention is implemented using Python, and the experimental environment is shown in Table 1. The entire network is trained using the SGD optimizer. During parameter initialization, the backbone network is loaded with weights pre-trained on ImageNet, while other network layers are initialized using the Kaiming initialization method. The pre-training phase of the model is performed on the synthetic dataset SynthText, with a fixed learning rate of 0.007, a batch size of 8, and a total of 2 epochs. The fine-tuning phase is performed on four benchmark datasets, with an initial learning rate of 0.007, and a total of 1200 epochs. Before training, image data augmentation is performed, with training images randomly cropped to 640×640 pixels, and horizontal flipping and random rotation are used to enhance the model's generalization ability.

[0118] Table 1 Experimental Environment

[0119]

[0120]

[0121] (3) Experimental results:

[0122] The proposed text detection method based on cross-level feature enhancement and auxiliary feature guidance is evaluated on four commonly used text detection benchmark datasets and compared with existing methods. The method is evaluated based on four metrics: precision, recall, H-means, and frames per second (FPS). Furthermore, ablation experiments are conducted on the MSRA-TD500 dataset to verify the effectiveness of each part of the invention.

[0123] The performance of this invention was compared with other text detection methods on a benchmark dataset. The evaluation results are shown in Tables 2 and 3. The best performance of each indicator in the tables is marked in bold.

[0124] Table 2 shows the experimental results using the benchmark datasets MSRA-TD500 and ICDAR 2015.

[0125]

[0126] Table 3 shows the experimental results for the benchmark datasets Total-Text and CTW1500.

[0127]

[0128]

[0129] As shown in Table 2, the proposed method achieved the highest accuracy on both the MSRA-TD500 and ICDAR2015 datasets. Particularly on the MSRA-TD500 dataset, all three metrics of the proposed method were optimal, fully demonstrating its superior performance in detecting text in multi-directional scenes. Table 3 presents the experimental results of various text detection methods on curved text datasets. As can be seen from the table, the proposed method achieved the highest recall on both natural scene text datasets and effectively balanced detection accuracy and speed.

[0130] The superior performance on the four basic datasets is attributed to the proposed method's unequal treatment of high-level and low-level features and its cross-level feature fusion, effectively addressing the challenge of large text scale differences and improving detection results. Furthermore, by generating auxiliary features to guide feature fusion, the model is helped to distinguish different text instances, preventing the model from incorrectly segmenting a single text instance into multiple text regions or detecting multiple text instances as a single text, thus improving the model's detection performance.

[0131] To further demonstrate the effectiveness of the method of the present invention, Tables 4, 5 and 6 show the ablation experiments conducted on the MSRA-TD500 dataset to illustrate the role of each component in the method of the present invention.

[0132] Table 4 Ablation Experiment Results of the Main Module

[0133] CFEM AWGM Precision Recall Hmean / / 88.97 79.04 83.71 √ / 91.46 82.82 86.93 / √ 90.75 82.65 86.51 √ √ 92.52 85.05 88.63

[0134] Table 5 shows the impact of two feature processing steps in AWGM on the experimental results.

[0135]

[0136] Table 6. The impact of different feature processing methods at different levels in CFEM on experimental results.

[0137] Low-level features High-level features Precision Recall Hmean / / 88.97 79.04 83.71 LTE / 92.28 80.07 85.74 LTE LTE 93.27 80.93 86.66 / HTE 91.90 79.90 85.48 HTE HTE 90.38 82.30 86.15 LTE HTE 91.46 82.82 86.93

[0138] Table 4 shows the ablation experiments conducted on the main modules of the method of this invention. As can be seen from the table, compared with the baseline network, the model with the addition of the Cross-Level Feature Enhancement Module (CFEM) improved the average accuracy by 3.21%, indicating that the CFEM proposed in this invention can effectively dynamically adjust the receptive field to improve the model's ability to locate text at different scales. When only the Auxiliary Weight Generation Module (AWGM) is added to the baseline model, the model's average accuracy is improved by 2.8%, indicating that generating auxiliary features through high-level features can effectively guide the fused features for text detection, more accurately assigning pixels to different text regions. When both CFEM and AWGM are added to the baseline model, the model's precision, recall, and average accuracy improve by 3.55%, 6.01%, and 4.92%, respectively. The experimental results fully demonstrate the effectiveness of the Cross-Level Feature Enhancement Module (CFEM) and the Auxiliary Weight Generation Module (AWGM) proposed in this invention.

[0139] Table 5 illustrates the importance of the first step of information extraction and further feature enhancement in AWGM for distinguishing text instances. To demonstrate the effectiveness of the unequal processing of features at different levels in the proposed CFEM, different enhancement methods were applied to features at different levels. The experimental results are shown in Table 6, where LTE represents the low-level feature enhancement method. Figure 2 In Chinese, HTE represents the processing method for low-level input features, while HTE represents the high-level feature enhancement method. Figure 2 The experimental results fully demonstrate the superior performance of each part of the CFEM proposed in this invention, which enhances the high-level features of the input.

[0140] The above description is only for the purpose of helping to understand the method and core essence of the present invention, but the scope of protection of the present invention is not limited thereto. For those skilled in the art, any equivalent substitutions or modifications made to the technical solution and inventive concept disclosed in the present invention within the scope of the technology disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A text detection method based on cross-level feature enhancement and auxiliary feature guidance, characterized in that, Includes the following steps: S1. Extract feature maps at different levels from the image to be detected; The image to be detected is processed by the backbone network to obtain feature maps at four different levels, from stage 2 to stage 5. The number of channels was adjusted to 256 using 1×1 convolutional layers. The adjusted multi-level features are represented as { X 2, X 3, X 4, X 5}; S2. For low-level features, the receptive field is dynamically adjusted by adjusting the size of the convolution kernel to obtain multi-scale information and generate enhanced features for low-level features. S3. For high-level features, differential convolution is used to enhance high-frequency information at text edges, generating enhanced features for high-level features. S4. Use cross-level feature enhancement to fuse features from different levels. The enhanced features of low-level features and the enhanced features of high-level features are fused one-to-one. The fusion is carried out gradually until all features from different levels are fused to generate fused features. First, cross-level features X 2 and X 4, X 3 and X 5. Perform the first step of feature enhancement and fusion to generate... and Then, the different levels of features obtained in the first step are... and The second step involves feature enhancement and fusion, which fuses the extracted multi-level features in a stepwise manner to generate fused features. X C ; Feature enhancement fusion is the process of adding the enhanced features of low-level features and the enhanced features of high-level features to obtain the fused feature. S5. Extract semantic information from high-level features and generate auxiliary features; S6. For the generated auxiliary features and fused features, the attention-guided method is used to determine the ownership of the text kernel pixels, and the attention-guided features are obtained. S7. Based on the attention-guided features, perform text-to-non-text prediction on pixels to generate text bounding boxes.

2. The text detection method based on cross-level feature enhancement and auxiliary feature guidance according to claim 1, characterized in that, S2 is specifically as follows: The low-level features are first dynamically adjusted by convolution operations with different kernel sizes to obtain multi-scale information. Then, feature maps with different receptive field sizes are concatenated along the channel dimension, and the number of channels is adjusted by 1×1 convolution to obtain multi-scale features. Subsequently, attention scores are learned to determine the channel importance of different receptive fields. Finally, the weights of the corresponding channels are multiplied by the multi-scale features to obtain the enhanced features of the low-level features. The process of generating enhanced features from low-level features is represented as follows: in, X i Indicates low-level features, X L [·] represents the enhancement feature of the low-level feature, and [·] represents the feature concatenation along the channel dimension. Conv k×k (·) represents a convolution operation with a kernel size of k×k. f GAP (·) indicates global average pooling. f FC (·) indicates a fully connected layer. ReLU (·) represents the ReLU activation function. σ (·) represents the Sigmoid activation function.

3. The text detection method based on cross-level feature enhancement and auxiliary feature guidance according to claim 2, characterized in that, S3 is specifically as follows: High-level features are enhanced using differential convolution. First, sub-pixel convolutions are applied to increase the resolution of the input features. Then, three convolutional layers are used to enhance the features. These layers use ordinary convolutions with 3×3 kernels to aggregate local features, central differential convolutions, and corner differential convolutions to enhance high-frequency edge information. Finally, the enhanced features from the three branches are summed pixel-by-pixel and then activated using the ReLU activation function to generate enhanced features for the high-level features. The functional representation of this process is as follows: in, X j Indicates high-level features, X H This represents an enhanced feature representing a higher-level feature. Conv 3×3 (·) indicates a convolution operation with a kernel size of 3×3. f sub×N (·) represents a subpixel convolution operation with an upsampling rate of N. Conv CD (·) denotes central difference convolution. Conv AD (·) denotes angle difference convolution. ReLU (·) represents the ReLU activation function.

4. The text detection method based on cross-level feature enhancement and auxiliary feature guidance according to claim 3, characterized in that, The feature enhancement fusion in S4 is represented as follows: in, X L This represents an enhancement feature representing a low-level feature. X H This represents an enhanced feature representing a higher-level feature. This represents the fused feature after the enhancement feature fusion.

5. The text detection method based on cross-level feature enhancement and auxiliary feature guidance according to claim 4, characterized in that, S5 is specifically as follows: Features X 5. First, semantic information is enriched through information extraction blocks, which consist of 1×1 convolutions, ReLU activation functions, and batch normalization operations. Then, sub-pixel convolutions are used to amplify the resolution, followed by features. X 4. Perform pixel-level subtraction to remove redundant information and obtain features. Its function is represented as follows: in, Conv 1×1 (·) represents a 1×1 convolution operation. f BN (·) indicates batch normalization. ReLU (·) represents the ReLU activation function. f sub×2 (·) indicates a subpixel convolution operation with an upsampling rate of 2; Features after noise suppression Further feature enhancement is performed; firstly, texture information is preserved through parallel global average pooling and global max pooling, and then the two pooled features are added at the pixel level before weight generation to obtain an attention map. X W Finally, attention maps and noise-suppressed features are utilized. Auxiliary features are obtained by performing multiplication operations and residual connections. X A ; indicates the following: in, f GAP (·) indicates global average pooling. f GMP (·) indicates global max pooling. f FC (·) indicates a fully connected layer. ReLU (·) represents the ReLU activation function. σ (·) represents the Sigmoid activation function. The transformation operation represents the reshaping of a vector into an attention map.

6. The text detection method based on cross-level feature enhancement and auxiliary feature guidance according to claim 5, characterized in that, S6 is specifically as follows: For auxiliary features and fused features, the resolution of the auxiliary features is first increased by a factor of 4 by upsampling using bilinear interpolation. Then, the auxiliary features are concatenated with the fused features along the channel dimension. Finally, attention weights are generated by convolution with a kernel size of 3×3 and activation functions. X S Finally, the attention weights are multiplied by the fused features and a residual connection is formed to obtain the guided features. X G ; indicates the following: in, f up×4 (·) indicates a bilinear interpolation upsampling operation with a scaling factor of 4, and [·] indicates feature concatenation along the channel dimension. Conv 3×3 (·) indicates a 3×3 convolutional layer. ReLU (·) represents the ReLU activation function. σ (·) represents the Sigmoid activation function.

7. The text detection method based on cross-level feature enhancement and auxiliary feature guidance according to claim 6, characterized in that, The specific details of S7 are as follows: Features X G The input is processed by two parallel branches to obtain single-channel threshold maps. X T and single-channel text probability graph X P The parallel branch includes upsampling operations and text-to-non-text prediction of pixels; subsequently, a post-processing process is performed to obtain a binary image, from which text bounding boxes are generated. The functions for prediction and generating the binary image are expressed as follows: in, Conv 3×3 (·) indicates a 3×3 convolutional layer. f deconv (·) indicates that upsampling is performed through deconvolution, which doubles the feature size. σ (·) represents the Sigmoid activation function. X B This represents the generated binary image, where (i,j) represents the pixel coordinates in the image.

8. A text detection system based on cross-level feature enhancement and auxiliary feature guidance applied to any one of the methods described in claims 1-7, characterized in that, include: The backbone network is constructed by replacing the 3×3 convolutional layers in stages 3, 4, and 5 of ResNet-50 with deformable convolutions to obtain feature maps from four different levels (stages 2-5). X 2, X 3, X 4, X 5}; The cross-level feature enhancement module dynamically adjusts the receptive field of low-level features to obtain multi-scale information, introduces differential convolution to enhance high-frequency edge information of high-level features, and adds the enhanced low-level and high-level features to output the cross-level enhanced fused feature. X C ; The auxiliary weight generation module is used to enhance the semantic information in high-level features and generate auxiliary features. X A ; The attention guidance module is used to guide the correct segmentation of text regions and obtain guided features. X G ; The output module uses upsampling operations and text-to-non-text prediction on pixels to generate text bounding boxes.

Images