Any scene text detection method based on context guidance and semantic compensation

Abstract

The invention discloses an arbitrary scene text detection method based on context guidance and semantic compensation, and belongs to the technical field of computer vision. The invention provides an arbitrary scene text detection method based on context guidance and semantic compensation, and aims to solve the problem of wrong segmentation of a text boundary region so as to accurately position a text instance in a scene image. The method mainly comprises a context-guided feature enhancement module and a high-level semantic information compensation module, the context-guided feature enhancement module learns local and global context information by combining convolution and attention, and complex text features are fully modeled; and the high-level semantic compensation module can make up for the missing high-level semantic information in the fusion features, so that the semantic richness and accuracy of feature expression are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for detecting text in arbitrary scenes based on context guidance and semantic compensation. Background Art

[0002] In recent years, with the widespread adoption and application of various mobile devices, such as modern smartphones, action cameras, and drones, the volume of natural scene images has exploded. Natural scene images often contain rich textual information, such as directions on traffic signs, sender and recipient information on express delivery slips, product descriptions, and advertisements on store signs. This textual information carries a wealth of semantic information. Therefore, understanding and analyzing textual information in scene images can significantly improve work efficiency in various application scenarios and assist other computer vision tasks. Extracting textual information from scene images can be divided into two steps: text detection and text recognition. Text detection locates and detects text regions in an image, while text recognition converts text in an image into a computer-processable sequence of characters or words. As the first step in text extraction, the accuracy of text detection results directly impacts the subsequent text recognition task. Therefore, designing a natural scene text detection algorithm with high detection accuracy, fast detection speed, and strong generalization is of great research value and practical significance.

[0003] Currently, scene text detection technology has been widely used in many fields. In the field of autonomous driving: through text detection and recognition technology, traffic signs, signs, license plates and other signs on the road are identified in real time. The autonomous driving system can make decisions such as adjusting the vehicle speed, choosing the correct lane or avoiding danger according to traffic rules. In the field of real-time translation: when faced with videos, menus, road signs, etc. containing foreign languages, text detection can automatically identify the text information therein and provide it to the translation system for translation into the target language, which greatly facilitates people's work, life and cultural exchanges. In the field of barrier-free technology: portable vision systems collect images of the scene around the user, perform text detection, obtain the text information therein and then voice broadcast it to the user. This technology can greatly facilitate the lives of people with visual impairments.

[0004] With the rapid development of deep learning technology, scene text detection has also achieved breakthroughs. A growing number of researchers have turned their attention to this field and proposed numerous innovative text detection algorithms. Deep learning-based text detection methods can be categorized into regression-based, segmentation-based, and image contour modeling-based text detection algorithms. Due to their unrestricted use of text shape and streamlined algorithmic workflow, segmentation-based text detection algorithms have become one of the mainstream algorithms in the field of natural scene text detection.

[0005] Inspired by tasks such as semantic segmentation and instance segmentation, researchers have transformed text detection into a pixel-level segmentation problem. To address the difficulty of separating dense text instances, Wang et al. proposed the Progressive Scale Expansion Network (PSENet) in 2019. This network generates a series of multi-scale text kernels for each text instance, gradually expanding from the smallest kernel to the complete text region, where the smallest kernel is used to separate adjacent text. However, the progressive scale expansion algorithm makes PSENet post-processing time-consuming and has difficulty processing smaller text instances. The Pixel Aggregation Network (PAN) predicts and calculates the similarity vector between pixels and text kernels, guiding text pixels to cluster towards the correct kernel, achieving accurate instance segmentation. In addition, PAN uses depthwise separable convolution instead of conventional convolution when fusing multi-scale features, which can significantly reduce the amount of computation and improve the detection rate. In 2020, Liao et al. proposed a differentiable binarization network (DBNet), an efficient and accurate text detection algorithm. DBNet designs a differentiable binarization function so that it can be optimized together with the segmentation network during the training cycle. The optimized network can adaptively set the threshold for binarization, which simplifies the post-processing process and improves the performance of text detection. However, the above algorithms directly use the coarse-grained boundary annotations provided by the dataset, ignoring the background noise in the annotations, resulting in inaccurate predicted text boundaries. In 2022, Zhang et al. proposed a base probability map segmentation network (Text Detection via Segmentation with Probability Maps, TextPMs). TextPMs designed a Sigmoid variant function to map the distance between the boundary and its internal pixels into a probability map, and generate a series of probability maps by adjusting the values of hyperparameters to describe the possible probability distribution. In the post-processing stage, a simple region growing algorithm is used to aggregate the probability maps into complete text instances.

[0006] Although the above-mentioned text detection algorithms have achieved good performance, these methods still face some problems. Text detection algorithms based on segmentation ideas usually face the problem of mis-segmentation of text boundary areas. Specifically, since there are no clearly defined closed geometric boundaries in scene texts, segmentation-based algorithms have difficulty separating adjacent text instances, and when processing text boundaries, they may cause over-segmentation or under-segmentation problems due to excessive refinement or insufficient division. In addition, in order to detect texts of varying sizes, most of the existing segmentation-based text detection algorithms introduce the use of a feature pyramid network (FPN) structure, which gradually fuses features of different levels upward through a top-down path, allowing the network to simultaneously utilize high-level semantic information and low-level detail information. Although this method can achieve good performance, it ignores the problem that high-level semantic information is gradually overwhelmed by low-level semantic information during the fusion process, and that there are semantic differences between features of different levels, resulting in mis-segmentation of text boundary areas.

[0007] In order to solve the above problems, the present invention proposes a text detection method for arbitrary scenes based on context guidance and semantic compensation. Summary of the Invention

[0008] The purpose of the present invention is to propose an arbitrary scene text detection method based on context guidance and semantic compensation to solve the problem of incorrect segmentation of text boundary areas and to accurately locate text instances in scene images.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] A method for detecting text in arbitrary scenes based on context guidance and semantic compensation is implemented by a system for detecting text in arbitrary scenes based on context guidance and semantic compensation. The system includes a backbone network, a context-guided feature enhancement module (CFEM), a high-level semantic compensation module (HSCM), and an output module.

[0011] The method comprises the following steps:

[0012] S1. Design a backbone network and use the improved backbone network to extract multi-level features.

[0013] S2. Based on convolution operation and attention mechanism, a context-attention-guided feature enhancement module (CFEM) is designed. The output features of the backbone network are fed into the context-attention-guided feature enhancement module (CFEM) to capture local and global contextual attention to enhance text feature representation.

[0014] S3, fuse the features of each level output by the contextual attention-guided feature enhancement module (CFEM) with the features output by the backbone network to obtain multi-scale fusion features;

[0015] S4. Design a high-level semantic information compensation module (HSCM) to feed the high-level features output by the contextual attention-guided feature enhancement module (CFEM) and the multi-scale fusion features obtained in S3 into the high-level semantic information compensation module (HSCM) to align the features through resampling technology, and further design a gating mechanism to dynamically adjust the weights of the aligned high-level features and the fusion features to achieve adaptive fusion;

[0016] S5. Generate text detection results through the output module to complete the text detection work.

[0017] Preferably, the S1 specifically includes the following contents:

[0018] ResNet-50 is used as the backbone network. The 3×3 convolutions in the 3rd to 5th stages of the backbone network are replaced with deformable convolutions, and the weight model pre-trained on ImageNet is loaded during training. The output features of the 2nd to 5th stages of the backbone network are labeled as {F2, F3, F4, F5}, and the resolution of the output features at each level is {1 / 2, 1 / 4, 1 / 8, 1 / 16} of the input image, respectively, and the number of channels is {256, 512, 1024, 2048}, respectively. The improved backbone network is used to extract multi-level features.

[0019] Preferably, the contextual attention-guided feature enhancement module (CFEM) includes a shape calibration branch based on wavelet convolution, a global self-attention branch, and a channel attention branch, wherein:

[0020] The wavelet convolution-based shape calibration branch performs pooling operations in the horizontal and vertical directions to capture axial context information. By adding the horizontal and vertical vectors, the text region is modeled as a rectangle. The wavelet convolution-based shape calibration branch further designs a shape calibration function to capture local context information, thereby calibrating the text region to make it more closely fit the text boundary contour. Specifically, the following contents are included:

[0021] Use k×1 convolution to adjust the elements of each row so that the modeling area is closer to the text shape in the horizontal direction;

[0022] Perform feature normalization through BN and increase nonlinearity through ReLU function;

[0023] The shape calibration function uses 1×k convolution to calibrate the modeled shape in the vertical direction;

[0024] The calibration features are mapped to the range of (0,1) through the Sigmoid function to obtain the shape calibration weight W B ;

[0025] The formula of the shape calibration function is as follows:

[0026]

[0027] Among them, δ represents the Sigmoid activation function; and Represent k×1 and 1×k convolutions respectively; β represents normalization; ReLU represents the ReLU activation function;

[0028] The global self-attention branch maps the features of each level output by the backbone network through the fully connected layer to obtain W i q , W i k , W i v Three weight matrices; the query matrix Q is obtained by multiplying the features at each level with the weight matrix i , key matrix K i , value matrix V i ;

[0029] Calculate the query matrix Q i and bond matrix K i The similarity between them is calculated by Q i With K i The dot product is implemented; the dot product result is scaled and divided by the key matrix K i Square root of dimension

[0030] The attention score of each query is normalized using the Softmax function and converted into a probability distribution W i a , thereby assigning a weight to each pixel, reflecting its degree of attention to other pixels;

[0031] The normalized attention weight matrix W i a and value matrix V i Perform weighted summation to obtain the output F of the global self-attention branch i G; The function of the above process is expressed as follows:

[0032] Q i =F i W i q ,K i =F i W i k ,V i =F i W i v

[0033]

[0034] F i G =W i a ·V i

[0035] The channel attention branch smoothes the feature map through a 3×3 convolutional layer;

[0036] The processed feature maps are subjected to global maximum pooling and global average pooling operations in the spatial dimension, respectively, to compress the features into a one-dimensional vector. Then, after passing through a fully connected layer and an activation function, pixel-by-pixel addition is performed to fuse the information of the two branches.

[0037] The vector is mapped to the range of (0,1) through the Sigmoid function to obtain the channel attention weight W i c ; Channel attention weight and input feature map F i Multiply pixel by pixel to get the channel attention feature F i C ; The function of the above process is expressed as follows:

[0038]

[0039] W i c =δ(F i CM +F i CA )

[0040] F i C =W i C ·F i

[0041] Among them, GMP S and GAP SRepresent the global maximum pooling and global average pooling operations in the spatial dimension respectively; and They represent the fully connected layer and the 3×3 convolutional layer respectively; δ represents the Sigmoid activation function.

[0042] Preferably, S2 specifically includes the following contents:

[0043] The output features {F3, F4, F5} of each stage from the 3rd to the 5th stage of the backbone network are fed into the context-guided feature enhancement module (CFEM). The shape calibration branch based on wavelet convolution, the global self-attention branch, and the channel attention branch capture local and global contextual attention, enhance the text feature representation through contextual information and expand the receptive field, and distinguish between text and background areas.

[0044] Preferably, the S3 specifically includes the following contents:

[0045] The features of each level output by the contextual attention guided feature enhancement module (CFEM) The feature F2 output from the second stage of the backbone network is fused top-down through pixel-level addition, and a 3×3 convolution layer is set to achieve smooth feature fusion;

[0046] The fourth level features The resolution is adjusted to 1 / 4 of the original image size by bilinear interpolation, and then the number of feature channels is reduced to 64 by a 3×3 convolution layer. The four-level features are then spliced together along the channel dimension to obtain the multi-scale fusion feature F * .

[0047] Preferably, the S4 specifically includes the following contents:

[0048] The output of the context-guided feature enhancement module (CFEM) and fusion feature F * The number of channels is unified to 256 through 3×3 convolution layers, standardized through batch normalization, and nonlinearity is increased through activation functions; high-level features are Upsample and fuse features F through bilinear interpolation * The same size; the function of the above process is expressed as:

[0049]

[0050] The High-Level Semantic Compensation Module (HSCM) groups the channel dimension into multiple sub-features, performs calibration and alignment operations, and integrates a gating mechanism to adaptively fuse and F * , specifically including:

[0051] Will The number of channels of F and F' is reduced to 128 through a 1×1 convolution layer, and the two are connected in series through channel splicing; they are sent to the weight generation block to learn two sets of offsets Δh and Δf and two gated masks α h and α f ; The calibration alignment operation is implemented using resampling technology. Assuming that each unknown spatial coordinate of the feature map F' is {(1,1), (1,2), Λ, (H, W)}, for the learned offset, use the coordinate (h+Δf h ,w+Δf w ) value to replace the original (h,w) value, where (h+Δf h ,w+Δf w ) is obtained by bilinear interpolation; similarly, the feature map Perform calibration and alignment. The calibration process function is expressed as follows:

[0052]

[0053]

[0054] The calibrated feature map is obtained, and the two are adaptively fused using a gated mask. The process function is expressed as follows:

[0055]

[0056] Among them, C h and C f represents the feature map after calibration; Represents a pixel-level multiplication operation.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The technology of the present invention aims to solve the problem of incorrect segmentation of text boundary areas in existing scene text detection algorithms based on segmentation ideas, and proposes a method for detecting text in arbitrary scenes based on context guidance and semantic compensation. The method proposed in the present invention is implemented based on the corresponding text detection system, and mainly includes a context-guided feature enhancement module and a high-level semantic information compensation module. The context-guided feature enhancement module learns local and global context information by combining convolution and attention, and fully models complex text features; and the high-level semantic compensation module can make up for the high-level semantic information missing in the fused features, thereby enhancing the semantic richness and accuracy of the feature expression. In addition, a large number of experiments on four text detection benchmark datasets have demonstrated the effectiveness and advantages of the method proposed in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1This is a diagram of the overall network structure of the arbitrary scene text detection method based on context guidance and semantic compensation mentioned in Example 1 of the present invention;

[0060] Figure 2 Detailed structural diagram of the context-guided feature enhancement module (CFEM) mentioned in Example 1 of the present invention;

[0061] Figure 3 Detailed structural diagram of the high-level semantic information compensation module (HSCM) mentioned in Example 1 of the present invention;

[0062] Figure 4 This is a comparison chart of the visualization results of this method and other methods mentioned in Example 2 of the present invention. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0064] First, the abbreviations and key term definitions mentioned in the present invention are explained, specifically:

[0065] PSENet: Progressive Scale Expansion Network

[0066] PAN: Pixel Aggregation Network

[0067] DBNet: Differentiable Binarization Network, Differentiable Binarization Network

[0068] TextPMs:Text Detection via Segmentation with Probability Maps,Probabilistic Graph Segmentation Network

[0069] CFEM: Context-guided Feature Enhancement Module

[0070] HSCM: High-level Semantic Compensation Module, based on the above content, is explained below with reference to relevant figures and specific examples.

[0071] Example 1:

[0072] This paper proposes a method for detecting text in any scene based on context guidance and semantic compensation. The overall network structure is as follows: Figure 1 As shown in the figure, it mainly consists of a backbone network, a context-guided feature enhancement module (CFEM), a high-level semantic information compensation module (HSCM) and an output module.

[0073] Specifically, the present invention first uses the backbone network ResNet-50 to extract multi-level features. In order to enhance the feature extraction capability of the backbone network, the 3×3 convolutions in the 3rd to 5th stages of the backbone network are replaced with deformable convolutions, and the weight model pre-trained on ImageNet is loaded during training. The output features of the backbone network from the 2nd to the 5th stage are marked as {F2, F3, F4, F5}, and the resolution of the output features at each level are {1 / 2, 1 / 4, 1 / 8, 1 / 16} of the input image, and the number of channels is {256, 512, 1024, 2048} respectively. The output features {F3, F4, F5} of the backbone network from the 3rd to the 5th stage are sent to the context-guided feature enhancement module (CFEM), which enhances the text feature representation through context information and expanding the receptive field to distinguish between text and background areas.

[0074] Then, the present invention performs the analysis of the features of each level output by CFEM. The feature F2 output from the second stage of the backbone network is fused from top to bottom using pixel-level addition, and a 3×3 convolutional layer is set to achieve smooth feature fusion.

[0075] Next, the four-level features Perform feature integration. The specific structure of feature integration is as follows Figure 2 Specifically, the resolution of the four-level features is first adjusted to 1 / 4 of the original image size through bilinear interpolation, and then the number of channels of the features is reduced to 64 through a 3×3 convolution layer. The four-level features are then spliced together along the channel dimension to obtain the multi-scale fusion feature F * .

[0076] The detailed structure of the context-guided feature enhancement module CFEM is as follows: Figure 2 As shown in Figure 2, the input of CFEM is the features {F3, F4, F5} obtained by the backbone network at each level. CFEM is divided into three branches: a shape calibration branch based on wavelet convolution, a global self-attention branch, and a channel attention branch.

[0077] In order to make the model features more concentrated in the text area, the shape calibration branch based on wavelet convolution performs pooling operations in the horizontal and vertical directions to capture axial context information in two directions. By adding these two vectors, the text area can be rectangularly modeled. Then, the branch designs a shape calibration function to capture local context information, thereby calibrating the text area to make it fit the text boundary contour more closely. Specifically, k×1 convolution is first used to adjust the elements of each row so that the modeling area is closer to the text shape in the horizontal direction. Feature normalization is then performed through BN, and nonlinearity is added through the ReLU function. Subsequently, the shape calibration function uses 1×k convolution to calibrate the modeling shape in the vertical direction. Finally, the calibration features are mapped to the range of (0,1) through the Sigmoid function to obtain the shape calibration weight W. B In this way, decoupling in both horizontal and vertical directions can be achieved, making the modeling adaptable to various shapes. The shape calibration function formula is as follows:

[0078]

[0079] Among them, δ represents the Sigmoid activation function; and β represents k×1 and 1×k convolutions, respectively; β represents normalization; and ReLU represents the ReLU activation function. Inspired by the large receptive field wavelet convolution WTConv can capture feature maps at different scales and further expand the receptive field, the wavelet convolution-based shape calibration branch uses wavelet convolution to capture local contextual information at different scales. Through wavelet convolution, the model can simultaneously capture local features at different scales, thereby enhancing the model's understanding of complex images and further expanding the receptive field.

[0080] In the global self-attention branch, the features at all levels output by the backbone network are first mapped through the fully connected layer to obtain W i q , W i k , W i v Three weight matrices. The query matrix Q is obtained by multiplying the features at each level with the weight matrix i , key matrix K i , value matrix V i Next, calculate the query matrix Q i and bond matrix K i The similarity between them is calculated by Q i and K i To improve numerical stability and ensure the validity of the calculation results, the dot product results are scaled and divided by the key matrix K i Square root of dimension Then, the attention score of each query is normalized using the Softmax function and converted into a probability distribution W i a , thereby assigning a weight to each pixel to reflect its degree of attention to other pixels. Finally, the normalized attention weight matrix W i a and value matrix V i Perform weighted summation to obtain the output F of the global self-attention branch i G The process of generating a global self-attention feature map is formulated as follows:

[0081] Q i =F i W i q ,K i =F i W i k ,V i =F i W i v

[0082]

[0083] F i G =W i a ·V i

[0084] In the channel attention branch, the feature map is first smoothed by a 3×3 convolutional layer. Subsequently, the processed feature map is subjected to global maximum pooling and global average pooling operations in the spatial dimension to compress the features into a one-dimensional vector. Then, the information of the two branches is fused by pixel-by-pixel addition after passing through a fully connected layer and an activation function. Finally, the vector is mapped to the range of (0, 1) through the Sigmoid function to obtain the channel attention weight W. i c . Channel attention weight and input feature map F i Multiply pixel by pixel to get the channel attention feature F i C The process of generating a global self-attention feature map can be expressed as follows:

[0085]

[0086] W i c =δ(F i CM +F iCA )

[0087] F i C =W i C ·F i

[0088] Among them, GMP S and GAP S Represent the global maximum pooling and global average pooling operations in the spatial dimension respectively; and Represent the fully connected layer and 3×3 convolutional layer respectively; represents the Sigmoid activation function.

[0089] The detailed structure of the high-level semantic information compensation module is as follows: Figure 3 Although some existing algorithms supplement high-level features to low-level features, they ignore the spatial dislocation and semantic gap between features at different levels. To alleviate this problem, the present invention proposes a high-level semantic information compensation module HSCM, which is based on the FPN structure and effectively aligns the high-level features output by the text outline enhancement module with the fusion feature F through resampling technology. * , thereby ensuring that the information in different features is accurately matched. The module then designs a gating mechanism to dynamically adjust the weights of the aligned high-level features and the fused features to achieve adaptive fusion. This process effectively compensates for the high-level semantic information missing in the fused features, thereby enhancing the semantic richness and accuracy of the feature expression.

[0090] In order to reduce computational overhead and improve computational efficiency, the context-guided feature enhancement module is first output and fusion feature F * The number of channels is unified to 256 through 3×3 convolution layers, standardized through batch normalization, and nonlinearity is increased through activation functions. Upsample and fuse features F through bilinear interpolation * The same size. This process can be expressed by the formula:

[0091]

[0092] However, due to the spatial misalignment and significant representation differences, directly fusing the two is not ideal. * Contains rich spatial details, and Contains more semantic information, and fusing the two after calibration of the channel dimension may have a negative impact on performance. To solve this problem, HSCM groups the channel dimension into multiple sub-functions, performs calibration and alignment operations separately, and integrates a gating mechanism to adaptively fuse and F * Specifically, first The number of channels of F and F' is reduced to 128 through a 1×1 convolution layer, and the two are connected in series through channel concatenation. After that, they are sent to the weight generation block to learn two sets of offsets Δh and Δf and two gated masks α h and α f In the calibration and alignment operation, this method uses resampling technology. Assuming that each unknown spatial coordinate of the feature map F' is {(1,1), (1,2), Λ, (H, W)}, the learned offset is expressed as the coordinate (h+Δf h ,w+Δf w ) value to replace the original (h,w) value, where (h+Δf h ,w+Δf w ) is obtained by bilinear interpolation. Similarly, the feature map Perform calibration and alignment. The calibration process can be expressed by the following formula.

[0093]

[0094] The calibrated feature map is obtained, and the two are adaptively fused using a gated mask. The process function is expressed as follows:

[0095]

[0096] Among them, C h and C f represents the feature map after calibration; Represents a pixel-level multiplication operation.

[0097] Example 2:

[0098] Based on Example 1, but different in that, a specific experiment is designed below to verify the performance of the arbitrary scene text detection method based on context guidance and semantic compensation proposed in the present invention. The specific content is as follows.

[0099] 1. Dataset

[0100] The context-guided and semantically compensated arbitrary scene text detection method proposed in Example 1 is experimentally verified on four benchmark datasets. The datasets used include:

[0101] (1) SynthText. SynthText is an artificially generated dataset containing more than 858,750 natural scene images. This dataset collects 8 million common English words and uses random transformations and other methods to render text onto natural images to synthesize text images. To make the generated text images closer to the real scene, deep learning and semantic segmentation are used to align the geometric shapes of the text with the background image when synthesizing the image. The method proposed in this paper is pre-trained only on this dataset.

[0102] (2) MSRA-TD500. MSRA-TD500 is a multilingual dataset containing Chinese and English. It contains 500 images, 300 of which are used as training images and the remaining 200 as test images. The image resolution ranges from 1296×864 to 1920×1280. In addition, due to the small amount of MSRA-TD500 training data, 400 training images from HUST-TR400 are added during network training.

[0103] (1) CTW1500. CTW1500 is a Chinese dataset of arbitrary-shaped text, consisting of 1,000 images for training and 500 images for testing. This dataset is highly diverse, including flat text, convex text, text in urban and rural street scenes, text in low-light conditions, text at a distance, and partially displayed text.

[0104] (4) Total-Text. The Total-Text dataset contains a total of 1555 images, of which 1255 are used for training and the remaining 300 are used for testing. The text in the dataset has various shapes, including horizontal, tilted, and curved text. More than half of the images contain text of two or more shapes. Curved text instances account for nearly half of the total text instances in the images. In addition, the dataset images contain a variety of font styles, text sizes, and complex backgrounds, which places higher demands on the performance of the text detection model.

[0105] 2. Experimental Setup

[0106] The model proposed in the present invention is implemented based on the Python language, and the experimental environment configured during the experiment is shown in Table 1. The SGD optimizer is used in the training process. The backbone network uses the ResNet-50 model weights pre-trained on ImageNet, and the parameters of other network layers are initialized using the Kaiming initialization method. In the data preprocessing stage, the image is uniformly resized to 640×640 by random cropping, and data enhancement methods such as random rotation and random flipping are also used. The pre-training stage is carried out on the artificial synthetic dataset SynthText, and 3 epochs are trained. The learning rate is fixed at 0.007 and the batch size is set to 12. After the pre-training is completed, fine-tuning is performed on four benchmark datasets for a total of 1500 epochs, and a "poly" strategy with an initial learning rate of 0.007 is adopted.

[0107] Table 1: Experimental environment

[0108]

[0109] 3. Experimental Results

[0110] The proposed method is compared with previous scene text detection methods on four benchmark datasets. Experiments were conducted on multi-oriented text datasets (MSRA-TD500 and ICDAR2015) and arbitrary-shaped text datasets (Total-Text and CTW1500). The performance of the proposed method was evaluated based on four metrics: precision, recall, mean average precision (Hmean), and frames per second (FPS). Ablation experiments were also conducted on the CTW1500 dataset to illustrate the role of each component of the proposed method.

[0111] Tables 2, 3, 4 and 5 show the quantitative evaluation results of various methods on text datasets, where the best indicators are bolded and the suboptimal performance is underlined. As can be seen from Tables 2 and 3, the method proposed in the present invention has achieved excellent average accuracy in the multi-directional text detection dataset. Tables 4 and 5 show the detection results of the method proposed in the present invention on arbitrary shape text datasets. The method proposed in the present invention has achieved optimal or suboptimal results on these two curved text datasets. This is due to the fact that the method proposed in the present invention uses local and global context information to effectively reduce the problems of false detection and missed detection caused by small text spacing. At the same time, by compensating high-level semantic information into the fusion features, the network's discrimination ability is enhanced, enabling the network to generate more accurate text contours.

[0112] Table 2: Experimental results on the MSRA-TD500 dataset. The best performance is marked in bold.

[0113]

[0114]

[0115] Table 3: Experimental results on the ICDAR2015 dataset. The best performance is marked in bold.

[0116]

[0117] Table 1: Experimental results on the Total-Text dataset, with the best performance highlighted in bold.

[0118]

[0119]

[0120] Table 2: Experimental results on the CTW1500 dataset. The best performance is marked in bold.

[0121]

[0122] In order to more intuitively demonstrate the effectiveness of the method proposed in this paper, the detection results of the method proposed in this paper are compared with those of the classic text detection methods DBNet and DBNet++. The visualization results of the real labels and each detection method are shown in Figure 2. Figure 4 As shown. Figure 4 It can be clearly seen that the method proposed in the present invention can accurately determine the text and non-text areas and generate a text box that accurately surrounds the text instance.

[0123] Table 6 presents an ablation experiment on the CTW1500 dataset using the main module of the proposed method. As shown in Table 6, the baseline model achieves precision, recall, and average precision of 86.90%, 80.20%, and 83.40%, respectively. When only CFEM is added to the baseline model, the average precision of the model improves by 2.32%. This result demonstrates that CFEM fully models complex text features by learning local and global contextual information, enabling more accurate prediction of text boundary regions. When only TFEM is added to the baseline model, the average precision of the model improves by 1.29%. This result demonstrates that the HSCM module effectively compensates for the high-level semantic information missing from the fused features, thereby enhancing the semantic richness and accuracy of the feature representation. When both CFEM and HSCM are added to the baseline model, the model ultimately achieves a precision of 88.71%, a recall of 85.04%, and an average precision of 86.83%, respectively improving the corresponding metrics of the baseline model by 1.81%, 4.84%, and 3.43%. The experimental results fully verify the effectiveness of the context-guided feature enhancement module (CFEM) and high-level semantic compensation module (HSCM) proposed in this paper.

[0124] Table 6: Ablation test results of each module, the best performance is bolded

[0125]

[0126] Tables 7, 8, and 9 present ablation experiments on the various components of the contextual attention-guided feature enhancement module. Table 7, an ablation experiment on the CTW1500 dataset, demonstrates the impact of various branches in CFEM on model performance. Table 8 demonstrates the rationale for using wavelet convolution, and Table 9, an ablation experiment on the CTW1500 dataset, compares the impact of the convolution kernel size of the shape self-calibration function in CFEM.

[0127] Table 7: The impact of various branches in CFEM on model performance. The best performance is bolded.

[0128]

[0129] Table 8: Effect of convolution selection on CFEM performance. The best performance is highlighted in bold.

[0130]

[0131] Table 9: The impact of convolution kernel size on model performance in the shape self-calibration function. The best performance is marked in bold.

[0132]

[0133] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for detecting text in arbitrary scenes based on context guidance and semantic compensation, characterized in that: The method is implemented by an arbitrary scene text detection system based on context guidance and semantic compensation, which includes a backbone network, a context-guided feature enhancement module, a high-level semantic information compensation module, and an output module; The method comprises the following steps: S1. Design a backbone network and use the improved backbone network to extract multi-level features. S2. Based on convolution operation and attention mechanism, a context-attention-guided feature enhancement module is designed. The output features of the backbone network are fed into the context-attention-guided feature enhancement module to capture local and global context attention to enhance text feature representation. S3, fuse the features of each level output by the feature enhancement module guided by contextual attention with the features output by the backbone network to obtain multi-scale fusion features; S4. Design a high-level semantic information compensation module, which feeds the high-level features output by the context-attention-guided feature enhancement module and the multi-scale fusion features obtained in S3 into the high-level semantic information compensation module to align the features through resampling technology, and further design a gating mechanism to dynamically adjust the weights of the aligned high-level features and the fusion features to achieve adaptive fusion; S5. Generate text detection results through the output module to complete the text detection work.

2. The method for detecting text in any scene based on context guidance and semantic compensation according to claim 1, characterized in that: The S1 specifically includes the following contents: ResNet-50 is used as the backbone network. The 3×3 convolutions in the 3rd to 5th stages of the backbone network are replaced with deformable convolutions, and the weight model pre-trained on ImageNet is loaded during training. The output features of the 2nd to 5th stages of the backbone network are labeled as {F2, F3, F4, F5}, and the resolution of the output features at each level is {1 / 2, 1 / 4, 1 / 8, 1 / 16} of the input image, respectively, and the number of channels is {256, 512, 1024, 2048}, respectively. The improved backbone network is used to extract multi-level features.

3. The method for detecting text in any scene based on context guidance and semantic compensation according to claim 2, characterized in that: The contextual attention-guided feature enhancement module includes a shape calibration branch based on wavelet convolution, a global self-attention branch, and a channel attention branch, wherein: The wavelet convolution-based shape calibration branch performs pooling operations in the horizontal and vertical directions to capture axial context information. By adding the horizontal and vertical vectors, the text region is modeled as a rectangle. The wavelet convolution-based shape calibration branch further designs a shape calibration function to capture local context information, thereby calibrating the text region to make it more closely fit the text boundary contour. Specifically, the following contents are included: Use k×1 convolution to adjust the elements of each row so that the modeling area is closer to the text shape in the horizontal direction; Perform feature normalization through BN and increase nonlinearity through ReLU function; The shape calibration function uses 1×k convolution to calibrate the modeled shape in the vertical direction; The calibration features are mapped to the range of (0,1) through the Sigmoid function to obtain the shape calibration weight W B ; The formula of the shape calibration function is as follows: Among them, δ represents the Sigmoid activation function; and Represent k×1 and 1×k convolutions respectively; β represents normalization; ReLU represents the ReLU activation function; The global self-attention branch maps the features of each level output by the backbone network through the fully connected layer to obtain W i q , W i k , W i v Three weight matrices; the query matrix Q is obtained by multiplying the features at each level with the weight matrix i , key matrix K i , value matrix V i ; Calculate the query matrix Q i and bond matrix K i The similarity between them is calculated by Q i With K i The dot product is implemented; the dot product result is scaled and divided by the key matrix K i Square root of dimension The attention score of each query is normalized using the Softmax function and converted into a probability distribution W i a , thereby assigning a weight to each pixel, reflecting its degree of attention to other pixels; The normalized attention weight matrix W i a and value matrix V i Perform weighted summation to obtain the output F of the global self-attention branch i G ; The function of the above process is expressed as follows: Q i =F i W i q ,K i =F i W i k ,V i =F i W i v F i G =W i a ·V i The channel attention branch smoothes the feature map through a 3×3 convolutional layer; The processed feature maps are subjected to global maximum pooling and global average pooling operations in the spatial dimension, respectively, to compress the features into a one-dimensional vector. Then, after passing through a fully connected layer and an activation function, pixel-by-pixel addition is performed to fuse the information of the two branches. The vector is mapped to the range of (0,1) through the Sigmoid function to obtain the channel attention weight W i c ; Channel attention weight and input feature map F i Multiply pixel by pixel to get the channel attention feature F i C ; The function of the above process is expressed as follows: W i c =δ(F i CM +F i CA ) F i C =W i C ·F i Among them, GMP S and GAP S Represent the global maximum pooling and global average pooling operations in the spatial dimension respectively; and They represent the fully connected layer and the 3×3 convolutional layer respectively; δ represents the Sigmoid activation function.

4. The method for detecting text in any scene based on context guidance and semantic compensation according to claim 3, characterized in that: The S2 specifically includes the following contents: The output features {F3, F4, F5} of each level from the 3rd to the 5th stage of the backbone network are fed into the context-guided feature enhancement module. The shape calibration branch, global self-attention branch, and channel attention branch based on wavelet convolution capture local and global contextual attention, enhance the text feature representation through contextual information and expand the receptive field, and distinguish between text and background areas.

5. The method for detecting text in any scene based on context guidance and semantic compensation according to claim 4, characterized in that: The S3 specifically includes the following contents: The features of each level output by the feature enhancement module guided by contextual attention {F3 w ,F4 w ,F5 w The features F2 output from the second stage of the backbone network are fused top-down through pixel-level addition, and a 3×3 convolutional layer is set to achieve smooth fusion of features; The fourth level features The resolution is adjusted to 1 / 4 of the original image size by bilinear interpolation, and then the number of feature channels is reduced to 64 by a 3×3 convolution layer. The four-level features are then spliced together along the channel dimension to obtain the multi-scale fusion feature F * .

6. The method for detecting text in any scene based on context guidance and semantic compensation according to claim 5, characterized in that: The S4 specifically includes the following contents: The output of the context-guided feature enhancement module and fusion feature F * The number of channels is unified to 256 through 3×3 convolution layers, standardized through batch normalization, and nonlinearity is increased through activation functions; high-level features are Upsample and fuse features F through bilinear interpolation * The same size; the function of the above process is expressed as: The high-level semantic information compensation module groups the channel dimension into multiple sub-features, performs calibration and alignment operations respectively, and integrates a gating mechanism to adaptively fuse and F * , specifically including: Will The number of channels of F and F' is reduced to 128 through a 1×1 convolution layer, and the two are connected in series through channel splicing; they are sent to the weight generation block to learn two sets of offsets Δh and Δf and two gated masks α h and α f ; The calibration alignment operation is implemented using resampling technology. Assuming that each unknown spatial coordinate of the feature map F' is {(1,1), (1,2), Λ, (H, W)}, for the learned offset, use the coordinate (h+Δf h ,w+Δf w ) value to replace the original (h,w) value, where (h+Δf h ,w+Δf w ) is obtained by bilinear interpolation; similarly, the feature map Perform calibration and alignment. The calibration process function is expressed as follows: The calibrated feature map is obtained, and the two are adaptively fused using a gated mask. The process function is expressed as follows: Among them, C h and C f represents the feature map after calibration; Represents a pixel-level multiplication operation.

Images