Scene text detection system

Abstract

The present invention relates to the field of text detection. The present invention provides a scene text detection system, comprising: an image acquisition unit, a feature extraction unit, a feature fusion unit and a differentiable binarization module, wherein the feature extraction unit is used to extract a feature map of an original image using Resnet, and a residual correction branch is embedded in the Resnet backbone network; the residual correction branch is used to form two branches after Resnet performs conventional convolution on the original image to obtain input features; one branch converts the input features into a low-dimensional embedding by downsampling, and the low-dimensional embedding is used to calibrate the convolution transformation of the convolution kernel in the other branch, and finally obtain the feature map of the original image; finally, the target text area in the image is determined according to the target feature map. The present invention introduces a residual correction branch (RCB) to expand the receptive field, to improve the ability to obtain contextual information, and thus obtain contextual information with a larger receptive field.

Description

Technical Field

[0001] The present invention belongs to the field of text detection, and in particular relates to a scene text detection system. Background Art

[0002] Text has become an indispensable means of transmitting information in the contemporary world. Various textual information exists in the social scenes of our daily lives. Natural scene text detection involves locating text regions within an image using a detection network and representing these regions with polygonal bounding boxes. Accurate detection results are beneficial for a wide range of practical applications, such as instant translation, image retrieval, scene parsing, geolocation, and license plate recognition, and have attracted considerable attention in the fields of computer vision and document analysis. In recent years, with the rapid development of convolutional neural networks (CNNs), scene text detection has made significant progress. Existing CNN-based text detection algorithms can be roughly divided into two categories: regression-based methods and segmentation-based methods.

[0003] Regression-based scene text detection algorithms typically use rectangular or quadrilateral boxes with specific orientations. While this provides fast detection speed and avoids errors accumulated across multiple stages, most existing regression-based methods are unable to accurately and effectively address the text detection problem due to the limited representation of text (axis-aligned rectangles, rotated rectangles, or quadrilaterals). This is particularly problematic when used to detect arbitrarily shaped text in datasets like Total-Text, which significantly impacts subsequent text recognition within the optical character recognition engine.

[0004] Segmentation-based scene text detection algorithms primarily locate text instances by classifying pixels. Although recent methods have achieved significant improvements in scene text detection tasks, and research focus has shifted from horizontal text to multi-oriented text and the more challenging arbitrary-shaped text (such as curved text), challenges remain in detecting arbitrary-shaped scene text due to the specific properties of scene text, such as significant variations in color, scale, orientation, aspect ratio, and shape, which make them significantly different from general target objects. Furthermore, natural image properties vary, such as image blur and lighting conditions.

[0005] Text in natural scenes possesses rich and clear semantic information. Using computer technology to quickly and accurately extract text from scene images is a hot research topic in computer vision and pattern recognition. Scene text detection technology is fundamental to text recognition and has widespread applications in daily life and production. Compared to traditional optical character recognition (OCR), text detection in natural scene images faces numerous difficulties and challenges, including complex backgrounds, diverse text scales and fonts, and uncertain image quality. In recent years, with the rapid development of deep learning technology, deep learning methods have achieved remarkable results in text detection tasks. While existing convolutional neural networks have demonstrated excellent representational capabilities, their receptive fields are limited, their localization capabilities are weak, and text localization is inaccurate, leading to false or missed detections when detecting long or large text. Furthermore, while feature pyramid networks can fuse features at different scales, high-level semantic information from small-scale text is lost at higher layers of the network, resulting in limited detection capabilities for multi-scale text.

[0006] Text information in natural scenes is usually diverse and irregular, and the detection of text of arbitrary shapes in natural scenes is complex. Due to the use of manually designed features, traditional natural scene text detection methods lack robustness, and existing deep learning-based text detection methods have the problem of losing important feature information during the feature extraction process of each layer of the network. Segmentation-based text detection methods are one of the most popular detection methods recently. The segmentation results can more intuitively describe scene text of various shapes. The original DB (Differentiable Binarization) algorithm uses a differentiable binarization algorithm to simplify the post-processing process, solves the problem of non-differentiable gradients caused by training, and improves the efficiency of scene text detection. However, it does not fully utilize the semantic and spatial information in the network, which limits the network's classification and positioning capabilities. Although segmentation-based algorithms have advantages in detecting text of arbitrary shapes, they can also cause false positives or missed detections due to the lack of sufficient contextual information. Summary of the Invention

[0007] The purpose of the present invention is to propose a scene text detection system to address the problems in the prior art where the DBNet text detection network does not fully utilize the semantic information and spatial information in the network, which limits the network's classification and positioning capabilities, and lacks sufficient contextual information, resulting in false positives or missed detections. This system enables the DBNet text detection network to obtain deeper semantic information and clarify key text features during the feature extraction process.

[0008] The present invention solves the above technical problems and adopts a technical solution of a scene text detection system, comprising: an image acquisition unit, a feature fusion unit and a differentiable binarization module, characterized in that:

[0009] The image acquisition unit is used to acquire the original image;

[0010] The feature extraction unit is configured to extract a feature map of the original image using Resnet; a residual correction branch is embedded in the Resnet backbone network; the residual correction branch is configured to form two branches after Resnet performs conventional convolution on the original image to obtain input features; one branch converts the input features into a low-dimensional embedding by downsampling, and the low-dimensional embedding is used to calibrate the convolution transformation of the convolution kernel in the other branch, ultimately obtaining a feature map of the original image;

[0011] The feature fusion unit is used to perform feature fusion on the feature map using FPN to finally obtain a target feature map;

[0012] The differentiable binarization module is used to determine the target text area in the image according to the target feature map.

[0013] In an embodiment of the present invention, the two branches of the residual correction branch are respectively a first branch and a second branch;

[0014] The first branch is used to perform conventional convolution on the input features to extract first branch features;

[0015] The second branch is used to perform average pooling and downsampling on the input features by a factor of r, then perform convolution and upsampling, and finally obtain the second branch features after passing the Sigmoid activation function;

[0016] The residual correction branch is also used to perform a dot product operation on the first branch feature and the second branch feature to obtain an output feature; after the output feature is added to the original image, a feature map of the original image is obtained after a Relu activation function.

[0017] In the embodiment of the present invention, average pooling is used to downsample by a factor of r, and the calculation formula is as follows:

[0018] x′2=AvgPool r (x2)

[0019] Among them, x2 is the input feature of the second branch; x′2 is the feature transformation of the second branch; r=4.

[0020] In the embodiment of the present invention, the calculation formula for obtaining the second branch feature after the Sigmoid activation function is as follows:

[0021]

[0022] Among them, y2 is the feature of the second branch; Up(·) is the nearest neighbor interpolation upsampling; x′2 is the feature transformation of the second branch; k2 represents the convolution operation.

[0023] In the embodiment of the present invention, the calculation formula of the first branch characteristic is as follows:

[0024]

[0025] Among them, y1 is the first branch feature; x1 is the input feature of the first branch; k1 represents the convolution operation.

[0026] In an embodiment of the present invention, a dual-branch attention feature fusion module is embedded in the FPN structure;

[0027] The dual-branch attention feature fusion module is used to enhance the feature expression of multi-scale scene text, thereby improving the accuracy of its detection.

[0028] In an embodiment of the present invention, the dual-branch attention feature fusion module includes a global feature channel and a local feature channel;

[0029] The FPN is used to initially fuse any two feature maps of the original image to obtain an initial fused feature;

[0030] The global feature channel is used to perform global average pooling on the initial fusion features and then perform convolution on them to extract the global feature channel attention;

[0031] The local feature channel is used to perform convolution on the initial fusion feature to extract the local feature channel attention;

[0032] The dual-branch attention feature fusion module is also used to add the global feature channel attention and the local feature channel attention, activate them, and then multiply them element-wise with the larger size feature map in the feature map of the original image to finally determine the target feature map.

[0033] In this embodiment of the present invention, the calculation formula of the global feature channel attention is as follows:

[0034] g(X)=B(PWConv2(δ(B(PWConv1(Avg(X))))))

[0035] Among them, g(X) represents the global feature channel attention; B represents the BatchNorm layer; PWConv represents point-by-point convolution; δ represents the ReLU activation function, X represents the initial fusion feature; Avg represents the global average pooling.

[0036] In this embodiment of the present invention, the calculation formula of the local feature channel attention is as follows:

[0037] L(X)=B(PWConv2(δ(B(PWConv1(X)))))

[0038] Among them, L(X) represents the local feature channel attention; B represents the BatchNorm layer; PWConv represents point-by-point convolution; δ represents the ReLU activation function, and X represents the initial fusion feature.

[0039] In this embodiment of the present invention, after adding the global feature channel attention and the local feature channel attention, the global feature channel attention is activated and then multiplied element-wise with the larger size feature map in the feature map of the original image to obtain the target feature map. The calculation formula is as follows:

[0040]

[0041] Among them, X′ represents the target feature map; Represents the attention weight; P represents the larger size feature map in the feature map of the original image; σ represents the Sigmoid activation function; g(X) represents the global feature channel attention; L(X) represents the local feature channel attention.

[0042] The beneficial effects of the present invention lie in that, based on the DBNet algorithm, the present invention improves the feature extraction network. The improved ResNet lightweight feature extraction network and the better feature fusion method effectively fuse features of different depths to guide segmentation. ResNet introduces a residual correction branch (RCB) to expand the receptive field and enhance the ability to obtain contextual information, thereby obtaining contextual information with a larger receptive field. At the same time, to improve the efficiency of feature utilization, a dual-branch attention feature fusion (TB-AFF) module is added to the FPN structure. By combining global and local attention mechanisms, the text region is precisely located and the text position in natural scenes is accurately detected. Finally, a differentiable binarization module is used to incorporate the binarization process into the model training process, adaptively set the binarization threshold, and convert the probability map generated by the segmentation method into a text region, achieving better text detection results. The entire model not only ensures the quality of feature extraction, but also, because it is a lightweight network, achieves a good balance between speed and accuracy. Without sacrificing speed, the network's receptive field is expanded, more detailed text position information is learned, and text regions are further accurately located. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a structural diagram of the scene text detection system in Example 1 of the present invention.

[0044] Figure 2 This is a structural diagram of the residual correction branch in Example 1 of the present invention.

[0045] Figure 3 This is a structural diagram of the dual-branch attention feature fusion module in Example 1 of the present invention.

[0046] Figure 4This is a structural diagram of the differentiable binarization in Example 1 of the present invention.

[0047] Figure 5 These are visualization results on different types of text examples in Example 2 of the present invention.

[0048] Figure 6 This is the visualization result of the Baseline in Example 2 of the present invention and the present invention.

[0049] Figure 7 The visualization results of different types of text examples in Example 2 of the present invention are displayed DETAILED DESCRIPTION

[0050] Example 1

[0051] The present invention aims to solve the problem that the DBNet text detection network in the prior art does not make full use of the semantic information and spatial information in the network, which limits the classification and positioning capabilities of the network, and lacks sufficient context information, resulting in false positives or missed detections. A scene text detection system is provided, comprising: an image acquisition unit, a feature extraction unit, a feature fusion unit, and a differentiable binarization module. The image acquisition unit is used to acquire the original image; the workflow diagram is as follows Figure 1 shown.

[0052] 1. The feature extraction unit is introduced as follows:

[0053] The system uses Resnet to extract feature maps of the original image. The Resnet backbone network is embedded with a residual correction branch (RCB). When the system is working, Resnet performs conventional convolution on the original image to obtain input features. The residual correction branch has two branches; one of the branches converts the input features into a low-dimensional embedding through downsampling, and uses this low-dimensional embedding to calibrate the convolution transformation of the convolution kernel in the other branch, ultimately obtaining the feature map of the original image. Specifically, the residual correction branch (RCB) does not simply perform traditional convolution on the input in the original space. Instead, it first converts the input into a low-dimensional embedding through downsampling, and uses this low-dimensional embedding to calibrate the convolution transformation of the convolution kernel in the other branch. Thanks to this convolution and communication between convolution kernels, each point in space has information about its nearby area and interactive information on the channel, avoiding interference from irrelevant areas in the entire global information. At the same time, it can also effectively expand the receptive field of each spatial position, thereby focusing on more contextual information.

[0054] Among them, the residual correction branch (RCB) has the following structure: Figure 2 As shown, it includes a first branch and a second branch;

[0055] The first branch is used to perform conventional convolution on the input features to extract first branch features;

[0056] The second branch is used to perform average pooling and downsampling on the input features by a factor of r, then perform convolution and upsampling, and finally obtain the second branch features after passing the Sigmoid activation function;

[0057] On the second branch, average pooling is first used to downsample by r times, and the calculation formula is as follows:

[0058] x′2=AvgPool r (x2)

[0059] Among them, x2 is the input feature of the second branch; x′2 is the feature conversion; r=4.

[0060] The calculation formula for the second branch feature after the Sigmoid activation function is as follows:

[0061]

[0062] Here, y2 represents the second branch feature; k2 represents the convolution operation; and Up(·) is the nearest neighbor interpolation upsampling, which aims to map the intermediate process from the small-scale space to the original feature space. The Sigmoid activation function can increase the nonlinearity of the neural network model, thereby improving the ability to fit nonlinear relationships between samples. Compared to the standard convolution in the original branch, the residual correction branch can adaptively establish dependencies between the surrounding environment of each channel and spatial location, allowing each channel and spatial location to adaptively treat its surrounding information environment as input from the latent space as a scalar in the response from the original scale space, thereby generating more discriminative features and extracting richer contextual information. Therefore, it can effectively expand the field of view of the network with the residual correction branch.

[0063] The calculation formula of the first branch characteristic is as follows:

[0064]

[0065] Where y1 is the first branch feature; x1 is the input feature of the first branch; and k1 represents the convolution operation. The input features of the first branch are the same as the input features of the second branch, both of which are obtained by ResNet performing regular convolution on the original image.

[0066] Then, the first branch feature and the second branch feature are multiplied to obtain the output feature; the output feature is added to the initial input of the module (i.e., residual connection), and the feature map of the original image is obtained after the ReLU activation function.

[0067] The residual correction branch (RCB) generates a global receptive field, fully capturing the contextual information of the segmented image. This significantly increases the field of view when applied to the convolutional layer, achieving the goal of expanding the convolutional receptive field and helping to effectively capture the entire discriminative region. It enables each spatial location to adaptively encode information context from the surrounding area, increasing the ability to extract feature information. It also takes into account the enhancement of inter-channel information, producing richer and more discriminative feature representations, thereby increasing the diversity of output features and improving the performance of the convolutional network.

[0068] On the other hand, the residual correction branch does not collect global context information, but only considers the context information around each channel and spatial position, thereby avoiding certain contamination information from irrelevant areas (non-text areas) to some extent. Therefore, the target object can be accurately located. And as can be seen from the figure, the residual correction branch module is highly versatile and easy to use, and can be easily applied to standard convolutional layers. In addition, it is well known that most attention-based or non-local methods generally require additional learnable parameters to construct the corresponding modules, and then insert them into the building block. In contrast, our residual correction branch does not rely on any additional learnable parameters, is applicable to a variety of tasks, and can be easily embedded in modern classification networks.

[0069] 2. The feature fusion unit is introduced as follows:

[0070] The feature fusion unit in this example is used to perform feature fusion on the feature map using FPN, and finally obtain a target feature map.

[0071] For the FPN structure, deeper features mean more channels, but the features of each layer are propagated from top to bottom when they are fused, so the top-level features are bound to reduce the number of channels. Since the reduction of feature channels will inevitably lead to the loss of contextual information, the top-level features tend to lose more information, and the contextual semantic information of the image plays a vital role in the segmentation network.

[0072] In order to retain more contextual information, this example uses FPN as an example of the most common scenario: long skip connections, and adds a dual-branch attention feature fusion (TB-AFF) module to FPN. This fully utilizes the features extracted from each layer of the network to cope with the scale changes of the text, which can retain more deep feature information and improve the performance of pyramid features. Specifically, TB-AFF is added to FPN to obtain an attention network, which we call a multi-scale attention fusion network (MSAFN). Its structure is as follows Figure 3 As shown in the figure, the dual-branch attention feature fusion (TB-AFF) module is embedded in the FPN structure, which can enhance the feature expression of multi-scale scene text and improve its detection accuracy.

[0073] The dual-branch attention feature fusion (TB-AFF) module is a feature fusion caused by long skip connections. It combines local features and global features in CNN, integrates the idea of ​​spatial attention, aggregates multi-scale feature context information within the attention module, and generates fusion weights that are the same size as the feature map, thereby dynamically selecting in an element-by-element manner, which is suitable for most common scenarios.

[0074] The dual-branch attention feature fusion (TB-AFF) module includes a global feature channel and a local feature channel; the global feature channel is based on SENet, but the fully connected layer is replaced with pointwise conv (point-by-point convolution), that is, a normal convolution with a convolution kernel of 1; the local feature channel uses pointwise conv (point-by-point convolution) to extract the channel attention of local features. For SENet, only the global channel attention is used, which is biased towards the global context, while the proposed TB-AFF also aggregates the local channel context attention, which helps the network contain less background clutter and is more conducive to the detection of small targets. By adding cross-layer connections to the dual-branch attention feature fusion (TB-AFF) module, the complementarity of multi-scale feature information can be achieved to obtain the final representation reflecting the context information.

[0075] The workflow is as follows: FPN performs an initial fusion on any two feature maps of the original image to obtain the initial fusion feature; in the global feature channel, the initial fusion feature is subjected to global average pooling, and then convolved to extract the global feature channel attention; in the local feature channel, the initial fusion feature is convolved to extract the local feature channel attention in order to preserve details. Then, the global feature channel attention and the local feature channel attention are added together, activated, and then element-wise multiplied with the larger feature map in the feature map of the original image to finally determine the target feature map. The global feature channel attention and the local feature channel attention are fused through the TB-AFF module, and attention is adjusted for each text position feature on the feature map. The features are updated by weighted summing of the aggregated features of all positions, focusing on the text area.

[0076] Among them, the calculation formula for performing global average pooling processing on the initial fusion feature and then performing convolution to extract the global channel context is as follows:

[0077] g(X)=B(PWConv2(δ(B(PWConv1(Avg(X))))))

[0078] Here, g(X) represents the global channel context; B represents the BatchNorm layer; PWConv represents point-by-point convolution; δ represents the ReLU activation function; X represents the initial fused features; and Avg represents global average pooling. The channel attention mechanism here uses point-by-point convolution to gradually compress channels and change the direction of convolution, assigning greater weight to channels with high response in text regions. Unlike L(X), this mechanism first performs a global average pooling (GAP) operation on the input X to obtain global attention information.

[0079] Similarly, the visual attention layer is used to enhance the extraction of local details. The calculation formula of the channel attention of local features is It is also extracted through point-by-point convolution. The calculation formula for extracting local channel context by convolution of the initial fusion feature is as follows:

[0080] L(X)=B(PWConv2(δ(B(PWConv1(X)))))

[0081] Where L(X) represents the local channel context; B represents the BatchNorm layer; PWConv represents the point-by-point convolution, δ represents the ReLU activation function, and X represents the initial fused features. L(X) has the same shape as the input features, which can preserve and highlight the subtle details in the low-level features.

[0082] Global and local attention are combined to identify the features that need attention. After adding the global channel context and the local channel context, they are activated and then multiplied element-wise with the larger size feature map in the feature map of the original image. The calculation formula for obtaining the target feature map is as follows:

[0083]

[0084] Where X′ represents the target feature map; T(X) represents the attention weight; and P represents the larger feature map in the feature map of the original image. Considering the limitations of the learned feature vector that highlights key regions, it is then element-wise added to the original input feature vector to learn more comprehensive features. σ represents the Sigmoid activation function. Using the sigmoid function to activate the attention channel, each element value is set between [0, 1]. This ensures that the attention module enhances useful image information and suppresses useless information.

[0085] This is because the global feature channel attention uses a global average pooling operation, so the resulting feature height and width shape is 1*1, while the local feature channel attention maintains the same height and width as the input feature, so the addition of the two requires a broadcast operation. It represents element-by-element multiplication, that is, the corresponding elements of the two feature maps are multiplied.

[0086] In summary, the dual-branch attention feature fusion (TB-AFF) module combines local and global feature information, as well as two input features, and uses two feature maps of different scales to extract attention weights. Its main contributions are as follows:

[0087] (1) The size problem in channel attention is raised. TB-AFF uses point-by-point convolution to focus on the scale of channels instead of convolution kernels of different sizes. Point-by-point convolution is used to make TB-AFF as lightweight as possible.

[0088] (2) TB-AFF aggregates global and local feature context information in the Feature Pyramid Attention Module (FPN) instead of in the backbone network.

[0089] 3. The differentiable binarization module is introduced as follows:

[0090] The system uses a segmentation network to segment the target feature map to generate a probability map P, P∈R H×W , where H and W represent the height and width of the input image respectively. To convert the probability map into a binary map, the binarization function is crucial. The standard binarization function is as follows:

[0091]

[0092] Pixels with a value of 1 are considered valid text areas. t is the set threshold, and (i, j) represents the coordinates of the points in the image. The standard binarization function is not differentiable, so it cannot be optimized with the segmentation network. To solve the problem of the non-differentiable binarization function, this example uses the following formula for binarization:

[0093]

[0094] Where B' is the approximate binary image, T is the adaptive threshold map learned from the network, and K is the amplification factor. During training, K amplifies the propagated gradient during backpropagation. This is beneficial for improving most mispredicted areas and producing more significant predictions. In this example, K is set to 50. This approximate binarization function is similar to the standard binarization function and is differentiable, allowing it to be optimized along with the segmentation network during training. Differentiable binarization allows for adaptive setting of the threshold T. This method not only effectively distinguishes foreground and background but also isolates closely connected text instances.

[0095] Specifically, the feature F is used to predict the probability map (P) and the threshold map (T), and the probability map and the threshold map are combined to obtain a binary map according to the differentiable binarization module, and the threshold of each position is adaptively predicted. Finally, the bounding box is formed to obtain the detection box of the text from the approximate binary map. The structure of the differentiable binarization is as follows Figure 4 Path 1 represents the standard binarization process, the dotted line only represents the inference process, and path 2 is the differentiable binarization used in this example, which adaptively predicts the threshold value for each position of the image.

[0096] Loss function plays a vital role in deep neural networks. This example uses L1 loss function and binary cross entropy loss function to optimize the network of this example. The loss function of this example consists of three parts during the training process: Probabilistic graph loss L s , Binarization image loss L b , adaptive threshold map loss L t , which is expressed as follows:

[0097] L=L S +α×L b +β×L t

[0098] Among them, α and β are weight parameters, α is set to 1, and β is set to 10. Among them, the probability map loss L s and binarization loss L b The binary cross entropy loss function is adopted, and its formula is as follows. Hard negative mining is also used to overcome the imbalance of positive and negative samples.

[0099]

[0100] Among them, S1 represents the sampling of the image with a positive and negative sample ratio of 1:3, and the adaptive threshold loss L t The L1 loss function is used, and its formula is:

[0101]

[0102] Among them, R d is the index of the pixel in the area, y * is the label of the adaptive threshold map.

[0103] In summary, the differentiable binarization module can effectively determine the target text area in the image based on the target feature map.

[0104] Example 2

[0105] In order to verify the effectiveness of the scene text detection system in this invention, this example also conducts experiments on three challenging public datasets, namely the multi-directional text dataset ICDAR2015, the curved text data Total-Text, and the multi-language text dataset MSRA-TD500. The visualization results of this method on different types of text examples are shown in the figure. Figure 5 As shown. Including curved (e) and (f), multi-directional (a) and (b), multi-language text (c) and (d). Figure 5 For each unit in , the probability map is in the second column, the threshold map is in the third column, and the binarization map is in the fourth column.

[0106] 1. Training Configuration

[0107] This experiment uses Python 3.7 as the programming language and PyTorch 1.5 as the deep learning framework. The Adam optimizer is used to train the model, with cosine learning rate decay as the learning rate schedule. The initial learning rate is 0.001, and the training batch size is 16. Data augmentation is performed on the training data using random rotation within the range of (-10°, 10°), random cropping, and flipping. All images are resized to 640×640. All experiments are conducted on a TITAN RTX. The initial learning rate is set to 0.007. Across all three datasets, all models are trained using the same strategy and tested under the same settings, which will not be detailed here.

[0108] 2. Experiments and Discussion

[0109] To better demonstrate the implementation of the various modules proposed in this example, detailed ablation studies were conducted on the ICDAR2015 multi-directional text dataset, the Total-Text curve text dataset, and the MSRA-TD500 multi-lingual text dataset. The model's detection performance was evaluated based on three key performance metrics: precision, recall, and a comprehensive evaluation metric. The impact of the residual correction branch (RCB) and two-branch attention feature fusion (TB-AFF) modules proposed in this example was demonstrated. Experiments were conducted under the same conditions during network training. Marked "√" indicates the use of this method. The results are listed in Table 1.

[0110] Table 1 Test results on the ICDAR2015 dataset

[0111]

[0112] Table 2 Test results on the Total-Text dataset

[0113]

[0114] Table 3 Test results on the MSRA-TD500 dataset

[0115]

[0116]

[0117] As shown in Tables 1, 2, and 3, the addition of the RCB module and / or the TB-AFF module significantly improves recall and comprehensive evaluation metrics on the ICDAR2015, Total-Text, and MSRA-TD500 datasets. Furthermore, it can be seen that the network combining the advantages of these two modules achieves superior detection performance compared to networks using either the RCB module or the TB-AFF module alone.

[0118] In the RCB module, an average pooling downsampling operation is introduced to achieve self-calibration. Average pooling establishes connections between locations within the entire pooling window, which better captures contextual information. Experimental results show that using an 18-layer backbone network with the proposed residual correction branch significantly improves baseline results. This phenomenon indicates that the network using the residual correction branch can generate richer and more discriminative feature representations than the original branch's plain convolution, helping to detect more complete target objects despite their small size. When the target objects are small, the network in this example can also better localize them to semantic regions. Furthermore, to overcome semantic and scale inconsistencies between input features, the dual-branch attention feature fusion (TB-AFF) module in this example adds local channel context to the global channel statistics. Experimental results show that the proposed TB-AFF-based network can improve the performance of state-of-the-art networks with a small parameter budget. This suggests that feature fusion in deep neural networks should be of interest, and that complex feature fusion attention mechanisms have the potential to produce superior results. This further demonstrates that, rather than blindly increasing network depth, it is more important to focus on the quality of feature fusion. Compared with linear methods (i.e., addition and concatenation), the Multi-Scale Attention Fusion Network (MSAFN) with a Two-Branch Attention Feature Fusion (TB-AFF) module consistently provides better performance.

[0119] Figure 6 The visualization results of the baseline and the method of the present invention are shown. For each unit in the figure, the probability map is in the second column, the threshold map is in the third column, and the binarization map is in the fourth column. From the experimental results, it can be seen that the residual correction branch (RCB) and the two-branch attention feature fusion (TB-AFF) module play an important role in feature extraction during model training, effectively enhancing the model's attention to text features, effectively utilizing the extracted text features, and improving the detection accuracy of scene text to a certain extent.

[0120] Figure 7 The visualization results of the present invention and the original DBNet on different types of text examples are shown. It is worth noting that the images here are randomly selected from three datasets, which can better demonstrate the robustness of this model. Figure 7 (a), compared with Baseline and Ours, Baseline missed a part of the text in the image (i.e., "CA"), while our method can detect it; Figure 7 (b) and Figure 7 (c) The Baseline misdetects non-text and detects non-text areas as text areas, while the method in this example can avoid misdetection well compared with the Baseline. Figure 7 (d), compared with Baseline and Ours, Baseline missed a part of the text in the image (i.e., "1"), while our method can detect it; Figure 7 (e), the Baseline missed the English text in the middle, but the method in this example can detect it accurately; Figure 7 (f),Baseline detects “COFFEE” as two parts of text, but in fact “COFFEE” represents a semantic information and should be detected as a whole text area.

[0121] Experimental results show that the present invention improves the detection capabilities on the multi-directional text dataset ICDAR2015, the curved text dataset Total-Text, and the multi-language text dataset MSRA-TD500. In this example, it can be seen that the present invention performs well in natural scene text detection datasets, with good performance, accuracy, recall rate and comprehensive evaluation index values. By adding the residual correction branch (RCB) and the dual-branch attention feature fusion (TB-AFF) module to enhance the feature information of text and orientation, the text detection field of view is expanded, and the detection effect of multi-scale text is effectively improved. The detection accuracy of the original algorithm is improved without losing detection efficiency, and to a certain extent, it is better than the current text detection in natural scenes. In various challenging scenes such as uneven lighting, low resolution, and complex background, the model in this example can effectively cope with the drastic scale changes of text and accurately detect scene text. On the one hand, the residual correction branch includes an adaptive response calibration operation, which helps to more accurately locate the exact position of the target object. ResNet with a residual correction branch can more accurately and completely locate the target object (text region), without excessively including background parts, even at low network depths. Meanwhile, the dual-branch attention feature fusion (TB-AFF) method offers superior performance and good versatility, enabling neural networks to extract features more efficiently. This effectively improves existing models by focusing on label-related objects, demonstrating its strong localization capabilities. This also demonstrates that early feature fusion also has a certain impact on attention feature fusion.

[0122] In summary, in order to make up for the defects of lightweight network in feature extraction ability and insufficient receptive field, the present invention embeds a residual correction branch (RCB) into the backbone network to enhance its feature extraction ability; and embeds a double-branch attention feature fusion (TB-AFF) module into the FPN to enhance the feature expression of multi-scale scene text, thereby improving the accuracy of its detection.

Claims

1. Scene text detection system, including: The image acquisition unit, feature extraction unit, feature fusion unit and differentiable binarization module are characterized by: The image acquisition unit is used to acquire the original image; The feature extraction unit is configured to extract a feature map of the original image using Resnet; a residual correction branch is embedded in the Resnet backbone network; the residual correction branch is configured to form two branches after Resnet performs conventional convolution on the original image to obtain input features; one branch converts the input features into a low-dimensional embedding by downsampling, and the low-dimensional embedding is used to calibrate the convolution transformation of the convolution kernel in the other branch, ultimately obtaining a feature map of the original image; The feature fusion unit is used to perform feature fusion on the feature map using FPN to finally obtain a target feature map; The differentiable binarization module is used to determine the target text area in the image according to the target feature map; The two branches of the residual correction branch are respectively a first branch and a second branch; The first branch is used to perform conventional convolution on the input features to extract first branch features; The second branch is used to perform average pooling and downsampling on the input features by a factor of r, then perform convolution and upsampling, and finally obtain the second branch features after passing the Sigmoid activation function; The residual correction branch is also used to perform a dot product operation on the first branch feature and the second branch feature to obtain an output feature; after the output feature is added to the original image, a feature map of the original image is obtained after a Relu activation function.

2. The scene text detection system according to claim 1, characterized in that Use average pooling to downsample r times, and the calculation formula is as follows: x′2=AvgPool r (x2) Among them, x2 is the input feature of the second branch; x′2 is the feature transformation of the second branch; r=4.

3. The scene text detection system according to claim 2, characterized in that The calculation formula for the second branch feature after the Sigmoid activation function is as follows: Among them, y2 is the feature of the second branch; Up(·) is the nearest neighbor interpolation upsampling; x′2 is the feature transformation of the second branch; k2 represents the convolution operation.

4. The scene text detection system according to claim 3, characterized in that The calculation formula of the first branch characteristic is as follows: Among them, y1 is the first branch feature; x1 is the input feature of the first branch; k1 represents the convolution operation.

5. The scene text detection system according to any one of claims 1 to 4, characterized in that: A dual-branch attention feature fusion module is embedded in the FPN structure; The dual-branch attention feature fusion module is used to enhance the feature expression of multi-scale scene text, thereby improving the accuracy of its detection.

6. The scene text detection system according to claim 5, characterized in that The dual-branch attention feature fusion module includes a global feature channel and a local feature channel; The FPN is used to initially fuse any two feature maps of the original image to obtain an initial fused feature; The global feature channel is used to perform global average pooling on the initial fusion features and then perform convolution on them to extract the global feature channel attention; The local feature channel is used to perform convolution on the initial fusion feature to extract the local feature channel attention; The dual-branch attention feature fusion module is also used to add the global feature channel attention and the local feature channel attention, activate them, and then multiply them element-wise with the larger size feature map in the feature map of the original image to finally determine the target feature map.

7. The scene text detection system according to claim 6, characterized in that The calculation formula of global feature channel attention is as follows: g(X)=B(PWConv2(δ(B(PWConv1(Avg(X)))))) in, g(X) represents global feature channel attention; B represents BatchNorm layer; PWConv represents point-by-point convolution; δ represents ReLU activation function, X represents initial fusion feature; Avg represents global average pooling.

8. The scene text detection system according to claim 7, characterized in that: The calculation formula of local feature channel attention is as follows: L(X)=B(PWConv2(δ(B(PWConv1(X))))) Among them, L(X) represents the local feature channel attention; B represents the BatchNorm layer; PWConv represents point-by-point convolution; δ represents the ReLU activation function, and X represents the initial fusion feature.

9. The scene text detection system according to claim 8, characterized in that: After adding the global feature channel attention and the local feature channel attention, they are activated and then multiplied element-by-element with the larger size feature map in the feature map of the original image. The calculation formula for obtaining the target feature map is as follows: Among them, X ′ Represents the target feature map; Represents the attention weight; P represents the larger size feature map in the feature map of the original image; σ represents the Sigmoid activation function; g(X) represents the global feature channel attention; L(X) represents the local feature channel attention.

Images