Arbitrary-shape scene text detection method based on contour feature enhancement

Abstract

The invention discloses an arbitrary-shape scene text detection method based on contour feature enhancement, and belongs to the technical field of computer vision. The invention provides an arbitrary-shape scene text detection method based on contour feature enhancement, and aims to solve the problems of edge contour missing and error detection in text detection so as to accurately position a text instance in a scene image. The method comprises the following steps: firstly, combining swin transform with a convolutional neural network, enhancing the perception of the network on a text region by utilizing rich detail information in a low-level feature map, and compensating for the missing of an edge contour; in addition, according to the method, the text area and the text-like area are distinguished through gradient information enhancement of the text edge, and therefore error detection is avoided. According to the method, a research scheme is provided for a deep learning method for natural scene text detection, and development of visual tasks such as image understanding is promoted to a certain extent.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method for detecting text in a scene of arbitrary shape based on contour feature enhancement. Background Art

[0002] The core content of computer vision is to process and analyze images or videos to complete deeper classification, detection, positioning, recognition and other tasks. As a classic task in the field of computer vision, the purpose of text detection is to detect text areas in natural images and locate them using bounding boxes or curves.

[0003] The widely distributed text information in natural scene images is a unique and important source of information in the image. Extracting text information from images has important research value and application value for image analysis and understanding. In recent years, text detection has been widely used in the fields of autonomous driving, image understanding / retrieval, machine translation, etc. In the field of autonomous driving: the detection and recognition of traffic signs is the key to accurately judge the road conditions. Detecting traffic signs, license plates, and any text information in the scene can help vehicles accurately navigate and locate the destination. In the field of image understanding / retrieval: by detecting and recognizing text information in scene images, it is helpful to understand the surrounding environment and objects, find the uniqueness between different objects, and further retrieve the desired objects from the environment, thereby improving the efficiency and accuracy of image retrieval. In the field of machine translation: in recent years, various machine translation software has emerged in an endless stream. Among them, reliably detecting and recognizing text from smartphone images can be more conveniently input into translation software, bringing convenience to people's lives. Therefore, it can be seen that natural scene text detection technology has profound research significance and practical value.

[0004] In recent years, with the development of deep learning in the field of computer vision, more and more scholars have paid attention to the field of text detection and have achieved excellent results. Text detection methods based on deep learning can be divided into regression-based text detection algorithms, transformer-based text detection algorithms, and segmentation-based text detection algorithms. Limited by the shape of fixed text boxes, regression-based text detection algorithms are difficult to achieve excellent results for arbitrary shape text detection; text detection algorithms based on segmentation ideas locate text areas through pixel-level predictions, are not limited by text shapes, and have become one of the mainstream algorithms in the field of text detection. Later, with the development and application of transformers in the field of target detection, considering that text detection is a special target detection, transformer-based text detection algorithms have also become the focus of text detection.

[0005] The segmentation-based text detection algorithm is mainly inspired by semantic segmentation. The algorithm first uses the network to generate pixel-level predictions and discriminate each pixel, and then reconstructs the text instance through pixel-level predictions and specific post-processing. In order to better process the features extracted by the backbone network and avoid the adhesion of adjacent text areas, in 2019, Wang et al. proposed the Progressive Scale Expansion Network (PSENet) to expand the detection area from a small core to a large core, and finally merge to form a complete text area, effectively separating text instances that are close to each other. The Pixel Aggregation Network (PAN) designs an efficient and accurate arbitrary shape text detector through a lightweight segmentation network. This method uses an efficient segmentation head to refine the features, predict the similarity vector between the text core and the surrounding pixels, and then aggregates the pixels belonging to the same text core through a learnable pixel aggregation algorithm. In 2020, Liao et al. proposed the Differentiable Binarization Network (DBNet), which inserts the differentiable binarization module into the segmentation network. The binarization module can be trained along with the network, which not only has simple network results and simplified post-processing, but also achieves good performance in real-time scene text detection. The Text Detection via Segmentation with Probability Maps (TextPMs) proposed a detection method for arbitrary-shaped text based on probability mapping. This method uses a set of function probability mappings to describe the possible probability distribution as comprehensively as possible, and optimizes the prediction of the probability map by iteratively learning the mapping relationship between continuous probability distributions. Then, the probability map is aggregated to generate a complete text instance.

[0006] Transformer-based text detection algorithms are mainly inspired by the application of transformers in object detection. DETR (Detection Transformer), the original framework of transformer in object detection, regards object detection as a direct set prediction problem and proposes a simple end-to-end framework without anchor generation and complex post-processing. The text detection task can be regarded as a special object detection problem. In 2021, transformer-based text detection methods introduced a loss function for multi-directional text detection, which mainly uses quadrilateral-based predictions to represent text regions. Feature Sampling and Grouping for Scene Text Detection (FSG) proposed a scene text detection framework based on feature sampling and grouping, which avoided background interference and reduced computational complexity. In 2023, Ye et al. designed an efficient dynamic point text detection transformer network (Text Detection with Dynamic Points in Transformer, DPText-DETR), which uses explicit point coordinates to generate location queries and dynamically updates them in a progressive manner to enhance training convergence.

[0007] Although the above text detection algorithms have achieved good performance, these methods still face some problems. Transformer-based text detection methods are affected by the self-attention mechanism and position encoding, and usually have high computational complexity, which affects the detection efficiency of the entire model. Segmentation-based text detection methods usually face two problems. The first problem is that the use of multiple convolutional layers will lead to unclear text edge information, resulting in the loss of text boundary contours. Another difficulty is that when using the feature pyramid network (FPN) to fuse multi-level features from top to bottom, it will blur the salient features between the text area and the background, which may cause the network to mistakenly classify non-text elements as text instances, resulting in misdetection.

[0008] In order to solve the above problems, the present invention proposes a method for detecting text in scenes of arbitrary shapes based on contour feature enhancement. Summary of the invention

[0009] The purpose of the present invention is to propose a method for detecting text in an arbitrary shape scene based on contour feature enhancement to solve the problems of missing edge contours and false detection in text detection. The present invention first combines a swin transformer with a convolutional neural network, and uses the rich detail information in a low-level feature map to enhance the network's perception of the text area and compensate for the missing edge contours. In addition, the present invention also uses the gradient information enhancement of the text edge to distinguish between the text area and the quasi-text area, thereby avoiding the occurrence of false detection. The present invention provides a research plan for a deep learning method for natural scene text detection, and to a certain extent promotes the development of visual tasks such as image understanding.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A method for detecting text in an arbitrary shape scene based on contour feature enhancement, the method is implemented by a system for detecting text in an arbitrary shape scene based on contour feature enhancement, the system is composed of a backbone network, a multi-scale feature fusion module and an output module, wherein the multi-scale fusion module includes a feature compensation module (FCM) and an orthogonal text attention module (OTAM);

[0012] The method comprises the following steps:

[0013] S1, use the backbone network to extract pyramid features and input the output of each stage into the multi-scale feature fusion module;

[0014] S2, adopt a top-down strategy to fuse features from different levels, input the lowest level features into the feature compensation module (FCM), and use a lightweight swin transformer combined with a convolutional neural network to enhance the model's perception of text areas with lower computational complexity and compensate for the lack of text edge information;

[0015] S3, using the attention module to adjust the information to enhance the interaction of feature map information in the channel dimension;

[0016] S4, perform top-down feature fusion on the newly obtained multi-layer pyramid features, and then upsample them through 3×3 convolution and bilinear interpolation of different proportions, adjust the resolution of multi-scale feature maps to the same size, and adjust the number of channels to 64; then splice the multi-level feature maps along the channel dimension to obtain the feature map F;

[0017] S5, using two parallel branches of the Orthogonal Text Attention Module (OTAM) to learn the attention weights of the orthogonal directions of the fused features, and then using residual connections to avoid the loss of detail information;

[0018] S6. Based on the operation described in S5, a pair of orthogonal edge texture feature maps are obtained, and they are spliced ​​along the channel dimension to obtain a new feature map F. C , then a nonlinear activation module is used to help gradient propagation to obtain the feature F*;

[0019] S7, for the orthogonal feature map obtained by S6, first divide it into two parts along the channel: α times the channel and (1-α) times the channel, and then process it through the activation module to generate attention weights; use convolution with a kernel size of 1×1 to compress the number of channels to improve computational efficiency; finally, multiply the attention weights with the input feature map and perform residual connection to obtain a feature map with enhanced edge information;

[0020] S8, input the feature map obtained in S7 to the output module for post-processing, wherein the output module is composed of two parallel branches with the same structure, each branch is composed of two deconvolution functions, and is used to predict the text probability map f P and threshold map f T , and then a binary mapping is obtained through a differentiable binarization function, and the predicted bounding box is obtained from it.

[0021] Preferably, the S1 specifically includes the following contents:

[0022] ResNet-50 is used as the backbone network to extract pyramid features, and the outputs {F2, F3, F4, F5} of stages 2, 3, 4, and 5 are input into the multi-scale feature fusion module. The resolution of the multi-scale features is {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the input image, and the number of channels is 256.

[0023] Preferably, S2 specifically includes the following contents:

[0024] The input high-resolution features{F2}first pass through two serial swin transformer blocks (STBs), which use window attention mechanism and multi-head self-attention mechanism to enhance the expressiveness of the model;

[0025] In the second block, based on the shift window method, boundary embedding (BE) is further introduced to encode the initial pixels of the feature map before segmentation, and the encoded information is embedded into the model to make it easier for the model to perceive text information; the process of processing features by a single swin transformer block (STB) is as follows:

[0026] F M =F E +f MSA (f LN(F E ))

[0027] F S =F M +f LN (f MLP (F M ))

[0028] Among them, f MSA represents long self-attention; f LN Representation layer normalization; F E represents the input features after layer normalization and multi-head self-attention module; F M represents the output features after layer normalization and multi-head self-attention module; f MLP Represents a multi-layer perceptron module; F S Represents the output of the swin transformer block.

[0029] Preferably, the information is adjusted using the attention module in S3, and its function is expressed as follows:

[0030] F′2=F S ×σ(Conv k×k (f GAP (F S )))

[0031] Among them, σ represents the Sigmoid activation function; f GAP Represents global average pooling; Conv k×k (·) represents a two-dimensional convolution with a kernel size of k×k, and “×” represents element-wise multiplication.

[0032] Preferably, the newly obtained multi-layer pyramid features in S4 are recorded as {F′2, F3, F4, F5}; the fused features are recorded as {P2, P3, P4, P5}; the functions of feature fusion and feature concatenation are expressed as follows:

[0033]

[0034] F=[f up×8 (P5), f up×4 (P4), f up×2 (P3), Conv 3×3 (P5)]

[0035] Among them, f up×N (·) represents the bilinear interpolation upsampling operation with a scaling factor of N; Conv 3×3 (·) indicates a 3×3 convolutional layer; [·] indicates special concatenation along the channel dimension.

[0036] Preferably, the S5 specifically includes the following contents:

[0037] In the left branch of the Orthogonal Text Attention Module (OTAM), global average pooling is used for feature aggregation in the horizontal direction, focusing only on the texture features along the horizontal direction, and then the feature map is expanded to the original size; in the right branch of the Orthogonal Text Attention Module (OTAM), the same operation is performed on the feature map in the vertical direction; the function of the above feature processing process is expressed as follows:

[0038] F h =f ex (f GAP_h (F)

[0039] F v =f ex (f GAP_v (F)

[0040] Among them, f ex (·) indicates expansion operation; f GAP_h (·) and f GAP_v (·) represents global average pooling in horizontal and vertical directions respectively; F h and F v Represents the output features in orthogonal directions.

[0041] Preferably, the function of S6 is expressed as:

[0042] F C =[F h ,F v ]

[0043] F * =ReLU(f BN (Conv 1×1 (F C )))

[0044] Among them, [·] indicates feature concatenation along the channel dimension; Conv 1×1 (·) indicates a convolution with a kernel size of 1×1; f BN (·) represents the batch normalization operation; ReLU(·) represents the ReLU activation function.

[0045] Preferably, the function of S7 is expressed as:

[0046]

[0047] F O =F+F×F S1 ×F S2

[0048] Where slice(·) represents the slicing operation; σ(·) represents the sigmoid activation function; Conv1×1 (·) indicates convolution with a kernel size of 1×1; “×” indicates element-wise multiplication.

[0049] Preferably, the differentiable binarization function in S8 is specifically:

[0050]

[0051] Among them, (i, j) represents the coordinate point of the pixel in the feature map.

[0052] Compared with the prior art, the present invention provides a method for detecting text in scenes of arbitrary shapes based on contour feature enhancement, which has the following beneficial effects:

[0053] The present invention proposes a method for detecting text in arbitrary-shaped scenes based on contour feature enhancement, aiming to solve the problem of blurred contours in text detection. The present invention includes a feature compensation module (FCM) and an orthogonal text attention module (OTAM); wherein the feature compensation module FCM uses a lightweight swin transformer to embed the details of the feature map in combination with the boundary, and then combines the attention module to compensate the text contour information of the feature map in the spatial and channel dimensions respectively; then, the orthogonal text attention module OTAM processes the fused feature map to enhance the feature difference between the text and the background, thereby reducing the occurrence of false positives and true negatives, and enabling the network to accurately locate the text boundary. A large number of experiments on four text detection benchmark datasets demonstrate the effectiveness and advantages of the method proposed in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The overall network structure diagram of the arbitrary shape scene text detection method based on contour feature enhancement mentioned in Example 1 of the present invention;

[0055] Figure 2 is a detailed structural diagram of the feature compensation module (FCM) mentioned in Example 1 of the present invention;

[0056] Figure 3 Detailed structural diagram of the orthogonal text attention module (OTAM) mentioned in Example 1 of the present invention;

[0057] Figure 4 This is a comparison chart of the detection results of this method and other methods mentioned in Example 2 of the present invention. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0059] First, the abbreviations and key term definitions mentioned in the present invention are explained, specifically:

[0060] PSENet: Progressive Scale Expansion Network

[0061] PAN: Pixel Aggregation Network

[0062] DBNet: Differentiable Binarization Network, Differentiable Binarization Network

[0063] TextPMs:Text Detection via Segmentation with Probability Maps,Probabilistic Graph Segmentation Network

[0064] FSG: Feature Sampling and Grouping for Scene Text Detection, feature sampling and grouping network

[0065] DPText-DETR: Text Detection with Dynamic Points in Transformer, dynamic point text detection network

[0066] FCM: Feature Compensation Module

[0067] OTAM: Orthogonal Text Attention Module

[0068] Based on the above content, specific examples are described below.

[0069] Embodiment 1:

[0070] The present invention proposes a method for detecting text in arbitrary shape scenes based on contour feature enhancement. The overall network structure is as follows: Figure 1As shown in Figure 1, it consists of three parts: backbone network, multi-scale feature fusion module and output module. The multi-scale feature fusion module includes feature compensation module (FCM) and orthogonal text attention module (OTAM).

[0071] Specifically, ResNet-50 is first used as the backbone network to extract pyramid features, and the outputs {F2, F3, F4, F5} of stage 2, stage 3, stage 4 and stage 5 are input into the multi-scale feature fusion module. The resolution of the multi-scale features is {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the input image, and the number of channels is 256. Afterwards, a top-down strategy is adopted to fuse features from different levels. However, only simple upsampling is used for top-down feature fusion, which usually leads to the loss of text edge information and affects the positioning of text boundaries. Therefore, the present invention inputs the features of the lowest level into the feature compensation module (FCM), and uses a lightweight swin transformer combined with a convolutional neural network to enhance the model's perception of the text area with lower computational complexity and compensate for the lack of text edge information.

[0072] The detailed structure of the feature compensation module FCM is as follows Figure 2 As shown in the figure, the input high-resolution features {F2} first pass through two serial swin transformer blocks (STB). The swin transformer uses the window attention mechanism and the multi-head self-attention mechanism to enhance the expressiveness of the model. In the second block, the shift window method is adopted. Our model sets the window size to 4×4 to reduce the computational complexity. However, the window block method may cause discontinuity of text within the same instance, which is not conducive to the network remote modeling, thus affecting the effectiveness of text detection. Although stacking more swin transformer blocks can partially alleviate this problem, it will greatly increase the computational complexity of the network. In order to solve this problem, boundary embedding (BE) is introduced to encode the initial pixels of the feature map before segmentation, and embed the encoded information into the model, making it easier for the model to perceive text information. The process of processing features by a single STB is as follows:

[0073] F M =F E +f MSA (f LN (F E ))

[0074] F S =F M+f LN (f MLP (F M ))

[0075] Among them, f MSA represents multi-head self-attention, f LN Representation layer normalization, F E and F M They represent the input features and output features after layer normalization and multi-head self-attention module, respectively, and f MLP Represents the multi-layer perceptron module, F S Represents the output of the STB.

[0076] Afterwards, in order to enhance the interaction of feature map information in the channel dimension, the attention module is used to adjust the information. The specific steps can be expressed as follows:

[0077] F′2=F S ×σ(Conv k×k (f GAP (F S )))

[0078] Among them, σ represents the Sigmoid activation function, f GAP represents global average pooling, Conv k×k (·) represents a two-dimensional convolution with a kernel size of k×k, where “×” represents element-wise multiplication.

[0079] Then, the newly obtained multi-layer pyramid features {F′2, F3, F4, F5} are fused from top to bottom to obtain {P2, P3, P4, P5}. After that, upsampling is performed through 3×3 convolution and bilinear interpolation of different proportions, the resolution of the multi-scale feature map is adjusted to the same size, the number of channels is adjusted to 64, and then the multi-level feature maps are spliced ​​along the channel dimension to obtain the feature map F. The feature fusion and feature splicing are shown as follows:

[0080]

[0081] F=[f up×8 (P5), f up×4 (P4), f up×2 (P3), Conv 3×3 (P5)]

[0082] Among them, f up×N (·) represents the bilinear interpolation upsampling operation, with a scaling factor of N. Conv 3×3 (·) represents a 3×3 convolutional layer, and [·] represents feature concatenation along the channel dimension.

[0083] It is well known that the feature difference between text edge information and complex background is the key to accurately locate text areas. However, the use of convolution with a kernel size of k×k when fusing multi-level features in a top-down path will lead to blurring of edge gradient information. In addition, targets with similar texture features to text areas will lead to false positives. Therefore, it is necessary to enhance the text contour features. The Orthogonal Text Attention Module (OTAM) proposed in the present invention adopts two parallel branches to learn the attention weights of the orthogonal directions of the fused features respectively, and then adopts residual connection to avoid the loss of detail information.

[0084] The detailed structure of the orthogonal text attention module OTAM is as follows Figure 3 As shown, in the left branch, global average pooling is used for feature aggregation in the horizontal direction, focusing only on the texture features along the horizontal direction, and then the feature map is expanded to the original size. In the right branch, the same operation is performed in the vertical direction to process the feature map. In this way, the model can perceive text boundary information from two directions, thereby improving the invention's ability to locate text instances. The entire feature processing process is shown below:

[0085] F h =f ex (f GAP_h (F)

[0086] F v =f ex (f GAP_v (F)

[0087] Among them, f ex (·) indicates expansion operation; f GAP_h (·) and f GAP_v (·) represents global average pooling in horizontal and vertical directions respectively; F h and F v Represents the output features in orthogonal directions.

[0088] After obtaining a pair of orthogonal edge texture feature maps, first concatenate them along the channel dimension to obtain the feature map

[0089] F C , then, a nonlinear activation module is used to help gradient propagation to obtain feature F * The process is as follows:

[0090] F C =[F h ,F v ]

[0091] F * =ReLU(f BN (Conv 1×1 (FC )))

[0092] Among them, [·] indicates feature concatenation along the channel dimension, Conv 1×1 (·) represents a convolution with a kernel size of 1×1, f BN (·) is the batch normalization operation, and ReLU(·) represents the ReLU activation function.

[0093] For the obtained orthogonal feature map, it is first divided into two parts along the channel: α times the channel and (1-α) times the channel, and then processed by the activation module to generate attention weights. In the method of the present invention, α is set to 0.5. In this part, in order to improve computational efficiency, a convolution with a kernel size of 1×1 is used to compress the number of channels. Finally, a feature map with enhanced edge information is obtained by multiplying the attention weights with the input feature map and performing a residual connection. By processing the fused features from an orthogonal direction, the feature distinguishability between the text area and the background can be effectively enhanced, which helps to more accurately predict the text area, thereby improving the detection accuracy of the network. The overall process of this part is as follows:

[0094]

[0095] F O =F+F×F S1 ×F S2

[0096] Among them, slice(·) represents the slicing operation, σ(·) represents the sigmoid activation function, Conv 1×1 (·) represents a 1×1 convolution, and “×” is element-wise multiplication.

[0097] Finally,

[0098] F O The output module of the present invention is composed of two parallel branches with the same structure, each branch mainly consists of two deconvolution functions, which are used to predict the text probability map f P and threshold map f T , and then a binary mapping is obtained through a differentiable binarization function, and the predicted bounding box is obtained from it. The binarization process can be shown in the following formula:

[0099]

[0100] Among them, (i, j) represents the coordinate point of the pixel in the feature map.

[0101] Embodiment 2:

[0102] Based on Example 1, but different in that, a specific experiment is designed below to verify the performance of the arbitrary shape scene text detection method based on contour feature enhancement proposed in the present invention, and the specific contents are as follows.

[0103] 1. Dataset

[0104] The arbitrary shape scene text detection method based on contour feature enhancement proposed in Example 1 is experimentally verified on four benchmark datasets. The datasets used include:

[0105] (1) Total-Text is a dataset mainly used for arbitrary shape text detection. The dataset is annotated with polygonal bounding boxes and consists of 1255 training images and 300 test images. These images contain horizontal, multi-directional, and curved text.

[0106] (2) CTW1500 is a dataset mainly composed of curved text, covering multiple languages ​​​​mainly Chinese and English. The dataset contains 1,000 training images and 500 test images, where text instances are annotated by polygons with 14 vertices.

[0107] (3) MSRA-TD500 is a multilingual dataset with complex background and multi-orientation, which includes 300 training images and 200 test images. Referring to the previous method, 400 training images from HUST-TR400 are also used in the training stage of the present invention.

[0108] (4) The ICDAR2015 dataset is collected by Google Glass and contains blurred text instances of different scales and orientations. The dataset contains 1,000 training images and 500 test images, and the text instances are annotated with rectangular bounding boxes.

[0109] (5) SynthText is a large dataset containing about 850,000 synthetic text images. The dataset includes images with different backgrounds from different scenes, which are annotated with characters, words, and lines. The proposed method is pre-trained only on this dataset.

[0110] 2. Experimental Setup

[0111] The whole experimental process is divided into a pre-training stage and a fine-tuning stage. First, two epochs of pre-training are performed on the artificial synthetic dataset SynthText dataset, with the learning rate fixed to 0.015 and the batch size set to 8; the fine-tuning stage is trained on four real scene datasets, with the initial learning rate set to 0.015, and a total of 1200 epochs are trained. The entire network is trained using the SGD optimizer. Data augmentation is performed on the images before training. The training images are randomly cropped to a size of 640×640. In addition, random horizontal flipping and random rotation are performed in the range of (-10°, 10°). The model proposed in this invention is implemented based on the Python language, and the experimental environment is shown in Table 1:

[0112] Table 1: Experimental environment

[0113]

[0114] 3. Experimental results

[0115] The proposed method is compared with previous scene text detection methods on four benchmark datasets. Experiments are conducted on multi-directional text datasets (MSRA-TD500 and ICDAR2015), arbitrary shape text datasets (Total-Text) and multi-language curve text datasets (CTW1500). The performance of the proposed method is evaluated from four indicators, including precision, recall, mean accuracy (Hmean) and frames per second (Frames Per Second, FPS). Ablation experiments are conducted on the MSRA-TD500 dataset and the CTW1500 dataset to illustrate the role of each component of the present invention.

[0116] Tables 2, 3, 4 and 5 show the quantitative evaluation results of each method on the text dataset, where the best indicators are bolded. It can be seen from Tables 2 and 3 that the method proposed in the present invention can well balance the detection accuracy and speed in the multi-directional text detection dataset, especially on the MSRA-TD500 dataset, the three indicators of the method of the present invention have achieved the highest performance, fully demonstrating the performance of the method of the present invention. Tables 4 and 5 show the detection results of the method of the present invention on arbitrary shape text datasets and multilingual curved text datasets. The method of the present invention has achieved the highest recall rate on these two curved text datasets. The superiority of the technology of the present invention is due to the fact that the method can effectively enhance the text contour features and distinguish the text area from the text-like background. At the same time, it can also compensate for the text edge information to obtain a more accurate text contour.

[0117] Table 2: Experimental results on the MSRA-TD500 dataset. The best performance is marked in bold.

[0118]

[0119] Table 3: Experimental results on the ICDAR2015 dataset. The best performance is marked in bold.

[0120]

[0121] Table 1: Experimental results on the Total-Text dataset, with the best performance in bold.

[0122]

[0123] Table 2: Experimental results on the CTW1500 dataset. The best performance is marked in bold.

[0124]

[0125]

[0126] In order to more intuitively demonstrate the effectiveness of the method proposed in this paper, the detection results of the method proposed in this paper and the classic text detection methods DBNet and DBNet++ are compared visually. The visualization results of the real labels and each detection method are shown in Figure 2. Figure 4 As shown. Figure 4 It can be clearly seen that the method proposed in the present invention can compensate for the boundary information of the text area and ensure the accurate positioning of the text area. In addition, the method of the present invention can also enhance the text contour features and prevent the complex background in the scene image from being mistakenly detected as a text instance. Figure 4 As shown in (b), the method proposed in the present invention can effectively detect multi-scale text and text of arbitrary shapes.

[0127] Table 6 shows an ablation experiment on the main module of the method of the present invention on the CTW1500 dataset. It should be noted that when verifying the effectiveness of the orthogonal text attention module, the initial learning rate is set to 0.007. As can be seen from Table 6, compared with the baseline network, the average accuracy of the model with the feature compensation module (FCM) added is improved by 2.05%. This shows that FCM can enrich the detailed features and enhance the network's perception of the text area, thereby compensating for the loss of contour information during the feature fusion process and more accurately locating the text area. When only the orthogonal text attention module (OTAM) is added to the baseline network, the average accuracy of the model is improved by 1.81%, which shows that the OTAM proposed in the present invention effectively captures text edge information from two orthogonal directions, enhances the contrast between the text area and the complex background, helps to accurately distinguish text from similar elements, and enables accurate detection of text instances. When SFCM and OTAM are added to the baseline network at the same time, the precision, recall and average accuracy of the model are improved by 5.3%, 1.77% and 3.52%, respectively. The experimental results fully demonstrate the effectiveness of the feature compensation module (FCM) and the orthogonal text attention module (OTAM) proposed in the present invention.

[0128] Table 6: Ablation experiment results of the main module, the best performance is bolded

[0129]

[0130] Tables 7, 8 and 9 conduct ablation experiments on the various components of the main module. The feature compensation module (FCM) is mainly based on the basic Swin Transformer block, with the addition of boundary embedding and attention modules. Table 7 is an ablation experiment on the CTW1500 dataset, which illustrates the role of each component in FCM in text contour feature compensation. Table 8 illustrates the influence of different levels of feature maps on the FCM feature compensation effect when fusing features from top to bottom, where F5 represents high-level features and F2 represents low-level features. From the experimental results, it can be seen that on the CTW1500 dataset, the average accuracy of the model using low-level features is improved by 1.01% compared with that using high-level features, which fully proves the effectiveness of FCM. Table 9 conducts an ablation experiment on the MSRA-TD500 dataset to illustrate the role of the slicing operation in the orthogonal text attention module (OTAM) and the division ratio when dividing into two parts along the channel.

[0131] Table 7: Ablation experiment results of feature compensation module, the best performance is bolded

[0132]

[0133] Table 8: Ablation experiment results of FCM processing feature maps at different levels, the best performance is bolded

[0134]

[0135] Table 9: Ablation experiment results of slicing operation and partition ratio in OTAM. The best performance is bolded, and w / o means no slicing operation

[0136]

[0137] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting text in arbitrary shape scenes based on contour feature enhancement, characterized in that: The method is implemented by an arbitrary shape scene text detection system based on contour feature enhancement, the system is composed of a backbone network, a multi-scale feature fusion module and an output module, wherein the multi-scale fusion module includes a feature compensation module and an orthogonal text attention module; The method comprises the following steps: S1, use the backbone network to extract pyramid features and input the output of each stage into the multi-scale feature fusion module; S2, adopt a top-down strategy to fuse features from different levels, input the lowest level features into the feature compensation module, and use a lightweight swin transformer and convolutional neural network to enhance the model's perception of text areas with low computational complexity and compensate for the lack of text edge information; S3, using the attention module to adjust the information to enhance the interaction of feature map information in the channel dimension; S4, perform top-down feature fusion on the newly obtained multi-layer pyramid features, and then upsample them through 3×3 convolution and bilinear interpolation of different proportions, adjust the resolution of multi-scale feature maps to the same size, and adjust the number of channels to 64; then splice the multi-level feature maps along the channel dimension to obtain the feature map F; S5, using the two parallel branches of the orthogonal text attention module to learn the attention weights of the orthogonal directions of the fusion features respectively, and then using residual connections to avoid the loss of detail information; S6. Based on the operation described in S5, a pair of orthogonal edge texture feature maps are obtained, and they are spliced ​​along the channel dimension to obtain a new feature map F. C , then a nonlinear activation module is used to help gradient propagation to obtain the feature F*; S7, for the orthogonal feature map obtained by S6, first divide it into two parts along the channel: α times the channel and (1-α) times the channel, and then process it through the activation module to generate attention weights; use convolution with a kernel size of 1×1 to compress the number of channels to improve computational efficiency; finally, multiply the attention weights with the input feature map and perform residual connection to obtain a feature map with enhanced edge information; S8, input the feature map obtained in S7 to the output module for post-processing, wherein the output module is composed of two parallel branches with the same structure, each branch is composed of two deconvolution functions, and is used to predict the text probability map f P and threshold map f T , and then a binary mapping is obtained through a differentiable binarization function, and the predicted bounding box is obtained from it.

2. The method for detecting text in an arbitrary shape scene based on contour feature enhancement according to claim 1, characterized in that: The S1 specifically includes the following contents: ResNet-50 is used as the backbone network to extract pyramid features. The outputs {F2, F3, F4, F5} of stage 2, stage 3, stage 4 and stage 5 are input into the multi-scale feature fusion module. The resolution of the multi-scale features is {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the input image, and the number of channels is 256.

3. The method for detecting text in an arbitrary shape scene based on contour feature enhancement according to claim 2, characterized in that: The S2 specifically includes the following contents: The input high-resolution features{F2}first pass through two serial swin transformer blocks, which use the window attention mechanism and multi-head self-attention mechanism to enhance the expressiveness of the model; In the second block, based on the shift window method, boundary embedding is further introduced to encode the initial pixels of the feature map before segmentation, and the encoded information is embedded into the model to make it easier for the model to perceive text information; the process of processing features by a single swintransformer block is as follows: F M =F E +f MSA (f LN (F E )) F S =F M +f LN (f MLP (F M )) Among them, f MSA represents long self-attention; f LN Representation layer normalization; F E represents the input features after layer normalization and multi-head self-attention module; F M represents the output features after layer normalization and multi-head self-attention module; f MLP Represents a multi-layer perceptron module; F S Represents the output of the swin transformer block.

4. The method for detecting text in an arbitrary shape scene based on contour feature enhancement according to claim 3, characterized in that: The information is adjusted using the attention module as described in S3, and its function is expressed as follows: F′2=F S ×σ(Conv k×k (f GAP (F S ))) Among them, σ represents the Sigmoid activation function; f GAP Represents global average pooling; Conv k×k (·) represents a two-dimensional convolution with a kernel size of k×k, and "×" represents element-by-element multiplication.

5. The method for detecting text in an arbitrary shape scene based on contour feature enhancement according to claim 4, characterized in that: The newly obtained multi-layer pyramid features described in S4 are recorded as {F2′, F3, F4, F5}; the fused features are recorded as {P2, P3, P4, P5}; the functions of feature fusion and feature concatenation are expressed as follows: F=[f up×8 (P5),f up×4 (P4),f up×2 (P3),Conv 3×3 (P5)] Among them, f up×N (·) represents the bilinear interpolation upsampling operation with a scaling factor of N; Conv 3×3 (·) indicates a 3×3 convolutional layer; [·] indicates special concatenation along the channel dimension.

6. The method for detecting text in an arbitrary shape scene based on contour feature enhancement according to claim 5, characterized in that: The S5 specifically includes the following contents: In the left branch of the orthogonal text attention module, global average pooling is used for feature aggregation in the horizontal direction, focusing only on the texture features along the horizontal direction, and then the feature map is expanded to the original size; in the right branch of the orthogonal text attention module, the same operation is performed in the vertical direction to process the feature map; the function of the above feature processing process is expressed as follows: F h =f ex (f GAP_h (F)) F v =f ex (f GAP_v (F)) Among them, f ex (·) indicates expansion operation; f GAP_h (·) and f GAP_v (·) represents global average pooling in horizontal and vertical directions respectively; F h and F v Represents the output features in orthogonal directions.

7. The method for detecting text in an arbitrary shape scene based on contour feature enhancement according to claim 6, characterized in that: The function of S6 is expressed as: F C =[F h ,F v ] F * =ReLU(f BN (Conv. 1×1 (F C ))) Among them, [·] indicates feature concatenation along the channel dimension; Conv 1×1 (·) indicates a convolution with a kernel size of 1×1; f BN (·) represents the batch normalization operation; ReLU(·) represents the ReLU activation function.

8. The method for detecting text in an arbitrary shape scene based on contour feature enhancement according to claim 7, characterized in that: The function of S7 is expressed as: F1 * ,F2 * =slice(F * ) F S1 =σ(Conv 1×1 (F1 * )) F S2 =σ(Conv 1×1 (F2 * )) F O =F+F×F S1 ×F S2 Where slice(·) represents the slicing operation; σ(·) represents the sigmoid activation function; Conv 1×1 (·) indicates convolution with a kernel size of 1×1; "×" indicates element-wise multiplication.

9. The method for detecting text in an arbitrary shape scene based on contour feature enhancement according to claim 8, characterized in that: The differentiable binarization function described in S8 is specifically: Among them, (i, j) represents the coordinate point of the pixel in the feature map.

Images