Image text segmentation method based on frequency separation

Abstract

The invention belongs to the field of image text segmentation in the field of computer vision, and discloses an image text segmentation method based on frequency separation. Firstly, a CNN and a Transform are used to construct a local feature coding branch and a global feature coding branch respectively, and local features and global features from an image are extracted preliminarily. In the frequency feature enhancement module, wavelet transform is used to decompose the frequency of the input feature. And then enhancing the obtained high-frequency component and low-frequency component by using different strategies. And finally, low-frequency information is utilized to guide the network to pay attention to local features near the text region, the influence of irrelevant background noise on a segmentation result is reduced, and finally an accurate segmentation result is obtained. According to the method, the problem of dependence on a text detection model in an image text segmentation task is solved, and a result output by the method has high integrity and accuracy.

Description

Technical Field

[0001] The invention belongs to the field of image text segmentation in the field of computer vision, and relates to an image text segmentation method based on frequency separation. Background Art

[0002] The image text segmentation task aims to generate pixel-level text masks from scene images. As a basic and key task in the field of text processing, it has attracted extensive attention and research in academia and industry in recent years. With the advancement of digital image processing technology and the increase in application needs, text-related research has continued to emerge and show a rapid development trend. For example, text erasure technology allows users to remove unnecessary text information from images, which is of great significance in fields such as image editing and copyright protection. In addition, interactive text image editing technology has also made great progress, allowing users to intuitively edit the text content in images. This has a wide range of applications in fields such as advertising design, social media, and personal entertainment. The development and application of these technologies all depend on a common prerequisite: the need to accurately extract pixel-level text masks from images. Therefore, in order to meet the growing demand, image text segmentation technology has gradually become the focus of researchers.

[0003] Unlike other image segmentation tasks that deal with semantic continuity (such as instance segmentation and semantic segmentation), image text segmentation tasks usually face the following challenges: 1) The text of interest is usually scattered at random locations in the image, and there may be huge scale differences between texts at different locations. Therefore, the model can easily ignore smaller texts located in the corners of the image. 2) Since the structure of text (especially Chinese characters composed of strokes) is loose and incoherent, the model can easily ignore small parts, such as the "dot" in Chinese characters or the "dot" in the lowercase letter "i" in English. These characteristics increase the complexity and difficulty of image text segmentation tasks, and specially designed models and algorithms are required to meet these challenges.

[0004] To address these challenges, Xixi Xu et al. proposed a two-stage network structure PGTSNet in their paper "BTS: A Bi-lingual Benchmark for Text Segmentation in the Wild". PGTSNet first uses a pre-trained text detector to identify text areas in the entire image, crops these areas, and then inputs them into the subsequent segmentation network for text segmentation. Although it has achieved good performance, it relies heavily on text detectors pre-trained on additional datasets. Yujin Ren et al. proposed in their paper "Looking from a Higher-level Perspective: Attention and Recognition Enhanced Multi-scale Scene Text Segmentation" that it is necessary to fully integrate global context information and local details to obtain better segmentation effects, but its feature encoder uses a CNN architecture. Therefore, it cannot capture enough global features. And it also uses a pre-trained text detector.

[0005] Based on the analysis of existing image text segmentation algorithms, this paper proposes an image text segmentation method based on frequency separation to address the problems faced by image text segmentation tasks. Summary of the invention

[0006] The technical problem to be solved by the present invention is how to accurately and without omission identify and locate the position of text in an image without using an additional text detection model, and obtain accurate segmentation results, providing a new method for image text segmentation that makes full use of different frequency information. The purpose of the present invention is to give a binary mask representing text pixels by the proposed method given a single RGB image.

[0007] The present invention is an image text segmentation method based on frequency separation, which is jointly implemented based on CNN and Transformer. The present invention first uses CNN and Transformer to construct local feature encoding branches and global feature encoding branches respectively, and preliminarily extracts local features and global features from the image. Then, in order to effectively enhance the global features obtained by the encoder, this paper proposes a frequency feature enhancement module. In the frequency feature enhancement module, wavelet transform is first used to decompose the frequency of the input feature. Then, different strategies are used to enhance the obtained high-frequency components and low-frequency components. Finally, the low-frequency information is further used to guide the network to focus on the local features near the text area, reduce the influence of irrelevant background noise on the segmentation results, and finally obtain accurate segmentation results.

[0008] The technical solution of the present invention:

[0009] A frequency separation-based image text segmentation method, whose overall network framework includes a dual-branch feature encoding module, a frequency feature enhancement module, a local detail enhancement module and a decoder; the specific steps are as follows:

[0010] Step 1: The dual-branch feature encoding module is mainly composed of a local feature encoding branch based on CNN and a global feature encoding branch based on Transformer, which are used to preliminarily extract the local features F1 and global features F of the target image respectively. i , i∈[2,5]; In order to reduce the computational complexity and training cost of the network, the local feature encoding branch is composed of a simple two-layer convolution with batch normalization and ReLU activation function, while the global feature encoding branch is completed using SwinTransformer.

[0011] Step 2: The global feature F of the target image obtained by the dual-branch feature encoding module is enhanced by the frequency feature enhancement module i , i∈[2,5] for targeted enhancement; the frequency feature enhancement module is mainly composed of a high-frequency component enhancement module and a low-frequency component enhancement module.

[0012] The frequency feature enhancement module first uses discrete wavelet transform to transform the global feature F i , i∈[2,4] is decomposed into high-frequency and low-frequency parts, and then the high-frequency component enhancement module and the low-frequency component enhancement module are used to enhance them respectively. Finally, the enhanced high-frequency component and low-frequency component are recombined by matrix addition to obtain the enhanced global features; the details are as follows:

[0013] First, the features are decomposed using discrete wavelet transform.

[0014]

[0015] Among them, F i represents the global features extracted by the dual-branch feature encoding module, D i represents the features recovered layer by layer in the decoder through convolution, batch normalization, ReLu activation function and two-fold upsampling, and Represents the high-frequency component in the global features of the image, and represents the low-frequency component in the global features of the image; ω(·) represents a learnable wavelet transform, whose coefficients are initialized by Haar wavelet.

[0016] Then the obtained low-frequency component and Input to the low-frequency component enhancement module, first calculate the spatial attention operation separately and The spatial attention weights M1 and M2 are:

[0017]

[0018] where Θ(·) represents the spatial attention operation. The weights M1 and M2 reflect the low-frequency components and The importance of each position in .

[0019] The attention weights M1 and M2 are then concatenated along the channel dimension and normalized using softmax:

[0020]

[0021] Among them, [·] represents the concatenation operation along the channel dimension, σ(·) represents the softmax function, and Represents the transformed spatial attention weight. By applying the softmax function, the low-frequency information from different feature maps is effectively integrated and balanced.

[0022] Finally, the optimized attention weights are applied to the original low-frequency components using element-by-element multiplication, thereby enhancing the low-frequency components:

[0023]

[0024] in and Represents the enhanced low-frequency component. In addition, the enhanced low-frequency component will be fused to guide the learning of high-frequency features in the local detail enhancement module, and the fusion is completed by simple convolution:

[0025]

[0026] Among them, G i represents the fused low-frequency features. Φ(·) is a convolutional layer with batch normalization and ReLU activation function.

[0027] For high frequency components and First, the cross-attention mechanism is used to guide it to focus on the edge and texture information of the image. This mechanism uses the local features extracted by the local feature encoder branch to enhance the high-frequency components in the global feature encoder branch:

[0028]

[0029]

[0030] in, and represents the high frequency component after cross attention enhancement. MHCA(·) represents the cross attention operation. Then, and The further enhanced high-frequency components are obtained through the same operations as in the low-frequency component enhancement module (spatial attention, concatenation along the channel dimension, softmax function, and element-wise multiplication).

[0031] Step 3: In order to enhance the network's attention to the features of the text region, the present invention constructs a local detail enhancement module. This module consists of three parallel cross-attention modules and a convolutional layer. The convolutional layer consists of convolution, batch normalization, and ReLU activation functions. In the cross-attention module, the low-frequency component G output by the low-frequency component enhancement module is used. i As the key and value, the local feature F1 extracted by the local feature encoder branch is used as the query. Then the local features enhanced by convolution fusion are used to further optimize the network's learning of text details:

[0032] fi=MHCA(F1,Gi,Gi)

[0033] Fd=Φ([f1,f2,f3])

[0034] Among them, f i Represents the local features enhanced by a single cross-attention branch, through the low-frequency component G i The guidance of makes the local features focus on the text area to enhance the task-related local details, F d It is the local features fused through the convolutional layer that ensure that the model can capture image details more accurately and achieve better text segmentation results.

[0035] In the image text segmentation task, the proportion of text pixels in the image is usually small, which will cause serious class imbalance problems. Therefore, the present invention adopts a combination of BCE loss and Dice loss to jointly supervise model training.

[0036] The beneficial effects of the present invention are as follows: the present invention solves the dependence on the text detection model in the image text segmentation task, and the result output by the method has high integrity and accuracy; the present invention can make full use of the respective advantages of high-frequency information and low-frequency information in the image, identify and locate the text scattered in the picture through the low-frequency information, thereby obtaining a rough segmentation result, and learn the detailed information of the target text through the high-frequency information to optimize the segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1It is the overall network structure of the present invention, the left half is a dual-branch feature encoding module, the middle part is a frequency feature enhancement module and a local detail enhancement module, and the right half is a decoder.

[0038] Figure 2 The detailed structure of the frequency characteristic enhancement module proposed in the present invention is as follows: represents pixel-by-pixel addition, ⊙ represents pixel-by-pixel multiplication, Indicates splicing along the channel, Represents the softmax operation.

[0039] Figure 3 The detailed structure of the local detail enhancement module proposed in the present invention is as follows: Represents matrix multiplication. DETAILED DESCRIPTION

[0040] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0041] (1) Building an overall network framework

[0042] Figure 1 The overall structure of the image text segmentation method based on frequency separation proposed in this invention is as follows: the target image is first downsampled twice and then sent to the global feature extraction branch composed of Swin Transformer to obtain global features F of different sizes. i , i∈[2,5], and at the same time, the target image is sent to the local feature branch consisting of two layers of convolution, batch normalization and ReLu activation function to obtain the local feature F1. After the global features of the image are extracted, they are sent to the frequency feature enhancement module. The frequency enhancement module first decomposes the feature map into high-frequency components and low-frequency components through discrete wavelet transform, and then enhances them using the high-frequency feature enhancement module and the low-frequency feature enhancement module respectively. Finally, the enhanced low-frequency features are used to guide the network to learn local features related to the text, reduce the interference of irrelevant background noise, and obtain the final binary segmentation result.

[0043] (2) Constructing frequency enhancement module

[0044] Figure 2The frequency enhancement module proposed in the present invention can use discrete wavelet transform to perform frequency separation on the feature map. After obtaining the high-frequency and low-frequency components of the feature map, the low-frequency features are first subjected to spatial attention operation to obtain their spatial attention weight map, and then the attention weight map obtained is spliced ​​along the channel dimension, and then a softmax operation is performed to exchange information from different feature maps. Finally, the spatial attention weight map is applied to the respective low-frequency components. At this time, the low-frequency components can better perceive the text information in the image. For the high-frequency components, they will be initially enhanced using cross-attention, and then enhanced in the same way as the low-frequency components.

[0045] (3) Constructing a local detail enhancement module

[0046] Figure 3 The local detail enhancement module proposed in the present invention utilizes the low-frequency components optimized in the frequency enhancement module through a cross-attention mechanism to guide the network to better learn the local features F1 related to the text area.

[0047] (4) Constructing the loss function

[0048] In the image text segmentation task, the proportion of text pixels in the image is relatively small. This will lead to serious class imbalance problems. Therefore, directly using binary cross entropy (BCE) loss may cause the model to overfit background pixels, thereby ignoring the accurate prediction of text pixels. Therefore, the present invention adopts a combination of BCE loss and Dice loss to jointly supervise model training.

[0049] BCE loss measures pixel-level prediction error and is applicable to most segmentation scenarios. It can be expressed as,

[0050]

[0051] where y i and p i Represent the manually labeled true value and the network's prediction results respectively.

[0052] Dice loss is widely used in medical image segmentation and can effectively solve the class imbalance problem. Its calculation formula is:

[0053]

[0054] The total loss is expressed as a combination of BCE loss and Dice loss, i.e.

[0055] Finally, inspired by the salient object detection method, in order to effectively supervise the boundary pixels of the text, the present invention also uses BCE loss and Dice loss to supervise the boundary pixels separately, that is, The total loss function is

[0056] (5) Dataset

[0057] The method is trained and tested on six image-text datasets: ICDAR13 FST, MLT-S, COCO-TS, Total-Text, TextSeg, and BTS. ICDAR13 FST focuses on text in natural scenes, such as road signs, billboards, menus, etc. The dataset contains 462 images, which is significantly smaller than other datasets. COCO-TS is an image dataset designed for text segmentation tasks. It is an extension of the COCO dataset and contains 14,690 images and 139,034 words. MLT-S is obtained from the MLT dataset through a weakly supervised learning method, and the annotation quality is relatively low. It contains 6,896 images and 30,691 words. Total-Text is a dataset that emphasizes text diversity, including horizontal, multi-directional, and curved text. It consists of 1,555 images and contains a total of 9,330 words. TextSeg contains 4,024 images, and the texts in these images are mainly divided into two categories: scene text and design text. Scene text usually refers to text that appears in natural environments, such as road signs, billboards, menus, etc., while design text refers to text that appears in design documents, diagrams, or product packaging. BTS is a dataset that focuses on Chinese scenes. It covers a variety of common Chinese scenes, including 14,287 images with rich content and fine annotations.

[0058] (6) Experimental details

[0059] The proposed model is trained on an 80GB Tesla A100 GPU using PyTorch. Swin-B pre-trained on ImageNet-22K is used as the low-frequency feature extraction branch. Other modules are initialized using normalized Gaussian distribution. Data augmentation operations such as random flipping, rotation, and color jittering are performed to avoid model overfitting. The Adam optimizer is used to train the model, with the initial learning rate set to 0.001, and the poly method is used to adjust the learning rate, with the default coefficient of 0.9 and the weight decay set to 0.001. During the training and testing phases, the size of the input image is adjusted to 768×768. The batch size is set to 8, and a total of 50 rounds of training are performed.

[0060] (7) Performance evaluation

[0061] The present invention uses fgIoU and F-score to evaluate the proposed method and compare it with 6 advanced methods. Among them, fgIoU is an indicator used to evaluate the foreground area segmentation effect. It represents the ratio of the intersection and union between the predicted foreground area and the true foreground area. Its calculation formula is,

[0062]

[0063] Where A represents the predicted foreground area, and B represents the real labeled foreground area.

[0064] F-score is a comprehensive evaluation indicator used for the overall performance of text segmentation methods. It combines the two indicators of precision and recall and obtains a comprehensive evaluation result by calculating their harmonic mean. Its calculation formula is:

[0065]

[0066] TP represents the number of samples that are actually positive and correctly predicted as positive, FP represents the number of samples that are actually negative but incorrectly predicted as positive. FN represents the number of samples that are actually positive but incorrectly predicted as negative. The higher the fgIoU and F-score, the higher the segmentation accuracy of the corresponding method.

[0067] Table 1 Quantitative comparison with other text segmentation algorithms on text datasets

[0068]

Claims

1. A frequency separation-based image text segmentation method, whose overall network framework includes a dual-branch feature encoding module, a frequency feature enhancement module, a local detail enhancement module and a decoder; the specific steps are as follows: Step 1: The dual-branch feature encoding module is mainly composed of a local feature encoding branch based on CNN and a global feature encoding branch based on Transformer, which are used to preliminarily extract the local features F1 and global features F of the target image respectively. i , i∈[2,5]; the local feature encoding branch is composed of two layers of convolution with batch normalization and ReLU activation function, and the global feature encoding branch is completed by Swin Transformer; Step 2: The global feature F of the target image obtained by the dual-branch feature encoding module is enhanced by the frequency feature enhancement module i , i∈[2,5] for targeted enhancement; The frequency characteristic enhancement module is mainly composed of a high-frequency component enhancement module and a low-frequency component enhancement module; The frequency feature enhancement module first uses discrete wavelet transform to transform the global feature F i , i∈[2,4] is decomposed into high-frequency and low-frequency parts, and then the high-frequency component enhancement module and the low-frequency component enhancement module are used to enhance them respectively. Finally, the enhanced high-frequency and low-frequency components are recombined through matrix addition to obtain the enhanced features. The details are as follows: First, the features are decomposed using discrete wavelet transform. Among them, F i represents the global features extracted by the dual-branch feature encoding module, D i represents the features recovered layer by layer in the decoder through convolution, batch normalization, ReLu activation function and two-fold upsampling, and Represents the high-frequency component in the global features of the image, and represents the low-frequency component in the global features of the image; ω(·) represents a learnable wavelet transform, whose coefficients are initialized by Haar wavelet; Then the obtained low-frequency component and Input to the low-frequency component enhancement module, first calculate the spatial attention operation separately and The spatial attention weights M1 and M2 are: Among them, Θ(·) represents the spatial attention operation, and the spatial attention weights M1 and M2 reflect the low-frequency components and The importance of each position in The spatial attention weights M1 and M2 are then concatenated along the channel dimension and normalized using softmax: Among them, [·] represents the concatenation operation along the channel dimension, σ(·) represents the softmax function, and Represents the transformed spatial attention weight; Finally, use element-by-element multiplication to apply the transformed attention weights to the low-frequency components. and To enhance the low-frequency components: in, and Represents the enhanced low-frequency component; in addition, the enhanced low-frequency component will be fused to guide the learning of high-frequency features in the local detail enhancement module, and the fusion is completed through convolution: Among them, G i represents the fused low-frequency features; Φ(·) is a convolutional layer containing batch normalization and ReLU activation function; For high frequency components and First, the cross-attention mechanism is used to guide it to focus on the edge and texture information of the target image. The cross-attention mechanism uses the local features extracted by the local feature encoding branch to enhance the high-frequency components in the global feature encoding branch: in, and represents the high-frequency component after cross-attention enhancement; MHCA(·) represents the cross-attention operation; then, and The further enhanced high-frequency components will be obtained through the same operation as in the low-frequency component enhancement module; Step 3: The local detail enhancement module is mainly composed of three parallel cross-attention modules and convolutional layers. The convolutional layer consists of convolution, batch normalization and ReLU activation function. In the cross-attention module, the low-frequency component G output by the low-frequency component enhancement module is used. i As the key and value, the local feature F1 extracted by the local feature encoding branch is used as the query; then the local features enhanced by convolutional fusion are used to further optimize the network's learning of text details: f i =MHCA(F1,G i ,G i ),i∈[1,3] Fd=Φ([f1,f2,f3]) Among them, f i Represents the local features enhanced by a single cross-attention branch, through the low-frequency component G i The guidance of makes the local features focus on the text area to enhance the task-related local details, F d It is the local features fused through the convolutional layer that ensure that the model can more accurately capture the local details of the text in the image, thereby achieving better text segmentation results.

Images