Text detection method in scene images based on multi-scale feature aggregation

Abstract

The present invention discloses a method for detecting text in scene images based on multi-scale feature aggregation. The method comprises the following steps: extracting text image feature representations at different scales and simultaneously obtaining a text embedding representation; aggregating the multi-scale text image feature representations and inputting them into a stacked Transformer-based encoder to obtain an enhanced multi-scale image feature representation; updating the text embedding representation using the enhanced multi-scale image feature representation; initializing a set of query vectors, restoring the enhanced multi-scale image feature representations to two-dimensional image features, inputting the query vectors, the two-dimensional image features, and the updated text embedding representation into a decoder to obtain an updated query vector; then calculating the updated query vector and the text embedding representation to obtain a text mask; and finally performing post-processing to obtain a text detection result. The method of the present invention has high accuracy for scene text detection, simple post-processing, and excellent detection performance.

Description

Technical Field

[0001] The present invention relates to the technical field of text detection in the process of scene text content understanding, and in particular to a scene image text detection method based on multi-scale feature aggregation. Background Art

[0002] Scene image text detection involves detecting and locating text in scene text images, determining the specific location of the text within the image. Images of natural scenes refer to a variety of everyday scenes, such as product packaging, store signs, menus, vehicles, screens, and posters. Scene text detection is the first step in text information extraction and has broad applications and research prospects in the field of text content understanding.

[0003] Inspired by common object detection methods, existing scene text detectors directly localize text instances by modifying the region proposal and bounding box regression modules of common object detectors. For example, the TextBoxes text detection method uses a one-stage common object detection method (SSD) to replace the default bounding box with a quadrilateral with varying aspect ratios to accommodate the text. The RRPN model incorporates rotation into the classic region proposal network, representing a text region as a rotated bounding box with a 5-tuple (x, y, h, w, θ), where (x, y) represents the geometric center of the bounding box, h represents the short side, w represents the long side, and θ represents the direction of the long side. Compared to common objects, text lines have a wide range of lengths and aspect ratios. This model performs poorly when text images contain curved, oriented, or long text. Furthermore, these methods use single-scale images as input, which, due to the limited receptive field size, cannot effectively detect the location of small characters.

[0004] Segmentation-based methods are currently very popular in the field of scene text detection. These methods first perform pixel-level classification, determining whether each pixel belongs to a text object, and then combine the adjacent pixel results into a text box. This method is well adapted to curved text. However, segmentation-based methods require complex post-processing to combine pixel-level results into text lines, which often results in high prediction overhead. Furthermore, these methods require additional design for operations between adjacent text, otherwise text adhesion is likely to occur.

[0005] At present, some text detection methods have adopted the feature aggregation strategy to improve the performance of text detection models. One strategy is to scale the text image to different scales, then use independent detection models to detect the text, and merge the detection results of each scale to obtain the final text detection result. Another strategy is Figure 1As shown in Figure 1, the feature extraction model extracts multi-scale features from the scene text image and then operates on these multi-scale features to produce the final text detection result. The text detection method mentioned in Strategy 1 requires independent detection models to obtain detection results at each scale, and then aggregates these results. This results in large model parameters and redundancy. Strategy 2 uses image features from text images of the same scale, resulting in relatively simple features and a tendency to miss parts of text in the detection results. Summary of the Invention

[0006] The present invention proposes a scene text detection method based on multi-scale feature aggregation, which mainly alleviates the problems of the current scene text detection method that it cannot handle curved text well, easily misses small characters, has cumbersome post-processing operations and text adhesion.

[0007] The technical solution of the present invention is specifically described as follows.

[0008] The present invention provides a scene text detection method based on multi-scale feature aggregation. The method performs detection based on a scene text detection model. The scene text detection model includes a feature extraction module, a stacked Transformer-based encoder, and a Transformer-based decoder. The specific steps are as follows:

[0009] (1) Scaling the original scene text image to different scales;

[0010] (2) Using the feature extraction module to extract feature representations from images of different scales, and at the same time obtain text embedding representations from the largest scale image;

[0011] (3) The image feature representations at different scales are fed into a stacked Transformer-based encoder together with the positional encoding, and the output is an enhanced multi-scale image feature representation;

[0012] (4) Perform variable attention calculation on the text embedding representation and the enhanced multi-scale image feature representation to obtain a text embedding representation with richer features;

[0013] (5) Initialize a set of query vectors and restore the enhanced multi-scale image feature representation to two-dimensional image features. Then, input the query vector, two-dimensional image features, and updated text embedding representation into the Transformer-based decoder, and output the updated query vector with text feature representation.

[0014] (6) The text mask is calculated based on the updated query vector and the updated text embedding representation, and then the detection result is obtained through post-processing operations.

[0015] In the present invention, in step (2), the feature extraction module is implemented by a ViT module, multiple convolutional layers or multiple residual convolution blocks.

[0016] In the present invention, in step (3), the encoder unit of each Transformer includes a multi-scale deformable attention module and a forward calculation module.

[0017] In the present invention, in step (6), the text mask calculation formula is as follows:

[0018]

[0019] Among them, M t (i, j) represents the value of the text mask at position (i, j) obtained by the t-th query vector, sigmoid refers to the sigmoid activation function, and MLP is used to enhance the query vector Q t The nonlinear ability of E(i,j) represents the value at position (i,j) in the updated text embedding representation.

[0020] In the present invention, in step (6), the post-processing operation is specifically: using a non-maximum suppression algorithm to filter out repeated text masks and masks with too small an area.

[0021] In the present invention, when the scene text detection model is trained, the loss function L is composed of the mask loss function L mask And the classification loss function L cls It consists of two parts; among them, L mask is the mask binarization loss of the text instance, which consists of bce loss and dice loss, and L cls is the loss of text and non-text classification prediction; the loss function is expressed as: L = L mask +λ cls L cls , where λ is used when the category is predicted to be text cls =0.4, when the prediction is non-text cls =0.02;

[0022] The individual losses are calculated as follows:

[0023]

[0024]

[0025] Where N and K are the number of query vectors and the sampling locations, and Represent the predicted value and the true value respectively;

[0026]

[0027] where l t and p t Represent the query vector Q t The true value of the category and the probability value of the category prediction.

[0028] Compared with the existing technology, the beneficial effects of the present invention are: multi-scale scene images can provide rich feature representations, which can improve the detection performance of small characters compared with the traditional single-scale image text detection method; a text embedding representation update strategy is designed to better represent image text features and further improve the performance of the detection model; the query vector and the updated text embedding representation are calculated to obtain a series of text masks, which can alleviate the problem of text adhesion; compared with the traditional text detection method that requires additional computational processing of the detection results, the present invention only needs to filter out repeated text masks and masks with too small areas, and the post-processing is simple. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 : Feature aggregation strategy 2.

[0030] Figure 2 : Result graph of text detection in scene image.

[0031] Figure 3 : Scene text detection method based on multi-scale feature aggregation.

[0032] Figure 4 : Flowchart for strengthening multi-scale image feature representation and updating text embedding representation.

[0033] Figure 5 : Decoding process diagram.

[0034] Figure 6 : The process of obtaining the final result. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0036] In the present invention, scene text detection is performed based on a scene text detection model, which includes a feature extraction module, an encoder based on a Transformer structure, and a decoder.

[0037] In a specific embodiment, the scene text detection model is first trained according to the following steps:

[0038] 1) The input text image is scaled to different scales, with scaling factors of {1 / 2, 1, 2}, and the scaled images are represented as {I1, I2, I3}

[0039] 2) The feature extraction module extracts feature representations of text images of different scales. The specific operation is: input image I∈R H×W×3 Perform block operation and represent the image block as in p is the size of the image patch, and then the feature extraction module is used to map the image patch into a feature embedding representation X p For images of different scales {I1,I2,I3}, we can get the feature embedding representation {X p1 , X p2 , X p3}.

[0040] 3) Represent the multi-scale features {X p1 , X p2 , X p3} together with the position code E pos (i.e., aggregation operation), the input is stacked in the Transformer-based encoder, each encoder unit contains a multi-scale deformable attention module and a forward calculation module, and the output is an enhanced multi-scale feature representation {X' p1 , X' p2 , X' p3}.

[0041] 4) Enhanced multi-scale feature representation {X' p1 , X' p2 , X' p3}Perform variable attention calculation with the text embedding representation and update the enhanced text embedding representation E.

[0042] 5) Given a set of query vectors Q. For the t-th query vector, use the enhanced text embedding representation E and the enhanced feature embedding representation {X' p1 , X' p2 , X' p3}, calculate the text mask M t , the corresponding calculation formula is as follows:

[0043]

[0044] Among them, sigmoid refers to the sigmoid activation function, and MLP is used to enhance the feature Q t Nonlinear capabilities;

[0045] Finally, the mask is post-processed to obtain the final text instance.

[0046] The loss function L of the model is composed of the mask loss function L mask And the classification loss function L cls It consists of two parts. maskis the mask binarization loss of the text instance, which consists of bce loss and dice loss, and L cls is the loss of text and non-text classification prediction. The loss function is expressed as: L = L mask +λ cls L cls , where λ is used when the category is predicted to be text cls =0.4, when the prediction is non-text cls =0.02. The individual losses are calculated as follows:

[0047]

[0048]

[0049] Where N and K are the number of query vectors and the sampling locations, and represent the predicted value and the true value respectively.

[0050]

[0051] where l t and p t Represent the query vector Q t The probability of the true value and the predicted value.

[0052] The visualization effect of scene text detection results is shown in the figure Figure 2 shown.

[0053] The main process of further detecting scene text based on the trained scene text detection model is as follows: Figure 3 The detailed steps are as follows:

[0054] 1) Scaling the original text image to different scales to obtain text images of different scales, with the scaling factors being {1 / 2, 1, 2} respectively;

[0055] 2) Input the scaled text image into the feature extraction model to obtain image feature representations at different scales and also generate text embedding representations;

[0056] 3) The multi-scale text image feature representation is input together with the position encoding into the stacked Transformer-based encoder to obtain an enhanced multi-scale feature representation; Figure 4As shown in the figure, the multi-scale text image features are aggregated and input into the Transformer-based encoder, which mainly performs variability attention calculation on the image features, adds and normalizes the calculation output and the input multi-scale image features, and then inputs the addition and normalization result into the forward calculation module. Finally, the input and output of the forward calculation module are added and normalized to obtain the enhanced image feature representation;

[0057] 4) Using enhanced multi-scale feature representation to update the text embedding representation, making the features of the text embedding representation richer; Figure 4 As shown in the figure, in the text embedding representation update part, the input text embedding representation is flattened and position encoding is added, and it is input into the multi-scale variable convolution module with the enhanced image feature representation. After addition, normalization, convolution and addition calculation, the updated text embedding representation is obtained;

[0058] 5) Reshape the enhanced multi-scale feature representation to obtain a two-dimensional feature map; at the same time, initialize a set of query vectors and input them into the Transformer-based decoder together with the updated text embedding representation. After multiple iterations, the query vector with text representation is obtained. The decoding process is as follows: Figure 5 As shown in the figure, the 2D feature map at scale 1 is flattened and fed into the Transformer-based decoder. Simultaneously, the query vector undergoes an MLP operation, multiplied by the updated text embedding, and thresholded to produce a text mask. Similarly, the 2D image features at scales 2 and 3 are processed accordingly, and the query vector is iteratively updated.

[0059] 6) The updated query vector and the text embedding representation are calculated to obtain the text mask, and the text mask is post-processed to obtain the final text instance result. The process of obtaining the result is as follows: Figure 6 As shown in the figure, the updated query vector is subjected to an MLP operation, multiplied with the updated text embedding representation, and thresholded to obtain a text mask. Finally, the text mask is post-processed to filter out duplicate text masks and masks with too small an area to obtain the final detection result.

[0060] The present invention evaluates scene text detection performance on public datasets (Total-Text, CTW1500, and MSRA-TD500). The proposed method achieves superior text detection performance to existing public text detection methods. Specifically, on the Total-Text dataset, the proposed method achieves 91.9% accuracy, 88.3% recall, and 90.1% F1 index; on the CTW1500 dataset, the proposed method achieves 89.7% accuracy, 87.9% recall, and 88.8% F1 index; and on the MSRA-TD500 dataset, the proposed method achieves 91.3% accuracy, 90.5% recall, and 90.9% F1 index.

Claims

1. A scene text detection method based on multi-scale feature aggregation, characterized in that: It performs detection based on a scene text detection model, which includes a feature extraction module, a stacked Transformer-based encoder, and a Transformer-based decoder. The specific steps are as follows: (1) Scaling the original scene text image to different scales; (2) Use the feature extraction module to extract feature representations from images of different scales, and obtain text embedding representations from the largest scale image; (3) The image feature representations of different scales are input together with the position encoding into the stacked Transformer-based encoder, and the output is the enhanced multi-scale image feature representation; (4) Perform deformable attention calculation on the text embedding representation and the enhanced multi-scale image feature representation to obtain a text embedding representation with richer features; (5) Initialize a set of query vectors and restore the enhanced multi-scale image feature representation to two-dimensional image features. Then, input the query vector, two-dimensional image features, and updated text embedding representation into the Transformer-based decoder, and output the updated query vector with text feature representation. (6) The text mask is calculated based on the last updated query vector and the updated text embedding representation, and then the detection result is obtained through post-processing operations; where: In step (3), each Transformer encoder unit contains a multi-scale deformable attention module and a forward calculation module; the image feature representations of different scales are input into the Transformer-based encoder together with the position code, and the multi-scale deformable attention module performs deformable attention calculation on the image features, adds and normalizes the calculation output and the input multi-scale image features, and then inputs the addition and normalization results into the forward calculation module, and finally adds and normalizes the input and output of the forward calculation module to obtain an enhanced multi-scale image feature representation; In step (6), the post-processing operation is specifically as follows: using the non-maximum suppression algorithm to filter out repeated text masks and masks with too small areas; When training the scene text detection model, the loss function L By mask loss function L mask And the classification loss function L cls It consists of two parts; L mask is the mask binarization loss of the text instance, which consists of bce loss and dice loss, and L cls is the loss of text and non-text classification prediction; the loss function is expressed as: , where, when the category prediction is text, λ cls =0.4, when the prediction is non-text, λ cls =0.02; The individual losses are calculated as follows: ， ， in, N and K is the number of query vectors and the sampling locations, and Represent the predicted value and the true value respectively; ， in l t and p t Represent the query vector Q t The true value of the category and the probability value of the category prediction.

2. The scene text detection method according to claim 1, characterized in that: In step (2), the feature extraction module is implemented by the ViT module, multiple convolutional layers or multiple residual convolution blocks.

3. The scene text detection method according to claim 1, wherein: In step (6), after the MLP operation is performed on the query vector obtained by the last update, it is multiplied with the updated text embedding representation and thresholded to obtain the text mask; the text mask calculation formula is as follows: ， Among them, M t (i, j) represents the value of the text mask at position (i, j) obtained by the t-th query vector, sigmoid refer to sigmoid activation function, MLP It is used to enhance the query vector Q t The nonlinear ability of Represents the value at position (i, j) in the updated word embedding representation.

Images