Arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement

Abstract

This invention discloses an arbitrary-shape text detection method based on iterative optimization and cross-knowledge enhancement, belonging to the field of natural scene text detection technology. Compared with previous text information detection methods, this invention improves upon the problems of existing boundary iteration methods, which have complex processing flows, only apply to features at a single scale, have high computational costs, and cannot meet the requirements of practical applications. It obtains representational prior information through semantic segmentation and generates initial boundary points. Furthermore, it fuses representational prior semantic information with cross-scale feature information through feature enhancement, giving the features rich semantic information. Finally, it uses a unified boundary iteration Transformer that can fully utilize multi-scale information to predict boundary offset values, thereby continuously optimizing the initial boundary points. The simple and efficient network process of this method can greatly reduce network complexity and efficiently utilize cross-scale information and representational semantic information, thus meeting the speed and performance requirements of real-time detection.

Description

Technical Field

[0001] The present invention relates to the technical field of natural scene text detection, and in particular to an arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement. Background Art

[0002] With the rapid development of information technology, various types of image and video equipment have been widely used in human social life, and the massive growth of image data has become the main source of information in today's society. While these devices generate a large number of images, they also contain important text information, and their specific semantic knowledge is crucial to the understanding of scene content. However, due to the comprehensive factors such as lighting, font size, shape, layout, language, and complex background in open scenes, the relatively mature optical character recognition (OCR) can no longer adapt to the text detection task in complex scenes. Therefore, how to break through the limitations of shooting randomness on text detection tasks in complex scenes and improve the accuracy and efficiency of text detection has become a scientific frontier issue for achieving scene semantic understanding and achieving human visual cognition. In recent years, with the development of text detection methods, they can be roughly divided into two categories: bottom-up (i.e., first detecting pixels or fragments, and then aggregating these components through various post-processing algorithms to produce the final detection results) and top-down (i.e., predicting each text target as a whole), but these methods cannot achieve satisfactory performance requirements. The existing document "Progressive contour regression for arbitrary-shape scene text detection. [C]. IEEE Conference on Computer vision and Pattern Recognition, 2021: 7393-7402." discloses a boundary iteration scheme. This scheme first uses the backbone network to extract features and enhances them to predict the horizontal boundary of the initial text boundary, and then adjusts the boundary to a rectangular boundary in any direction through the corner point prediction module, and finally predicts boundary points of arbitrary shapes through the boundary point generation module. Although this method achieves more efficient generation of arbitrary shape boundaries, it is not efficient in terms of network processing flow and feature utilization due to the complex processing flow and the application of features only at a single scale.

[0003] In order to solve the above problems, the present invention proposes an arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement to solve the complex processing flow and utilize a simple and efficient network structure to improve the detection performance and reduce the amount of calculation. Summary of the invention

[0004] The purpose of the present invention is to propose an arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement to solve the problems raised in the background technology:

[0005] Existing text detection methods have complex processing procedures, are only applied to features at a single scale, and have a large amount of calculation, which cannot meet the requirements of practical applications.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement includes the following steps:

[0008] S1: Multi-scale feature extraction: Input the image into the backbone network ResNet50 to obtain different scale features C i , through FPN to C i Fusion to get UP j ;

[0009] S2: Initial boundary generation: semantic segmentation of multi-scale features is performed through three layers of dilated convolution to obtain a priori semantic feature map, and the closed area in the text confidence map in the priori semantic feature map that is greater than the threshold 0.85 is used to generate a boundary candidate box and sample several boundary points as core points Pc;

[0010] S3: Feature enhancement: downsample the prior semantic feature maps in S2 and combine them with the UP in S1 j The enhanced feature F is obtained by information fusion of features k ;

[0011] S4: Boundary iterative optimization: Use Pc to obtain point features Pf on F1 as the initial point features. Each Pf q-1 Both get the next layer’s feature Pf through the boundary Transformer q To optimize the lower boundary point coordinates Pc q .

[0012] Preferably, the FPN in S1 is C i Fusion to get UP j The fusion process is as follows:

[0013]

[0014] Among them, UP j UP j-1 represents the fusion features of the j-th and j-1-th layers of images; concat() is the connection function; Represents the j-th layer feature C j Perform linear interpolation, where j≤i; UpConv jIndicates that the jth DeConv x2 and an upsampling module with two layers of 2-D convolution; DeConv x2 Represents a deconvolution module with two layers of 2-D deconvolution.

[0015] Preferably, the prior semantic feature map in S2 includes a text confidence map, a distance map and a two-dimensional direction map.

[0016] Preferably, the information fusion process in S3 is specifically as follows:

[0017]

[0018] Among them, the function Linear interpolation is used to combine the prior semantic feature map with the UP k Size alignment.

[0019] Preferably, in S4, Pf q-1 Get the feature Pf of the next layer through the boundary Transformer q The process of optimizing the boundary point coordinates is as follows:

[0020] Pf q-1 The feature Pf of the next layer is obtained through self-attention and multi-scale cross attention, i.e. SCA q :

[0021]

[0022] in, represents the addition operation; q represents the number of layers of the current boundary Transformer; F represents all the enhanced features F1, F2, F3 in S3; Pc q-1 Indicates the core coordinates of the boundary points of the previous layer;

[0023] The final Pf q The offset value Po is obtained by using an MLP with three fully connected layers and 1×1 one-dimensional convolution with ReLU activation function. q Prediction:

[0024] Po q =MLP(Pf q )

[0025] Then compare it with the boundary core point coordinates Pc of the previous layer q-1 Add to get the next layer of boundary core point coordinates Pc q :

[0026] Pc q =Pc q-1 +θ(Po q-1 )

[0027] Among them, the function θ(Po q-1 ) means to change the boundary core point coordinates Po of the previous layer q-1 The value is normalized within 16 pixels.

[0028] Preferably, each of the self-attention and multi-scale cross-attention has the following unified structure: a head-to-head self-attention, i.e., MHSA, a cross-knowledge enhancement, i.e., ACKA, and two forward feedback networks, i.e., FFNs, as follows:

[0029]

[0030] in, represents an addition operation; FFN1 and FFN2 represent the first and second independent FFNs, respectively, which are equipped with several fully connected layers with residual connections;

[0031] The ACKA is specifically as follows:

[0032]

[0033] Where t represents the number of queries; l represents the number of layers of the feature map; n represents the nth attention point; represents the learning weight, where C is the feature dimension of the corresponding F, C v is C / T; and Δk tln Respectively represent the weight and offset value of the t-th search of the n-th point on the l-th layer feature map; F l represents the enhanced features of the lth layer in S3; σ l (Pc q-1 ) represents the normalized Pc q-1 The coordinates correspond to the position on the feature map of layer l.

[0034] Compared with the prior art, the present invention provides an arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement, which has the following beneficial effects:

[0035] The present invention proposes an efficient natural scene text detection method. In view of the complex processing flow and inefficient feature utilization of the existing boundary iteration optimization method, a cross-knowledge enhancement method based on boundary iteration optimization is proposed. It obtains representation prior information through semantic segmentation and generates initial boundary points, and fuses the representation prior semantic information with cross-scale feature information through feature enhancement so that its features have rich semantic information. It predicts the boundary offset value through a unified boundary iteration Transformer that can fully utilize multi-scale information, thereby continuously optimizing the initial boundary points. The simple and efficient network process of this method can greatly reduce the complexity of the network and efficiently utilize cross-scale information and representation semantic information so that it can take into account the speed and performance requirements of real-time detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the method structure mentioned in Example 1 of the present invention;

[0037] Figure 2 This is a schematic diagram of the feature extraction and feature enhancement structure mentioned in Example 1 of the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of the boundary Transformer module based on cross-knowledge enhancement mentioned in Example 1 of the present invention;

[0039] Figure 4 It is a schematic diagram of the visualization effect of the method of the present invention mentioned in Example 3 of the present invention and the existing advanced method. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0041] The present invention proposes an efficient natural scene text detection method. In view of the complex processing flow and inefficient feature utilization of the existing boundary iterative optimization method, a cross-knowledge enhancement method based on boundary iterative optimization is proposed. It obtains representation prior information through semantic segmentation and generates initial boundary points, and fuses the representation prior semantic information with cross-scale feature information through feature enhancement so that its features have rich semantic information. It predicts the boundary offset value through a unified boundary iterative Transformer that can fully utilize multi-scale information, thereby continuously optimizing the initial boundary points. The simple and efficient network process of this method can greatly reduce the complexity of the network and efficiently utilize cross-scale information and representation semantic information so that it can take into account the speed and performance requirements of real-time detection. Specifically include the following contents.

[0042] Embodiment 1:

[0043] See also Figure 1-3 The present invention provides an arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement, comprising the following steps:

[0044] S1: Multi-scale feature extraction: Input the image into the backbone network ResNet50 to obtain different scale features C i , through FPN to C i Fusion to get UP j ; The details are as follows:

[0045] Stage 1: Multi-scale feature extraction

[0046] At this stage, if Figure 2 As shown, the picture will be input into the backbone network ResNet50 to obtain different scale features C1, C2, C3, C4, C5, whose scales are 1 / 4, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 times of the original image respectively. Then, C1, C2, C3, C4, and C5 are fused through FPN to obtain UP1, UP2, and UP3, whose scales are 1 / 1, 1 / 2, and 1 / 4 of the original image respectively. The fusion process is as follows:

[0047]

[0048] Among them, UP j UP j-1 represents the fusion features of the j-th and j-1-th layers of images; concat() is the connection function; Represents the j-th layer feature C j Perform linear interpolation; UpConv j Indicates that the jth DeConv x2 and an upsampling module with two layers of 2-D convolution; DeConv x2 Represents a deconvolution module with two layers of 2-D deconvolution.

[0049] S2: Initial boundary generation: semantic segmentation of multi-scale features is performed through three layers of dilated convolution to obtain a priori semantic feature map, and the closed area in the text confidence map in the priori semantic feature map that is greater than the threshold 0.85 is used to generate a boundary candidate box and sample several boundary points as core points Pc; the details are as follows:

[0050] Phase 2: Initial boundary generation

[0051] In this stage, the features of stage 1 are converted into a priori semantic feature maps of the same size (text confidence map, distance map and two-dimensional direction map) through three layers of dilated convolution, and the closed area with text confidence map greater than the threshold of 0.85 is used to generate boundary candidate boxes and 25 boundary points are sampled as core points Pc.

[0052] S3: Feature enhancement: downsample the prior semantic feature maps in S2 and combine them with the UP in S1 j The enhanced feature F is obtained by information fusion of features k ; The details are as follows:

[0053] Stage 3: Feature Enhancement

[0054] Since the prior information (text confidence map, direction field map and distance map) used to calculate the initial boundary points contains the implicit relationship between text instances, the present invention fuses the prior information with multi-scale image features. Figure 2 As shown, in this stage, the 4-dimensional prior feature map of stage 2 is downsampled and information fused with the UP1, UP2, and UP3 features of stage 1 to obtain enhanced features F1, F2, and F3. The information fusion process is shown in the formula:

[0055]

[0056] Among them, the function Linear interpolation is used to combine the prior semantic feature map with the UP k Size alignment.

[0057] S4: Boundary iterative optimization: Use Pc to obtain point features Pf on F1 as the initial point features. Each Pf q-1 Both get the next layer’s feature Pf through the boundary Transformer q To optimize the lower boundary point coordinates Pc q The details are as follows:

[0058] Phase 4: Boundary Iteration Optimization

[0059] In order to make each boundary point fully utilize the feature information of different scales to guide more accurate boundary point coordinate optimization, the present invention also designs a cross-knowledge enhanced boundary iteration module to predict the offset value of the boundary point multiple times to gradually optimize the boundary point. Figure 3 As shown, in this stage, first, use Pc to obtain point features on F1 as initial point features, and each Pf q-1 The boundary Transformer will obtain the features of the lower layer to optimize the boundary point coordinates. The features of the next layer Pf will be obtained through self-attention and multi-scale cross attention (SCA) q , the formula is as follows:

[0060]

[0061] in, represents the addition operation; q represents the number of layers of the current boundary Transformer; F represents all the enhanced features F1, F2, F3 in S3; Pc q-1 Indicates the core coordinates of the boundary points of the previous layer;

[0062] Each self-attention and multi-scale cross-attention has a unified structure, including a multi-head self-attention (MHSA), cross-knowledge enhancement (ACKA), and two feed-forward networks (FFN), the formula of which is as follows.

[0063]

[0064] in, represents an addition operation; FFN1 and FFN2 represent the first and second independent FFNs, respectively, which are equipped with several fully connected layers with residual connections;

[0065] The specific details of ACKA are as follows:

[0066]

[0067] Where t represents the number of queries; l represents the number of layers of the feature map; n represents the nth attention point; represents the learning weight, where C is the feature dimension of the corresponding F, C v is C / T; and Δk tln Respectively represent the weight and offset value of the t-th search of the n-th point on the l-th layer feature map; F l represents the enhanced features of the lth layer in S3; σ l (Pc q-1 ) represents the normalized Pc q-1 The coordinates correspond to the position on the feature map of layer l.

[0068] The final Pf q The offset value Po is obtained by using an MLP with three fully connected layers and 1×1 one-dimensional convolution with ReLU activation function. i The prediction is then compared with the boundary core point coordinates Pc of the previous layer. q-1 Add to get the next layer of boundary core point coordinates Pc q , the specific formula is as follows:

[0069] Po q =MLP(Pf q )

[0070] Pc q =Pc q-1 +θ(Po q-1 )

[0071] Among them, the function θ(Po q-1 ) means to Po q-1 The value is normalized within 16 pixels.

[0072] Embodiment 2:

[0073] As shown in Table 1, the detection effects of the proposed method and the existing advanced methods are evaluated from the perspective of quantitative index evaluation. The higher the values ​​of the four indicators, namely, precision, recall rate, F1 score and frames per second, the better the performance. It can be seen that in terms of quantitative evaluation, the detection effect of the proposed method is superior to that of the existing advanced methods.

[0074] Table 1 Detection effect of the method of the present invention and existing advanced methods

[0075] method Accuracy Recall F1 score frames per second DRRG 86.5 84.9 85.7 - TextPMs 89.9 87.6 88.7 6.9 Proposed method 91.9 89.1 90.5 13.0

[0076] Embodiment 3:

[0077] like Figure 4 As shown in FIG. 1 , the effects of the proposed method and the existing advanced methods are evaluated from the perspective of detection image visualization. It can be seen that the proposed method can accurately detect the boundary position of text of any shape in natural scene images under different complex environments.

[0078] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. Arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement, characterized in that: The steps include: S1: Multi-scale feature extraction: Input the image into the backbone network ResNet50 to obtain features of different scales , through FPN Fusion ; S2: Initial boundary generation: semantic segmentation of multi-scale features is performed through three layers of dilated convolution to obtain a priori semantic feature map, and the closed area in the text confidence map in the priori semantic feature map that is greater than the threshold of 0.85 is used to generate boundary candidate boxes and sample several boundary points as core points ; S3 : Feature enhancement: The prior semantic feature maps in S2 are downsampled and combined with those in S1 The enhanced features are obtained by information fusion ; S4: Boundary Iteration Optimization: Utilizing exist Get point features on As the initial point feature, each Both get the features of the next layer through the boundary Transformer To optimize the coordinates of the lower boundary points ; The FPN in S1 is Fusion The fusion process is as follows: in, , Indicates , Layer image fusion features; is the connection function; Indicates Layer Features Perform linear interpolation, where ; Indicates Has and two layers Convolutional upsampling module; It means it has two layers Deconvolution module for deconvolution; The information fusion process in S3 is as follows: Among them, the function Use linear interpolation to combine the prior semantic feature map with Size alignment; In S4, Get the features of the next layer through the boundary Transformer The process of optimizing the boundary point coordinates is as follows: Obtain the features of the next layer through self-attention and multi-scale cross attention, i.e. SCA : in, Represents addition operation; Indicates the number of layers of the current boundary Transformer; Represents all enhanced features in S3 , , ; Indicates the core coordinates of the boundary points of the previous layer; Final The offset value is obtained through an MLP with three fully connected layers and 1×1 one-dimensional convolution with ReLU activation function. Prediction: Then compare it with the boundary core point coordinates of the previous layer Add to get the coordinates of the next layer's boundary core points : Among them, the function Indicates that the boundary core point coordinates of the previous layer The value is normalized within 16 pixels.

2. The arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement according to claim 1 is characterized in that: The prior semantic feature map described in S2 includes a text confidence map, a distance map, and a two-dimensional direction map.

3. The arbitrary shape text detection method based on iterative optimization and cross-knowledge enhancement according to claim 1 is characterized in that: Each of the self-attention and multi-scale cross-attention has the following unified structure: a head-to-head self-attention, i.e., MHSA, cross-knowledge enhancement, i.e., ACKA, and two forward feedback networks, i.e., FFN, as follows: in, Represents addition operation; , Respectively represent the first and second independent FFNs, which are equipped with several fully connected layers with residual connections; The ACKA is specifically as follows: in, Indicates the number of queries; Indicates the number of layers of the feature map; Indicates Attention points; represents the learning weight, where For the corresponding The characteristic dimension of for ; and Respectively represent The point is The first layer feature map The weight and bias values ​​for the search; Indicates the first Layer enhanced features; Represents the normalized The coordinates are The corresponding position on the layer feature map.

Images