A method and system for detecting text in a focused feature and classification map

Abstract

The application discloses a kind of focused feature and classification chart text detection method and detection system, through the convolution feature fusion module in detection system, constructs the text attention mixing mechanism of focused feature and classification chart, text attention mixing mechanism is used to solve false positive problem, joint classification chart input feature and the angle of classification chart, proposed the text attention mixing mechanism of focused feature and classification chart, for simultaneously enhancing the discriminability of classification chart input feature and classification chart, and better performance is obtained;Text attention mixing mechanism can effectively overcome the false positive detection problem existing in the current SSTDTAM attention mechanism, while the performance of text attention mixing mechanism applied to text detector can significantly exceed the performance of difference loss function applied to text detector classification chart branch alone, and can significantly inhibit false positive detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of scene text detection, and particularly relates to a text detection method and a detection system focusing on features and classification graphs. BACKGROUND

[0002] The purpose of scene text detection is to accurately locate the text boundary. Based on a convolutional neural network, a method first extracts convolutional features from an input image, then performs feature fusion, and finally classifies the text and background based on the features and outputs the boundary box of the text. Because natural scene images have high diversity and complexity, some features of the background and the text are similar. In the case of similar text and background, if the classification graph of the text detector and the classification graph input feature are weakly distinguishable for the text and the background, the text detector will output false positive detection results. False positive text detection refers to the detection algorithm classifying the background as text.

[0003] To solve the false positive problem of text detection, SSTD proposes a text detection attention mechanism. The mechanism uses an auxiliary loss function to learn an attention distribution graph, which is used to effectively identify the text and background regions in the image. Then, the mechanism encodes the attention distribution graph into the classification graph input feature, which significantly suppresses the inference of the background region and highlights the features of the text region. Therefore, the mechanism enhances the distinguishability of the features for the text and the background from the feature perspective. The SSTD text attention mechanism first performs two 2 times upsampling operations on the classification graph input feature, then connects the convolutional layer to output the attention distribution graph, and uses a difference loss function to train the distribution graph. Then, the attention distribution graph is 4 times down-sampled and multiplied with the classification graph input feature for fusion. Finally, the classification graph branch and the rotated rectangle branch of EAST are connected.

[0004] When SSTDTAM is applied to the text detector EAST, its performance is lower than that of the difference loss function applied to the classification graph branch of EAST alone. This situation shows that SSTDTAM cannot improve the performance. The reason for the low performance of SSTDTAM is that it only enhances the distinguishability of the classification graph input feature for the text and the background, but does not enhance the distinguishability of the classification graph.

[0005] Therefore, in view of the false positive detection problem of the existing SSTD text attention mechanism applied to the text detector, the application discloses a text detection method and a detection system focusing on features and classification graphs. SUMMARY

[0006] The application discloses a text detection method and system of a focused feature and a classification graph, constructs a text attention hybrid mechanism of the focused feature and the classification graph, and is used for solving a false positive problem.

[0007] The application is achieved by the following technical solutions:

[0008] The text detection method of the focused feature and the classification graph comprises the following steps:

[0009] Step S1, input an original image, and extract feature images with different resolutions through a residual convolutional neural network;

[0010] Step S2, fuse the feature images through upsampling operation and series operation to obtain original classification graph input features;

[0011] Step S3, introduce a text attention hybrid mechanism, embed the text attention hybrid mechanism into a text detector, and in the process of detecting text by the text detector, perform upsampling operation on the original classification graph input features through the text attention hybrid mechanism to obtain attention distribution graph input features; perform single-channel convolution operation on the attention distribution graph input features through the text attention hybrid mechanism to obtain an attention distribution graph; perform downsampling operation and exponential power operation on the attention distribution graph input features, and then fuse the original classification graph input features to obtain hybrid classification graph input features;

[0012] Step S4, perform single-channel convolution on the hybrid classification graph input features to obtain a single-channel classification graph, perform multi-channel convolution on the classification graph input features to obtain a minimum prediction rectangular frame, the minimum prediction rectangular frame comprises a direction boundary and a rotation angle; perform downsampling on the attention distribution graph, and fuse the single-channel classification graph and the attention distribution graph downsampling result to obtain a hybrid classification graph;

[0013] Step S5, perform threshold filtering on pixels in the hybrid classification graph, retain pixels greater than a classification graph threshold in the hybrid classification graph as reference pixels, calculate reference direction boundaries and reference rotation angles of the reference pixels based on the minimum prediction rectangular frame, and calculate minimum prediction rectangular frame coordinates corresponding to the remaining pixels in the hybrid classification graph as text coordinates based on the reference direction boundaries and the reference rotation angles of the reference pixels.

[0014] In order to better achieve the application, further, the step S3 specifically comprises the following steps:

[0015] Step 3.1, linearly doubling the sampling operation is performed on the classification map input feature to obtain an attention distribution map input feature;

[0016] Step 3.2, a single-channel 1x1 convolution operation is performed on the attention distribution map feature to obtain a feature map, and a sigmoid excitation function is used to convert the feature map into an attention distribution map;

[0017] Step 3.3, a down-sampling operation is performed on the attention distribution map feature, and the classification map input feature is multiplied by the down-sampling result of the attention distribution map feature to obtain a mixed classification map input feature.

[0018] In order to better realize the present application, further, the step 3.3 specifically comprises the following steps:

[0019] Step 3.3.1, a four-fold down-sampling operation is performed on the attention distribution map feature to match the resolution of the classification map input feature;

[0020] Step 3.3.2, an exponential power operation is performed on the four-fold down-sampling operation result of step 3.3.1;

[0021] Step 3.3.3, the classification map input feature is multiplied by the exponential power operation result in step 3.3.2 to obtain a mixed classification map input feature.

[0022] In order to better realize the present application, further, the step S4 specifically comprises the following steps:

[0023] Step 4.1, a single-channel 1x1 convolution operation is performed on the mixed classification map input feature to obtain a single-channel classification map, and the original image is judged through the single-channel classification map.

[0024] Step 4.2, a 5-channel 1x1 convolution operation is performed on the mixed classification map input feature to obtain a minimum prediction rectangular frame covering the text area, and the coordinates of the text are represented through the minimum prediction rectangular frame;

[0025] Step 4.3, a four-fold down-sampling operation is performed on the attention distribution map, and the single-channel classification map is multiplied by the down-sampling result of the attention distribution map to obtain a mixed classification map.

[0026] In order to better realize the present application, further, the step 4.2 specifically comprises:

[0027] Step 4.2.1, the text detector generates a minimum prediction rectangular frame covering the text area by using a minAreaRect function;

[0028] Step 4.2.2, the distance from each pixel in the minimum prediction rectangular frame to the four boundaries of the minimum prediction rectangular frame is calculated;

[0029] Step 4.2.3, calculate the rotation angle of the minimum prediction rectangular frame relative to the horizontal axis.

[0030] In order to better realize the present application, further, the distance of the pixel to the four boundaries of the minimum prediction rectangular frame is characterized by an IOU loss function, and the IOU loss function is specifically:

[0031] ;

[0032] Wherein: IOU loss function is represented by; The prediction rectangular frame is represented by; The true value rectangular frame is represented by; The IOU value between the prediction rectangular frame and the true value rectangular frame is represented by.

[0033] In order to better realize the present application, further, the rotation angle of the minimum prediction rectangular frame relative to the horizontal axis is characterized by a rotation angle loss function, and the rotation angle loss function is specifically:

[0034] ;

[0035] Wherein: Rotation angle loss function is represented by; The predicted value of the rotation angle is represented by; The true value of the rotation angle is represented by.

[0036] In order to better realize the present application, further, the residual convolutional neural network comprises a ResNet50 network, the ResNet50 network comprises a first convolutional module, a second convolutional module, a third convolutional module and a fourth convolutional module, the resolution of the feature image extracted by the first convolutional module is one fourth of the original image resolution, the resolution of the feature image extracted by the second convolutional module is one eighth of the original image resolution, the resolution of the feature image extracted by the third convolutional module is one sixteenth of the original image resolution, and the resolution of the feature image extracted by the fourth convolutional module is one thirty-second of the original image resolution.

[0037] In order to better realize the present application, further, the pixel value of the pixel in the attention distribution map is as follows:

[0038] 0≤k≤1, wherein: k represents the pixel value of the pixel in the attention distribution map.

[0039] The text detection system of a focused feature and a classification graph is used for realizing the text detection method, and comprises a feature extraction module, a convolution feature fusion module, an attention mechanism module, a detection output module and a final processing module, the feature extraction module is used for introducing a residual convolutional neural network to extract feature images with different resolutions, the convolution feature fusion module is used for fusing the feature images to obtain original classification graph input features, the attention mechanism module comprises a text detector, and the attention mechanism module is used for embedding a text attention hybrid mechanism in the text detector, the detection output module comprises a classification graph branch module and a rotated rectangle branch module, the classification graph branch module is used for performing single-channel convolution on the mixed classification graph input features to obtain a single-channel classification graph, the rotated rectangle branch module is used for performing multi-channel convolution on the classification graph input features to obtain a minimum predicted rectangle frame, and the final processing module calculates rotated rectangle coordinates of text in the minimum predicted rectangle frame based on a direction boundary and a rotation angle of the minimum predicted rectangle frame, and obtains coordinates of the text after maximum suppression on the rotated rectangle coordinates.

[0040] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0041] (1) The present application proposes a text attention hybrid mechanism of focused features and classification graphs, which is used to solve the problem of false positive detection. From the perspective of classification graph input features, the present application constructs a text attention mechanism of focused features, which is used to enhance the distinguishability of classification graph input features for text and background. From the perspective of classification graphs, the present application constructs a text attention mechanism of focused classification graphs, which is used to enhance the distinguishability of classification graphs. The present application combines the perspectives of classification graph input features and classification graphs, and proposes a text attention hybrid mechanism of focused features and classification graphs, which is used to simultaneously enhance the distinguishability of classification graph input features and classification graphs, and better performance is achieved.

[0042] (2) The text attention hybrid mechanism of focused features and classification graphs proposed by the present application solves the shortcomings of the SSTDTAM text attention mechanism in the prior art, that is, when SSTDTAM is applied to the text detector EAST, the performance is lower than that of DL applied to the EAST classification graph branch alone. This situation shows that SSTDTAM cannot improve the performance. The performance of the text attention hybrid mechanism applied to EAST proposed by the present application can obviously exceed that of DL applied to the EAST classification graph branch alone, and false positive detection can be obviously inhibited. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 It is a flow step schematic diagram of the text detection method of focused features and classification graphs.

[0044] Figure 2 It is a scene text detection model of FSFTAHM embedded in EAST.

[0045] Figure 3 Architecture diagram of the text attention mechanism for focusing on features;

[0046] Figure 4 Architecture diagram of the text attention mechanism for focusing on classification maps;

[0047] Figure 5 Architecture diagram of the text attention hybrid mechanism;

[0048] Figure 6 Architecture diagram of the convolution feature fusion module;

[0049] Figure 7 Diagram for qualitatively comparing the role of the FFTAM;

[0050] Figure 8 Diagram for qualitatively comparing the role of the SFTAM;

[0051] Figure 9 Diagram for qualitatively comparing the role of the FSFTAHM. DETAILED DESCRIPTION

[0052] Embodiment 1:

[0053] A text detection method of focusing on features and classification maps according to the embodiment includes the following steps, as shown in the following figure: Figure 1

[0054] Step S1, input an original image, and extract feature images with different resolutions through a residual convolutional neural network;

[0055] Step S2, fuse the feature images through upsampling operation and concatenation operation to obtain original classification map input features;

[0056] Step S3, introduce a text attention hybrid mechanism, as shown in the following figure, embed the text attention hybrid mechanism into a text detector, and in the process of detecting text by the text detector, perform upsampling operation on the original classification map input features through the text attention hybrid mechanism to obtain attention distribution map input features; perform single-channel convolution operation on the attention distribution map input features through the text attention hybrid mechanism to obtain an attention distribution map; perform downsampling operation and exponential power operation on the attention distribution map input features through the text attention hybrid mechanism, and then fuse the original classification map input features to obtain hybrid classification map input features; the architecture of the text attention hybrid mechanism is as shown in the following figure: Figure 2 Figure 5

[0057] ​​​Step S4, single-channel convolution is performed on the mixed classification map input feature to obtain a single-channel classification map, and multi-channel convolution is performed on the classification map input feature to obtain a minimum prediction rectangular frame, the minimum prediction rectangular frame including a direction boundary and a rotation angle; the attention distribution map is down-sampled, and the single-channel classification map is fused with the down-sampled result of the attention distribution map to obtain a mixed classification map;

[0058] Step S5, threshold filtering is performed on the pixels in the mixed classification map, the pixels greater than a classification map threshold in the mixed classification map are reserved as reference pixels, a reference direction boundary and a reference rotation angle of the reference pixels are calculated based on the minimum prediction rectangular frame, and coordinates of a minimum prediction rectangular frame corresponding to the remaining pixels in the mixed classification map are calculated as coordinates of the text based on the reference direction boundary and the reference rotation angle of the reference pixels.

[0059] Further, the step S3 specifically includes the following steps:

[0060] Step 3.1, linear two times up-sampling operation is performed on the classification map input feature to obtain an attention distribution map input feature;

[0061] Step 3.2, single-channel 1*1 convolution operation is performed on the attention distribution map feature to obtain a feature map, and the feature map is converted into an attention distribution map by using a sigmoid excitation function;

[0062] Step 3.3, down-sampling operation is performed on the attention distribution map feature, and the down-sampling result of the classification map input feature and the attention distribution map feature is multiplied and fused to obtain a mixed classification map input feature.

[0063] Further, the step 3.3 specifically includes the following steps:

[0064] Step 3.3.1, four times down-sampling operation is performed on the attention distribution map feature to match the resolution of the classification map input feature;

[0065] Step 3.3.2, exponential power operation is performed on the four times down-sampling operation result of step 3.3.1;

[0066] Step 3.3.3, the classification map input feature is multiplied and fused with the exponential power operation result in step 3.3.2 to obtain a mixed classification map input feature.

[0067] Further, the step S4 specifically includes the following steps:

[0068] Step 4.1, single-channel 1*1 convolution operation is performed on the mixed classification map input feature to obtain a single-channel classification map, and the text and the background in the original image are determined through the single-channel classification map;

[0069] Step 4.2, performing a 5-channel 1x1 convolution operation on the mixed classification map input features to obtain a minimum predicted rectangular frame covering the text area, and representing the coordinates of the text through the minimum predicted rectangular frame;

[0070] Step 4.3, performing a four times down-sampling operation on the attention distribution map, and multiplying and fusing the single-channel classification map and the down-sampling result of the attention distribution map to obtain a mixed classification map.

[0071] Further, the step 4.2 specifically comprises:

[0072] Step 4.2.1, the text detector generates a minimum predicted rectangular frame covering the text area by using a minAreaRect function;

[0073] Step 4.2.2, calculating the distance of each pixel in the minimum predicted rectangular frame to the four boundaries of the minimum predicted rectangular frame;

[0074] Step 4.2.3, calculating the rotation angle of the minimum predicted rectangular frame relative to the horizontal axis.

[0075] Further, the distance of the pixel to the four boundaries of the minimum predicted rectangular frame is represented by an IOU loss function, and the IOU loss function is specifically:

[0076] ;

[0077] Wherein: represents the IOU loss function; represents the predicted rectangular frame; represents the true value rectangular frame; represents the IOU value between the predicted rectangular frame and the true value rectangular frame.

[0078] The width and the height of the intersection of the predicted rectangular frame and the true value rectangular frame are respectively:

[0079] ;

[0080] ;

[0081] Wherein: , , , respectively represent the true value distance from the pixel point to the upper boundary, the right boundary, the lower boundary and the left boundary of the minimum predicted rectangular frame.

[0082] The width and the height The intersection area of the minimum prediction rectangular frame and the true value rectangular frame is:

[0083] ;

[0084] The union area of the prediction rectangular frame and the true value rectangular frame is: The union area of the minimum prediction rectangular frame and the true value rectangular frame is:

[0085] ;

[0086] Further, the rotation angle loss function is used to represent the rotation angle of the minimum prediction rectangular frame relative to the horizontal axis, and the rotation angle loss function is specifically:

[0087] ;

[0088] Wherein: represents the rotation angle loss function; represents the predicted value of the rotation angle; represents the true value of the rotation angle.

[0089] The total loss of the minimum prediction rectangular frame is:

[0090] ;

[0091] Wherein: represents the total loss function of the minimum prediction rectangular frame; is a weight parameter, which is set to 20.

[0092] Then the loss of the character coordinates represented by the minimum prediction rectangular frame is:

[0093] ;

[0094] Wherein: represents the loss function of the character coordinates represented by the minimum prediction rectangular frame; is a weight parameter, which is set to 1; represents the loss CBCE function of the single-channel classification map.

[0095] Further, as Figure 6 ​As shown, the residual convolutional neural network includes a ResNet50 network, the ResNet50 network includes a first convolutional module Res1, a second convolutional module Res2, a third convolutional module Res3, and a fourth convolutional module Res4, the first convolutional module extracts a feature image with a resolution of one fourth of the original image resolution, the second convolutional module extracts a feature image with a resolution of one eighth of the original image resolution, the third convolutional module extracts a feature image with a resolution of one sixteenth of the original image resolution, and the fourth convolutional module extracts a feature image with a resolution of one thirty-second of the original image resolution.

[0096] Further, the pixel value of the pixel in the attention distribution map is as follows:

[0097] 0≤k≤1, wherein k represents the pixel value of the pixel in the attention distribution map.

[0098] A text detection system focusing on feature and classification map, comprising a feature extraction module, a convolutional feature fusion module, an attention mechanism module, a detection output module, and a final processing module, the feature extraction module is used to introduce a residual convolutional neural network to extract feature images with different resolutions; the convolutional feature fusion module is used to fuse the feature images to obtain original classification map input features; the attention mechanism module includes a text detector, and the attention mechanism module is used to embed a text attention hybrid mechanism in the text detector; the detection output module includes a classification map branch module and a rotated rectangle branch module, the classification map branch module is used to perform single-channel convolution on the hybrid classification map input features to obtain a single-channel classification map, and the rotated rectangle branch module is used to perform multi-channel convolution on the classification map input features to obtain a minimum predicted rectangular frame; the final processing module calculates the rotated rectangular coordinates of the text in the minimum predicted rectangular frame based on the direction boundary and the rotation angle of the minimum predicted rectangular frame, and performs maximum suppression on the rotated rectangular coordinates to obtain the coordinates of the text.

[0099] Embodiment 2

[0100] The embodiment discloses a text detection method focusing on feature and classification map, which is further optimized on the basis of embodiment 1 and comprises the following steps:

[0101] Step S1, input an original image, and extract feature images with different resolutions through a residual convolutional neural network; specifically, the original image is input into a ResNet50 network, the ResNet50 network includes a convolutional layer, a pooling layer and a batch normalization layer, the ResNet50 network uses a first convolutional module Res1, a second convolutional module Res2, a third convolutional module Res3 and a fourth convolutional module Res4 to extract four features in the original image, and the resolutions of the four features are respectively one fourth, one eighth, one sixteenth and one thirty-second of the resolution of the input original image.

[0102] Step S2, the feature images are fused by using upsampling operation and concatenation operation to obtain original classification map input features; for the four features obtained in step S1, the four features are gradually fused by using upsampling operation and concatenation operation, and the fusion is performed from the feature with small resolution to the feature with large resolution in turn, and the classification map input features are obtained after the fusion.

[0103] Step S3, a text attention mixing mechanism is introduced, the text attention mixing mechanism is embedded into a text detector, and in the process of detecting the text by the text detector, the original classification map input features are subjected to upsampling operation by the text attention mixing mechanism to obtain attention distribution map input features; the text attention mixing mechanism performs single-channel convolution operation on the attention distribution map input features to obtain an attention distribution map, and the text attention mixing mechanism performs downsampling operation, exponential power operation on the attention distribution map input features, and then fuses the original classification map input features to obtain mixed classification map input features.

[0104] Specifically,

[0105] The text attention mixing mechanism is constructed and introduced, and is used for simultaneously enhancing the distinguishability of the classification map and the classification map input features to the text and the background. The architecture of the text attention mixing mechanism is shown in Figure 5 , wherein Unpool represents a 2-fold upsampling operation, and DownSample represents a downsampling operation. Assuming that the height of an input image is and the width is , then is the classification map input feature of the input image, and the resolution of the classification map input feature is one fourth of the resolution of the original image. The output attention distribution map is denoted as , the attention distribution map is a three-order tensor, and the attention distribution map is calculated according to the following formula:

[0106] ;

[0107] Among them: represents the attention distribution map input feature; represents a 2-fold upsampling operation; ​The classification map input feature is represented as F.

[0108] The text attention mixing mechanism performs a single-channel convolution operation and an excitation function operation on the attention distribution map input feature to obtain an attention distribution map A, and the calculation formula is as follows:

[0109] ;

[0110] Wherein: represents a single-channel feature map; represents a single-channel 1x1 convolution operation on the attention distribution map input feature.

[0111] ;

[0112] Wherein: A represents an attention distribution map; represents an excitation function operation on the single-channel feature map.

[0113] Each pixel value in the attention distribution map A is between 0 and 1, representing the probability of the corresponding image pixel being classified as text. The text attention mixing mechanism uses DL to train the attention distribution map.

[0114] The text attention mixing mechanism performs a 4-fold downsampling operation on the attention distribution map input feature to match the resolution of the classification map input feature , and then performs an exponential power operation and a multiplication fusion operation with the classification map input feature to obtain a mixed classification map input feature. The mixed classification map input feature is calculated according to the following formula:

[0115] ;

[0116] Wherein: represents the attention distribution map input feature after the downsampling operation; represents the downsampling operation on the attention distribution map input feature; represents the exponential power of , and the exponential power parameter is used to adjust the strength of the attention effect. The larger the parameter , the stronger the attention effect, and vice versa. represents the mixed classification map input feature.

[0117] A multi-channel convolution with a convolution kernel of and a channel number of 5 is connected to the mixed classification map input feature to form a rotated rectangular branch. The rotated rectangular branch outputs four distance maps and an angle map, which are used to represent the coordinates of the text. A multi-channel convolution with a convolution kernel of , a single-channel convolution with a channel number of 1 is used to form a classification map branch, which is used to output a single-channel classification map. In order to further enhance the discrimination ability of the text detector EAST for text and background, the text attention mixing mechanism uses an attention distribution map to enhance the discriminability of the classification map. Specifically, the text attention mixing mechanism performs a four-fold downsampling operation on the attention distribution map to match the resolution of the single-channel classification map. Then, the attention distribution map after downsampling is multiplied by the single-channel classification map to obtain a mixed classification map, which is calculated according to the following formula:

[0118] ;

[0119] wherein: represents the attention distribution map after downsampling; represents the downsampling operation on the attention distribution map A; represents the mixed classification map, and the loss function thereof is CBCE; represents the single-channel classification map.

[0120] The text attention mixing mechanism uses the attention distribution map input feature to enhance the discriminability of the classification map input feature, and uses the attention distribution map to enhance the discriminability of the classification map. Because the text attention mixing mechanism simultaneously enhances the discriminability of the classification map input feature and the discriminability of the classification map for text and background, the performance of the text attention mixing mechanism applied to the text detector EAST exceeds the performance of the DL applied to the classification map branch of the text detector EAST. As shown in Figure 3 , a scene detection model embedding the text detector EAST with the text attention mixing mechanism.

[0121] The other parts of this embodiment are the same as those of Embodiment 1, and will not be described again.

[0122] Embodiment 3:

[0123] This embodiment discloses a text detection method focusing on features and classification maps, which is further optimized on the basis of Embodiment 1. A text attention mechanism focusing on features (FFTAM) is embedded in the text detector, and the architecture thereof is shown in Figure 3 , wherein Unpool represents a 2-fold upsampling operation, and DownSample represents a downsampling operation.

[0124] Taking the classification map input feature as input, the text attention mechanism focusing on features learns an attention distribution map representing a rough spatial region of text, and then the attention distribution map input feature is multiplied by the classification map input feature to fuse and output a classification map input feature focusing on features. The classification map input feature focusing on features is connected to the classification map branch and the rotated rectangle branch, respectively.

[0125] It is assumed that the height of the input image is , width is , then is the classification map input feature of the input image, whose resolution is one quarter of the original image. The output attention distribution map is denoted as The attention distribution map is a third-order tensor, and the attention distribution map is calculated as follows:

[0126] ;

[0127] wherein: represents the attention distribution map input feature; represents a two-fold upsampling operation; represents the classification map input feature.

[0128] The text attention mixing mechanism performs a single-channel convolution operation and an excitation function operation on the attention distribution map input feature to obtain the attention distribution map A, and the calculation formula is as follows:

[0129] ;

[0130] wherein: represents a single-channel feature map; represents a single-channel 1x1 convolution operation on the attention distribution map input feature.

[0131] ;

[0132] wherein A represents the attention distribution map; represents an excitation function operation on the single-channel feature map.

[0133] Each pixel value in the attention distribution map A is between 0 and 1, representing the probability that the corresponding image pixel is classified as text. The text attention mixing mechanism uses a loss function DL to train the attention distribution map.

[0134] After outputting the attention distribution map A, the text attention mechanism of the focus feature uses a loss function DL to train the attention distribution map. The text attention mechanism of the focus feature multiplies and fuses the attention distribution map input feature and the classification map input feature . The calculation formula is as follows:

[0135] ;

[0136] wherein: represents the attention distribution map input feature after the downsampling operation; represents a downsampling operation on the attention distribution map input feature; represents the classification map input feature of the focus feature.

[0137] Thus, the classification map input feature of the focus feature has strong discriminative ability to distinguish the text region and the background region.

[0138] A convolution kernel of 1 is connected to the classification map input feature of the focus feature to form a multi-channel convolution with a channel number of 5 to constitute a rotated rectangle branch, and the rotated rectangle branch outputs four distance maps and an angle map for representing the coordinates of the text. A convolution kernel of 1 is connected to the mixed classification map input feature to form a single-channel convolution with a channel number of 1 to constitute a classification map branch for outputting a single-channel classification map.

[0139] When the focus feature text attention mechanism (FFTAM) is embedded into the text detector EAST, the loss function CBCE of the single-channel classification map is:

[0140] ;

[0141] wherein: CBCE represents the loss function of the single-channel classification map; represents the classification map prediction value of the text and the non-text; represents the classification map true value of the text and the non-text; represents a balance factor between positive and negative samples.

[0142] Because the number of text pixels is far less than the number of background pixels, the parameter is very close to 1, and the parameter is extremely close to 0, resulting in insufficient training of negative samples, so that the discriminability of the classification map and the classification map input feature corresponding to the loss function CBCE to the text and the background is weak. When the classification threshold of the classification map is low, the recall rate of the text detector EAST is high, and the precision is low; when the classification threshold of the classification map is high, the recall rate of the text detector EAST is low, and the precision is high.

[0143] The loss function for training the attention distribution map is DL, denoted as:

[0144] ;

[0145] wherein: and represent the true value and the prediction value of the attention distribution map, respectively, and the values of the corresponding pixels ; the true value of the attention distribution map ​​is a binary image at the pixel level, where the pixel value in the text region is 1, and the pixel value in the background region is 0. Because the positive sample loss weight and the negative sample loss weight of the loss function DL are both 1, the negative samples of the attention distribution graph corresponding to the loss function DL can be fully trained, so the attention distribution graph and the attention distribution graph input feature have strong discrimination for text and background.

[0146] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described again.

[0147] Embodiment 4:

[0148] This embodiment discloses a text detection method of focusing on features and classification graphs. On the basis of Embodiment 1, a text attention mechanism (SFTAM) of focusing on classification graphs is embedded in the text detector, and the architecture is as shown in Figure 4 , wherein Unpool represents a 2x upsampling operation, and DownSample represents a downsampling operation.

[0149] Suppose the height of the input image is , and the width is , then is the classification graph input feature of the input image, and the resolution is one fourth of the original image. The output attention distribution graph is denoted as A, and the attention distribution graph is a three-order tensor, and the attention distribution graph is calculated according to the following formula:

[0150] ;

[0151] , wherein: represents the attention distribution graph input feature; represents a 2x upsampling operation; represents the classification graph input feature.

[0152] The text attention mixing mechanism performs a single-channel convolution operation and an excitation function operation on the attention distribution graph input feature to obtain the attention distribution graph A, and the calculation formula is as follows:

[0153] ;

[0154] , wherein: represents a single-channel feature map; represents a single-channel 1x1 convolution operation on the attention distribution graph input feature.

[0155] ;

[0156] , wherein A represents the attention distribution graph; represents an excitation function operation on the single-channel feature map. ​

[0157] Each pixel value in the attention distribution map A is between 0 and 1, representing the probability of the corresponding image pixel being classified as a text, and the text attention mixing mechanism trains the attention distribution map using a loss function DL.

[0158] After outputting the attention distribution map A, the text attention mechanism of the focus classification map trains the attention distribution map using a loss function DL. The text attention mechanism of the focus classification map inputs the classification map input feature SMIF into a single-channel convolutional layer with a convolution kernel of and a channel number of 1 to form a classification map branch, which outputs a single-channel classification map SM. The text attention mechanism of the focus classification map directly multiplies the attention distribution map and the single-channel classification map. Specifically, the text attention mechanism of the focus classification map first performs a 4-fold downsampling operation on the attention distribution map, and then directly multiplies the result of the downsampling operation with the single-channel classification map, calculated as follows:

[0159]

[0160] wherein: represents the attention distribution map after the downsampling operation; represents the downsampling operation on the attention distribution map A; represents the single-channel classification map; represents the classification map input feature of the focus classification map.

[0161] The text attention mechanism of the focus classification map inputs the classification map input feature SMIF into a multi-channel convolutional layer with a convolution kernel of and a channel number of 5 to form a rotated rectangle branch, which outputs four distance maps and an angle map for representing the coordinates of the text. The text attention mechanism of the focus classification map inputs the classification map input feature SMIF into a single-channel convolutional layer with a convolution kernel of and a channel number of 1 to form a classification map branch, which is used to output a single-channel classification map.

[0162] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described again.

[0163] Embodiment 5:

[0164] ​​The three evaluation indexes of the text detection method adopted by the present application are respectively: recall, precision and F-score value. The recall represents the proportion of the successfully detected true value frame relative to all true value frames. The precision measures the proportion of the successfully detected true value frame relative to all detection frames. In performance evaluation, if the IOU value of the detection frame and the true value frame is greater than 0.5, the detection frame is a correct detection frame, otherwise it is an incorrect detection frame. The F-score value is the harmonic mean of the recall and the precision. When comparing the performance of the scene text detection method, the greater the F-score value of the method, the better the performance.

[0165] The data set used for evaluating the text detection method of the present application is three scene text detection data sets, which are respectively: ICDAR2015, MSRA-TD500 and CASIA-10K.

[0166] In the subsequent experimental chart, EAST+CBCE represents the version of the re-implemented text detector EAST framework, that is, the benchmark model, using CBCE as the loss function of the classification graph; EAST+SSTDTAM is the model embedding SSTDTAM into EAST; EAST+FFTAM is the model embedding FFTAM into EAST; EAST+SFTAM represents the model embedding SFTAM into EAST; EAST+FSFTAM represents the model embedding FSFTAM into EAST; and EAST+DL represents the EAST model with DL as the classification graph loss function.

[0167] The detection performance of the above various text detection methods on the ICDAR2015 data set is shown in Table 1:

[0168] Table 1

[0169]

[0170] The detection performance of various text detection methods on the MSRA-TD500 data set is shown in Table 2:

[0171] Table 2

[0172]

[0173] The detection performance of various text detection methods on the CASIA-10K data set is shown in Table 3, and “ * ” represents multi-scale testing.

[0174] Table 3

[0175]

[0176] Table 1 shows the detection performance of different algorithms on the ICDAR A2015 dataset. From Table 1, it can be seen that the FSFTAHM proposed in this chapter effectively improves the detection performance of EAST. Compared with the detection performance of EAST+CBCE, the detection performance of EAST+SSTDTAM reaches a F-score value of 0.8153, with a performance gain of 1.73%, among which the performance gain of recall rate is 2.7% and the performance gain of precision is 0.54%. Compared with the detection performance of EAST+CBCE, the detection performance of EAST+FSFTAHM obtains a F-score value of 0.8249, with a performance gain of 2.69%, among which the performance gain of recall rate is 2.31% and the performance gain of precision is 3.16%. From Table 1, it can be seen that the detection performance of EAST+FSFTAHM significantly exceeds the detection performance of EAST+SSTDTAM, with a performance gain of 0.96% in F-score value. The exponential power parameter M of EAST+FSFTAHM is 2.

[0177] The detection performance of SSTDTAM and FSFTAHM on the MSRA-TD500 dataset is shown in Table 2. SSTDTAM makes the F-score value of EAST+CBCE increase by 1.71%, among which the recall rate increases by 2.06% and the precision increases by 1.22%. The F-score value gain of FSFTAHM to EAST+CBCE is 2.83%, among which the recall rate increases by 1.72% and the precision increases by 4.36%. The detection performance of EAST+FSFTAHM significantly exceeds the detection performance of EAST+SSTDTAM, with a performance gain of 1.12% in F-score value, an increase of 3.14% in precision, and a slight decrease in recall rate. The exponential power parameter M of EAST+FSFTAHM is 1.

[0178] The detection results of the CASIA-10K dataset are shown in Table 3. Compared with the detection performance of EAST+CBCE, the detection performance of EAST+SSTDTAM reaches a F-score value of 0.7162, with a performance gain of 1.1%, among which the performance gain of recall rate is 1.54%. Compared with the detection performance of EAST+CBCE, the detection performance of EAST+FSFTAHM obtains a F-score value of 0.7366, with a performance gain of 3.14%, among which the performance gain of recall rate is 3.89% and the performance gain of precision is 1.99%. The detection performance of EAST+FSFTAHM significantly exceeds the detection performance of EAST+SSTDTAM, with a performance gain of 2.04% in F-score value, an increase of 2.35% in recall rate, and an increase of 1.57% in precision. The exponential power parameter M of EAST+FSFTAHM is 1.

[0179] The detection performance of SSTDTAM, FFTAM, SFTAM and FSFTAHM on the ICDAR2015 dataset is shown in Table 4.

[0180] Table 4

[0181]

[0182] The detection performance of SSTDTAM, FFTAM, SFTAM and FSFTAHM on the MSRA-TD500 dataset is shown in Table 5.

[0183] Table 5

[0184]

[0185] The detection performance of SSTDTAM, FFTAM, SFTAM and FSFTAHM on the CASIA-10K dataset is shown in Table 6.

[0186] Table 6

[0187]

[0188] From Table 1, Table 2 and Table 3, it can be seen that FSFTAHM can obviously improve the recall rate and precision of EAST+CBCE, which shows that FSFTAHM can obviously enhance the distinguishability of text detector to text and background, and can recall more text while suppressing false positive background. Since SSTDTAM only enhances the distinguishability of classification map input features, the detection performance of EAST+SSTDTAM is lower than that of EAST+DL on ICDAR2015 and CASIA-10K datasets. The F-score values of EAST+SSTDTAM on ICDAR2015 and CASIA-10K datasets are 0.8153 and 0.7162 respectively, and the F-score values of EAST+DL are 0.8162 and 0.7273 respectively, as shown in Table 4 and Table 6. However, the detection performance of SSTDTAM on MSRA-TD500 exceeds that of EAST+DL, and the F-score value of EAST+SSTDTAM is 0.7922, and the F-score value of EAST+DL is 0.7855, as shown in Table 5, so SSTDTAM still has a certain effect, but its effect is limited. Since FSFTAHM enhances the distinguishability of classification map and classification map input features at the same time, the detection performance of FSFTAHM exceeds that of EAST+DL, and the F-score values of EAST+FSFTAHM on ICDAR2015, MSRA-TD500 and CASIA-10K datasets are 0.8249, 0.8034 and 0.7366 respectively, and the F-score values of EAST+DL are 0.8162, 0.7855 and 0.7273 respectively, as shown in Table 4, Table 5 and Table 6.

[0189] Because FFTAM only enhances the distinguishability of the input features of the classification map to the text and the background, the performance of EAST+FFTAM cannot obviously exceed or cannot exceed the performance of EAST+DL on the ICDAR2015 and CASIA-10K datasets, the F-score values of EAST+FFTAM on ICDAR2015, MSRA-TD500 and CASIA-10K are 0.8164, 0.7937 and 0.7243 respectively, which obviously exceed the performance of EAST+CBCE, the F-score values of EAST+CBCE on the three datasets are 0.798, 0.7751 and 0.7052 respectively. Similarly, because SFTAM only enhances the distinguishability of the classification map to the text and the background, the performance of EAST+SFTAM is lower than the performance of EAST+DL on the ICDAR2015 and CASIA-10K datasets, the F-score values of EAST+SFTAM on ICDAR2015, MSRA-TD500 and CASIA-10K are 0.8114, 0.794 and 0.7205 respectively, which obviously exceed the performance of EAST+CBCE, as shown in Tables 4, 5 and 6. On the basis of FFTAM and SFTAM, FSFTAHM combines the structure and function of the first two mechanisms, so it achieves more superior performance.

[0190] In order to qualitatively analyze the effects of FFTAM, SFTAM and FSFTAHM, the present application embeds FFTAM, SFTAM and FSFTAHM on the benchmark model EAST+CBCE respectively. Because the attention mechanisms in this chapter are all experimented on the EAST text detector, the original EAST text detector model EAST+CBCE is qualitatively compared and analyzed, Figure 9 The effects of FFTAM, SFTAM and FSFTAHM are shown. The FFTAM, SFTAM and FSFTAHM proposed by the present application can help EAST to achieve better detection results.

[0191] The output of the benchmark model EAST+CBCE Figure 7 (a) detects two background regions as text, causing false positive detection, the output of the model EAST+FFTAM Figure 7 (c) removes the two background regions, which shows that FFTAM can effectively reduce false positive detection. Figure 7 (b) from the benchmark model EAST+CBCE, the subgraph does not detect another English character, the output of the model EAST+FFTAM Figure 7 (d) recalls the English character, which shows that FFTAM can recall more text.

[0192] Figure 8(a) is output by the baseline model EAST+CBCE, which does not recall the text in the subgraph, and the output of the model EAST+SFTAM Figure 8 (c) can detect Figure 8 (a) the missed text, which indicates that SFTAM can effectively recall more text. The baseline model EAST+CBCE detects the grid background as text, resulting in false positive detection, as shown in Figure 8 (b), and the output of the model EAST+SFTAM Figure 8 (d) can remove this grid background, which indicates that SFTAM can reduce false positive detection.

[0193] Figure 9 (a) is the detection result of the baseline model EAST+CBCE, which does not detect the two difficult texts with large inclinations. The detection result output by the EAST+FSFTAHM model is shown in Figure 9 (c), which recalls the text missed in Figure 9 (a). This indicates that FSFTAHM can help the text detector EAST to recall more challenging texts. The baseline model EAST+CBCE is prone to detect background areas as text, resulting in false positive detection, as shown in Figure 9 (b). The detection result output by the EAST+FSFTAHM model is shown in Figure 9 (d), which removes the background area in Figure 9 (b). This indicates that FSFTAHM can help the scene text detector EAST to remove the background and reduce false positive detection results.

[0194] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification or equivalent change based on the technical essence of the present application to the above embodiment falls within the protection scope of the present application.

Claims

1. A text detection method focusing on features and classification graphs, characterized in that: The following steps are involved: Step S1: input the original image and extract feature images with different resolutions through the residual convolutional neural network; Step S2: fusing the feature images using upsampling and concatenation operations to obtain the original classification map input features; Step S3: Introduce a text attention hybrid mechanism and embed it into the text detector. During the text detection process, the text attention hybrid mechanism is used to upsample the original classification map input features to obtain the attention distribution map input features. The text attention hybrid mechanism performs a single-channel convolution operation on the attention distribution map input features to obtain the attention distribution map. The text attention hybrid mechanism performs downsampling and exponential power operations on the attention distribution map input features and then fuses them with the original classification map input features to obtain the mixed classification map input features. Step S4: performing single-channel convolution on the mixed classification map input features to obtain a single-channel classification map, performing multi-channel convolution on the classification map input features to obtain a minimum prediction rectangle, wherein the minimum prediction rectangle includes a direction boundary and a rotation angle; downsampling the attention distribution map, and fusing the single-channel classification map and the attention distribution map downsampling results to obtain a mixed classification map; Step S5: perform threshold filtering on the pixels in the mixed classification map, retain the pixels in the mixed classification map that are greater than the classification map threshold as reference pixels, calculate the reference direction boundary and reference rotation angle of the reference pixel based on the minimum prediction rectangular box, and calculate the minimum prediction rectangular box coordinates corresponding to the remaining pixels in the mixed classification map based on the reference direction boundary and reference rotation angle of the reference pixel as the coordinates of the text.

2. The method for detecting text by focusing on features and classification diagrams according to claim 1, wherein: The step S3 specifically includes the following steps: Step 3.1: Perform a linear double upsampling operation on the classification map input features to obtain the attention distribution map input features; Step 3.2: Perform a single-channel 1×1 convolution operation on the attention distribution map feature to obtain a feature map, and use the sigmoid activation function to convert the feature map into an attention distribution map; Step 3.3: Downsample the attention distribution map features, and multiply and fuse the classification map input features with the downsampled results of the attention distribution map features to obtain the mixed classification map input features.

3. The method for detecting text by focusing on features and classification diagrams according to claim 2, wherein: The step 3.3 specifically includes the following steps: Step 3.3.

1. Downsample the attention map features by a factor of four to match the resolution of the classification map input features. Step 3.3.2, performing an exponential power operation on the result of the fourfold downsampling operation in step 3.3.1; Step 3.3.3: Multiply and fuse the classification map input features with the result of the exponential power operation in step 3.3.2 to obtain the mixed classification map input features.

4. The method for detecting text by focusing on features and classification diagrams according to claim 3, wherein: The step S4 specifically includes the following steps: Step 4.1: Perform a single-channel 1×1 convolution operation on the mixed classification map input features to obtain a single-channel classification map, and use the single-channel classification map to determine the text and background in the original image; Step 4.2: Perform a 5-channel 1×1 convolution operation on the mixed classification map input features to obtain the minimum predicted rectangle covering the text area, and use the minimum predicted rectangle to represent the coordinates of the text; Step 4.3: Perform a four-fold downsampling operation on the attention distribution map, and multiply and fuse the single-channel classification map with the downsampling result of the attention distribution map to obtain a mixed classification map.

5. The method for detecting text by focusing on features and classification diagrams according to claim 4, wherein: The step 4.2 specifically includes: Step 4.2.1: The text detector uses the minAreaRect function to generate the minimum predicted rectangle covering the text area. Step 4.2.2: Calculate the distance between each pixel in the minimum prediction rectangle and the four edges of the minimum prediction rectangle. Step 4.2.3: Calculate the rotation angle of the minimum prediction rectangle relative to the horizontal axis.

6. The method for detecting text by focusing on features and classification diagrams according to claim 5, characterized in that: The IOU loss function is used to characterize the distance from the pixel to the four boundaries of the minimum prediction rectangle. The IOU loss function is specifically: ； in: Represents the IOU loss function; Represents the predicted rectangular box; Represents the true value rectangle; Represents the IOU value between the predicted rectangle and the true rectangle.

7. The method for detecting text by focusing on features and classification diagrams according to claim 5, wherein: The rotation angle loss function is used to characterize the rotation angle of the minimum prediction rectangle relative to the horizontal axis. The rotation angle loss function is specifically: ； in: represents the rotation angle loss function; represents the predicted value of the rotation angle; The true value representing the rotation angle.

8. A method for detecting text based on focus features and classification diagrams according to any one of claims 1 to 7, characterized in that: The residual convolutional neural network includes a ResNet50 network, and the ResNet50 network includes a first convolution module, a second convolution module, a third convolution module, and a fourth convolution module. The resolution of the feature image extracted by the first convolution module is one-fourth of the resolution of the original image, the resolution of the feature image extracted by the second convolution module is one-eighth of the resolution of the original image, the resolution of the feature image extracted by the third convolution module is one-sixteenth of the resolution of the original image, and the resolution of the feature image extracted by the fourth convolution module is one-thirty-second of the resolution of the original image.

9. A method for detecting text by focusing on features and classification diagrams according to any one of claims 1 to 7, characterized in that: The pixel values ​​of the pixels in the attention distribution map are as follows: 0≤k≤1, where k represents the pixel value of the pixel in the attention distribution map.

10. A text detection system focusing on features and classification diagrams, used to implement the detection method according to any one of claims 1 to 9, characterized in that: It includes a feature extraction module, a convolutional feature fusion module, an attention mechanism module, a detection output module, and a final processing module. The feature extraction module is used to introduce a residual convolutional neural network to extract feature images with different resolutions; the convolutional feature fusion module is used to fuse the feature images to obtain the original classification map input features; The attention mechanism module includes a text detector, and the attention mechanism module is used to embed a text attention hybrid mechanism in the text detector; the detection output module includes a classification map branch module and a rotation rectangle branch module, the classification map branch module is used to perform single-channel convolution on the mixed classification map input features to obtain a single-channel classification map, and the rotation rectangle branch module is used to perform multi-channel convolution on the classification map input features to obtain a minimum prediction rectangular frame; the final processing module calculates the rotated rectangle coordinates of the text in the minimum prediction rectangular frame based on the direction boundary and rotation angle of the minimum prediction rectangular frame, and obtains the coordinates of the text after maximum suppression of the rotated rectangle coordinates.

Images