Character detection method and detection system for focusing feature and classification graph

Abstract

The invention discloses a focussing feature and classification graph character detection method and detection system, a focussing feature and classification graph character attention mixing mechanism is constructed through a convolution feature fusion module in the detection system, the character attention mixing mechanism is used for solving a false positive problem, classification graph input features and classification graph angles are combined, and the focussing feature and classification graph character recognition accuracy is improved. A character attention mixing mechanism of a focusing feature and a classification graph is provided and is used for simultaneously enhancing the discrimination of the input feature of the classification graph and the classification graph, and better performance is obtained; the character attention mixing mechanism can effectively overcome the problem of false positive detection existing in an existing SSTDTAM attention mechanism, meanwhile, the performance of the character attention mixing mechanism applied to a character detector can obviously exceed the performance of a difference loss function independently applied to classification graph branches of the character detector, and false positive detection can be obviously inhibited.

Description

Technical Field

[0001] The present invention belongs to the technical field of scene text detection, and specifically relates to a text detection method and detection system focusing on features and classification graphs. Background Art

[0002] The goal of scene text detection is to accurately locate text boundaries. Convolutional neural network-based methods first extract convolutional features from the input image, then perform feature fusion. Finally, based on these features, they classify the text and background and output the text's bounding box. Because natural scene images are highly diverse and complex, certain features of the background and text are similar. In situations where text and background are similar, if the text detector's classification map and the input features of the classification map only weakly distinguish between text and background, the text detector will output a false positive detection result. A false positive text detection occurs when the detection algorithm classifies background as text.

[0003] To address the false positive problem in text detection, SSTD proposes an attention mechanism for text detection. This mechanism uses an auxiliary loss function to learn an attention distribution map to effectively distinguish text and background areas in an image. The mechanism then encodes the attention distribution map into the classification map input features, significantly suppressing the inference of background areas and highlighting the features of text areas. Therefore, the mechanism enhances the distinguishability of features between text and background from a feature perspective. The SSTD text attention mechanism first performs two 2x upsampling operations on the classification map input features, then connects the convolutional layer output attention distribution map and trains the distribution map using a difference loss function. The attention distribution map is then downsampled 4x and multiplied with the classification map input features. Finally, it is connected to the classification map branch and the rotated rectangle branch of EAST.

[0004] When SSTDTAM is applied to the text detector EAST, its performance is lower than that of the difference loss function applied alone to the EAST classification map branch. This indicates that SSTDTAM does not provide performance improvement. The reason for SSTDTAM's poor performance is that it only enhances the classification map input features to distinguish text from background, but does not enhance the classification map's discriminability.

[0005] Therefore, in order to address the false positive detection problem that exists when the existing SSTD text attention mechanism is applied to a text detector, the present invention discloses a text detection method and detection system that focuses on features and classification maps. Summary of the Invention

[0006] The present invention discloses a text detection method and detection system that focuses on features and classification maps, and constructs a text attention hybrid mechanism that focuses on features and classification maps to solve the false positive problem. The text attention hybrid mechanism can effectively overcome the problem of false positive detection in the existing SSTDTAM attention mechanism. At the same time, the performance of the text attention hybrid mechanism applied to the text detector can significantly exceed the performance of the difference loss function applied alone to the classification map branch of the text detector, and can significantly suppress false positive detection.

[0007] The present invention is achieved through the following technical solutions: A method for detecting text by focusing on features and classification graphs, comprising the following steps: Step S1: input the original image and extract feature images with different resolutions through the residual convolutional neural network; Step S2: fusing the feature images using upsampling and concatenation operations to obtain the original classification map input features; Step S3: Introduce a text attention hybrid mechanism and embed it into a text detector. During the text detector's text detection process, the text attention hybrid mechanism performs an upsampling operation on the original classification map input features to obtain an attention distribution map input feature. The text attention hybrid mechanism performs a single-channel convolution operation on the attention distribution map input feature to obtain an attention distribution map. The text attention hybrid mechanism performs a downsampling operation and an exponential power operation on the attention distribution map input feature and then fuses the original classification map input feature to obtain a mixed classification map input feature. Step S4: performing single-channel convolution on the mixed classification map input features to obtain a single-channel classification map, performing multi-channel convolution on the classification map input features to obtain a minimum prediction rectangle, wherein the minimum prediction rectangle includes a direction boundary and a rotation angle; downsampling the attention distribution map, and fusing the single-channel classification map and the attention distribution map downsampling results to obtain a mixed classification map; Step S5: perform threshold filtering on the pixels in the mixed classification map, retain the pixels in the mixed classification map that are greater than the classification map threshold as reference pixels, calculate the reference direction boundary and reference rotation angle of the reference pixel based on the minimum prediction rectangular box, and calculate the minimum prediction rectangular box coordinates corresponding to the remaining pixels in the mixed classification map based on the reference direction boundary and reference rotation angle of the reference pixel as the coordinates of the text.

[0008] In order to better implement the present invention, further, step S3 specifically includes the following steps: Step 3.1: Perform a linear double upsampling operation on the classification map input features to obtain the attention distribution map input features; Step 3.2: Perform a single-channel 1×1 convolution operation on the attention distribution map feature to obtain a feature map, and use the sigmoid activation function to convert the feature map into an attention distribution map; Step 3.3: Downsample the attention distribution map features, and multiply and fuse the classification map input features with the downsampled results of the attention distribution map features to obtain the mixed classification map input features.

[0009] In order to better implement the present invention, further, the step 3.3 specifically includes the following steps: Step 3.3.1. Downsample the attention map features by a factor of four to match the resolution of the classification map input features. Step 3.3.2, performing an exponential power operation on the result of the four-fold downsampling operation in step 3.3.1; Step 3.3.3: Multiply and fuse the classification map input features with the result of the exponential power operation in step 3.3.2 to obtain the mixed classification map input features.

[0010] In order to better implement the present invention, further, step S4 specifically includes the following steps: Step 4.1: Perform a single-channel 1×1 convolution operation on the mixed classification map input features to obtain a single-channel classification map, and use the single-channel classification map to determine the text and background in the original image; Step 4.2: Perform a 5-channel 1×1 convolution operation on the mixed classification map input features to obtain the minimum predicted rectangle covering the text area, and use the minimum predicted rectangle to represent the coordinates of the text; Step 4.3: Perform a four-fold downsampling operation on the attention distribution map, and multiply and fuse the single-channel classification map with the downsampling result of the attention distribution map to obtain a mixed classification map.

[0011] In order to better implement the present invention, further, the step 4.2 specifically includes: Step 4.2.1: The text detector uses the minAreaRect function to generate the minimum predicted rectangle covering the text area. Step 4.2.2: Calculate the distance between each pixel in the minimum prediction rectangle and the four edges of the minimum prediction rectangle. Step 4.2.3: Calculate the rotation angle of the minimum prediction rectangle relative to the horizontal axis.

[0012] In order to better implement the present invention, an IOU loss function is further used to characterize the distance from the pixel to the four boundaries of the minimum prediction rectangle. The IOU loss function is specifically: ; in: Represents the IOU loss function; Represents the predicted rectangular box; Represents the true value rectangle; Represents the IOU value between the predicted rectangle and the true rectangle.

[0013] In order to better implement the present invention, a rotation angle loss function is further used to characterize the rotation angle of the minimum prediction rectangle relative to the horizontal axis. The rotation angle loss function is specifically: ; in: represents the rotation angle loss function; represents the predicted value of the rotation angle; The true value representing the rotation angle.

[0014] In order to better implement the present invention, further, the residual convolutional neural network includes a ResNet50 network, and the ResNet50 network includes a first convolution module, a second convolution module, a third convolution module, and a fourth convolution module. The resolution of the feature image extracted by the first convolution module is one-fourth of the resolution of the original image, the resolution of the feature image extracted by the second convolution module is one-eighth of the resolution of the original image, the resolution of the feature image extracted by the third convolution module is one-sixteenth of the resolution of the original image, and the resolution of the feature image extracted by the fourth convolution module is one-thirty-second of the resolution of the original image.

[0015] In order to better implement the present invention, further, the pixel values of the pixels in the attention distribution map are as follows: 0≤k≤1, where k represents the pixel value of the pixel in the attention distribution map.

[0016] A text detection system focusing on features and classification maps is used to implement the above-mentioned text detection method, including a feature extraction module, a convolutional feature fusion module, an attention mechanism module, a detection output module, and a final processing module. The feature extraction module is used to introduce a residual convolutional neural network to extract feature images with different resolutions; the convolutional feature fusion module is used to fuse the feature images to obtain the original classification map input features; the attention mechanism module includes a text detector, and the attention mechanism module is used to embed a text attention hybrid mechanism in the text detector; the detection output module includes a classification map branch module and a rotation rectangle branch module, the classification map branch module is used to perform single-channel convolution on the mixed classification map input features to obtain a single-channel classification map, and the rotation rectangle branch module is used to perform multi-channel convolution on the classification map input features to obtain a minimum prediction rectangle; the final processing module calculates the rotated rectangle coordinates of the text in the minimum prediction rectangle based on the direction boundary and rotation angle of the minimum prediction rectangle, and obtains the coordinates of the text after maximum suppression of the rotated rectangle coordinates.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) The present invention proposes a hybrid mechanism of text attention focusing on features and classification graphs to solve the problem of false positive detection. From the perspective of classification graph input features, the present invention constructs a text attention mechanism focusing on features to enhance the discrimination of classification graph input features for text and background; from the perspective of classification graphs, the present invention constructs a text attention mechanism focusing on classification graphs to enhance the discrimination of classification graphs; the present invention combines the perspectives of classification graph input features and classification graphs to propose a hybrid mechanism of text attention focusing on features and classification graphs to simultaneously enhance the discrimination of classification graph input features and classification graphs, and achieves better performance; (2) The hybrid text attention mechanism of focusing on features and classification graphs proposed in the present invention solves the shortcomings of the SSTDTAM text attention mechanism in the prior art, that is, when SSTDTAM is applied to the text detector EAST, its performance is lower than the performance of DL applied alone to the EAST classification graph branch. This situation shows that SSTDTAM cannot bring performance improvement; the performance of the hybrid text attention mechanism proposed in the present invention applied to EAST can significantly exceed the performance of DL applied alone to the EAST classification graph branch, and can significantly suppress false positive detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the process steps of a text detection method focusing on features and classification graphs; Figure 2 Embed the EAST scene text detection model for FSFTAHM; Figure 3Schematic diagram of the text attention mechanism architecture for focusing features; Figure 4 Schematic diagram of the text attention mechanism architecture for focusing on classification images; Figure 5 This is a schematic diagram of the text attention hybrid mechanism architecture; Figure 6 This is a schematic diagram of the architecture of the convolutional feature fusion module; Figure 7 Schematic diagram for qualitative comparison of the effects of FFTAM; Figure 8 Schematic diagram for qualitative comparison of the effects of SFTAM; Figure 9 Schematic diagram for qualitative comparison of the effects of FSFTAHM. DETAILED DESCRIPTION

[0019] Example 1: A text detection method of this embodiment focusing on features and classification diagrams, such as Figure 1 As shown, the following steps are included: Step S1: input the original image and extract feature images with different resolutions through the residual convolutional neural network; Step S2: fusing the feature images using upsampling and concatenation operations to obtain the original classification map input features; Step S3: Introduce the text attention hybrid mechanism, such as Figure 2 As shown in the figure, the text attention hybrid mechanism is embedded in the text detector. In the process of text detector detecting text, the original classification map input feature is upsampled by the text attention hybrid mechanism to obtain the attention distribution map input feature; the text attention hybrid mechanism performs a single-channel convolution operation on the attention distribution map input feature to obtain the attention distribution map, and the text attention hybrid mechanism performs a downsampling operation and an exponential power operation on the attention distribution map input feature and then fuses the original classification map input feature to obtain the mixed classification map input feature; the architecture of the text attention hybrid mechanism is shown in the figure. Figure 5 As shown; Step S4: performing single-channel convolution on the mixed classification map input features to obtain a single-channel classification map, performing multi-channel convolution on the classification map input features to obtain a minimum prediction rectangle, wherein the minimum prediction rectangle includes a direction boundary and a rotation angle; downsampling the attention distribution map, and fusing the single-channel classification map and the attention distribution map downsampling results to obtain a mixed classification map; Step S5: perform threshold filtering on the pixels in the mixed classification map, retain the pixels in the mixed classification map that are greater than the classification map threshold as reference pixels, calculate the reference direction boundary and reference rotation angle of the reference pixel based on the minimum prediction rectangular box, and calculate the minimum prediction rectangular box coordinates corresponding to the remaining pixels in the mixed classification map based on the reference direction boundary and reference rotation angle of the reference pixel as the coordinates of the text.

[0020] Furthermore, the step S3 specifically includes the following steps: Step 3.1: Perform a linear double upsampling operation on the classification map input features to obtain the attention distribution map input features; Step 3.2: Perform a single-channel 1×1 convolution operation on the attention distribution map feature to obtain a feature map, and use the sigmoid activation function to convert the feature map into an attention distribution map; Step 3.3: Downsample the attention distribution map features, and multiply and fuse the classification map input features with the downsampled results of the attention distribution map features to obtain the mixed classification map input features.

[0021] Furthermore, the step 3.3 specifically includes the following steps: Step 3.3.1. Downsample the attention map features by a factor of four to match the resolution of the classification map input features. Step 3.3.2, performing an exponential power operation on the result of the four-fold downsampling operation in step 3.3.1; Step 3.3.3: Multiply and fuse the classification map input features with the result of the exponential power operation in step 3.3.2 to obtain the mixed classification map input features.

[0022] Furthermore, the step S4 specifically includes the following steps: Step 4.1: Perform a single-channel 1×1 convolution operation on the mixed classification map input features to obtain a single-channel classification map, and use the single-channel classification map to determine the text and background in the original image; Step 4.2: Perform a 5-channel 1×1 convolution operation on the mixed classification map input features to obtain the minimum predicted rectangle covering the text area, and use the minimum predicted rectangle to represent the coordinates of the text; Step 4.3: Perform a four-fold downsampling operation on the attention distribution map, and multiply and fuse the single-channel classification map with the downsampling result of the attention distribution map to obtain a mixed classification map.

[0023] Furthermore, the step 4.2 specifically includes: Step 4.2.1: The text detector uses the minAreaRect function to generate the minimum predicted rectangle covering the text area. Step 4.2.2: Calculate the distance between each pixel in the minimum prediction rectangle and the four edges of the minimum prediction rectangle. Step 4.2.3: Calculate the rotation angle of the minimum prediction rectangle relative to the horizontal axis.

[0024] Furthermore, the IOU loss function is used to characterize the distance from the pixel to the four boundaries of the minimum prediction rectangle. The IOU loss function is specifically: ; in: Represents the IOU loss function; Represents the predicted rectangular box; Represents the true value rectangle; Represents the IOU value between the predicted rectangle and the true rectangle.

[0025] Predicted rectangle and the true value rectangle The width of the intersection and height They are: ; ; in: 、 、 、 Represents the pixel points The ground truth distance to the upper, right, lower, and left boundaries of the minimum predicted rectangle.

[0026] Predicted rectangle and the true value rectangle The intersection area is: ; Predicted rectangle and the true value rectangle The area of the union of is: ; Furthermore, a rotation angle loss function is used to characterize the rotation angle of the minimum prediction rectangle relative to the horizontal axis. The rotation angle loss function is specifically: ; in: represents the rotation angle loss function; represents the predicted value of the rotation angle; The true value representing the rotation angle.

[0027] The total loss of the minimum predicted rectangle is: ; in: Represents the total loss function of the minimum predicted rectangle; is the weight parameter, set to 20.

[0028] The loss of representing the text coordinates through the minimum predicted rectangular box is: ; in: Represents the loss function that represents the text coordinates through the minimum predicted rectangle; is the weight parameter, set to 1; Represents the loss CBCE function for a single-channel classification map.

[0029] Further, such as Figure 6 As shown, the residual convolutional neural network includes a ResNet50 network, and the ResNet50 network includes a first convolution module Res1, a second convolution module Res2, a third convolution module Res3, and a fourth convolution module Res4. The resolution of the feature image extracted by the first convolution module is one-fourth of the resolution of the original image, the resolution of the feature image extracted by the second convolution module is one-eighth of the resolution of the original image, the resolution of the feature image extracted by the third convolution module is one-sixteenth of the resolution of the original image, and the resolution of the feature image extracted by the fourth convolution module is one-thirty-second of the resolution of the original image.

[0030] Furthermore, the pixel values of the pixels in the attention distribution map are as follows: 0≤k≤1, where k represents the pixel value of the pixel in the attention distribution map.

[0031] A text detection system focusing on features and classification maps includes a feature extraction module, a convolutional feature fusion module, an attention mechanism module, a detection output module, and a final processing module. The feature extraction module is used to introduce a residual convolutional neural network to extract feature images with different resolutions; the convolutional feature fusion module is used to fuse the feature images to obtain the original classification map input features; the attention mechanism module includes a text detector, and the attention mechanism module is used to embed a text attention hybrid mechanism in the text detector; the detection output module includes a classification map branch module and a rotation rectangle branch module, the classification map branch module is used to perform single-channel convolution on the mixed classification map input features to obtain a single-channel classification map, and the rotation rectangle branch module is used to perform multi-channel convolution on the classification map input features to obtain a minimum prediction rectangle; the final processing module calculates the rotated rectangle coordinates of the text in the minimum prediction rectangle based on the direction boundary and rotation angle of the minimum prediction rectangle, and obtains the coordinates of the text after maximum suppression of the rotated rectangle coordinates.

[0032] Example 2: This embodiment discloses a text detection method focusing on features and classification graphs, which is further optimized based on the first embodiment and includes the following steps: Step S1, input the original image, and extract feature images with different resolutions through the residual convolutional neural network; specifically, the original image is input into the ResNet50 network, the ResNet50 network includes a convolution layer, a pooling layer and a batch normalization layer, and the ResNet50 network uses the first convolution module Res1, the second convolution module Res2, the third convolution module Res3 and the fourth convolution module Res4 to extract four features in the original image. The resolutions of the four features are one-quarter, one-eighth, one-sixteenth and one-thirty-second of the resolution of the input original image, respectively.

[0033] Step S2: Use upsampling and concatenation operations to fuse the feature images to obtain the original classification map input features; for the four features obtained in step S1, use upsampling and concatenation operations to gradually fuse the four features, from the features with small resolution to the features with large resolution, and obtain the classification map input features after fusion.

[0034] Step S3: Introduce a text attention hybrid mechanism and embed the text attention hybrid mechanism into the text detector. In the process of detecting text in the text detector, the original classification map input features are upsampled by the text attention hybrid mechanism to obtain the attention distribution map input features; the text attention hybrid mechanism performs a single-channel convolution operation on the attention distribution map input features to obtain the attention distribution map; the text attention hybrid mechanism performs a downsampling operation and an exponential power operation on the attention distribution map input features and then fuses the original classification map input features to obtain the mixed classification map input features.

[0035] Specifically: Construct and introduce a text attention hybrid mechanism to simultaneously enhance the classification map and the classification map input features to distinguish text and background. The architecture of the text attention hybrid mechanism is as follows Figure 5 As shown, Unpool represents a 2x upsampling operation, and DownSample represents a downsampling operation. Assume that the height of the input image is , with a width of ,but Input features for the classification map of the input image, whose resolution is one quarter of the original image. The output attention distribution map is recorded as , the attention distribution map is a The third-order tensor of , the attention distribution map is calculated according to the following formula: ; in: Represents the attention distribution map input features; Indicates a two-fold upsampling operation; Represents the classification map input features.

[0036] The text attention hybrid mechanism performs a single-channel convolution operation and an activation function operation on the input features of the attention distribution map to obtain the attention distribution map A. The calculation formula is as follows: ; in: Represents the feature map of a single channel; Indicates a single-channel 1×1 convolution operation on the attention distribution map input feature.

[0037] ; Where: A represents the attention distribution map; Indicates that the activation function is operated on the feature map of a single channel.

[0038] Each pixel value in the attention distribution map A is between 0 and 1, indicating the probability that the corresponding image pixel is classified as text. The text attention hybrid mechanism uses DL to train the attention distribution map.

[0039] Text attention hybrid mechanism for attention distribution map input features Perform a 4x downsampling operation to match the classification map input features The resolution is then exponentially operated and compared with the classification map input feature Perform multiplication and fusion operations to obtain mixed classification map input features. The mixed classification map input features are calculated according to the following formula: ; in: Represents the input features of the attention distribution map after downsampling operation; Indicates downsampling of the input features of the attention distribution map; express The exponential power of Used to adjust the strength of attention effect, parameter The larger it is, the stronger the effect of attention is, and vice versa; Categorical graph representing a mixture of input features.

[0040] Connect a convolution kernel to the mixed classification map input feature as , a multi-channel convolution with 5 channels forms a rotating rectangular branch, which outputs four distance maps and one angle map to represent the coordinates of the text. A convolution kernel is connected to the mixed classification map input feature as , a single-channel convolution with a channel number of 1 constitutes a classification map branch, which is used to output a single-channel classification map. In order to further enhance the text detector EAST's ability to distinguish text and background, the text attention hybrid mechanism uses the attention distribution map to enhance the distinguishability of the classification map. Specifically, the text attention hybrid mechanism performs a four-fold downsampling operation on the attention distribution map to match the resolution of the single-channel classification map, and then multiplies the downsampled attention distribution map with the single-channel classification map to obtain a hybrid classification map. The hybrid classification map is calculated according to the following formula: ; in: Represents the attention distribution map after downsampling; Indicates downsampling operation on the attention distribution map A; represents a mixed classification graph, whose loss function is CBCE; Represents a single-channel classification map.

[0041] The text attention hybrid mechanism uses the attention distribution map input features to enhance the discriminability of the classification map input features, and uses the attention distribution map to enhance the discriminability of the classification map. Because the text attention hybrid mechanism simultaneously enhances the classification map input features and the classification map's ability to distinguish text from background, the performance of the text attention hybrid mechanism applied to the text detector EAST exceeds the performance of DL applied alone to the classification map branch of the text detector EAST. Figure 3 As shown, the text attention hybrid mechanism is embedded in the scene detection model of the text detector EAST.

[0042] The rest of this embodiment is the same as that of embodiment 1, so it will not be described again.

[0043] Example 3: This embodiment discloses a text detection method based on focused features and classification graphs. Based on the first embodiment, a feature-focused text attention mechanism (FFTAM) is embedded in the text detector. The architecture is as follows: Figure 3 As shown, Unpool represents a 2x upsampling operation, and DownSample represents a downsampling operation.

[0044] Taking the classification map input features as input, an attention distribution map is learned through the text attention mechanism of the focused features to represent the rough spatial area of the text. Then, the attention distribution map input features are multiplied and fused with the classification map input features to output the classification map input features of the focused features. The classification map input features of the focused features are then connected to the classification map branch and the rotated rectangle branch respectively.

[0045] Assume that the height of the input image is , with a width of ,but Input features for the classification map of the input image, whose resolution is one quarter of the original image. The output attention distribution map is recorded as , the attention distribution map is a The third-order tensor of , the attention distribution map is calculated according to the following formula: ; in: Represents the attention distribution map input features; Indicates a two-fold upsampling operation; Represents the classification map input features.

[0046] The text attention hybrid mechanism performs a single-channel convolution operation and an activation function operation on the input features of the attention distribution map to obtain the attention distribution map A. The calculation formula is as follows: ; in: Represents the feature map of a single channel; Indicates a single-channel 1×1 convolution operation on the attention distribution map input feature.

[0047] ; Where: A represents the attention distribution map; Indicates that the activation function is operated on the feature map of a single channel.

[0048] Each pixel value in the attention distribution map A is between 0 and 1, indicating the probability that the corresponding image pixel is classified as text. The text attention hybrid mechanism uses the loss function DL to train the attention distribution map.

[0049] After outputting the attention distribution map A, the text attention mechanism of the focused feature uses the loss function DL to train the attention distribution map. The text attention mechanism of the focused feature inputs the attention distribution map into the feature Input features with classification graph Perform multiplication and fusion. Calculate according to the following formula: ; in: Represents the input features of the attention distribution map after downsampling operation; Indicates downsampling of the input features of the attention distribution map; Classification map representing focused features of input features.

[0050] Therefore, the classification map input features of the focused features have a strong discriminative ability to distinguish text areas from background areas.

[0051] The input feature of the classification map of the focused feature is connected to a convolution kernel as , a multi-channel convolution with 5 channels forms a rotating rectangular branch, which outputs four distance maps and one angle map to represent the coordinates of the text. A convolution kernel is connected to the mixed classification map input feature as , a single-channel convolution with a channel number of 1 constitutes a classification map branch, which is used to output a single-channel classification map.

[0052] When the feature-focused text attention mechanism (FFTAM) is embedded in the text detector EAST, the loss function CBCE of the single-channel classification map is: ; in: Represents the loss function of a single-channel classification map; The predicted values of the classification map representing text and non-text; The true value of the classification map representing text and non-text; Indicates the balance factor between positive and negative samples.

[0053] Because the number of text pixels is much smaller than the number of background pixels, the parameter is very close to 1, and the parameter The value is extremely close to 0, resulting in insufficient negative sample training. This makes the classification map and the input features of the classification map corresponding to the loss function CBCE weakly distinguish between text and background. When the classification threshold of the classification map is low, the recall rate of the text detector EAST is high and the precision is low. When the classification threshold of the classification map is high, the recall rate of the text detector EAST is low and the precision is high.

[0054] The loss function for training the attention distribution map is DL, denoted as : ; in: and Represents the true value of the attention distribution map and predicted values Corresponding pixels The value of attention distribution map; the true value It is a pixel-level binary image, where the pixel value inside the text area is 1 and the pixel value in the background area is 0. Because the positive sample loss weight and negative sample loss weight of the loss function DL are both 1, the negative samples of the attention distribution map corresponding to the loss function DL can be fully trained. Therefore, the attention distribution map and the attention distribution map input features have strong discrimination between text and background.

[0055] The rest of this embodiment is the same as that of embodiment 1, so it will not be described again.

[0056] Example 4: This embodiment discloses a method for detecting text by focusing on features and classification graphs. Based on the first embodiment, a text attention mechanism (SFTAM) focusing on classification graphs is embedded in the text detector. The architecture is as follows: Figure 4 As shown, Unpool represents a 2x upsampling operation, and DownSample represents a downsampling operation.

[0057] Assume that the height of the input image is , with a width of ,but Input features for the classification map of the input image, whose resolution is one quarter of the original image. The output attention distribution map is denoted as A, and the attention distribution map is a The third-order tensor of , the attention distribution map is calculated according to the following formula: ; in: Represents the attention distribution map input features; Indicates a two-fold upsampling operation; Represents the classification map input features.

[0058] The text attention hybrid mechanism performs a single-channel convolution operation and an activation function operation on the input features of the attention distribution map to obtain the attention distribution map A. The calculation formula is as follows: ; in: Represents the feature map of a single channel; Indicates a single-channel 1×1 convolution operation on the attention distribution map input feature.

[0059] ; Where: A represents the attention distribution map; Indicates that the activation function is operated on the feature map of a single channel.

[0060] Each pixel value in the attention distribution map A is between 0 and 1, indicating the probability that the corresponding image pixel is classified as text. The text attention hybrid mechanism uses the loss function DL to train the attention distribution map.

[0061] After outputting the attention distribution map A, the text attention mechanism focusing on the classification map uses the loss function DL to train the attention distribution map. The text attention mechanism focusing on the classification map inputs the classification map into the feature map. Connect a convolution kernel to , a single-channel convolutional layer with a channel number of 1 constitutes the classification map branch and outputs a single-channel classification map SM. The text attention mechanism focusing on the classification map directly multiplies the attention distribution map with the single-channel classification map. Specifically, the text attention mechanism focusing on the classification map first downsamples the attention distribution map by a factor of 4, and then directly multiplies the downsampling result with the single-channel classification map, calculated according to the following formula: ; in: Represents the attention distribution map after downsampling operation; Indicates downsampling operation on the attention distribution map A; Represents a single-channel classification map; Classification map input features representing focused classification maps.

[0062] Connect a convolution kernel to the classification map input feature SMIF , a multi-channel convolution with 5 channels forms a rotating rectangular branch, which outputs four distance maps and one angle map to represent the coordinates of the text. A convolution kernel is connected to the input feature SMIF of the classification map as , a single-channel convolution with a channel number of 1 constitutes a classification map branch, which is used to output a single-channel classification map.

[0063] The rest of this embodiment is the same as that of embodiment 1, so it will not be described again.

[0064] Example 5: The three evaluation indicators of the text detection method adopted in the present invention are: recall rate (Recall), precision (Precision) and F-score value. The recall rate represents the ratio of the successfully detected true value frame to all the true value frames. The precision measures the ratio of the successfully detected true value frame to all the detection frames. In the performance evaluation, if the IOU value between the detection frame and the true value frame is greater than 0.5, the detection frame is the correct detection frame, otherwise it is an incorrect detection frame. The F-score value is the harmonic mean of the recall rate and the precision. When comparing the performance of scene text detection methods, the larger the F-score value of the method, the better its performance.

[0065] The datasets used to evaluate the text detection method of the present invention are three scene text detection datasets, namely: ICDAR2015, MSRA-TD500 and CASIA-10K.

[0066] In the subsequent experimental charts, EAST+CBCE represents the version of the EAST framework that re-implements the text detector, that is, the baseline model, which uses CBCE as the loss function of the classification map; EAST+SSTDTAM is a model that embeds SSTDTAM into EAST; EAST+FFTAM is a model that embeds FFTAM into EAST; EAST+SFTAM represents a model that embeds SFTAM into EAST; EAST+FSFTAHM represents a model that embeds FSFTAM into EAST; EAST+DL represents the EAST model with DL as the loss function of the classification map.

[0067] The detection performance of the above-mentioned various text detection methods on the ICDAR2015 dataset is shown in Table 1: Table 1 The detection performance of various text detection methods on the MSRA-TD500 dataset is shown in Table 2: Table 2 The detection performance of various text detection methods on the CASIA-10K dataset is shown in Table 3. * ” indicates multi-scale testing.

[0068] Table 3 Table 1 shows the detection performance of different algorithms on the ICDARA2015 dataset. Table 1 shows that the FSFTAHM proposed in this chapter effectively improves the detection performance of EAST. Compared with the detection performance of EAST+CBCE, the detection performance of EAST+SSTDTAM achieves an F-score of 0.8153, a performance gain of 1.73%, with a performance gain of 2.7% in recall and 0.54% in precision. Compared with the detection performance of EAST+CBCE, the detection performance of EAST+FSFTAHM achieves an F-score of 0.8249, a performance gain of 2.69%, with a performance gain of 2.31% in recall and 3.16% in precision. Table 1 shows that the detection performance of EAST+FSFTAHM significantly exceeds that of EAST+SSTDTAM, with an F-score performance gain of 0.96%. The exponential parameter M of EAST+FSFTAHM is 2.

[0069] The detection performance of SSTDTAM and FSFTAHM on the MSRA-TD500 dataset is shown in Table 2. SSTDTAM increases the F-score of EAST+CBCE by 1.71%, with a 2.06% increase in recall and a 1.22% increase in precision. FSFTAHM increases the F-score of EAST+CBCE by 2.83%, with a 1.72% increase in recall and a 4.36% increase in precision. The detection performance of EAST+FSFTAHM significantly exceeds that of EAST+SSTDTAM, with a 1.12% increase in F-score and a 3.14% increase in precision, with a slight decrease in recall. The exponential parameter M of EAST+FSFTAHM is 1.

[0070] The detection results on the CASIA-10K dataset are shown in Table 3. Compared with the detection performance of EAST+CBCE, EAST+SSTDTAM achieved an F-score of 0.7162, a performance gain of 1.1%, with a performance gain of 1.54% in recall. Compared with the detection performance of EAST+CBCE, EAST+FSFTAHM achieved an F-score of 0.7366, a performance gain of 3.14%, with a performance gain of 3.89% in recall and 1.99% in precision. The detection performance of EAST+FSFTAHM significantly surpassed that of EAST+SSTDTAM, with an F-score gain of 2.04%, an increase in recall of 2.35%, and an increase in precision of 1.57%. The exponential parameter M of EAST+FSFTAHM was 1.

[0071] The detection performance of SSTDTAM, FFTAM, SFTAM and FSFTAHM on the ICDAR2015 dataset is shown in Table 4.

[0072] Table 4 The detection performance of SSTDTAM, FFTAM, SFTAM and FSFTAHM on the MSRA-TD500 dataset is shown in Table 5.

[0073] Table 5 The detection performance of SSTDTAM, FFTAM, SFTAM and FSFTAHM on the CASIA-10K dataset is shown in Table 6.

[0074] Table 6 As shown in Tables 1, 2, and 3, FSFTAHM significantly improves the recall and precision of EAST+CBCE. This indicates that FSFTAHM significantly enhances the discriminability of text detectors between text and background, enabling them to recall more text while suppressing false positive background detection. Because SSTDTAM only enhances the discriminability of the input features of the classification map, the detection performance of EAST+SSTDTAM is lower than that of EAST+DL on the ICDAR2015 and CASIA-10K datasets. On the ICDAR2015 and CASIA-10K datasets, the F-score values of EAST+SSTDTAM are 0.8153 and 0.7162, respectively, while the F-score values of EAST+DL are 0.8162 and 0.7273, respectively, as shown in Tables 4 and 6. However, SSTDTAM's detection performance on MSRA-TD500 exceeds that of EAST+DL. The F-score of EAST+SSTDTAM is 0.7922, while that of EAST+DL is 0.7855, as shown in Table 5. Therefore, SSTDTAM still has a certain role, but its role is limited. Because FSFTAHM enhances the discriminability of both the classification map and the input features of the classification map, the detection performance of FSFTAHM exceeds that of EAST+DL. On the ICDAR2015, MSRA-TD500, and CASIA-10K datasets, the F-scores of EAST+FSFTAHM are 0.8249, 0.8034, and 0.7366, respectively, while the F-scores of EAST+DL are 0.8162, 0.7855, and 0.7273, respectively, as shown in Tables 4, 5, and 6.

[0075] Because FFTAM only enhances the distinguishability of the classification map input features for text and background, the performance of EAST+FFTAM cannot significantly exceed or cannot exceed the performance of EAST+DL on the ICDAR2015 and CASIS-10K datasets. The F-score values of EAST+FFTAM on ICDAR2015, MSRA-TD500 and CASIA-10K are 0.8164, 0.7937 and 0.7243, respectively, which significantly exceeds the performance of EAST+CBCE. The F-score values of EAST+CBCE on the three datasets are 0.798, 0.7751 and 0.7052, respectively. Similarly, because SFTAM only enhances the discrimination between text and background in the classification map, EAST+SFTAM performs worse than EAST+DL on the ICDAR2015 and CASIS-10K datasets. EAST+SFTAM achieves F-scores of 0.8114, 0.794, and 0.7205 on ICDAR2015, MSRA-TD500, and CASIA-10K, respectively, significantly exceeding the performance of EAST+CBCE, as shown in Tables 4, 5, and 6. Building on FFTAM and SFTAM, FSFTAHM combines the structures and functions of the first two mechanisms to achieve even superior performance.

[0076] In order to qualitatively analyze the effects of FFTAM, SFTAM and FSFTAHM, this paper embeds FFTAM, SFTAM and FSFTAHM respectively on the benchmark model EAST+CBCE. Because the attention mechanism in this chapter is all experimented on the EAST text detector, a qualitative comparative analysis is performed with the original EAST text detector model EAST+CBCE. Figure 9 The effects of FFTAM, SFTAM and FSFTAHM are shown. The FFTAM, SFTAM and FSFTAHM proposed in the present invention can help EAST achieve better detection results.

[0077] The output of the benchmark model EAST+CBCE Figure 7 (a) Detecting two background regions as text leads to false positive detections. The output of the model EAST+FFTAM is Figure 7 (c) The two background regions are removed, which shows that FFTAM can effectively reduce false positive detections. Figure 7 (b) From the baseline model EAST+CBCE, this subgraph does not detect another English character, and the model EAST+FFTAM outputs Figure 7 (d) The English character was recalled, which shows that FFTAM is able to recall more words.

[0078] Figure 8(a) Output by the baseline model EAST+CBCE, this subgraph does not have the text in the recall graph, and the model EAST+SFTAM outputs Figure 8 (c) Ability to detect Figure 8 (a) Missed text, which shows that SFTAM can effectively recall more text. The baseline model EAST+CBCE detects the grid background as text, resulting in false positive detections, such as Figure 8 (b) shows the EAST+SFTAM model. Figure 8 (d) The grid background can be removed, which shows that SFTAM can reduce false positive detections.

[0079] Figure 9 (a) is the detection result of the benchmark model EAST+CBCE. This sub-image does not detect two difficult characters with large inclinations. The detection results output by the EAST+FSFTAHM model are as follows: Figure 9 As shown in (c), this subgraph recalls Figure 9 (a) The text is missed. This shows that FSFTAHM can help the text detector EAST recall more challenging text. The baseline model EAST+CBCE is prone to detecting background areas as text, resulting in false positive detections, such as Figure 9 (b) The detection results output by the EAST+FSFTAHM model are shown as follows Figure 9 As shown in (d), the subgraph removes Figure 9 (b) The background area. This shows that FSFTAHM can help the scene text detector EAST remove the background and reduce false positive detection results.

[0080] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A text detection method focusing on features and classification graphs, characterized in that: The following steps are involved: Step S1: input the original image and extract feature images with different resolutions through the residual convolutional neural network; Step S2: fusing the feature images using upsampling and concatenation operations to obtain the original classification map input features; Step S3: Introduce a text attention hybrid mechanism and embed it into the text detector. During the text detection process, the text attention hybrid mechanism is used to upsample the original classification map input features to obtain the attention distribution map input features. The text attention hybrid mechanism performs a single-channel convolution operation on the attention distribution map input features to obtain the attention distribution map. The text attention hybrid mechanism performs downsampling and exponential power operations on the attention distribution map input features and then fuses them with the original classification map input features to obtain the mixed classification map input features. Step S4: performing single-channel convolution on the mixed classification map input features to obtain a single-channel classification map, performing multi-channel convolution on the classification map input features to obtain a minimum prediction rectangle, wherein the minimum prediction rectangle includes a direction boundary and a rotation angle; downsampling the attention distribution map, and fusing the single-channel classification map and the attention distribution map downsampling results to obtain a mixed classification map; Step S5: perform threshold filtering on the pixels in the mixed classification map, retain the pixels in the mixed classification map that are greater than the classification map threshold as reference pixels, calculate the reference direction boundary and reference rotation angle of the reference pixel based on the minimum prediction rectangular box, and calculate the minimum prediction rectangular box coordinates corresponding to the remaining pixels in the mixed classification map based on the reference direction boundary and reference rotation angle of the reference pixel as the coordinates of the text.

2. The method for detecting text by focusing on features and classification diagrams according to claim 1, wherein: The step S3 specifically includes the following steps: Step 3.1: Perform a linear double upsampling operation on the classification map input features to obtain the attention distribution map input features; Step 3.2: Perform a single-channel 1×1 convolution operation on the attention distribution map feature to obtain a feature map, and use the sigmoid activation function to convert the feature map into an attention distribution map; Step 3.3: Downsample the attention distribution map features, and multiply and fuse the classification map input features with the downsampled results of the attention distribution map features to obtain the mixed classification map input features.

3. The method for detecting text by focusing on features and classification diagrams according to claim 2, wherein: The step 3.3 specifically includes the following steps: Step 3.3.

1. Downsample the attention map features by a factor of four to match the resolution of the classification map input features. Step 3.3.2, performing an exponential power operation on the result of the four-fold downsampling operation in step 3.3.1; Step 3.3.3: Multiply and fuse the classification map input features with the result of the exponential power operation in step 3.3.2 to obtain the mixed classification map input features.

4. The method for detecting text by focusing on features and classification diagrams according to claim 3, wherein: The step S4 specifically includes the following steps: Step 4.1: Perform a single-channel 1×1 convolution operation on the mixed classification map input features to obtain a single-channel classification map, and use the single-channel classification map to determine the text and background in the original image; Step 4.2: Perform a 5-channel 1×1 convolution operation on the mixed classification map input features to obtain the minimum predicted rectangle covering the text area, and use the minimum predicted rectangle to represent the coordinates of the text; Step 4.3: Perform a four-fold downsampling operation on the attention distribution map, and multiply and fuse the single-channel classification map with the downsampling result of the attention distribution map to obtain a mixed classification map.

5. The method for detecting text by focusing on features and classification diagrams according to claim 4, wherein: The step 4.2 specifically includes: Step 4.2.1: The text detector uses the minAreaRect function to generate the minimum predicted rectangle covering the text area. Step 4.2.2: Calculate the distance between each pixel in the minimum prediction rectangle and the four edges of the minimum prediction rectangle. Step 4.2.3: Calculate the rotation angle of the minimum prediction rectangle relative to the horizontal axis.

6. The method for detecting text by focusing on features and classification diagrams according to claim 5, characterized in that: The IOU loss function is used to characterize the distance from the pixel to the four boundaries of the minimum prediction rectangle. The IOU loss function is specifically: ； in: Represents the IOU loss function; Represents the predicted rectangular box; Represents the true value rectangle; Represents the IOU value between the predicted rectangle and the true rectangle.

7. The method for detecting text by focusing on features and classification diagrams according to claim 5, wherein: The rotation angle loss function is used to characterize the rotation angle of the minimum prediction rectangle relative to the horizontal axis. The rotation angle loss function is specifically: ； in: represents the rotation angle loss function; represents the predicted value of the rotation angle; The true value representing the rotation angle.

8. A method for detecting text based on focus features and classification diagrams according to any one of claims 1 to 7, characterized in that: The residual convolutional neural network includes a ResNet50 network, and the ResNet50 network includes a first convolution module, a second convolution module, a third convolution module, and a fourth convolution module. The resolution of the feature image extracted by the first convolution module is one-fourth of the resolution of the original image, the resolution of the feature image extracted by the second convolution module is one-eighth of the resolution of the original image, the resolution of the feature image extracted by the third convolution module is one-sixteenth of the resolution of the original image, and the resolution of the feature image extracted by the fourth convolution module is one-thirty-second of the resolution of the original image.

9. A method for detecting text by focusing on features and classification diagrams according to any one of claims 1 to 7, characterized in that: The pixel values of the pixels in the attention distribution map are as follows: 0≤k≤1, where k represents the pixel value of the pixel in the attention distribution map.

10. A text detection system focusing on features and classification diagrams, used to implement the detection method according to any one of claims 1 to 9, characterized in that: It includes a feature extraction module, a convolutional feature fusion module, an attention mechanism module, a detection output module, and a final processing module. The feature extraction module is used to introduce a residual convolutional neural network to extract feature images with different resolutions; the convolutional feature fusion module is used to fuse the feature images to obtain the original classification map input features; The attention mechanism module includes a text detector, and the attention mechanism module is used to embed a text attention hybrid mechanism in the text detector; the detection output module includes a classification map branch module and a rotation rectangle branch module, the classification map branch module is used to perform single-channel convolution on the mixed classification map input features to obtain a single-channel classification map, and the rotation rectangle branch module is used to perform multi-channel convolution on the classification map input features to obtain a minimum prediction rectangular frame; the final processing module calculates the rotated rectangle coordinates of the text in the minimum prediction rectangular frame based on the direction boundary and rotation angle of the minimum prediction rectangular frame, and obtains the coordinates of the text after maximum suppression of the rotated rectangle coordinates.

Images