Differential binary text detection method and system based on background suppression

Abstract

The invention relates to the technical field of text detection, in particular to a differential binaryzation text detection method and system based on background suppression, the method constructs an improved DBNet text detection model, the improved DBNet text detection model adds prediction of a background probability graph at a feature graph F of text detection of the DBNet model, and after the result is negated, the background probability graph is added to the feature graph F of the text detection of the DBNet model, and then the DBNet model is obtained. And performing element-by-element product operation on the predicted text probability graph and the obtained predicted text probability graph to generate a background-suppressed text probability graph, and performing differential binarization operation on the background-suppressed text probability graph and the text threshold graph to obtain a corresponding approximate binary graph. According to the method, the background learning branch is introduced into the DBNet text detection network model to predict the background image, and the text probability graph of background suppression is obtained by combining the predicted text probability graph, so that background region interference is suppressed to obtain the corresponding approximate binary image, and the robustness and precision of text detection in a complex scene are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text detection, and in particular to a differentiable binarization text detection method and system based on background suppression. Background Art

[0002] Natural scene text detection is an important research direction in the field of computer vision. It aims to accurately locate and extract text regions of arbitrary shapes from complex background images. Since the text in the scene contains high semantic value, its application covers multiple practical scenarios and has a wide range of applications in scene understanding, document analysis, and intelligent information processing.

[0003] In recent years, text detection methods based on deep learning have made significant progress. EAST[Zhou X, Yao C, Wen H, et al. East: an efficient and accurate scene text detector. Proceedings of the Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, F, 2017[C].] improves the speed and efficiency of detection by directly outputting the location of the text region by regressing the offset of the boundary or vertex to the current point, and can accurately locate the inclined or arbitrary direction text. LOMO[Zhang C, Liang B, Huang Z, et al. Look more than once: An accurate detector for text of arbitrary shapes. Proceedings of the Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, F, 2019[C].] solves the problem of incomplete detection of long text and inaccurate positioning of curved text by using a direct regressor, an iterative optimization module, a shape expression module, and an arbitrary shape expression. Pixellink[Deng D, Liu H, Li X, et al. Pixellink: Detecting scene text via instance segmentation. Proceedings of the AAAI conference on artificial intelligence, F, 2018[C].] simplifies the process and improves the detection accuracy of small text regions by predicting the probability of each pixel belonging to text and the connection state with its field pixels to generate the boundary contour.TextSnake[LONG S,RUAN J,ZHANG W,et al.Textsnake:A flexible representation for detecting text of arbitrary shapes;proceedings of theProceedings of the European conference on computer vision(ECCV),F,2018[C].] adopts a flexible and efficient way to describe and detect text regions, which can well represent horizontal, multi-directional, curved and other multi-shaped texts. DBNet[Liao M,Wan Z,Yao C,et al.Real-time scene text detection withdifferentiable binarization[C] / / Proceedings ofthe AAAI conference onartificial intelligence.2020,34(07):11474-11481.] has performed well on multiple public datasets due to its efficient segmentation capability and real-time performance, becoming the current mainstream text detection algorithm.

[0004] Although DBNet[Liao M, Wan Z, Yao C, et al. Real-time scene text detection with differentiable binarization[C] / / Proceedings of the AAAI conference on artificial intelligence. 2020, 34(07): 11474-11481.] has achieved good detection results in public datasets, text images in natural scenes often contain complex and diverse background areas, which poses a major challenge to the accuracy of text detection. DBNet is significantly affected by background interference in complex scenes, resulting in false detection or missed detection, thereby reducing detection accuracy. Summary of the Invention

[0005] Purpose of the invention: To address the problem in the prior art that the DBNet model has poor detection effect under complex backgrounds, the present invention provides a differentiable binarization text detection method based on background suppression. The present invention also discloses a differentiable binarization text detection system based on background suppression.

[0006] Technical solution: In a first aspect, the present invention provides a differentiable binarization text detection method based on background suppression, the method comprising:

[0007] Obtain a certain number of text images to form sample data, and divide the sample data into training set and test set according to a certain ratio;

[0008] An improved DBNet text detection model is constructed, and the training set is input into the improved DBNet text detection model for model training to obtain an approximate binary image; the improved DBNet text detection model adds a prediction of a background probability map to a feature map F of text detection of the DBNet model, negates the result, and performs an element-by-element product operation with the obtained predicted text probability map to generate a background-suppressed text probability map, and the background-suppressed text probability map is subjected to a differentiable binarization operation with a text threshold map to obtain a corresponding approximate binary image;

[0009] The improved DBNet text detection model is tested using a test set, thereby generating a text detection result based on the obtained approximate binary image.

[0010] Further, including:

[0011] The background probability map is predicted using a background probability map prediction model, which is expressed as:

[0012] I BP =deconv1(deconv1(WDconv(F)));

[0013] in, is the predicted background probability map, deconv1(·) represents the deconvolution operation with a convolution kernel of k2×k2 and a step size of 2, and WDconv(·) represents the weighted deformable convolution operation with a convolution kernel of k3×k3 and a step size of 1.

[0014] Further, including:

[0015] The operation process of the weighted deformable convolution operation WDconv(·) includes:

[0016] Two additional convolutional layers are used to predict the offset of the original sampling points and the region of interest metric in the feature map F of the input text image respectively;

[0017] Dynamically adjust the sampling position of the convolution kernel according to the predicted offset, and weight these sampling points in combination with the region of interest measurement factor;

[0018] The convolution kernel is convolved with the feature map F of the input text image according to the adjusted sampling points to obtain the enhanced output feature map Y.

[0019] Further, including:

[0020] The convolution operation is performed on the convolution kernel and the feature map F of the input text image according to the adjusted sampling points, which is expressed as:

[0021]

[0022] Among them, F represents the feature map of the input text image, Y represents the enhanced output feature map, and x represents the current arbitrary position in F. Represents the number of convolution points in the convolution kernel, c k Represents the value corresponding to the kth convolution point in the convolution kernel, x k Represents the offset of the kth convolution point relative to the center of the convolution kernel, Vx k Represents the predicted offset of the k-th sampling point relative to the k-th convolution point, w k ∈[0,1] represents F(x+x k +Vx k ) is the region of interest measurement factor, namely: F(x+x k +Vx k ) belongs to the region of interest.

[0023] Further, including:

[0024] The text probability map is predicted using a text probability map prediction model, which is expressed as:

[0025] I TP =deconv2(deconv2(conv2(F)));

[0026] in, is the predicted text probability map, deconv2(·) represents the deconvolution operation with a convolution kernel of k4×k4 and a step size of 2, and conv2(·) represents the convolution operation with a convolution kernel of k5×k5 and a step size of 1.

[0027] Further, including:

[0028] The improved DBNet text detection model also includes a predicted text threshold map, and the corresponding prediction model is expressed as:

[0029] I T =deconv3(deconv3(conv3(F)));

[0030] in, is the predicted text threshold map, deconv3(·) represents the deconvolution operation with a convolution kernel of k6×k6 and a step size of 2, and conv3(·) represents the convolution operation with a convolution kernel of k7×k7 and a step size of 1.

[0031] Further comprising:

[0032] Training the background probability map prediction model comprises:

[0033] A background ground truth map is collected as a supervision signal and input into the background probability map prediction model, a background probability map prediction model loss function is used to measure the difference between the predicted output and the expected output of the model, and then an optimization direction is found, the background ground truth map is obtained by negating the text ground truth map, and the text ground truth map is obtained by shrinking the manually annotated text box.

[0034] Further comprising:

[0035] The background probability map prediction model loss function is used to measure the difference between the predicted output and the expected output of the model, and the loss function is represented as:

[0036]

[0037] Wherein, L b represents the loss function corresponding to the background probability map prediction model, N=HxW represents the number of pixel points in the current predicted background probability map, x i represents the gray value of the i-th pixel point in the current predicted background probability map, y i represents the gray value of the i-th pixel point in the background ground truth map, and s is a small constant value to prevent the denominator from being zero.

[0038] Further comprising:

[0039] The training set is input into the improved DBNet text detection model for model training, comprising: using a constructed overall loss function to measure the difference between the predicted output and the expected output of the improved DBNet text detection model, wherein the overall loss function is represented as:

[0040] L=β1L b +β2L p +β3L ab +β4L t ;

[0041] Wherein, L p represents the loss function corresponding to the text probability map prediction model, L ab represents the loss function corresponding to the predicted approximate binary map, L t represents the loss function corresponding to the text threshold value map prediction model, and β1, β2, β3, β4 represent the weight coefficients of L b , L p , L ab , L t , respectively.

[0042] In another aspect, the present application provides a differentiable binarization text detection system based on background suppression, comprising:

[0043] a collection module configured to obtain a certain amount of text image sample data, and split the sample data into a training set and a test set according to a certain proportion;

[0044] a model construction and training module configured to construct an improved DBNet text detection model, and input the training set into the improved DBNet text detection model for model training to obtain an approximate binary graph; the improved DBNet text detection model is constructed by adding a prediction of a background probability graph at a feature graph F of text detection of the DBNet model, taking the inverse of the result, and performing an element-by-element multiplication operation on the obtained predicted text probability graph, thereby generating a background-suppressed text probability graph; the background-suppressed text probability graph is subjected to a differentiable binarization operation with a text threshold graph to obtain a corresponding approximate binary graph;

[0045] a prediction module configured to test the improved DBNet text detection model using the test set, so as to generate a text detection result according to the obtained approximate binary graph.

[0046] Advantages: Compared with the prior art, the present application has the following advantages:

[0047] Firstly, the present application adds a prediction of a background probability graph in the DBNet text detection network model, combines the predicted background probability graph with the predicted text probability graph to obtain a background-suppressed text probability graph, and further obtains an approximate binary graph that effectively suppresses the interference of the background area, and the text detection result obtained based on the approximate binary graph has better robustness and precision in complex scenes.

[0048] Secondly, the present application constructs a background probability graph prediction model when predicting the background probability graph, which uses two deconvolutions to gradually increase the size of the background probability graph, so that the predicted background probability graph is consistent with the width and height size of the input text image, instead of using the traditional up-sampling interpolation operation to achieve this purpose. The convolution kernel parameters required by the deconvolution operation are adaptively determined through the training of the background probability graph prediction model, while the up-sampling interpolation operation expands the size of the background probability graph through a fixed mode set by hand. Therefore, using two deconvolutions to gradually increase the size of the background probability graph can effectively reduce the prediction error of the background probability graph. Similarly, the text probability graph prediction model and the text threshold graph prediction model also use two deconvolutions to gradually increase the size of the text probability graph and the text threshold graph, so they also have the same effect.

[0049] Finally, the present invention constructs a corresponding loss function when predicting the background probability map. The design idea of ​​this loss function is to measure the overall similarity between the background probability map and the background truth map by the scalar product of the two, which is consistent with the requirement that the overall outlines of the background probability map and the background truth map are consistent. The existing methods usually use binary cross entropy to construct the loss function, and the binary cross entropy focuses more on the similarity at the local level. Therefore, the loss function of the background probability map prediction model proposed in the present invention can improve the prediction accuracy of the background probability map. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of the differentiable binarization text detection method based on background suppression according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the structure of the improved DBNet text detection model described in an embodiment of the present invention;

[0052] Figure 3 Schematic diagram of the weighted deformable convolution operation structure according to an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the principle of obtaining a background truth map, a text truth map, and a boundary truth map according to an embodiment of the present invention;

[0054] Figure 5 This is a structural diagram of the differentiable binarization text detection system based on background suppression described in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention and not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] like Figure 1 As shown, the present invention provides a differentiable binarization text detection method based on background suppression, which includes the following steps:

[0057] S1 obtains a certain number of text images to form sample data, and divides the sample data into a training set and a test set according to a certain ratio.

[0058] S2 constructs an improved DBNet text detection model, inputs the training set into the improved DBNet text detection model for model training, and obtains an approximate binary graph; the improved DBNet text detection model is that a background probability graph prediction is added at a feature graph F of text detection of the DBNet model, the result is negated, and an element-by-element multiplication operation is performed on the obtained predicted text probability graph, thereby generating a background-suppressed text probability graph, the background-suppressed text probability graph is subjected to a differentiable binarization operation with a text threshold graph, and a corresponding approximate binary graph is obtained.

[0059] As shown in Figure 2 In this embodiment, the improved DBNet text detection model is improved based on the DBNet text detection model, mainly including adding a background prediction branch to predict a background probability graph and setting a corresponding prediction model and a loss function corresponding to training the prediction model, and setting a corresponding prediction model for text probability graph and text threshold graph prediction.

[0060] Here, the process of processing the text image is described in detail in combination with the improved DBNet model:

[0061] Feature extraction: first, a text image with a size of HxWx3 is input into a ResNet18 backbone network to obtain five levels of feature graphs, which are marked as Next, the feature graph used for text detection can be obtained by the following formula The specific relationship is as follows:

[0062] C5=P5 (1)

[0063] C4=P4+up2(C5) (2)

[0064] C3=P3+up2(C4) (3)

[0065] C2=P2+up2(C3) (4)

[0066] F=concat(up8(conv(C5)),up4(conv(C4)),up2(conv(C3)),conv(C2)) (5)

[0067] Wherein, H and W represent the height and width of the input image respectively, up2(·), up4(·), up8(·) represent 2, 4, 8 times up-sampling operations respectively, conv(·) represents a convolution operation with a convolution kernel size of k1xk1 and a step of 1, concat(·) represents a concatenation operation along the channel direction, is an intermediate feature graph used to generate F.

[0068] Differential binarization of background suppression: input the intermediate feature map F into the background probability map prediction model, the text probability map prediction model and the text threshold map prediction model respectively, set the corresponding supervision signals and loss functions, train the above models respectively, obtain the corresponding background probability map, text probability map and text threshold map respectively, take the inverse of the background probability map and perform element-wise multiplication operation with the text probability map to obtain the corresponding background-suppressed text probability map, and perform differential binarization operation on the background-suppressed text probability map and the text threshold map to obtain the corresponding approximate binary map I B .

[0069] Specifically, the background probability map prediction model in the embodiment is represented as:

[0070] I BP = deconv1(deconv1(WDconv(F))) (6)

[0071] Wherein, deconv1(·) represents an inverse convolution operation with a convolution kernel of k2xk2 and a step size of 2, and WDconv(·) represents a weighted deformable convolution operation with a convolution kernel of k3xk3 and a step size of 1.

[0072] In the embodiment, the prediction model uses two inverse convolutions to gradually increase the size of the background probability map, so that the predicted background probability map is consistent with the width and height size of the input text image, instead of using a traditional upsampling interpolation operation to achieve this purpose. The convolution kernel parameters required by the inverse convolution operation are adaptively determined through the training of the background probability map prediction model, while the upsampling interpolation operation expands the size of the background probability map through a fixed mode set by humans. Therefore, using two inverse convolutions to gradually increase the size of the background probability map can effectively reduce the prediction error of the background probability map.

[0073] The principle of WDconv(·) is as follows: as shown in Figure 3 , first, for the input feature map, WDconv(·) predicts the offset of the original sampling point and the RoI (Region of Interest) measurement factor through two additional convolution layers. Second, the sampling position of the convolution kernel is dynamically adjusted according to the predicted offset, and the dynamic sampling points are weighted combined with the RoI measurement factor. Finally, the adjusted sampling points are convolved with the input feature map to output an enhanced feature map.

[0074] The convolution operation of WDconv(·) can be expressed as:

[0075]

[0076] Among them, F represents the feature map of the input text image, Y represents the enhanced output feature map, and x represents the current arbitrary position in F. Represents the number of convolution points in the convolution kernel, c k Represents the value corresponding to the kth convolution point in the convolution kernel, x k Represents the offset of the kth convolution point relative to the center of the convolution kernel, Vx k Represents the predicted offset of the k-th sampling point relative to the k-th convolution point, w k ∈[0,1] represents F(x+x k +Vx k ) is the region of interest measurement factor, namely: F(x+x k +Vx k ) belongs to the region of interest.

[0077] In this embodiment, the convolution point is the sampling point in the convolution kernel. If the size of the convolution kernel is 3×3, each small square corresponds to a convolution point, and the small square in the middle is the center point of the convolution kernel. Then x k is the offset of another small square relative to the center point of the middle small square, and F is the feature map of the text image, in which a certain calculation point is called a sampling point, then Vx k Represents the predicted offset of the k-th sampling point in F relative to the k-th convolution point in the convolution kernel.

[0078] As can be seen from formula (7), by dynamically learning the offset, WDconv(·) can flexibly adjust the sampling position of the convolution kernel to adapt to the irregular shape of the background area. At the same time, the importance of the sampling points is weighted by the RoI metric factor, effectively highlighting key features. Therefore, by introducing WDconv(·), the background prediction branch can significantly improve the feature extraction ability of irregularly shaped background areas, thereby improving the prediction accuracy of the background probability map.

[0079] For text probability map The prediction model can be obtained by the following formula:

[0080] I TP =deconv2(deconv2(conv2(F))) (8)

[0081] Among them, deconv2(·) represents the deconvolution operation with a convolution kernel of k4×k4 and a stride of 2, and conv2(·) represents the convolution operation with a convolution kernel of k5×k5 and a stride of 1.

[0082] Text Threshold Map The prediction process and I TP The prediction process is similar to that of

[0083] I T =deconv3(deconv3(conv3(F))) (9)

[0084] in, is the predicted text threshold map, deconv3(·) represents the deconvolution operation with a convolution kernel of k6×k6 and a step size of 2, and conv3(·) represents the convolution operation with a convolution kernel of k7×k7 and a step size of 1.

[0085] In getting I BP , I TP , I T After that, the differentiable binarization of background suppression can be performed, and the process can be expressed by the following formula:

[0086]

[0087] in, represents the negated background probability map, Text probability map representing background suppression, represents an approximate binary image, represents the element-wise product operation, and m represents the modulation factor.

[0088] In model training, the supervisory signal required for text detection model training is first obtained. The present invention expands and contracts the manually annotated text boxes respectively, defines the text truth map with the contracted text boxes, and jointly defines the boundary truth map with the contracted and expanded text boxes.

[0089] In this embodiment, the loss functions corresponding to the background probability map prediction model, the text probability map prediction model, the text threshold map prediction model, and the approximate binary map prediction model are respectively expressed as L b , L p , L t , L ab .

[0090] Among them, L b The definition of is as follows:

[0091]

[0092] Where N = H × W represents I BP The number of pixels in x i Representative I BP The gray value of the i-th pixel, y i Represents the grayscale value of the i-th pixel in the background truth image. s is a small constant used to prevent the denominator from being 0.

[0093] The loss function L in this embodiment bThe design idea is to measure the overall similarity between the background probability map and the background truth map by the scalar product of the two, which is consistent with the requirement that the overall contours of the background probability map and the background truth map are consistent. The existing methods usually use binary cross entropy to construct the loss function, and the binary cross entropy focuses more on the similarity at the local level. Therefore, the loss function of the background probability map prediction model proposed in this invention can improve the prediction accuracy of the background probability map.

[0094] Among them, L p , L ab , L t The definition of can be found in DBNet[Liao M,WanZ,Yao C,et al.Real-timescene text detection with differentiable binarization[C] / / Proceedings of theAAAI conference on artificial intelligence.2020,34(07):11474-11481.].

[0095] In this embodiment, the overall loss function corresponding to the improved DBNet text detection model is expressed as:

[0096] L=β1L b +β2L p +β3L ab +β4L t (14)

[0097] Among them, L b Represents the loss function corresponding to the predicted background probability map, L p Represents the loss function corresponding to the predicted text probability map, L ab Represents the loss function corresponding to the predicted approximate binary image, L t Represents the loss function corresponding to the predicted text threshold map, β1, β2, β3, and β4 represent L b , L p , L ab , L t After obtaining the overall loss function L in this embodiment, the text detection model can be trained by backpropagation.

[0098] S3 uses the test set to test the improved DBNet text detection model to obtain the corresponding prediction results. B To generate a text box as the text detection result.

[0099] In order to verify the effectiveness of the improved model of the present invention, the following specific cases are given:

[0100] Step 1: Feature extraction. First, the text image with a size of 640×640×3 is fed into the ResNet18 backbone network to obtain 5 levels of feature maps, which are marked as Next, the feature map for text detection can be obtained by the following formula

[0101] C5=P5 (1)

[0102] C4=P4+up2(C5) (2)

[0103] C3=P3+up2(C4) (3)

[0104] C2=P2+up2(C3) (4)

[0105] F=concat(up8(conv(C5)),up4(conv(C4)),up2(conv(C3)),conv(C2)) (5)

[0106] Among them, 640×640 represents the height and width of the input image, up2(·), up4(·), and up8(·) represent 2, 4, and 8 times upsampling operations respectively, conv(·) represents the convolution operation with a convolution kernel size of 3×3 and a stride of 1, and concat(·) represents the splicing operation along the channel direction, such as Figure 1 As shown, is the intermediate feature map used to generate F.

[0107] Step 2: Differentiable binarization with background suppression. This paper uses DBnet[Liao M, Wan Z, Yao C, et al. Real-time scene text detection with differentiable binarization[C] / / Proceedings of the AAAI conference on artificial intelligence. 2020, 34(07): 11474-11481.] as the basic model and adds a background prediction branch to predict the background probability map. The prediction process can be expressed as follows:

[0108] I BP =deconv1(deconv1(WDconv(F))) (6)

[0109] Among them, deconv1(·) represents the deconvolution operation with a convolution kernel of 2×2 and a stride of 2, and WDconv(·) represents the weighted deformable convolution operation with a convolution kernel of 3×3 and a stride of 1. The principle of WDconv(·) is as follows:

[0110] First, for the input feature map, WDconv(·) predicts the offset of the original sampling point and the RoI metric factor respectively through two additional convolutional layers.

[0111] Secondly, the sampling position of the convolution kernel is dynamically adjusted according to the predicted offset, and these dynamic sampling points are weighted in combination with the RoI metric factor. Finally, the adjusted sampling points are convolved with the input feature map to output the enhanced output feature map.

[0112] The convolution operation of WDconv(·) can be expressed as follows:

[0113]

[0114] Among them, x represents the current arbitrary position in F, Represents the number of convolution points in the convolution kernel, c k Represents the value corresponding to the kth convolution point in the convolution kernel, x k Represents the offset of the kth convolution point relative to the center of the convolution kernel, Vx k Represents the predicted offset of the k-th sampling point relative to the k-th convolution point, w k ∈[0,1] represents F(x+x k +Vx k ) is the region of interest measurement factor, namely: F(x+x k +Vx k ) belongs to the region of interest.

[0115] As can be seen from formula (7), by dynamically learning the offset, WDconv(·) can flexibly adjust the sampling position of the convolution kernel to adapt to the irregular shape of the background area. At the same time, the importance of the sampling points is weighted by the RoI metric factor, effectively highlighting key features. Therefore, by introducing WDconv(·), the background prediction branch can significantly improve the feature extraction ability of irregularly shaped background areas, thereby improving the prediction accuracy of the background probability map.

[0116] For text probability map The prediction process can be obtained by the following formula:

[0117] I TP =deconv2(deconv2(conv2(F))) (8)

[0118] Among them, deconv2(·) represents the deconvolution operation with a convolution kernel of 2×2 and a stride of 2, and conv2(·) represents the convolution operation with a convolution kernel of 3×3 and a stride of 1.

[0119] For text threshold map The prediction process can be obtained by the following formula:

[0120] I TP =deconv3(deconv3(conv3(F))) (9)

[0121] Among them, deconv3(·) represents the deconvolution operation with a convolution kernel of 2×2 and a stride of 2, and conv3(·) represents the convolution operation with a convolution kernel of 3×3 and a stride of 1.

[0122] In getting I BP , I TP , I T After that, the differentiable binarization of background suppression can be performed, and the process can be expressed by the following formula:

[0123]

[0124] in, represents the negated background probability map, Text probability map representing background suppression, represents an approximate binary image, represents the element-by-element product operation, and m=50 represents the modulation factor.

[0125] Step 3: Get text detection results.

[0126] Based on the approximate binary image I obtained in the previous step B To generate a text box as the text detection result, the technology involved in generating the text box is performed with reference to DBNet [Liao M, Wan Z, Yao C, et al. Real-time scene text detection with differentiable binarization [C] / / Proceedings of the AAAI conference on artificial intelligence. 2020, 34 (07): 11474-11481.].

[0127] Step 4: Obtain the supervisory signal required for text detection model training.

[0128] The text truth map and boundary truth map of this application are obtained as follows: Figure 4As shown in the figure, first, the manually annotated text box, marked as a light-colored rectangle, is expanded and contracted respectively. The expanded text box is marked as a dotted rectangle, and the contracted text box is marked as a dark rectangle. The contracted text box is used to define the text truth map, that is, the grayscale value of all pixels in the dark rectangle is 1, and the grayscale value of the remaining pixels is 0. The contracted and expanded text boxes are used to jointly define the boundary truth map, that is, the grayscale value of all pixels between the dark rectangle and the dotted rectangle is in the range [0.3, 0.7], and the grayscale value of the remaining pixels is 0. The background truth map is obtained by inverting the text truth map, that is, the grayscale value of all pixels in the dark rectangle is 0, and the grayscale value of the remaining pixels is 1.

[0129] For details on obtaining text truth maps and boundary truth maps, please refer to DBNet[Liao M, Wan Z, Yao C, et al. Real-time scene text detection with differentiable binarization[C] / / Proceedings of the AAAI conference on artificial intelligence. 2020, 34(07): 11474-11481.].

[0130] Obtaining the background truth map: Perform the inverse operation on the text truth map to obtain the background truth map (the grayscale value of all pixels in the dark rectangle is 0, and the grayscale value of the remaining pixels is 1).

[0131] Step 5: Training of text detection model:

[0132] (1) Overall loss function of the text detection model. The overall loss function of the text detection model proposed in this invention is defined as:

[0133] L=β1L b +β2L ab +β3L p +β4L t (13)

[0134] Among them, L b Represents the loss function corresponding to the predicted background probability map, L ab Represents the loss function corresponding to the predicted approximate binary image, L p Represents the loss function corresponding to the predicted text probability map, L ab Represents the loss function corresponding to the predicted approximate binary image, L t Represents the loss function corresponding to the predicted text threshold map, β1=1, β2=1, β3=5, β4=10 represent L b , L ab, L p , L t The weight coefficient of .

[0135] (2)L b Definition of L b The definition of is as follows:

[0136]

[0137] Among them, N = 640 × 640 = 409600 represents I BP The number of pixels in x i Representative I BP The gray value of the i-th pixel, y i Represents the grayscale value of the i-th pixel in the background truth image, s = 1 × 10 -6 is a small constant used to prevent the denominator from being zero.

[0138] (3)L p , L ab , L t For the definition, see DBNet [Liao M, WanZ, Yao C, et al.Real-time scenetext detection with differentiable binarization[C] / / Proceedings of the AAAIconference on artificial intelligence.2020,34(07):11474-11481.].

[0139] (4) After obtaining L according to the above steps, the text detection model can be trained through backpropagation. For details, see DBNet[Liao M, Wan Z, Yao C, et al. Real-time scene text detection with differentiable binarization[C] / / Proceedings of the AAAI conference on artificial intelligence. 2020, 34(07):11474-11481.].

[0140] The hardware configuration for implementing this invention is as follows: CPU: E5-2650V4 (2.2GHz 12x2 cores), 512GB of memory, and a single NVIDIA RTX 2080Ti (11GB of video memory) graphics card. The software platform configuration is: Ubuntu 16.04, Python 3.8, PyTorch 1.8.1.

[0141] In order to better demonstrate the performance of the present invention, a comparative experiment was conducted on the public dataset ICDAR2015 [KARATZASD, GOMEZ-BIGORDA L, NICOLAOU A, et al. ICDAR 2015 competition on robust reading; proceedings of the 201513th international conference on document analysis and recognition (ICDAR), F, 2015[C]. IEEE.].

[0142] As shown in Table 1, the present invention is compared with four popular algorithms, namely: TextBoxes++[LIAO M,SHI B,BAI XJITO IP.Textboxes++:A single-shot oriented scene text detector[J].2018,27(8):3676-90.], EAST[ZHOU X,YAO C,WEN H,et al.East:anefficient and accurate scene text detector;proceedings of the Proceedings ofthe IEEE conference on Computer Vision and Pattern Recognition,F,2017[C].], RRD[Liao M,Zhu Z,Shi B,et al.Rotation-sensitive regression for oriented scenetext detection[C] / / Proceedings of the IEEE conference on computer vision andpattern recognition.2018:5909-5918.], DBNet[Liao M,Wan Z,Yao C,et al.Real-timescene text detection with differentiable binarization[C] / / Proceedings of the AAAI conference on artificial intelligence.2020,34(07):11474-11481.].

[0143] Table 1 Comparative experimental results of the proposed method and four popular methods on the ICDAR2015 dataset

[0144]

[0145] As can be seen from Table 1, the accuracy (%), recall (%) and F1 (%) scores of the present invention reached 88.1, 78.9 and 83.2 respectively, which are the best compared with the other four text detection methods.

[0146] In the second aspect, the present invention provides a differentiable binarization text detection system based on background suppression, such as Figure 5 As shown, the system includes:

[0147] The acquisition module is used to obtain a certain number of text images to form sample data, and divide the sample data into training sets and test sets according to a certain ratio.

[0148] The model construction and training module is used to construct an improved DBNet text detection model, and input the training set into the improved DBNet text detection model for model training to obtain an approximate binary image; the improved DBNet text detection model adds the prediction of the background probability map to the feature map F of the text detection of the DBNet model, inverts the result, and performs an element-by-element multiplication operation with the obtained predicted text probability map to generate a background-suppressed text probability map, and the background-suppressed text probability map is subjected to a differentiable binarization operation with the text threshold map to obtain a corresponding approximate binary image.

[0149] The prediction module is used to test the improved DBNet text detection model using a test set, thereby generating predicted text area information based on the obtained approximate binary image.

[0150] The other technical features of the differentiable binarization text detection system based on background suppression described in this invention are similar to those of the corresponding differentiable binarization text detection method based on background suppression, and will not be repeated here.

[0151] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0153] Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if such changes and modifications of the embodiments of the present invention fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A differentiable binarization text detection method based on background suppression, characterized in that: The method includes: Obtain a certain number of text images to form sample data, and divide the sample data into training set and test set according to a certain ratio; An improved DBNet text detection model is constructed, and the training set is input into the improved DBNet text detection model for model training to obtain an approximate binary image; the improved DBNet text detection model adds a prediction of a background probability map to a feature map F of text detection of the DBNet model, negates the result, and performs an element-by-element product operation with the obtained predicted text probability map to generate a background-suppressed text probability map; the background-suppressed text probability map is subjected to a differentiable binarization operation with a text threshold map to obtain a corresponding approximate binary image; The improved DBNet text detection model is tested using a test set, thereby generating a text detection result based on the obtained approximate binary image.

2. The differentiable binarization text detection method based on background suppression according to claim 1 is characterized in that: Also includes: The background probability map is predicted using a background probability map prediction model, which is expressed as: I BP =deconv1(deconv1(WDconv(F))); in, is the predicted background probability map, deconv1(·) represents the deconvolution operation with a convolution kernel of k2×k2 and a step size of 2, and WDconv(·) represents the weighted deformable convolution operation with a convolution kernel of k3×k3 and a step size of 1.

3. The differentiable binarization text detection method based on background suppression according to claim 2 is characterized in that: The operation process of the weighted deformable convolution operation WDconv(·) includes: Two additional convolutional layers are used to predict the offset of the original sampling points and the region of interest metric in the feature map F of the input text image respectively; Dynamically adjust the sampling position of the convolution kernel according to the predicted offset, and weight these sampling points in combination with the region of interest measurement factor; The convolution kernel is convolved with the feature map F of the input text image according to the adjusted sampling points to obtain the enhanced output feature map Y.

4. The differentiable binarization text detection method based on background suppression according to claim 3 is characterized in that: The convolution operation is performed on the convolution kernel and the feature map F of the input text image according to the adjusted sampling points, which is expressed as: Among them, F represents the feature map of the input text image, Y represents the enhanced output feature map, and x represents the current arbitrary position in F. Represents the number of convolution points in the convolution kernel, c k Represents the value corresponding to the kth convolution point in the convolution kernel, x k Represents the offset of the kth convolution point relative to the center of the convolution kernel, Vx k Represents the predicted offset of the k-th sampling point relative to the k-th convolution point, w k ∈[0,1] represents F(x+x k +Vx k ) is the region of interest measurement factor, namely: F(x+x k +Vx k ) belongs to the region of interest.

5. The differentiable binarization text detection method based on background suppression according to claim 1 is characterized in that: Also includes: The text probability map is predicted using a text probability map prediction model, which is expressed as: I TP =deconv2(deconv2(conv2(F))); in, is the predicted text probability map, deconv2(·) represents the deconvolution operation with a convolution kernel of k4×k4 and a step size of 2, and conv2(·) represents the convolution operation with a convolution kernel of k5×k5 and a step size of 1.

6. The differentiable binarization text detection method based on background suppression according to claim 1 is characterized in that: The improved DBNet text detection model also includes a predicted text threshold map, and the corresponding prediction model is expressed as: I T =deconv3(deconv3(conv3(F))); in, is the predicted text threshold map, deconv3(·) represents the deconvolution operation with a convolution kernel of k6×k6 and a step size of 2, and conv3(·) represents the convolution operation with a convolution kernel of k7×k7 and a step size of 1.

7. The differentiable binarization text detection method based on background suppression according to claim 2 is characterized in that: Also includes: Training the background probability map prediction model includes: A background truth map is collected as a supervisory signal and input into the background probability map prediction model. The loss function of the constructed background probability map prediction model is used to measure the difference between the predicted output and the expected output of the model, thereby finding the optimization direction. The background truth map is obtained by inverting the text truth map, and the text truth map is obtained by shrinking the manually annotated text box.

8. The differentiable binarization text detection method based on background suppression according to claim 7 is characterized in that: The constructed background probability map prediction model loss function is used to measure the difference between the predicted output and the expected output of the model. The loss function is expressed as: Among them, L b represents the loss function corresponding to the background probability map prediction model, N = H × W represents the number of pixels in the current predicted background probability map, x i Represents the grayscale value of the i-th pixel in the currently predicted background probability map, y i Represents the grayscale value of the i-th pixel in the background truth image. s is a small constant used to prevent the denominator from being 0.

9. The differentiable binarization text detection method based on background suppression according to claim 5, characterized in that: Inputting the training set into the improved DBNet text detection model for model training includes: The constructed overall loss function is used to measure the difference between the predicted output and the expected output of the improved DBNet text detection model, wherein the overall loss function is expressed as: L=β1L b +β2L p +β3L ab +β4L t ; Among them, L p Represents the loss function corresponding to the text probability graph prediction model, L ab Represents the loss function corresponding to the predicted approximate binary image, L t Represents the loss function corresponding to the text threshold graph prediction model, β1, β2, β3, and β4 represent L b 、L p 、L ab 、L t The weight coefficient of .

10. A differentiable binarization text detection system based on background suppression, characterized in that: The system includes: The acquisition module is used to obtain a certain number of text images to form sample data, and divide the sample data into training set and test set according to a certain ratio; A model construction and training module is used to construct an improved DBNet text detection model, and input the training set into the improved DBNet text detection model for model training to obtain an approximate binary image; the improved DBNet text detection model adds a prediction of a background probability map to the feature map F of text detection of the DBNet model, inverts the result, and performs an element-by-element product operation with the obtained predicted text probability map to generate a background-suppressed text probability map, and the background-suppressed text probability map is subjected to a differentiable binarization operation with a text threshold map to obtain a corresponding approximate binary image; The prediction module is used to test the improved DBNet text detection model using a test set, thereby generating a text detection result based on the obtained approximate binary image.

Images