Text detection method and device

By performing single-word prediction processing and binary processing on the target image, combined with the backbone network and segmentation network, single-word-level detection and text line positioning are realized, solving the problem of high text detection complexity in the existing technology and improving detection accuracy and informative richness.

CN119091455APending Publication Date: 2024-12-06ZHUHAI KINGSOFT OFFICE SOFTWARE +2
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310667721.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-06
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing text detection methods based on deep learning are mainly used to locate text lines, and cannot meet the detection needs of single-word levels, resulting in high complexity in downstream tasks when processing a large number of text pages.

Method used

By obtaining the target image, performing single-word prediction processing, obtaining the probability map of the single-word box and the connected components, and performing binarization processing, positioning the single-word box and its text lines to which it belongs, and using the backbone network, feature fusion network and segmented network for feature extraction and fusion to improve detection accuracy.

Benefits of technology

It realizes single-word detection and text line positioning, improves the accuracy of detection single-word boxes and text lines, provides more information to assist in the processing of downstream tasks, and reduces processing complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091455A_ABST
    Figure CN119091455A_ABST
Patent Text Reader

Abstract

The invention provides a text detection method and device. The text detection method comprises the steps of obtaining a to-be-detected target image; performing single character prediction processing on the target image to obtain a prediction result; wherein the prediction result comprises a probability graph of a single word frame in the target image and a probability graph of a connection component; the connecting component is connected between two adjacent single word frames in the same text line of the target image; performing binarization processing on the probability graph of the single word frame to obtain a first target binary image; performing binarization processing on the probability graph of the connection component to obtain a second target binary graph; and according to the first target binary image and the second target binary image, positioning a single word frame in the target image and a text line to which the single word frame belongs. According to the embodiment of the invention, accurate detection of the single word box and the text line can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a text detection method and device. Background Art

[0002] With the rapid development of deep learning, deep learning-based methods have been widely used in the field of text detection.

[0003] In the prior art, text detection based on deep learning is mainly used to locate text lines. However, since it is only used to locate text lines, it cannot meet the needs of obtaining more precise single-word level detection. In addition, if single-word detection is simply performed, the processing of downstream tasks (for example, text recognition) will need to be based on single-word level processing, which will lead to high complexity in downstream task processing when processing pages with a large amount of text. Summary of the invention

[0004] The embodiments of the present disclosure provide a text detection method and device.

[0005] According to a first aspect of an embodiment of the present disclosure, a text detection method is provided, the method comprising:

[0006] Acquire the target image to be detected;

[0007] Performing single-word prediction processing on the target image to obtain a prediction result; wherein the prediction result includes: a probability map of single-word frames in the target image and a probability map of connected components; the connected component is connected between two adjacent single-word frames in the same text line of the target image;

[0008] Binarizing the probability map of the single-word frame to obtain a first target binary map;

[0009] Binarizing the probability map of the connected components to obtain a second target binary map;

[0010] A single-character frame in the target image and a text line to which the single-character frame belongs are located according to the first target binary image and the second target binary image.

[0011] In one embodiment, the target image is processed by a text detection model, wherein the text detection model includes a backbone network, a feature fusion network, and a segmentation network; the prediction result also includes: a threshold map of a single word frame in the target image and a threshold map of a connected component;

[0012] The step of performing single word prediction processing on the target image to obtain a prediction result includes:

[0013] Inputting the target image into the backbone network for feature extraction to obtain a multi-scale feature map of a single word frame in the target image and a multi-scale feature map of a connected component;

[0014] Inputting the multi-scale feature map of the single word frame and the multi-scale feature map of the connected component into the feature fusion network to obtain a fused feature map output by the feature fusion network;

[0015] The fused feature map is input into the segmentation network to obtain the probability map of the single word box, the threshold map of the single word box, the probability map of the connected component and the threshold map of the connected component output by the segmentation network.

[0016] In one embodiment, the binarization of the probability map of the single-word frame to obtain a first target binary map includes:

[0017] Binarizing the probability map of the single-word frame according to the threshold map of the single-word frame to obtain the first target binary map;

[0018] The binarization process is performed on the probability map of the connected component to obtain a second target binary map, including:

[0019] The probability map of the connected component is binarized according to the threshold map of the connected component to obtain the second target binary map.

[0020] In one embodiment, locating a single-word frame in the target image and a text line to which the single-word frame belongs according to the first target binary image and the second target binary image comprises:

[0021] Determine each of the single-word frames in the target image according to the first target binary image;

[0022] Determine the adjacent relationship between the single-word frames in the target image according to each of the single-word frames in the target image and the second target binary image; wherein the existence of an adjacent relationship between two of the single-word frames indicates that the two single-word frames are connected by the connection component;

[0023] The text line in the target image is located according to the adjacent relationship between the single-word frames.

[0024] In one embodiment, the text detection model is trained by the following steps:

[0025] Acquire a sample image and label data; the label data includes: a label probability map of a sample single word frame in the sample image, a label threshold map of the sample single word frame, a label probability map of a sample connection component, and a label threshold map of the sample connection component; the sample connection component is connected between two adjacent sample single word frames in the same text line;

[0026] The text detection model to be trained is trained according to the sample image and the label data.

[0027] In one embodiment, the obtaining of sample images and label data includes:

[0028] Acquire the annotation information of each word in the sample image and the annotation information of the text line where each word is located, wherein the annotation information of the word is the sample word frame;

[0029] Determining the adjacent relationship between sample single word frames in the same text line in the sample image according to the annotation information of each single word and the annotation information of each text line;

[0030] Generating a sample connected component in the sample image according to the adjacent relationship between the sample single-word frames in the same text line in the sample image;

[0031] The label data is obtained according to the sample single word frame and the sample connected component in the sample image.

[0032] In one embodiment, training the text detection model to be trained according to the sample image and the label data includes:

[0033] Input the sample image into the text detection model to be trained for prediction, and obtain a prediction probability map of the sample single word frame in the sample image, a prediction threshold map of the sample single word frame, a prediction probability map of the sample connected component, and a prediction threshold map of the sample connected component;

[0034] Obtaining a loss value of the text detection model for the sample single-word frame according to the prediction probability map of the sample single-word frame, the prediction threshold map of the sample single-word frame, the label probability map of the sample single-word frame, and the label threshold map of the sample single-word frame;

[0035] Obtaining a loss value of the text detection model for the sample connected component according to the predicted probability map of the sample connected component, the predicted threshold map of the sample connected component, the label probability map of the sample connected component, and the label threshold map of the sample connected component;

[0036] The model parameters of the text detection model are adjusted according to the loss value of the text detection model for the sample single-word frame and the loss value of the sample connected component.

[0037] In one embodiment, obtaining the loss value of the text detection model for the sample single word frame according to the prediction probability map of the sample single word frame, the prediction threshold map of the sample single word frame, the label probability map of the sample single word frame, and the label threshold map of the sample single word frame includes:

[0038] Determining a first loss value of the text detection model for the sample single-word frame according to the prediction probability map of the sample single-word frame and the label probability map of the sample single-word frame;

[0039] Determining a second loss value of the text detection model for the sample single-word frame according to the prediction threshold map of the sample single-word frame and the label threshold map of the sample single-word frame;

[0040] Determine a third loss value of the text detection model for the sample single-word frame according to the first predicted binary map and the label probability map of the sample single-word frame; wherein the first predicted binary map is obtained by binarizing the predicted probability map of the sample single-word frame according to the predicted threshold map of the sample single-word frame;

[0041] According to the first loss value, the second loss value and the third loss value, the loss value of the text detection model for the sample single-word frame is obtained.

[0042] In one embodiment, obtaining the loss value of the text detection model for the sample connection component according to the prediction probability map of the sample connection component, the prediction threshold map of the sample connection component, the label probability map of the sample connection component, and the label threshold map of the sample connection component includes:

[0043] Determining a fourth loss value of the text detection model for the sample connected component according to the predicted probability map of the sample connected component and the label probability map of the sample connected component;

[0044] Determining a fifth loss value of the text detection model for the sample connected component according to the prediction threshold map of the sample connected component and the label threshold map of the sample connected component;

[0045] Determine a sixth loss value of the text detection model for the sample connection component according to the second predicted binary map and the label probability map of the sample connection component; wherein the second predicted binary map is obtained by binarizing the predicted probability map of the sample connection component according to the predicted threshold map of the sample connection component;

[0046] According to the fourth loss value, the fifth loss value and the sixth loss value, a loss value of the text detection model for the sample connected component is obtained.

[0047] According to a second aspect of an embodiment of the present disclosure, a text detection model training method is provided, the method comprising:

[0048] Acquire a sample image and label data; the label data includes: a label probability map of a sample single word frame in the sample image, a label threshold map of the sample single word frame, a label probability map of a sample connection component, and a label threshold map of the sample connection component; the sample connection component is connected between two adjacent sample single word frames in the same text line;

[0049] The text detection model to be trained is trained according to the sample image and the label data.

[0050] According to a third aspect of an embodiment of the present disclosure, a text detection device is provided, the device comprising:

[0051] An acquisition module, used for acquiring a target image to be detected;

[0052] A prediction module is used to perform single-word prediction processing on the target image to obtain a prediction result; wherein the prediction result includes: a probability map of a single-word frame in the target image and a probability map of a connected component; the connected component is connected between two adjacent single-word frames in the same text line of the target image;

[0053] A first processing module is used to perform a binarization process on the probability map of the single-word frame to obtain a first target binary map;

[0054] A second processing module is used to perform binarization processing on the probability map of the connected component to obtain a second target binary map;

[0055] A positioning module is used to locate a single-word frame in the target image and a text line to which the single-word frame belongs according to the first target binary image and the second target binary image.

[0056] According to a fourth aspect of an embodiment of the present disclosure, a text detection model training device is provided, the device comprising a training module, the training module comprising:

[0057] An acquisition unit is used to acquire a sample image and label data; the label data includes: a label probability map of a sample single word frame in the sample image, a label threshold map of the sample single word frame, a label probability map of a sample connection component, and a label threshold map of the sample connection component; the sample connection component is connected between two adjacent sample single word frames in the same text line;

[0058] A training unit is used to train the text detection model to be trained according to the sample image and the label data.

[0059] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the text detection method described in any one of the first aspect or the text detection model training method described in the second aspect is implemented.

[0060] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the text detection method described in any one of the first aspect or the text detection model training method described in the second aspect is implemented.

[0061] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0062] By acquiring a target image to be detected, performing single-word prediction processing on the target image, obtaining a probability map of a single-word frame in the target image and a probability map of a connected component; the connected component is connected between two adjacent single-word frames in the same text line of the target image; binarizing the probability map of the single-word frame to obtain a first target binary map, and binarizing the probability map of the connected component to obtain a second target binary map, and finally locating the single-word frame in the target image and the text line to which the single-word frame belongs according to the first target binary map and the second target binary map, so that not only single-word level detection can be achieved, but also text line positioning can be achieved; in addition, by introducing the binarization processing of the image in the text detection process, the accuracy of detecting single-word frames and text lines can be effectively improved. Compared with the text detection scheme in the prior art, the scheme provided by the embodiment of the present disclosure can provide more information for downstream tasks (e.g., text recognition) and can better assist the processing of downstream tasks.

[0063] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0065] Figure 1 A flowchart of a text detection method provided by an embodiment of the present disclosure is shown;

[0066] Figure 2A schematic diagram of a single-word frame and connection components provided by an embodiment of the present disclosure is shown;

[0067] Figure 3 A flowchart of a text detection method provided by an embodiment of the present disclosure is shown;

[0068] Figure 4 A flowchart of a text detection method provided by an embodiment of the present disclosure is shown;

[0069] Figure 5 A flowchart of a text detection model training method provided by an embodiment of the present disclosure is shown;

[0070] Figure 6 A flowchart of a text detection model training method provided by an embodiment of the present disclosure is shown;

[0071] Figure 7 A flowchart of a text detection model training method provided by an embodiment of the present disclosure is shown;

[0072] Figure 8a A flowchart of a text detection model training method provided by an embodiment of the present disclosure is shown;

[0073] Figure 8b A flowchart of a text detection model training method provided by an embodiment of the present disclosure is shown;

[0074] Figure 9a A schematic diagram of a labeled sample image provided by an embodiment of the present disclosure is shown;

[0075] Figure 9b A schematic diagram of the structure of a text detection model provided by an embodiment of the present disclosure is shown;

[0076] Fig.9c A schematic diagram showing an image to be detected and a text detection result provided by an embodiment of the present disclosure is shown;

[0077] Fig.10 A schematic diagram of the structure of a text detection device provided by an embodiment of the present disclosure is shown;

[0078] Fig.11 A structural schematic diagram of a text detection model training device provided by an embodiment of the present disclosure is shown;

[0079] Fig.12 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0080] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the embodiments of the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present disclosure as detailed in the appended claims.

[0081] The terms used in the disclosed embodiments are only for the purpose of describing specific embodiments and are not intended to limit the disclosed embodiments. The singular forms of "a", "said" and "the" used in the disclosed embodiments and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0082] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the disclosed embodiments, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the disclosed embodiments, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0083] It can be understood that the description of various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0084] Figure 1 A flowchart of a text detection method provided by an embodiment of the present disclosure is shown. Figure 1 As shown, the method may include the following steps:

[0085] 101: Acquire a target image to be detected;

[0086] 102: Perform single word prediction processing on the target image to obtain a prediction result; wherein the prediction result includes: a probability map of single word frames in the target image and a probability map of connected components; the connected component connects two adjacent single word frames in the same text line of the target image;

[0087] 103: binarizing the probability map of the single-word frame to obtain a first target binary map, and binarizing the probability map of the connected component to obtain a second target binary map;

[0088] 104: Locate a single-word frame and a text line to which the single-word frame belongs in the target image according to the first target binary image and the second target binary image.

[0089] In the embodiments of the present disclosure, the text detection method may be executed by an electronic device, which includes a terminal and / or a server. The terminal may include, but is not limited to, a mobile phone, a tablet computer, a computer, a smart camera, a smart home device, and / or a smart car device. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server.

[0090] In some examples, in the above step 101, the target image captured by an image capturing device may be obtained, or the target image stored in a storage device (such as a hard disk, etc.) may be obtained, or the target image may be obtained from the Internet.

[0091] For example, the target image may be a natural scene image. The text information in the natural scene image may include but is not limited to at least one of the following: license plate information, road sign information, promotional slogans, store names, etc. in the natural scene.

[0092] In some examples, the text lines in the target image may be arranged in any manner, such as horizontal arrangement, tilted arrangement, flipped arrangement, or even curved arrangement.

[0093] In some examples, the text in the target image may be in printed form or in handwritten form.

[0094] In the present disclosure, the target image contains text information in Chinese and / or English.

[0095] Here, the single word box, which may also be called a single word text box, a single word detection box or a single word bounding box, is used to indicate the area of ​​a single word in an image. The single word may be a single word or a single character (e.g., a number or a letter). The shape of the single word box may be a quadrilateral, such as a parallelogram.

[0096] Here, the connection component is used to connect two adjacent single-word frames in the same text line. For example, the connection component may refer to a quadrilateral frame formed by two sides of the nearest neighbors of two adjacent single-word frames in the same text line.

[0097] In the present disclosure, the connection assembly may also be referred to as a “connection component” or a “connection frame”.

[0098] For example, Figure 2As shown, the single-word frames of some characters in the target image are single-word frame a1, single-word frame a2 and single-word frame a3, wherein the connection component connected between the single-word frame a1 and the single-word frame a2 is the connection component b1, and the connection component connected between the single-word frame a2 and the single-word frame a3 is the connection component b2.

[0099] In the present disclosure, the value of each pixel in the probability map of the single-word frame is used to characterize the probability of whether the pixel or image block at the corresponding position of the target image belongs to the single-word frame. The value range of the pixel in the probability map of the single-word frame is 0 to 1, and the closer the value of the pixel is to 1, the higher the possibility of belonging to the single-word frame.

[0100] In the present disclosure, the value of each pixel in the probability map of the connected component is used to represent the probability of whether the pixel or image block at the corresponding position of the target image belongs to the connected component. The value range of the pixel in the probability map of the connected component is 0 to 1, and the closer the value of the pixel is to 1, the higher the possibility of belonging to the connected component.

[0101] In some examples, in the above step 102, a preset text detection algorithm may be used to detect each single word box and each connected component in the target image to obtain a prediction result.

[0102] In some examples, the preset text detection algorithm may be, for example, a fusion of one or more of a candidate box-based text detection algorithm, a connected domain-based text detection algorithm, a connected component-based text detection algorithm, and a segmentation-based text detection algorithm. Here, the connected component-based text detection algorithm may be, for example, a CRAFT algorithm. The segmentation-based text detection algorithm may be, for example, a DBnet algorithm or a PSENet algorithm.

[0103] In some examples, the target image can be input into a preset text detection model for single-word prediction processing to obtain a prediction result.

[0104] For example, the text detection model can be used to detect each single word box and each connected component in the target image using a preset text detection algorithm to obtain a prediction result.

[0105] The text detection model may be a trained deep learning model, which is constructed based on a deep neural network. For example, the deep neural network may be a lightweight neural network, such as SqueezeNet or MobileNet, so as to reduce the amount of computation of the model.

[0106] In some examples, the text detection model can be a model such as UNet or U2Net suitable for performing segmentation tasks.

[0107] In some examples, the implementation process of step 103 may include:

[0108] The probability map of the single-word frame is binarized according to a preset first probability threshold to obtain a first target binary map, and the probability map of the connected component is binarized according to a preset second probability threshold to obtain a second target binary map.

[0109] The first probability threshold and the second probability threshold may be determined by expert experience.

[0110] In this embodiment, both the first target binary image and the second target binary image include a foreground area and a background area, the pixel value of the background area is 0, and the pixel value of the foreground area is 1.

[0111] In some examples, in the above step 104, each single-word box in the target image can be determined based on the first target binary map, and the determined single-word boxes can be combined with the second target binary map to determine the single-word boxes with adjacent relationships connected by connection components in the target image, and the single-word boxes with adjacent relationships through the connection components can be merged to form text lines.

[0112] Here, the connected domain analysis is used to determine the pixels in the foreground area of ​​the first target binary image that are connected. A single word frame includes an area composed of pixels in the foreground area that have the same pixel value and are adjacent in position.

[0113] In this embodiment, by locating the single-word frame in the target image, the position and size of the single word in the single-word frame can be accurately determined, and by locating the text line to which the single-word frame belongs, the single-word frames belonging to the same text line can be accurately determined.

[0114] The disclosed embodiment provides a text detection method, by acquiring a target image to be detected, performing single-word prediction processing on the target image, obtaining a probability map of a single-word frame in the target image and a probability map of a connected component; the connected component is connected between two adjacent single-word frames in the same text line of the target image, and a first target binary map obtained by binarizing the probability map of the single-word frame and a second target binary map obtained by binarizing the probability map of the connected component are obtained, and finally, according to the first target binary map and the second target binary map, the single-word frame in the target image and the text line to which the single-word frame belongs are located, so that not only single-word level detection can be achieved, but also text line positioning can be achieved; in addition, by introducing the binary processing of the image in the text detection process, the accuracy of detecting single-word frames and text lines can be effectively improved. Compared with the text detection scheme in the prior art, the scheme provided by the disclosed embodiment can provide more information for downstream tasks (e.g., text recognition) and can better assist the processing of downstream tasks.

[0115] In one embodiment, the prediction result may further include: a threshold map of a single-word box in the target image and a threshold map of a connected component.

[0116] In this embodiment, the value of each pixel in the threshold map of the single-character frame is used to binarize the pixel at the corresponding position in the probability map of the single-character frame to complete the binarization process of the probability map of the single-character frame.

[0117] For example, when the value of a pixel point in the probability map of a single-word box is greater than the value of the pixel point at the corresponding position in the threshold map of the single-word box, the pixel point in the probability map of the single-word box will be assigned a pixel value as the foreground area (for example, the pixel value is 1); otherwise, the pixel point in the probability map of the single-word box will be assigned a pixel value as the background area (for example, the pixel value is 0), thereby obtaining a first target binary map.

[0118] Each pixel point in the threshold map of the single-word frame may correspond one-to-one to each pixel point in the probability map of the single-word frame.

[0119] In this embodiment, the value of each pixel point in the threshold map of the connected component is used to binarize the pixel point at the corresponding position in the probability map of the connected component to complete the binarization processing of the probability map of the connected component.

[0120] For example, when the value of a pixel point in the probability map of a connected component is greater than the value of the pixel point at the corresponding position in the threshold map of the connected component, the pixel point in the probability map of the connected component will be assigned a pixel value as the foreground area (for example, a pixel value of 1); otherwise, the pixel point in the probability map of the connected component will be assigned a pixel value as the background area (for example, a pixel value of 0), thereby obtaining a second target binary map.

[0121] Each pixel point in the threshold map of the connected component can correspond one-to-one to each pixel point in the probability map of the connected component.

[0122] Here, both the threshold map of the single-word frame and the threshold map of the connected component can be adaptive threshold maps.

[0123] Here, each pixel in the adaptive threshold map has a separate value as a binarization threshold, and not all pixels share a fixed value. In this way, using the adaptive threshold map of the single-word frame to perform binary processing on the probability map of the single-word frame can make the segmentation effect of the foreground area and the background area in the probability map of the single-word frame better; using the adaptive threshold map of the connected component to perform binary processing on the probability map of the connected component can make the segmentation effect of the foreground area and the background area in the probability map of the connected component better, thereby further improving the accuracy of detecting single-word frames and text lines.

[0124] In one embodiment, in the above step 102, the single word prediction process can be performed on the target image through a text detection model; the text detection model includes: a backbone network, a feature fusion network and a segmentation network.

[0125] In this embodiment, the text detection model can be a convolutional neural network model or other types of models. It only needs to be able to process the sample image to obtain the probability map and threshold map of the single word box of the sample image, and the probability map and threshold map of the connection box. This embodiment does not specifically limit the type of text detection model.

[0126] In some examples, the text detection model can apply the method used in the DBNet algorithm to solve the text box heat map to the single-word boxes and connected components in the target image to obtain the probability map and threshold map of the single-word boxes and the probability map and threshold map of the connected components.

[0127] like Figure 3 As shown, the single word prediction processing of the target image to obtain the prediction result may include the steps of:

[0128] 201: Input the target image into the backbone network in the text detection model for feature extraction, and obtain a multi-scale feature map of a single word box in the target image and a multi-scale feature map of a connected component;

[0129] 202: Input the multi-scale feature map of the single word box and the multi-scale feature map of the connected component into the feature fusion network of the text detection model to obtain a fused feature map output by the feature fusion network;

[0130] 203: Input the fused feature map into the segmentation network of the text detection model to obtain the probability map of the single-word box, the threshold map of the single-word box, the probability map of the connected component, and the threshold map of the connected component output by the segmentation network.

[0131] Here, the text detection model may be a pre-trained model, wherein the training process of the text detection model will be described in detail later and will not be described in detail here.

[0132] In some examples, the backbone network may be ResNet or MobileNet, etc. The feature fusion network may be a feature pyramid network (FPN). The segmentation network may be a convolutional network, which is used to perform convolution processing on the multi-scale feature map output by the feature fusion network to obtain a prediction result.

[0133] In this embodiment, after the target image is subjected to feature extraction by the backbone network in the text detection model, a multi-scale feature map of a single-word frame and a multi-scale feature map of a connected component can be obtained, and the resolutions of the feature maps of different scales can be 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the sample image, respectively. The feature fusion network in the text detection model can perform feature fusion processing on the multi-scale feature map of the single-word frame and the multi-scale feature map of the connected component to obtain a fused feature map. The segmentation network in the text detection model performs convolution processing on the fused feature map to obtain a probability map and a threshold map of a single-word frame, and a probability map and a threshold map of a connected component.

[0134] In some examples, the number of fused feature maps is one.

[0135] In the above step 202, the multi-scale feature map of the single word frame and the multi-scale feature map of the connected component can be input into the feature fusion network, and the feature fusion network fuses the multi-scale feature map of the single word frame and the multi-scale feature map of the connected component to obtain a fused feature map.

[0136] Accordingly, in the above step 203, the fused feature map output by the feature fusion network can be processed by the segmentation network to obtain the probability map of the single word box, the threshold map of the single word box, the probability map of the connected components and the threshold map of the connected components.

[0137] Exemplarily, the segmentation network may include two subnetworks, one of which is used to perform convolution processing on the fused feature map to obtain a probability map and a threshold map of a single word box, and the other subnetwork is used to perform convolution processing on the fused feature map to obtain a probability map and a threshold map of a connected component.

[0138] As another example, in order to reduce the amount of calculation and improve the detection speed, the segmentation network can use multiple segmentation heads to perform segmentation processing of the probability map and the threshold map respectively, for example, it can include the first to fourth segmentation heads, wherein the first segmentation head is used to perform convolution processing on the fused feature map to obtain the probability map of the single word box. The second segmentation head is used to perform convolution processing on the fused feature map to obtain the threshold map of the single word box. The third segmentation head is used to perform convolution processing on the fused feature map to obtain the probability map of the connected component. The fourth segmentation head is used to perform convolution processing on the fused feature map to obtain the threshold map of the connected component.

[0139] In some other examples, the fused feature map includes: a first fused feature map and a second fused feature map.

[0140] In the above step 202, the multi-scale feature map of the single word frame and the multi-scale feature map of the connected component are input into the feature fusion network to obtain a first fused feature map obtained by the feature fusion network fusing the multi-scale feature map of the single word frame, and a second fused feature map obtained by the feature fusion network fusing the multi-scale feature map of the connected component.

[0141] Correspondingly, in the above step 203, the first fused feature map and the second fused feature map are input into the segmentation network, and the probability map of the single-word box and the threshold map of the single-word box are obtained by the segmentation network in the text detection model processing the first fused feature map, and the probability map of the connected component and the threshold map of the connected component are obtained by the segmentation network processing the second fused feature map.

[0142] In some examples, the segmentation network may include two subnetworks, one of which is used to perform convolution processing on the first fused feature map to obtain a probability map and a threshold map of a single word box, and the other subnetwork is used to perform convolution processing on the second fused feature map to obtain a probability map and a threshold map of a connected component.

[0143] In other examples, in order to reduce the amount of calculation and improve the detection speed, the segmentation network can use multiple segmentation heads to perform segmentation processing of the probability map and the threshold map respectively, for example, it can include the first to fourth segmentation heads, wherein the first segmentation head is used to perform convolution processing on the first fused feature map to obtain the probability map of the single word box. The second segmentation head is used to perform convolution processing on the first fused feature map to obtain the threshold map of the single word box. The third segmentation head is used to perform convolution processing on the second fused feature map to obtain the probability map of the connected component. The fourth segmentation head is used to perform convolution processing on the second fused feature map to obtain the threshold map of the connected component.

[0144] As an example, the segmentation head can be composed of depth-wise separated convolutions (e.g., 7×7 convolution kernels).

[0145] In the disclosed embodiment, multi-scale feature maps of single-word frames and multi-scale feature maps of connected components are extracted through the backbone network, and feature fusion is performed through the feature fusion network. In this way, potential feature information of single-word frames and connected components can be extracted, making the extracted feature information richer, and enabling the segmentation network to more accurately segment the background and foreground, thereby further improving the accuracy of text detection.

[0146] In one embodiment, in the above step 103, the binarization of the probability map of the single-word frame to obtain the first target binary map may include the following steps:

[0147] The probability map of the single-word frame is binarized according to the threshold map of the single-word frame to obtain the first target binary map.

[0148] In the above step 103, the binary processing of the probability map of the connected components to obtain the second target binary map may include:

[0149] The probability map of the connected component is binarized according to the threshold map of the connected component to obtain the second target binary map.

[0150] Here, the probability map of the single-word frame can be differentiably binarized according to the threshold map of the single-word frame to obtain a first target binary map, and the probability map of the connected component can be differentiably binarized according to the threshold map of the connected component to obtain a second target binary map.

[0151] In this embodiment, the probability map of the single-word box is binarized by using the threshold map of the single-word box. In this way, the standard binarization operation in the general text detection model is replaced by a differentiable binarization operation. The binarization operation can be combined with network training, and the threshold of each pixel point can be adaptively predicted, which can effectively improve the segmentation performance and greatly reduce the complexity of post-processing.

[0152] In this embodiment, since the threshold map of the single-word box and the threshold map of the connected component are both obtained through the segmentation network in the text detection model, that is, the threshold map is learned by the network instead of using a fixed threshold value, the corresponding probability map is binarized using the learned threshold map, which can well separate the background from the foreground, thereby further improving the accuracy of text detection.

[0153] In one embodiment, Figure 4 As shown, in the above step 104, locating the single-word frame and the text line to which the single-word frame belongs in the target image according to the first target binary image and the second target binary image may include the steps of:

[0154] 301: Determine each single word frame in the target image according to the first target binary image;

[0155] 302: Determine, according to each single-word frame in the target image and the second target binary image, an adjacent relationship between the single-word frames in the target image; wherein the existence of an adjacent relationship between two single-word frames indicates that a connected component is connected between the two single-word frames;

[0156] 303: Locate the text line in the target image according to the adjacent relationship between the single-word boxes.

[0157] In some examples, in the above step 301, a connected domain analysis may be performed on the first target binary image, and each single word frame in the target image may be determined according to the connected domain analysis result.

[0158] Here, connected domain analysis can be used to determine the pixels with a connected state in the foreground area of ​​the target binary image.

[0159] An image block composed of pixel points having the same pixel value and adjacent positions in the foreground area of ​​the first target binary image corresponds to a single-word frame in the target image.

[0160] In some examples, in the above step 302, a connected domain analysis can be performed based on the second target binary graph, and based on the connected domain analysis results and the individual single-word frames in the target image, the single-word frames connected by the connection components in the target image are determined, that is, the adjacent relationship between the single-word frames is obtained.

[0161] In the above step 303, the adjacent single-word frames can be combined into text lines based on the adjacent relationship between the single-word frames, that is, the position information of the text lines in the target image can be obtained.

[0162] In the disclosed embodiment, each single word frame in the target image can be accurately located according to the first target binary map. The adjacent relationship between the single word frames in the target image is determined by combining the second target binary map with each single word frame, and the text line in the target image can be accurately located according to the adjacent relationship between the single word frames, thereby improving the accuracy of text line detection.

[0163] In one embodiment, a text detection model training method is provided, such as Figure 5 As shown, the text detection model training method may include the following steps:

[0164] 401: Obtain a sample image and label data; the label data includes: a label probability map of a sample single word frame in the sample image, a label threshold map of the sample single word frame, a label probability map of a sample connected component, and a label threshold map of a sample connected component; the sample connected component is connected between two adjacent sample single word frames in the same text line;

[0165] 402: Train the text detection model to be trained based on the sample images and label data.

[0166] In the disclosed embodiments, the text detection model training method may be executed by an electronic device, which may include a terminal and / or a server. The terminal may include, but is not limited to, a mobile phone, a tablet computer, a computer, a smart camera, a smart home device, and / or a smart vehicle device. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server.

[0167] In some examples, the sample image contains text information in Chinese and / or English, etc. The sample image may be an image captured by an image capturing device, or an image stored in a storage device (such as a hard disk), or an image downloaded from the Internet.

[0168] For example, the sample image may be a natural scene image. The text information in the natural scene image may include, but is not limited to: license plate information, road sign information, promotional slogans, store names, etc. in the natural scene.

[0169] In some examples, the text lines in the sample image may be arranged in any manner, such as horizontal arrangement, tilted arrangement, flipped arrangement, or even curved arrangement.

[0170] In some examples, the text in the sample image may be in printed form or in handwritten form.

[0171] In the above step 401, the label probability map (which may be referred to as “GT probability map”) and the label threshold map (which may be referred to as “GT threshold map”) of the sample single word box in the sample image, wherein GT is Ground Truth.

[0172] In some examples, a segmentation-based text detection algorithm can be used to detect each sample single-word frame and each sample connected frame in a sample image to obtain a label threshold map of the sample single-word frame and a label threshold map of the sample connected component.

[0173] It is understandable that the value of each pixel in the sample single word frame and the label probability map of the sample connected component can only be 0 or 1. The value range of each pixel in the sample single word frame and the label threshold map of the sample connected component is 0-1.

[0174] In some examples, the implementation process of step 402 may include:

[0175] The sample image is input into the text detection model for prediction to obtain the prediction result of the sample image; the prediction result of the sample image includes: the prediction probability map and prediction threshold map of the sample single word box, the prediction probability map and prediction threshold map of the sample connected component; according to the loss value between the label data of the sample image and the prediction result, the text detection model is iteratively trained until the training stop condition is reached, and a trained text detection model is obtained.

[0176] In the disclosed embodiments, a text detection model is trained based on sample images, label probability maps and label threshold maps of sample single-word boxes, and label probability maps and label threshold maps of sample connected components. By utilizing the probability maps and threshold maps, binarization processing can be introduced during the training of the text detection model, which can effectively improve the training effect of the text detection model, thereby improving the accuracy of the model in detecting single-word boxes and text lines.

[0177] In one embodiment, Figure 6 As shown, in the above step 401, obtaining sample images and label data may include the following steps:

[0178] 501: Obtaining the annotation information of each word in the sample image and the annotation information of the text line where each word is located, wherein the annotation information of the word is a sample word frame;

[0179] 502: Determine the adjacent relationship between sample single word frames in the same text line in the sample image according to the annotation information of each single word and the annotation information of each text line;

[0180] 503: Generate a sample connected component in the sample image according to the adjacent relationship between sample single word boxes in the same text line in the sample image;

[0181] 504: Obtain label data according to the sample single word box and the sample connected component in the sample image.

[0182] In some examples, in the above step 501, each word and each text line in the sample image can be annotated by an automatic annotation tool or a manual annotation method to obtain a sample word frame of each word and annotation information of the text line where each word is located.

[0183] Here, the shape of the sample word frame may be a quadrilateral, such as a parallelogram.

[0184] Here, the annotation information of the text line may include: color information, position information or number information corresponding to the text line, wherein different text lines correspond to different color information, different text lines correspond to different number information, and the position information of the text line may be coordinate information.

[0185] In some examples, in the above step 502, multiple candidate word frames around the sample word frame of the word can be determined for each word in the sample image; based on the annotation information of the text line where the word is located, among the multiple candidate word frames adjacent to the sample word frame of the word, the sample word frame belonging to the same text line as the sample word frame of the word is determined to obtain the adjacent relationship between the sample word frames in the same text line.

[0186] It is understandable that a sample single word frame may have two or more adjacent sample single word frames in a sample image, while the sample single word frame may have only one or two adjacent sample single word frames in the same text line.

[0187] In some examples, in the above step 503, two edges of the nearest neighbors of two adjacent sample single-word frames in the same text line are determined based on the adjacent relationship between the sample single-word frames in the same text line in the sample image; a quadrilateral frame is formed based on the two determined nearest neighbor edges to serve as a sample connected component.

[0188] In some examples, in the above step 504, the DBnet algorithm can be used to detect each sample single word box and each sample connected component in the sample image to obtain the label probability map and label threshold map of the sample single word box, and the label probability map and label threshold map of the sample connected component.

[0189] In one embodiment, Figure 7 As shown, in the above step 402, training the text detection model to be trained according to the sample image and the label data may include the following steps:

[0190] 601: Input the sample image into the text detection model to be trained for prediction, and obtain a prediction probability map of a sample single word box in the sample image, a prediction threshold map of the sample single word box, a prediction probability map of a sample connected component, and a prediction threshold map of the sample connected component;

[0191] 602: Obtain a loss value of the text detection model for the sample single word frame according to the prediction probability map of the sample single word frame, the prediction threshold map of the sample single word frame, the label probability map of the sample single word frame, and the label threshold map of the sample single word frame;

[0192] 603: Obtaining a loss value of the text detection model for the sample connected component according to the predicted probability map of the sample connected component, the predicted threshold map of the sample connected component, the label probability map of the sample connected component, and the label threshold map of the sample connected component;

[0193] 604: Adjust model parameters of the text detection model according to the loss value of the text detection model for the sample single-word box and the loss value of the sample connected component.

[0194] Here, the predicted probability map and predicted threshold map of the sample single-word box in the sample image can refer to the relevant description of the probability map and threshold map of the single-word box in the target image in the above embodiments, and will not be repeated here.

[0195] Here, the predicted probability map and predicted threshold map of the sample connected component in the sample image can refer to the relevant description of the probability map and threshold map of the connected component in the target image in the above embodiments, which will not be repeated here.

[0196] Here, the implementation process of the above step 601 can refer to the relevant description content of step 102 in the above embodiment, which will not be repeated here.

[0197] In some examples, in the above step 602, the predicted probability map of the sample single word box can be binarized according to the predicted threshold map of the sample single word box to obtain the predicted binary map of the sample single word box, and then the loss value of the text detection model for the sample single word box is obtained based on the predicted binary map of the sample single word box and the label probability map.

[0198] In some examples, in the above step 603, the predicted probability map of the sample connected component can be binarized according to the predicted threshold map of the sample connected component to obtain the predicted binary map of the sample connected component; then, based on the predicted binary map of the sample connected component and the label probability map, the loss value of the text detection model for the sample connected component is obtained.

[0199] It should be noted that the embodiment of the present disclosure does not specifically limit the execution order of step 602 and step 603, and executing step 602 and step 603 simultaneously is a preferred solution.

[0200] In some examples, in the above step 604, the text detection model can be trained according to the total loss value obtained by weighted summing the loss value of the text detection model for the sample single-word box and the loss value for the sample connected component until the training stop condition is reached.

[0201] Here, the weight values ​​corresponding to the sample single word frame and the sample connection component can be preset according to application needs, and this embodiment does not specifically limit this. For example, the loss value of the sample single word frame and the loss value of the sample connection component can be summed according to their respective corresponding predetermined ratios to obtain the total loss value of the text detection model.

[0202] Among them, the training stop condition can be, for example: when the total loss value is less than or equal to a preset threshold, or the error between the total loss value obtained in this round of training and the total loss value obtained in the previous round of training is less than a preset value.

[0203] In one embodiment, Figure 8a As shown, in the above step 602, according to the prediction probability map of the sample single word frame, the prediction threshold map of the sample single word frame, the label probability map of the sample single word frame and the label threshold map of the sample single word frame, obtaining the loss value of the text detection model for the sample single word frame may include the steps of:

[0204] 701: Determine a first loss value of a text detection model for a sample single word frame according to a prediction probability map of the sample single word frame and a label probability map of the sample single word frame;

[0205] 702: Determine a second loss value of the text detection model for the sample single word frame according to the prediction threshold map of the sample single word frame and the label threshold map of the sample single word frame;

[0206] 703: Determine a third loss value of the text detection model for the sample single word frame according to the first predicted binary map and the label probability map of the sample single word frame; wherein the first predicted binary map is obtained by binarizing the predicted probability map of the sample single word frame according to the predicted threshold map of the sample single word frame;

[0207] 704: Obtain a loss value of the text detection model for the sample single-word box according to the first loss value, the second loss value, and the third loss value.

[0208] It should be noted that the embodiment of the present disclosure does not specifically limit the execution order of step 701, step 702 and step 703, and it is a preferred solution to execute step 701, step 702 and step 703 simultaneously.

[0209] In this embodiment, the cross entropy function can be used as the loss function of the text detection model, and the loss calculation is performed on the predicted probability map and the label probability map of the sample single word frame to obtain a first loss value for the sample single word frame by the text detection model. The loss calculation is performed on the predicted threshold map and the label threshold map of the sample single word frame to obtain a second loss value for the sample single word frame by the text detection model. Moreover, the loss calculation is performed on the predicted binary map and the label probability map of the sample single word frame to obtain a third loss value for the sample single word frame.

[0210] In the above step 704, the first loss value, the second loss value and the third loss value may be weightedly summed, and the total loss value obtained by the weighted sum is used as the loss value of the text detection model for the sample single-word box.

[0211] Here, the weight values ​​corresponding to the first loss value, the second loss value and the third loss value can be preset according to application requirements, and this embodiment does not specifically limit this.

[0212] In the disclosed embodiment, loss calculations are performed respectively based on the predicted probability map and label probability map of the sample single-word box, the predicted threshold map and label threshold map, the predicted binary map and label probability map, and then multiple loss values ​​are integrated to obtain the loss value of the file detection model for the sample single-word box. In this way, the parameters of the text detection model can be better adjusted according to the final loss value, which can help improve the model performance.

[0213] In one embodiment, Figure 8b As shown, in the above step 603, according to the prediction probability map of the sample connection component, the prediction threshold map of the sample connection component, the label probability map of the sample connection component and the label threshold map of the sample connection component, obtaining the loss value of the text detection model for the sample connection component may include the steps of:

[0214] 801: Determine a fourth loss value of the text detection model for the sample connected component according to the predicted probability map of the sample connected component and the label probability map of the sample connected component;

[0215] 802: Determine a fifth loss value of the text detection model for the sample connected component according to the prediction threshold map of the sample connected component and the label threshold map of the sample connected component;

[0216] 803: Determine a sixth loss value of the text detection model for the sample connected component according to the second predicted binary map and the label probability map of the sample connected component; wherein the second predicted binary map is obtained by binarizing the predicted probability map of the sample connected component according to the predicted threshold map of the sample connected component;

[0217] 804: Obtain the loss value of the text detection model for the sample connected component according to the fourth loss value, the fifth loss value, and the sixth loss value.

[0218] It should be noted that the embodiment of the present disclosure does not specifically limit the execution order of step 801, step 802 and step 803, and it is a preferred solution to execute step 801, step 802 and step 803 simultaneously.

[0219] In this embodiment, the cross entropy function can be used as the loss function of the text detection model, and the loss calculation is performed on the predicted probability map and the label probability map of the sample connection component to obtain the fourth loss value for the sample connection component by the text detection model, and the loss calculation is performed on the predicted threshold map and the label threshold map of the sample single-word box to obtain the fifth loss value for the sample connection component by the text detection model, and the loss calculation is performed on the predicted binary map and the label probability map of the sample connection component to obtain the sixth loss value for the sample connection component.

[0220] In the above step 804, the fourth loss value, the fifth loss value and the sixth loss value may be weightedly summed, and the total loss value obtained by the weighted sum is used as the loss value of the text detection model for the sample connected component.

[0221] Here, the weight values ​​corresponding to the fourth loss value, the fifth loss value and the sixth loss value can be preset according to application requirements, and this embodiment does not specifically limit this.

[0222] In the disclosed embodiment, loss calculations are performed respectively based on the predicted probability map and label probability map, the predicted threshold map and label threshold map, the predicted binary map and label probability map of the sample connection component, and then multiple loss values ​​are integrated to obtain the loss value of the file detection model for the sample connection component. In this way, the parameters of the text detection model can be better adjusted according to the final loss value, which can help improve the model performance.

[0223] It should be noted that the method provided in the embodiments of the present disclosure may be executed alone or together with some methods in the embodiments of the present disclosure or some methods in related technologies.

[0224] In order to further explain any embodiments of the present disclosure, some specific embodiments are provided below.

[0225] In the related art, most of the text detection algorithms are marked line by line and detected line by line. However, the granularity of such marking is not fine enough, and people cannot obtain the position information at the word level. The reality is that sometimes people need more detailed text position information and line information at the single word level while obtaining the text line to facilitate further processing of the single word. On the other hand, simply performing single word detection requires subsequent processing based on single word level processing, which makes the complexity of subsequent processing very high when processing pages with a large amount of text. And the positioning at the single word level does not conform to the human reading habits of text. Therefore, it is necessary to judge the line relationship of each single word so that subsequent downstream tasks can be processed line by line.

[0226] The disclosed embodiment provides a text detection method, which can use a pre-trained text detection model to predict a model to be detected. According to the prediction result, the position and size of each word (expressed as a single-word text box) can be accurately obtained, and at the same time, it can be determined which text boxes belong to the same text line.

[0227] The training process of the text detection model may include:

[0228] 1. Marking stage:

[0229] The sample image is annotated with single words and text lines. The annotated result of a single word is a single word text box, and it is also annotated which single word text boxes are in the same line.

[0230] like Figure 9a As shown, each English word in the sample image can be marked with a single word frame, and each text line in the sample image can be marked (for example, with color information). The color information of different text lines is different.

[0231] 2. Training phase:

[0232] (1) Data processing:

[0233] The sample image and all the single-word frames in the sample image are read, and the row relationship between the marked single-word frames is determined.

[0234] For all single-word frames in the same row, the adjacent relationship between the single-word frames is determined, and a connected component is generated at the same time. The connected component is defined as a quadrilateral frame formed by the two edges of the nearest neighbors of two adjacent single-word frames.

[0235] According to the single word frame and the connected component, the method for solving the text box heat map and the text box threshold map in the DBNet algorithm is applied to the single word frame and the connected component, and then the GT single word heat map (for example, the label probability map of the sample single word frame in the above embodiment), the GT single word threshold map (for example, the label threshold map of the sample single word frame in the above embodiment), the GT connected component heat map (for example, the label probability map of the sample connected component in the above embodiment), and the GT connected component threshold map (for example, the label threshold map of the sample connected component in the above embodiment) can be obtained. Among them, the GT single word heat map and the GT single word threshold map are used to learn to determine the single word frame, and the GT connected component heat map and the GT connected component threshold map are used to learn to locate the connected component.

[0236] At this point, the processing of data for one page is completed.

[0237] (2) Deep learning training

[0238] Model network structure: The model network structure can use the backbone network used for segmentation tasks, such as unet, u2net and other networks, or it can use the backbone network combined with the feature pyramid network (FPN).

[0239] The model network structure can be found in Figure 9bAs shown, after the input image is subjected to feature extraction by the backbone network, a multi-scale feature map of a single-word frame and a multi-scale feature map of a connected component can be obtained, and the resolution of the feature map of different scales can be 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image in turn. The multi-scale feature map of the single-word frame output by the backbone network is input into the feature pyramid network for feature fusion, and a fused feature map of the single-word frame can be obtained. The multi-scale feature map of the connected component output by the backbone network is input into the feature pyramid network for feature fusion, and a fused feature map of the connected component can be obtained. The fused feature map of the single-word frame and the fused feature map of the connected component are respectively input into the segmentation network for prediction, and four label maps of the network output can be obtained, which are respectively recorded as: a single-word heat map (e.g., a predicted probability map of the sample single-word frame in the above embodiment), a single-word threshold map (e.g., a predicted threshold map of the sample single-word frame in the above embodiment), a connected component heat map (e.g., a predicted probability map of the sample connected component in the above embodiment), and a connected component threshold map (e.g., a predicted threshold map of the sample connected component in the above embodiment).

[0240] Model training process: Based on the network-predicted single-word heat map, single-word threshold map, GT single-word heat map, and GT single-word threshold map, the loss value of the single word can be calculated using the loss function in the DBNet algorithm. Based on the network-predicted connected component heat map, connected component threshold map, GT connected component heat map, and GT connected component threshold map, the loss value of the connected component can be calculated using the loss function in the DBNet algorithm. The deep learning model is trained based on the total loss value obtained by adding the loss value of the single word to the loss value of the connected component.

[0241] The text detection process may include: an inference stage.

[0242] After reading the image to be detected and passing it through the model network inference, the single-word heat map, single-word threshold map, connected component heat map, and connected component threshold map of the image to be detected are obtained. The differential binarization in the DBnet algorithm is applied to the single-word heat map and the single-word threshold map to calculate the single-word binary map. The differential binarization in the DBnet algorithm is applied to the connected component heat map and the connected component threshold map to calculate the connected component binary map. Each connected domain in the single-word binary map represents a word component. Each connected domain in the connected component binary map represents a connected component.

[0243] The inference method in the DBnet algorithm is used on the single-word binary graph to determine each single-word frame.

[0244] Based on the single word binary graph and the connected component binary graph, determine which two single word frames are adjacent single word frames. If two word components are connected by a connected component, the two words are adjacent two words. Then the single word frames corresponding to the two word components are adjacent single word frames in the same row.

[0245] According to the adjacent relationship between two single-word frames, all adjacent single-word frames are combined into a text line.

[0246] As an example, the text detection method provided by the embodiment of the present disclosure can be applied to Fig.9c The image to be detected is shown in the right figure in the figure. By detecting the image to be detected, we can get the following Fig.9c The detection result shown in the left figure in , that is, each single word frame in the image to be detected and the text line formed by connecting each single word frame through the connection component.

[0247] The technical solution provided by the embodiments of the present disclosure has at least the following beneficial effects:

[0248] 1. It can detect text lines based on full-page document images, and determine the text line position and text line height;

[0249] 2. It can locate each word on the page and determine the font height of each word;

[0250] 3. More detailed label information and better performance;

[0251] 4. Compared with simple text line detection or single word box detection, it can provide more information for downstream tasks and better assist the processing of downstream tasks;

[0252] 5. It has good scene versatility and can be applied to Chinese, English (both letter level and word level) and other language scenarios with single-word level concepts. It can also be used for text detection in natural scenes.

[0253] Fig.10 A schematic diagram of the structure of a text detection device provided by an embodiment of the present disclosure is shown. Fig.10 As shown, the text detection device 100 includes:

[0254] An acquisition module 110 is used to acquire a target image to be detected;

[0255] The prediction module 120 is used to perform single-word prediction processing on the target image to obtain a prediction result; wherein the prediction result includes: a probability map of the single-word frame in the target image and a probability map of the connected component; the connected component is connected between two adjacent single-word frames in the same text line of the target image;

[0256] A first processing module 130 is used to perform a binarization process on the probability map of the single-word frame to obtain a first target binary map;

[0257] A second processing module 140 is used to perform a binarization process on the probability map of the connected component to obtain a second target binary map;

[0258] The positioning module 150 is used to locate the single-word frame in the target image and the text line to which the single-word frame belongs according to the first target binary image and the second target binary image.

[0259] In one embodiment, the prediction module 120 is used to perform the single word prediction processing on the target image through a text detection model, and the text detection model includes a backbone network, a feature fusion network and a segmentation network; the prediction result also includes: a threshold map of the single word box in the target image and a threshold map of the connected component;

[0260] The prediction module 120 is used to:

[0261] Inputting the target image into the backbone network for feature extraction to obtain a multi-scale feature map of a single word frame in the target image and a multi-scale feature map of a connected component;

[0262] Inputting the multi-scale feature map of the single word frame and the multi-scale feature map of the connected component into the feature fusion network to obtain a fused feature map output by the feature fusion network;

[0263] The fused feature map is input into the segmentation network to obtain the probability map of the single word box, the threshold map of the single word box, the probability map of the connected component and the threshold map of the connected component output by the segmentation network.

[0264] In one embodiment, the first processing module 130 is used to:

[0265] Binarizing the probability map of the single-word frame according to the threshold map of the single-word frame to obtain the first target binary map;

[0266] The second processing module 140 is used to:

[0267] The probability map of the connected component is binarized according to the threshold map of the connected component to obtain the second target binary map.

[0268] In one embodiment, the positioning module 150 is used to:

[0269] Determine each of the single-word frames in the target image according to the first target binary image;

[0270] Determine the adjacent relationship between the single-word frames in the target image according to each of the single-word frames in the target image and the second target binary image; wherein the existence of an adjacent relationship between two of the single-word frames indicates that the two single-word frames are connected by the connection component;

[0271] The text line in the target image is located according to the adjacent relationship between the single-word frames.

[0272] In one embodiment, the apparatus further comprises a training module, wherein the training module comprises:

[0273] An acquisition unit is used to acquire a sample image and label data; the label data includes: a label probability map of a sample single word frame in the sample image, a label threshold map of the sample single word frame, a label probability map of a sample connection component, and a label threshold map of the sample connection component; the sample connection component is connected between two adjacent sample single word frames in the same text line;

[0274] A training unit is used to train the text detection model to be trained according to the sample image and the label data.

[0275] In one embodiment, the acquisition unit is used to:

[0276] Acquire the annotation information of each word in the sample image and the annotation information of the text line where each word is located, wherein the annotation information of the word is the sample word frame;

[0277] Determining the adjacent relationship between sample single word frames in the same text line in the sample image according to the annotation information of each single word and the annotation information of each text line;

[0278] Generating a sample connected component in the sample image according to the adjacent relationship between the sample single-word frames in the same text line in the sample image;

[0279] The label data is obtained according to the sample single word frame and the sample connected component in the sample image.

[0280] In one embodiment, the training unit is used to:

[0281] Input the sample image into the text detection model to be trained for prediction, and obtain a prediction probability map of the sample single word frame in the sample image, a prediction threshold map of the sample single word frame, a prediction probability map of the sample connected component, and a prediction threshold map of the sample connected component;

[0282] Obtaining a loss value of the text detection model for the sample single-word frame according to the prediction probability map of the sample single-word frame, the prediction threshold map of the sample single-word frame, the label probability map of the sample single-word frame, and the label threshold map of the sample single-word frame;

[0283] Obtaining a loss value of the text detection model for the sample connected component according to the predicted probability map of the sample connected component, the predicted threshold map of the sample connected component, the label probability map of the sample connected component, and the label threshold map of the sample connected component;

[0284] The model parameters of the text detection model are adjusted according to the loss value of the text detection model for the sample single-word frame and the loss value of the sample connected component.

[0285] In one embodiment, the training unit is used to:

[0286] Determining a first loss value of the text detection model for the sample single-word frame according to the prediction probability map of the sample single-word frame and the label probability map of the sample single-word frame;

[0287] Determining a second loss value of the text detection model for the sample single-word frame according to the prediction threshold map of the sample single-word frame and the label threshold map of the sample single-word frame;

[0288] Determine a third loss value of the text detection model for the sample single-word frame according to the first predicted binary map and the label probability map of the sample single-word frame; wherein the first predicted binary map is obtained by binarizing the predicted probability map of the sample single-word frame according to the predicted threshold map of the sample single-word frame;

[0289] According to the first loss value, the second loss value and the third loss value, the loss value of the text detection model for the sample single-word frame is obtained.

[0290] In one embodiment, the training unit is used to:

[0291] Determining a fourth loss value of the text detection model for the sample connected component according to the predicted probability map of the sample connected component and the label probability map of the sample connected component;

[0292] Determining a fifth loss value of the text detection model for the sample connected component according to the prediction threshold map of the sample connected component and the label threshold map of the sample connected component;

[0293] Determine a sixth loss value of the text detection model for the sample connected component according to the second predicted binary map and the label probability map of the sample connected component; wherein the second predicted binary map is obtained by binarizing the predicted probability map of the sample connected component according to the predicted threshold map of the sample connected component;

[0294] According to the fourth loss value, the fifth loss value and the sixth loss value, a loss value of the text detection model for the sample connected component is obtained.

[0295] Regarding the text detection device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the text detection method, and will not be elaborated here.

[0296] Fig.11 A structural diagram of a text detection model training device provided by an embodiment of the present disclosure is shown. Fig.11 As shown, the text detection model training device 200 includes a training module 210, and the training module 210 includes:

[0297] An acquisition unit is used to acquire a sample image and label data; the label data includes: a label probability map of a sample single word frame in the sample image, a label threshold map of the sample single word frame, a label probability map of a sample connection component, and a label threshold map of the sample connection component; the sample connection component is connected between two adjacent sample single word frames in the same text line;

[0298] A training unit is used to train the text detection model to be trained according to the sample image and the label data.

[0299] Here, the specific manner in which the training module performs operations may refer to the training module in the text detection device in the above embodiment, which will not be described in detail here.

[0300] Regarding the text detection model training device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the text detection model training method, and will not be elaborated here.

[0301] An embodiment of the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the text detection method or text detection model training method provided in any embodiment of the present disclosure is implemented.

[0302] The device may be, but is not limited to: a terminal and / or a server. The terminal may be, but is not limited to, a mobile phone, a tablet computer, a computer, a wearable device, a vehicle-mounted device, a smart home device and / or a smart vehicle-mounted device.

[0303] Here, the server may be various types of servers; for example, the server may be, but is not limited to, a cloud server or a local server.

[0304] The memory may include various types of storage media, which are non-transitory computer storage media that can continue to memorize information stored thereon after the communication device loses power.

[0305] The processor may be connected to the memory via a bus or the like, and is used to read an executable program stored in the memory, for example, to implement the text detection method or text detection model training method provided in any embodiment of the present disclosure.

[0306] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores an executable program, wherein when the executable program is executed by a processor, the text detection method or text detection model training method provided by any embodiment of the present disclosure is implemented.

[0307] like Fig.12 FIG. 1 is a diagram showing a structure of an electronic device 300 according to an exemplary embodiment. For example, the electronic device 300 may be provided as a network side device. Fig.12 The electronic device 300 includes a processing component 310, which further includes one or more processors, and a memory resource represented by a memory 320 for storing instructions executable by the processing component 310, such as an application. The application stored in the memory 320 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 310 is configured to execute instructions to perform the text detection method or the text detection model training method in the above-mentioned embodiment.

[0308] The electronic device 300 may further include a power supply component 330 configured to perform power management of the electronic device 300, a wired or wireless network interface 340 configured to connect the electronic device 300 to a network, and an input / output (I / O) interface 350. The electronic device 300 may operate based on an operating system stored in the memory 320, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0309] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0310] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A text detection method, characterized in that: The method comprises: Acquire the target image to be detected; Performing single-word prediction processing on the target image to obtain a prediction result; wherein the prediction result includes: a probability map of single-word frames in the target image and a probability map of connected components; the connected component is connected between two adjacent single-word frames in the same text line of the target image; Binarizing the probability map of the single-word frame to obtain a first target binary map; Binarizing the probability map of the connected components to obtain a second target binary map; A single-word box in the target image and a text line to which the single-word box belongs are located according to the first target binary image and the second target binary image.

2. The method according to claim 1, characterized in that The target image is subjected to the single-word prediction processing by a text detection model, wherein the text detection model includes a backbone network, a feature fusion network, and a segmentation network; the prediction result also includes: a threshold map of a single-word frame in the target image and a threshold map of a connected component; The step of performing single word prediction processing on the target image to obtain a prediction result includes: Inputting the target image into the backbone network for feature extraction to obtain a multi-scale feature map of a single word frame in the target image and a multi-scale feature map of a connected component; Inputting the multi-scale feature map of the single word frame and the multi-scale feature map of the connected component into the feature fusion network to obtain a fused feature map output by the feature fusion network; The fused feature map is input into the segmentation network to obtain the probability map of the single word box, the threshold map of the single word box, the probability map of the connected component and the threshold map of the connected component output by the segmentation network.

3. The method according to claim 2, characterized in that The binarization process of the probability map of the single-word frame to obtain a first target binary map includes: Binarizing the probability map of the single-word frame according to the threshold map of the single-word frame to obtain the first target binary map; The binarization process is performed on the probability map of the connected component to obtain a second target binary map, including: The probability map of the connected component is binarized according to the threshold map of the connected component to obtain the second target binary map.

4. The method according to any one of claims 1 to 3, characterized in that: The method of locating a single-word frame in the target image and a text line to which the single-word frame belongs according to the first target binary image and the second target binary image comprises: Determine each of the single-word frames in the target image according to the first target binary image; Determine the adjacent relationship between the single-word frames in the target image according to each of the single-word frames in the target image and the second target binary image; wherein the existence of an adjacent relationship between two of the single-word frames indicates that the two single-word frames are connected by the connection component; The text line in the target image is located according to the adjacent relationship between the single-word frames.

5. The method according to claim 2, characterized in that: The text detection model is trained by the following steps: Acquire a sample image and label data; the label data includes: a label probability map of a sample single word frame in the sample image, a label threshold map of the sample single word frame, a label probability map of a sample connection component, and a label threshold map of the sample connection component; the sample connection component is connected between two adjacent sample single word frames in the same text line; The text detection model to be trained is trained according to the sample image and the label data.

6. The method according to claim 5, characterized in that The obtaining of sample images and label data includes: Acquire the annotation information of each word in the sample image and the annotation information of the text line where each word is located, wherein the annotation information of the word is the sample word frame; Determining the adjacent relationship between sample single word frames in the same text line in the sample image according to the annotation information of each single word and the annotation information of each text line; Generating a sample connected component in the sample image according to the adjacent relationship between the sample single-word frames in the same text line in the sample image; The label data is obtained according to the sample single word frame and the sample connected component in the sample image.

7. The method according to claim 5 or 6, characterized in that: The step of training the text detection model to be trained according to the sample image and the label data includes: Input the sample image into the text detection model to be trained for prediction, and obtain a prediction probability map of the sample single word frame in the sample image, a prediction threshold map of the sample single word frame, a prediction probability map of the sample connected component, and a prediction threshold map of the sample connected component; Obtaining a loss value of the text detection model for the sample single-word frame according to the prediction probability map of the sample single-word frame, the prediction threshold map of the sample single-word frame, the label probability map of the sample single-word frame, and the label threshold map of the sample single-word frame; Obtaining a loss value of the text detection model for the sample connected component according to the predicted probability map of the sample connected component, the predicted threshold map of the sample connected component, the label probability map of the sample connected component, and the label threshold map of the sample connected component; The model parameters of the text detection model are adjusted according to the loss value of the text detection model for the sample single-word frame and the loss value of the sample connected component.

8. The method according to claim 7, characterized in that The step of obtaining the loss value of the text detection model for the sample single word frame according to the prediction probability map of the sample single word frame, the prediction threshold map of the sample single word frame, the label probability map of the sample single word frame, and the label threshold map of the sample single word frame comprises: Determining a first loss value of the text detection model for the sample single-word frame according to the prediction probability map of the sample single-word frame and the label probability map of the sample single-word frame; Determining a second loss value of the text detection model for the sample single-word frame according to the prediction threshold map of the sample single-word frame and the label threshold map of the sample single-word frame; Determine a third loss value of the text detection model for the sample single-word frame according to the first predicted binary map and the label probability map of the sample single-word frame; wherein the first predicted binary map is obtained by binarizing the predicted probability map of the sample single-word frame according to the predicted threshold map of the sample single-word frame; According to the first loss value, the second loss value and the third loss value, the loss value of the text detection model for the sample single-word frame is obtained.

9. The method according to claim 7, characterized in that: The step of obtaining the loss value of the text detection model for the sample connection component according to the prediction probability map of the sample connection component, the prediction threshold map of the sample connection component, the label probability map of the sample connection component, and the label threshold map of the sample connection component comprises: Determining a fourth loss value of the text detection model for the sample connected component according to the predicted probability map of the sample connected component and the label probability map of the sample connected component; Determining a fifth loss value of the text detection model for the sample connected component according to the prediction threshold map of the sample connected component and the label threshold map of the sample connected component; Determine a sixth loss value of the text detection model for the sample connection component according to the second predicted binary map and the label probability map of the sample connection component; wherein the second predicted binary map is obtained by binarizing the predicted probability map of the sample connection component according to the predicted threshold map of the sample connection component; According to the fourth loss value, the fifth loss value and the sixth loss value, a loss value of the text detection model for the sample connected component is obtained.

10. A text detection device, characterized in that: The device comprises: An acquisition module, used for acquiring a target image to be detected; A prediction module is used to perform single-word prediction processing on the target image to obtain a prediction result; wherein the prediction result includes: a probability map of a single-word frame in the target image and a probability map of a connected component; the connected component is connected between two adjacent single-word frames in the same text line of the target image; A first processing module is used to perform a binarization process on the probability map of the single-word frame to obtain a first target binary map; A second processing module is used to perform binarization processing on the probability map of the connected component to obtain a second target binary map; A positioning module is used to locate a single-word frame in the target image and a text line to which the single-word frame belongs according to the first target binary image and the second target binary image.

Citation Information

Patent Citations

  • Model training method, device, text detection method, device and lightweight network model

    CN113780283A

  • Text detection model training method and device and electronic equipment

    CN114359927A

  • Target detection model training method, text group line method and related products

    CN114821230A