Text detection and model training method, device, system and readable storage medium

Abstract

The application provides a character detection method and model training method, device and system and readable storage medium method. The model training method comprises the following steps: obtaining a sample feature map of a sample picture, inputting the sample feature map into a first prediction network to obtain a first probability map and a first threshold value map with a size smaller than the sample picture; performing a differentiable binarization process on the first probability map and the first threshold value map to obtain an approximate binary map; and performing supervised learning on the approximate binary map based on a sample label of the sample picture to train a character detection model. By using the above method, the post-processing efficiency of the character detection model can be obviously improved.

Description

Technical Field

[0001] This invention belongs to the field of character recognition, specifically relating to a character detection and its model training method, apparatus, system, and readable storage medium. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention as set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] As the demand for edge text recognition applications increases, the requirements for the efficiency of models running on hardware devices are also increasing. Therefore, optimizing and accelerating the model is particularly important. Common methods for accelerating the model include pruning, quantization, distillation, and modifying the network structure.

[0004] Existing text detection networks are mainly based on regression and segmentation methods from convolutional neural networks. Regression-based algorithms generally borrow from general object detection algorithms, either by setting regression detection boxes or by directly performing pixel regression. The advantage of this type of method is that it performs well in detecting text with regular shapes, but the disadvantage is that it performs relatively poorly in detecting text with irregular shapes.

[0005] Therefore, how to optimize and accelerate the model while ensuring its accuracy is an urgent problem to be solved. Summary of the Invention

[0006] To address the problems existing in the prior art, a method, apparatus, system, and computer-readable storage medium for text detection and its model training are proposed. Using this method, apparatus, and computer-readable storage medium, the aforementioned problems can be solved.

[0007] The present invention provides the following solutions.

[0008] In a first aspect, a method for training a text detection model is provided, comprising: acquiring a sample feature map of a sample image; inputting the sample feature map into a first prediction network to obtain a first probability map and a first threshold map with a size smaller than that of the sample image; performing differentiable binarization processing on the first probability map and the first threshold map to obtain an approximate binary map; and performing supervised learning on the approximate binary map based on the sample labels of the sample image to train and generate a text detection model.

[0009] The method further includes:

[0010] In one implementation, the method further includes: inputting a sample feature map into a second prediction network, wherein the output feature size of the second prediction network is greater than or equal to the size of the sample image, and using the output features of the second prediction network to perform intermediate supervised learning on the text detection model.

[0011] In one implementation, the method further includes: inputting a sample feature map into a second prediction network to obtain a second probability map and a second threshold map, wherein the size of the second probability map and the second threshold map is greater than or equal to the size of the sample image; calculating the probabilistic map loss of the first probability map using the second probability map; calculating the threshold map loss of the first threshold map using the second threshold map; and performing intermediate supervised learning on the text detection model using the probabilistic map loss and the threshold map loss.

[0012] In one implementation, the second probability map, the second threshold map, and the sample image are the same size.

[0013] In one implementation, the probabilistic graph loss and / or threshold graph loss are calculated using the KL divergence loss function.

[0014] In one implementation, intermediate supervision learning is performed using the following KL divergence loss function formula:

[0015] Wherein, pred refers to the output of the first prediction network, pred′ refers to the output of the second prediction network, i refers to the pixel index, and x... i "N" refers to the number of pixels, and "N" refers to the total number of pixels.

[0016] In one implementation, the size of the first probability map and the first threshold map is half the size of the sample image.

[0017] In one implementation, the first prediction network performs the following operations: convolving the sample feature map to obtain a first intermediate map; batch normalizing and activating the first intermediate map to obtain a second intermediate map; deconvolving the second intermediate map to output a third intermediate map; convolving the third intermediate map to output a fourth intermediate map with 1 channel; and outputting a first probability map and a first threshold map based on the sigmoid function and the fourth intermediate map.

[0018] In one implementation, the second prediction network performs the following operations: convolving the sample feature map to obtain a fifth intermediate map; batch normalizing and activating the fifth intermediate map to obtain a sixth intermediate map; deconvolving the sixth intermediate map to output a seventh intermediate map; deconvolving the seventh intermediate map to output an eighth intermediate map with 1 channel; and outputting a second probability map and a second threshold map based on the sigmoid function and the fourth intermediate map.

[0019] In one implementation, obtaining a sample feature map of a sample image includes: obtaining a training sample set, which includes multiple sample images carrying sample labels; and inputting the sample images into a feature extraction network to obtain a sample feature map.

[0020] In one implementation, supervised learning of the approximate binary image based on the sample labels of the sample images further includes: downsampling the sample labels so that their size is the same as that of the approximate binary image.

[0021] In one implementation, the text detection model includes at least: a trained feature extraction network and a probabilistic graph branch of a first prediction network.

[0022] Secondly, a text detection method is provided, comprising: acquiring an image to be detected; inputting the image to be detected into a text detection model trained by the method in the first aspect; outputting a probability map of the image to be detected; binarizing the probability map using a fixed threshold or a threshold map output by the text detection model to obtain a binary map; and using the binary map to perform text segmentation on the image to be detected.

[0023] Thirdly, a text detection model training device is provided, configured to perform the method of the first aspect, comprising: a feature extraction module for inputting a sample image into a feature extraction network to obtain a sample feature map; a first prediction module for inputting the sample feature map into a first prediction network to obtain a first probability map and a first threshold map with a size smaller than the sample image; a differentiable binarization module for performing differentiable binarization processing on the first probability map and the first threshold map to obtain an approximate binary map; and a training module for performing supervised learning on the approximate binary map based on the sample labels of the sample image to train and generate a text detection model.

[0024] Fourthly, a text detection apparatus is provided, configured to perform the method of the second aspect. The apparatus includes: a detection module for acquiring an image to be detected, inputting the image to be detected into a text detection model trained by the method of the first aspect, and outputting a probability map of the image to be detected; a binarization module for binarizing the probability map using a fixed threshold or a threshold map output by the text detection model to obtain a binary map; and a text segmentation module for segmenting the image to be detected using the binary map.

[0025] Fifthly, a text detection model training system is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform: the method of the first aspect.

[0026] A sixth aspect provides a text detection system, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform: the method of the second aspect.

[0027] In a seventh aspect, a computer-readable storage medium is provided, which stores a program that, when executed by a processor, causes the processor to perform the method as described in the first or second aspect.

[0028] One of the advantages of the above implementation method is that by training with probability maps and threshold maps smaller than the sample images, the post-processing efficiency of the trained text detection model can be significantly improved.

[0029] Other advantages of the present invention will be explained in more detail below with reference to the accompanying drawings.

[0030] It should be understood that the above description is merely an overview of the technical solution of the present invention, so as to enable a clearer understanding of the technical means of the present invention, and thus to implement it in accordance with the contents of the specification. To make the above and other objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are illustrated below. Attached Figure Description

[0031] By reading the detailed description of the exemplary embodiments below, those skilled in the art will understand the advantages and benefits described herein, as well as other advantages and benefits. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0032] Figure 1 This is a schematic diagram of the structure of a text detection model training device according to an embodiment of the present invention;

[0033] Figure 2 This is a flowchart illustrating a text detection model training method according to an embodiment of the present invention.

[0034] Figure 3 This is a flowchart illustrating another text detection model training method according to an embodiment of the present invention;

[0035] Figure 4 This is a flowchart illustrating another text detection model training method according to an embodiment of the present invention.

[0036] Figure 5 This is a flowchart illustrating another text detection model training method according to an embodiment of the present invention.

[0037] Figure 6 This is a schematic diagram of the calculation process of a first prediction network according to an embodiment of the present invention;

[0038] Figure 7 This is a schematic diagram of the calculation process of the second prediction network according to an embodiment of the present invention;

[0039] Figure 8 This is a flowchart illustrating a text detection method according to an embodiment of the present invention;

[0040] Figure 9 This is a flowchart illustrating another text detection method according to an embodiment of the present invention;

[0041] Figure 10 This is a schematic diagram of the structure of a text detection model training device according to an embodiment of the present invention;

[0042] Figure 11 This is a schematic diagram of the structure of a text detection device according to an embodiment of the present invention.

[0043] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0044] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0045] In the description of embodiments of this application, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, numbers, steps, behaviors, components, portions or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence of one or more other features, numbers, steps, behaviors, components, portions or combinations thereof.

[0046] Unless otherwise stated, " / " means "or". For example, A / B can mean A or B. In this article, "and / or" is merely a way of describing the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0047] The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0048] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] First see Figure 1The illustration schematically shows an environment 100 in which an exemplary implementation according to the present disclosure can be used.

[0050] Figure 1 A schematic diagram of an example of a computing device 100 according to an embodiment of the present disclosure is shown. It should be noted that... Figure 1 This is a schematic diagram of the hardware operating environment for the text detection model training method. The text detection model training device in this embodiment can be a PC, portable computer, or other terminal device.

[0051] like Figure 1 As shown, the text detection model training device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0052] Those skilled in the art will understand that Figure 1 The structure of the text detection model training device shown in the figure does not constitute a limitation on the text detection model training device. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0053] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a text detection model training program. The operating system is a program that manages and controls the hardware and software resources of the text detection model training device, supporting the operation of the text detection model training program and other software or programs.

[0054] exist Figure 1 In the text detection model training device shown, the user interface 1003 is mainly used to receive requests and data sent by the first terminal, the second terminal, and the monitoring terminal; the network interface 1004 is mainly used to connect the backend server and communicate with it; and the processor 1001 can be used to call the text detection model training program stored in the memory 1005 and perform the following operations:

[0055] The process involves obtaining sample feature maps from sample images, inputting these feature maps into a first prediction network to obtain a first probability map and a first threshold map smaller than the sample image. Differentiable binarization is then applied to the first probability map and the first threshold map to obtain approximate binary maps. Supervised learning is then performed on these approximate binary maps based on the sample labels of the sample images to train a text detection model. Therefore, training with probability maps and threshold maps smaller than the sample image significantly improves the post-processing efficiency of the trained text detection model.

[0056] The aforementioned invention also lies in optimizing the neural network structure of the first prediction network, adaptively adjusting its convolution kernel and weight size so that the output size of the first prediction network is smaller than the first probability map and the first threshold map of the sample image. Because the output size is smaller, the processing speed of the first prediction network will be improved. The effect is also that the output size of the probability map and threshold map smaller than the sample image can reduce the amount of computation and improve the efficiency of text detection in post-processing. Post-processing refers to performing binarization, contour calculation, contour expansion, bounding rectangle calculation, etc. based on the first probability map and the first threshold map to generate segmentation features for text detection.

[0057] Figure 2 A flowchart illustrating a method for training a text detection model according to an embodiment of the present disclosure is shown. This method can, for example, be performed by... Figure 1 The method is executed by the computing device 100 shown. It should be understood that the method 200 may also include additional boxes not shown and / or the boxes shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0058] Step 210: Obtain the sample feature map of the sample image;

[0059] Step 220: Input the sample feature map into the first prediction network to obtain the first probability map and the first threshold map.

[0060] Specifically, the size of the first probability map and the first threshold map is smaller than the size of the sample image; for example, the size of the first probability map and the first threshold map can be 1 / 2 of the sample image.

[0061] Step 230: Differentiable binarization is performed on the first probability map and the first threshold map to obtain an approximate binary map.

[0062] Step 240: Based on the sample labels of the sample images, supervised learning is performed on the probability map, threshold map, and approximate binary map to train and generate a text detection model.

[0063] refer to Figure 3In the training process of the text detection model, sample images are input into the model to be trained. This text detection model is based on the DBNET network structure and includes a feature extraction network, a first prediction network, and a differentiable binarization module. The feature extraction network includes an FPN network and a concat operation module. In step 210, the sample image is processed through the FPN network structure of the feature extraction network to obtain four feature maps, which are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 the size of the sample image, respectively. Then, the four feature maps are upsampled and fused, and then a concat operation is performed to obtain... Figure 3 The sample feature map F is 1 / 4 the size of the original sample image. The first prediction network may include a pred' prediction head. The feature extraction network described above is not limited to the feature extraction steps described above, and the size of the final extracted feature map is not limited to 1 / 4 of the original sample image; it can also be 1 / 8, 1 / 16, etc.

[0064] In step 220, the sample feature map is input into the first prediction network. A series of convolutional operations, batch normalization, ReLU operations, deconvolution operations, and sigmoid function operations are performed on the 1 / 4-sized feature map through the pred' prediction head, outputting a first probability map and a first threshold map of 1 / 2 the size of the sample image, respectively. Furthermore, DBnet proposes a differentiable binarization (DB) module, which can perform binarization processing in the segmentation network. The differentiable binarization method with adaptive thresholding can not only distinguish text regions from the background but also separate closely connected text instances. Based on this, in step 230, the generated first probability map and first threshold map are input into the differentiable binarization (DB) module, outputting an approximate binary map. Subsequently, in step 240, a loss function 1 is calculated based on the output probability map, threshold map, approximate binary map, and sample labels of the sample image. Based on the loss calculation results, the parameters of the feature extraction network, the first prediction network, and the differentiable binarization module in the text detection model are adjusted until the model converges.

[0065] The sample label is a sample label map, which includes a sample probability map, a sample threshold map, and a sample binary map, such as a binary map that can clearly mark the foreground and background parts of the sample image.

[0066] Probability Map: The value of each pixel represents the probability that the location belongs to the text region; during model training, it corresponds to the sample probability map. Threshold Map: The value of each pixel represents the binarization threshold at that location; during model training, it corresponds to the sample threshold map. Approximate Binary Map: The value of each pixel is either 0 or 1, calculated using the DB algorithm based on the probability map and threshold map; during model training, it corresponds to the sample binary map.

[0067] Before calculating the loss function using the sample labels, probability map, threshold map, and approximate binary map of the sample images, the sample labels of the sample images can be downsampled first. This ensures that the size of the downsampled sample labels matches that of the probability map, threshold map, and approximate binary map, which is more conducive to model training.

[0068] It is understandable that because the output probability map and threshold map are smaller than the original sample image, the efficiency of the trained text detection model can be significantly improved in post-processing calculations such as binarization, contour calculation, contour expansion, and bounding rectangle calculation.

[0069] In one implementation, in step 210 above, to obtain the sample feature map, the following operations can be performed: First, a training sample set is collected, wherein the training sample set includes multiple sample images carrying sample labels; the sample labels may be text contour annotation information of the sample image. Then, the sample images are input into a feature extraction network to obtain the sample feature map.

[0070] Figure 4 This is a flowchart illustrating a text detection model method according to another exemplary embodiment of the present invention. This embodiment... Figure 2 Based on the implementation method shown, the detection accuracy is further corrected.

[0071] The sample feature map is input into the second prediction network. The output feature size of the second prediction network is greater than or equal to the size of the sample image. The output features of the second prediction network are used to perform intermediate supervised learning on the text detection model. The output features of the second prediction network, which has been trained, meets the accuracy requirements, and has converged, supervise the first prediction network. Because the output feature size of the second prediction network is large enough and its accuracy is sufficient for accurate text detection, it can provide a reference for the first prediction network, which has a smaller output size. This allows for adjustments to the parameters of the feature extraction network, the first prediction network, and the differentiable binarization module in the text detection model, ensuring that the output of the first prediction network meets the accuracy requirements and can accurately detect text.

[0072] like Figure 4 As shown, the method provided in this embodiment may further include the following steps:

[0073] Step 251: Input the sample feature map into the second prediction network to obtain the second probability map and the second threshold map.

[0074] Among them, the size of the second probability map and the second threshold map is greater than or equal to the size of the sample image;

[0075] Step 252: Calculate the probability graph loss of the first probability graph using the second probability graph;

[0076] Step 253: Calculate the threshold map loss of the first threshold map using the second threshold map;

[0077] Step 254: Perform intermediate supervised learning on the text detection model using probabilistic graphical loss and threshold graphical loss.

[0078] refer to Figure 5 The second prediction network may include a pred prediction head. In step 251, the sample feature map is input into the second prediction network. A series of convolutional operations, batch normalization operations, ReLU operations, deconvolution operations, convolutional operations, and sigmoid function operations are performed through the pred prediction head to output a second probability map and a second threshold map with a size greater than or equal to that of the sample image. In steps 242 and 253, after downsampling the second probability map and the second threshold map or upsampling the first probability map and the first threshold map, the probability map loss between the second probability map and the first probability map, and the threshold map loss between the second threshold map and the first threshold map are calculated, respectively. Loss function 2 is calculated, and the parameters of the feature extraction network, the first prediction network, and the differentiable binarization module in the text detection model are adjusted based on the loss calculation results.

[0079] It is understandable that the second probability map and the second threshold map mentioned above have the same size as the sample image and are therefore more accurate. By using this loss function 2, the more accurate segmentation edge information of the high resolution can be effectively transferred to the segmentation map of the low resolution output, thereby correcting the results of the low resolution prediction.

[0080] Optionally, the second probability map and the second threshold map mentioned above are the same size as the sample image. This preserves sufficiently high map accuracy and can effectively correct the results of low-resolution predictions.

[0081] Optionally, the first probability map and the first threshold map mentioned above are half the size of the sample image. This significantly reduces the amount of computational data and effectively improves post-processing efficiency.

[0082] Optionally, during training, cross-training can be performed using loss function 1 and loss function 2. Alternatively, loss function 1 and loss function 2 can be weighted and merged into a single loss function before overall training. This application does not impose specific limitations in this regard.

[0083] It should be noted that steps not described in detail in this embodiment can be referred to... Figure 2 The descriptions of the relevant steps in the illustrated implementation are not repeated here.

[0084] In one implementation, in steps 252 and 253 above, the KL divergence loss function can be used to calculate the probability map loss and / or threshold map loss. The KL divergence calculates the expected value of the logarithmic difference between the probabilities of the original distribution and the approximate distribution of the data, which can effectively represent the degree of distribution difference between the less accurate first probability map / first threshold map and the more accurate second probability map / second threshold map.

[0085] Specifically, intermediate supervised learning can be performed using the following KL divergence loss function formula:

[0086]

[0087] Where, pred′(x i ) refers to the output of the first prediction network, which is the output of the first probability map / the first threshold map corresponding to x. i The value of a pixel, pred(x) i ) refers to the output of the second prediction network, which is the output of the second probability map / second threshold map corresponding to x. i The value of a pixel, where i refers to the pixel index and N refers to the total number of pixels.

[0088] Alternatively, other loss functions that can represent the degree of distributional difference can be used to calculate the probabilistic graph loss and / or threshold graph loss, and this application does not impose specific restrictions on this.

[0089] In one implementation, to obtain a first probability map and a first threshold map with a size of half that of the sample image, the first prediction network may perform the following operations:

[0090] (1) Perform a convolution operation on the sample feature map to obtain the first intermediate map;

[0091] (2) Perform batch standardization and activation processing on the first intermediate graph to obtain the second intermediate graph;

[0092] (3) Perform deconvolution on the second intermediate image to output the third intermediate image;

[0093] (4) Perform a convolution operation on the third intermediate image to output a fourth intermediate image with 1 channel;

[0094] (5) Based on the sigmoid function and the fourth intermediate graph, output the first probability graph and the first threshold graph.

[0095] refer to Figure 6For example, first, a (3*3) convolution operation is performed on the sample feature map to compress the channels to 1 / 4 of the input. Then, after batch normalization and activation processing, a feature map of size (batch size, 64, 1 / 4W, 1 / 4H) is obtained, which is the second intermediate map. The second intermediate map is then deconvolved with a (2*2) kernel to obtain a feature map of size (batch size, 256, 1 / 2W, 1 / 2H), which is the third intermediate map. At this time, the size of the third intermediate map is 1 / 2 of the original image. The third intermediate map is then convolved with a (3*3) convolution operation to output a feature map with 1 channel and a size of (batch size, W / 2, H / 2), which is the fourth intermediate map. Finally, the fourth intermediate map is processed by the sigmoid function to output a first probability map P and a first threshold map T, which are 1 / 2 the size of the original image. The neural network structure of the first prediction network is not limited to the above embodiments. The key is to adjust the structure and then adaptively adjust its convolutional kernels and weights to output a feature map with 1 channel and a size of (batch size, W / 2, H / 2), i.e., the fourth intermediate image. Then, using the pre-trained second prediction network and sample labels, supervised learning is performed on the first prediction network to adjust its parameters. The parameters of the feature extraction network are also adjusted to train a text detection model that meets the accuracy requirements. Finally, it can output a first probability map and a first threshold map that are smaller than the original image size and have higher accuracy, which are then used in post-processing calculations to speed up text detection time.

[0096] In one implementation, in order to obtain a second probability map and a second threshold map of the same size as the sample image, the second prediction network can perform the following operations:

[0097] (1) Perform a convolution operation on the sample feature map to obtain the fifth intermediate map;

[0098] (2) Perform batch standardization and activation processing on the fifth intermediate graph to obtain the sixth intermediate graph;

[0099] (3) Perform deconvolution on the sixth intermediate image to output the seventh intermediate image;

[0100] (4) Perform deconvolution on the seventh intermediate image to output the eighth intermediate image with 1 channel;

[0101] (5) Based on the sigmoid function and the fourth intermediate graph, output the second probability graph and the second threshold graph.

[0102] refer to Figure 7First, a feature map of 1 / 4 the size of the sample image is obtained through a feature extraction network, with dimensions (batch size, 256, 1 / 4W, 1 / 4H). This is then processed by the pred prediction head to obtain the second probability map and the second threshold map. Specifically, a (3*3) convolution operation is performed on the sample feature map to compress the channels to 1 / 4 of the input. Then, batch normalization and activation processing are performed to obtain a feature map of size (batch size, 64, 1 / 4W, 1 / 4H), which is the sixth intermediate image. The sixth intermediate image is then inverted... A convolution operation with a 2x2 kernel is performed to obtain a feature map of size (batch size, 256, 1 / 2W, 1 / 2H), which is the seventh intermediate image. This seventh intermediate image is half the size of the original image. A 2x2 deconvolution operation is then performed on the seventh intermediate image, outputting a feature map with 1 channel and size (batch size, W, H), which is the eighth intermediate image. Finally, the eighth intermediate image is passed through a sigmoid function to output a second probability map P2 and a second threshold map T2 of the original image size. The neural network structure of the second prediction network is not limited to the above embodiment. It can be understood that the second prediction network is a relatively mature and well-trained model for text detection in the existing technology. Its output second probability map and second threshold map already meet the accuracy requirements and can achieve good text detection results. Therefore, its output second probability map and second threshold map are used to supervise the first probability map and first threshold map of the first prediction network to adjust the parameters of the first prediction network. The parameters of the feature extraction network correspondingly arranged at the front end of the first prediction network can also be adjusted.

[0103] In one implementation, the text detection model includes: a trained feature extraction network and a probabilistic graph branch of a first prediction network. This saves computation on the threshold graph portion in subsequent text detection models, allowing for calculations using a fixed threshold, thus effectively improving post-processing efficiency.

[0104] Alternatively, the text detection model can also include a trained feature extraction network and a complete first prediction network. In this case, detection accuracy can be better guaranteed.

[0105] In the description of this specification, the references to terms such as "some possible implementations," "some implementations," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that implementation or example is included in at least one implementation or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same implementation or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more implementations or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different implementations or examples described in this specification, as well as the features of different implementations or examples.

[0106] The method flowcharts for embodiments of this application describe certain operations as different steps performed in a certain order. Such flowcharts are illustrative and not restrictive. Some steps described herein may be grouped together and performed in a single operation, some steps may be divided into multiple sub-steps, and some steps may be performed in an order different from that shown herein. The various steps shown in the flowcharts may be implemented in any way by any circuit structure and / or tangible mechanism (e.g., software running on a computer device, hardware (e.g., logic functions implemented by a processor or chip), and / or any combination thereof).

[0107] Based on the same technical concept, the present invention also provides a text detection method, which is a reasoning method for text contour detection using a text detection model trained by any of the above embodiments. Figure 8 A flowchart of a text detection method provided for an embodiment of the present invention.

[0108] refer to Figure 8 The method 800 includes:

[0109] Step 810: Obtain the image to be detected, input the image to be detected into the text detection model trained by the method in the above embodiment, and output the probability map of the image to be detected.

[0110] Step 820: Use a fixed threshold or a threshold map output by a text detection model to binarize the probability map to obtain a binary map;

[0111] Step 830: Perform text segmentation on the image to be detected using the binary image. This determines the background and text portions of the image, forming text detection boxes. Before text segmentation, the binary image needs to be upsampled to match the size of the image to be detected.

[0112] refer to Figure 9For example, the image to be detected can be input into a text detection model. A feature map is generated by a trained feature extraction network. This feature map is then input into a first prediction network, which uses the probability map branch to generate a probability map. This probability map is then binarized using a fixed threshold to obtain an approximate binary image. This approximate binary image can then undergo post-processing. Finally, the post-processed image is upsampled to the original image size. Then, based on the upsampled post-processed image, the background and text portions of the image to be detected are identified, and the detection boxes are determined, resulting in an image with detection boxes. Using the fixed threshold method can save computation and improve processing efficiency.

[0113] Alternatively, instead of the fixed threshold method described above, a threshold map can be generated based on the threshold map branch of the first prediction network, and then binarized based on the threshold map and the probability map to obtain the approximate binary map. This can achieve more accurate detection results, and this application does not limit this approach.

[0114] Based on the same technical concept, embodiments of the present invention also provide a text detection model training device for executing the text detection model training method provided in any of the above embodiments. Figure 10 This is a schematic diagram of a text detection model training device provided for an embodiment of the present invention.

[0115] refer to Figure 10 The device 100 includes:

[0116] Feature extraction module 101 is used to input sample images into the feature extraction network to obtain sample feature maps;

[0117] The first prediction module 102 is used to input the sample feature map into the first prediction network to obtain a first probability map and a first threshold map with a size smaller than the sample image.

[0118] Differentiable binarization module 103 is used to perform differentiable binarization processing on the first probability map and the first threshold map to obtain an approximate binary map;

[0119] Training module 104 is used to supervise the learning of approximate binary images based on sample labels of sample images, and train a text detection model.

[0120] Based on the same technical concept, embodiments of the present invention also provide a text detection device for performing the text detection method provided in any of the above embodiments. Figure 11 This is a schematic diagram of a text detection device provided for an embodiment of the present invention.

[0121] refer to Figure 11 The device includes:

[0122] The detection module 111 is used to acquire the image to be detected, input the image to be detected into the text detection model trained by the method of any one of claims 1-10, and output the probability map of the image to be detected.

[0123] Binarization module 112 is used to binarize the probability map using a fixed threshold or a threshold map output by a text detection model to obtain an approximate binary map.

[0124] The text segmentation module 113 is used to segment text in the image to be detected using an approximate binary image.

[0125] It should be noted that the apparatus in the embodiments of this application can implement each process of the aforementioned method and achieve the same effect and function, which will not be elaborated here.

[0126] According to some embodiments of this application, a non-volatile computer storage medium for a text detection model training method or a text detection method is provided, having stored computer-executable instructions configured to be executed by a processor: the method described in the above embodiments.

[0127] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the descriptions of the apparatus, device, and computer-readable storage medium embodiments are simplified because they are substantially similar to the method embodiments; relevant details can be found in the descriptions of the method embodiments.

[0128] The apparatus, device, and computer-readable storage medium provided in this application correspond one-to-one with the method. Therefore, the apparatus, device, and computer-readable storage medium also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the apparatus, device, and computer-readable storage medium will not be repeated here.

[0129] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus (devices or systems), or computer-readable storage media. Therefore, the present invention can take the form of a completely hardware implementation, a completely software implementation, or an implementation combining software and hardware aspects. Furthermore, the present invention can take the form of a computer-readable storage medium embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-readable program code.

[0130] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices or systems), and computer-readable storage media according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0135] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. Furthermore, although the operations of the method of the invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0136] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for training a text detection model, characterized in that, The method comprises: obtaining a sample feature map of a sample picture, inputting the sample feature map into a first prediction network to obtain a first probability map and a first threshold map with a size smaller than the sample picture; performing differentiable binarization processing on the first probability map and the first threshold map to obtain an approximate binary map; performing supervised learning on the approximate binary map based on a sample label of the sample picture to train a text detection model; inputting the sample feature map into a second prediction network to obtain a second probability map and a second threshold map with a size greater than or equal to the size of the sample picture; based on the second probability map and the second threshold map, respectively calculating a first probability map loss and a first threshold map loss to perform intermediate supervised learning on the text detection model.

2. The method of claim 1, wherein, The second probability map, the second threshold map and the sample picture have the same size.

3. The method of claim 1, wherein, Further comprising: calculating the probability map loss and / or the threshold map loss using a KL divergence loss function.

4. The method of claim 3, wherein, The intermediate supervised learning is performed using the following KL divergence loss function formula: ， Wherein, the is the output of the first prediction network, and the is the output of the second prediction network, and the is the pixel point sequence number, and the is the pixel point, is the total pixel quantity.

5. The method of claim 1, wherein, The size of the first probability map and the first threshold map is 1 / 2 of the size of the sample picture.

6. The method of claim 1, wherein, The first prediction network performs: performing convolution operation on the sample feature map to obtain a first intermediate map; performing batch normalization processing and activation processing on the first intermediate map to obtain a second intermediate map; performing deconvolution operation on the second intermediate map to output a third intermediate map; performing convolution operation on the third intermediate map to output a fourth intermediate map with a channel number of 1; outputting the first probability map and the first threshold map according to a sigmoid function and the fourth intermediate map.

7. The method of claim 6, wherein, The second prediction network performs: performing convolution operation on the sample feature map to obtain a fifth intermediate map; performing batch normalization processing and activation processing on the fifth intermediate map to obtain a sixth intermediate map; performing deconvolution operation on the sixth intermediate map to output a seventh intermediate map; performing deconvolution operation on the seventh intermediate map to output an eighth intermediate map with a channel number of 1; outputting the second probability map and the second threshold map according to a sigmoid function and the fourth intermediate map.

8. The method of claim 1, wherein, The method of obtaining a sample feature map of a sample picture comprises: obtaining a training sample set comprising a plurality of sample pictures carrying sample labels; inputting the sample pictures into a feature extraction network to obtain sample feature maps.

9. The method of claim 1, wherein, The method of performing supervised learning on the approximate binary map based on a sample label of the sample picture further comprises: performing down-sampling processing on the sample label to make its size the same as that of the approximate binary map.

10. The method of claim 8, wherein, The text detection model at least comprises a trained feature extraction network and a probability map branch of the first prediction network.

11. A character detection method characterized by comprising: The method comprises: obtaining a to-be-detected image, inputting the to-be-detected image into a text detection model trained by the method of any one of claims 1-10 to output a probability map of the to-be-detected image; performing binarization processing on the probability map using a fixed threshold or a threshold map output by the text detection model to obtain a binary map; performing text segmentation on the to-be-detected image using the binary map.

12. A character detection model training apparatus, characterized by comprising: The apparatus is configured to perform the method of any one of claims 1-10. The feature extraction module is configured to input a sample picture into a feature extraction network to obtain a sample feature map; The first prediction module is configured to input the sample feature map into a first prediction network to obtain a first probability map and a first threshold map with a size smaller than the sample picture; The differentiable binarization module is configured to perform differentiable binarization processing on the first probability map and the first threshold map to obtain an approximate binary map; The training module is configured to perform supervised learning on the approximate binary map based on a sample label of the sample picture to train a text detection model.

13. A character detection apparatus characterized by comprising: The apparatus configured to perform the method of claim 11 comprises: The detection module is configured to obtain a picture to be detected, input the picture to be detected into the text detection model trained by the method of any one of claims 1-10, and output a probability map of the picture to be detected; The binarization module is configured to perform binarization processing on the probability map using a fixed threshold or a threshold map output by the text detection model to obtain a binary map; The text segmentation module is configured to perform text segmentation on the picture to be detected using the binary map. 14.A system for training a text detection model, the system comprising: comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

15. A character detection system, characterized by, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of claim 11.

16. A computer readable storage medium, the computer readable storage medium storing a program, when the program is executed by a processor, causing the processor to perform the method of any one of claims 1-10.

Images