A character detection method, a terminal and a readable storage medium

Abstract

The embodiment of the present application relates to the field of computer vision, and discloses a character detection method, a terminal and a readable storage medium.In the present application, the character detection method comprises: forming marks on sample pictures in a data set, wherein the marks at least comprise reduced text box marks, and the marks further comprise distance distribution marks and / or boundary distribution marks; inputting the sample pictures after the marks are formed into a character detection model to be trained, to obtain a trained character detection model, and the trained character detection model is used for character detection.The character detection method, the terminal and the readable storage medium provided by the present application can improve the accuracy of character region detection, and further improve the accuracy of character recognition results.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a text detection method, terminal, and readable storage medium. Background Technology

[0002] Text appearing in natural scenes often possesses rich and precise semantic information. The ability to identify this text from natural scenes has high application potential, such as in autonomous driving, scene understanding, and assistive technology for the blind. Scene text detection involves using algorithms to directly discover and locate text regions within scene images. Scene text detection plays a crucial role as a preliminary step in scene text recognition, directly impacting the effectiveness of subsequent text recognition. In recent years, with the rapid development of deep learning, deep learning-based text detection has made breakthrough progress, achieving detection and recognition results far exceeding traditional methods. Currently, deep learning-based text detection methods include segmentation-based methods. However, due to the fuzziness and sparsity at text boundaries, segmentation-based scene text detection methods still suffer from low accuracy in text region detection, leading to lower accuracy in text recognition results. Summary of the Invention

[0003] The purpose of this invention is to provide a text detection method, terminal, and readable storage medium that can improve the accuracy of text region detection, thereby improving the accuracy of text recognition results.

[0004] To address the aforementioned technical problems, embodiments of the present invention provide a text detection method, comprising: forming markers on sample images within a dataset, wherein the markers include at least reduced text box markers, and the markers further include distance distribution markers and / or boundary distribution markers; inputting the marked sample images into a text detection model to be trained to obtain a trained text detection model, wherein the trained text detection model is used for text detection.

[0005] Embodiments of the present invention also provide a terminal, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the text detection method as described above.

[0006] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described text detection method.

[0007] Compared with the prior art, the embodiments of the present invention utilize distance distribution markers and boundary distribution markers to more accurately describe the ambiguity and sparsity at text boundaries, enhance the supervision information required for model training, thereby improving the accuracy of text region detection and thus improving the accuracy of text recognition results. Attached Figure Description

[0008] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0009] Figure 1 This is a flowchart illustrating a text detection method provided in one embodiment of the present invention;

[0010] Figure 2 This is a schematic diagram of the prediction network structure of the trained text detection model;

[0011] Figure 3 This is a flowchart illustrating a text detection method provided in another embodiment of the present invention;

[0012] Figure 4 This is a schematic diagram of the structure of a terminal provided in one embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of the present invention to enable the reader to better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0014] Segmentation-based scene text detection uses a neural network model to predict a probability map containing positional and shape information, and then uses this probability map to generate predicted text boxes. To avoid adjacent text regions from overlapping during prediction, related techniques use reduced text box markers to train the neural network model.

[0015] The inventors discovered two problems with this method: First, there is a difference between the reduced marker and the actual text region. During the training phase, this difference makes it difficult for the model to fit, thus affecting the training effect. Second, the reduced marker is not good for smaller text regions. Because smaller text regions contain very few positive examples to begin with, further reducing the text region on this basis makes it difficult to achieve the effect of supervised training for excessively small text regions, causing the model to easily miss smaller text regions during prediction.

[0016] In addition, there are two phenomena in real text markup: First, it is difficult to determine whether the pixels at the boundary of the text box are positive or negative examples, that is, the ambiguity of the boundary; Second, when manually marking text boxes, in order to ensure that the text box can better frame the text area, the text box is usually slightly larger, and the strokes of the characters at the boundary of the text box are usually sparser, that is, the sparseness of the boundary.

[0017] This invention relates to a text detection method. The method involves forming markers on sample images within a dataset. These markers include at least reduced text box markers, and further include distance distribution markers and / or boundary distribution markers. The marked sample images are then input into a text detection model to be trained, resulting in a trained text detection model. This trained model can be used for more accurate and efficient text detection. Utilizing distance distribution markers and / or boundary distribution markers can more accurately describe the ambiguity and sparsity at text boundaries, enhancing the supervisory information needed for model training, thereby improving the accuracy of text region detection and ultimately improving the accuracy of text recognition results.

[0018] The following is a detailed description of the implementation details of the text detection method in this embodiment. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0019] According to one embodiment of this application, a text detection method is provided, such as... Figure 1 As shown, the following operations may be included, but are not limited to. Of course, in other embodiments, fewer than the following steps may be required.

[0020] Operation S11: Create labels on sample images within the dataset, wherein the labels include at least reduced text box labels, and may also include distance distribution labels and / or boundary distribution labels.

[0021] In an exemplary embodiment, forming distance distribution markers on sample images within a dataset may include the following steps: generating an original text box and a reduced text box on the sample image; marking pixels in the sample image located inside the reduced text box as a first marker value, marking pixels in the sample image located outside the original text box as a second marker value, and uniformly marking pixels in the sample image located between the reduced text box and the original text box as transitional real numbers between the second marker value and the first marker value according to the distance variation, so as to form a distance distribution marker on the sample image with high internal values ​​and low external values.

[0022] In some embodiments, forming boundary distribution markers on sample images within a dataset may include the following steps: generating an original text box, a reduced text box, and an enlarged text box on the sample image; marking pixels in the sample image located inside the reduced text box and outside the enlarged text box as second marker values; marking pixels in the sample image located on the original text box as first marker values; and uniformly marking pixels in the sample image located between the reduced text box and the original text box, and between the original text box and the enlarged text box, as transitional real numbers between the second marker value and the first marker value according to the change in distance, so as to form boundary distribution markers on the sample image with the highest values ​​at the boundaries and the values ​​gradually decreasing in both inward and outward directions.

[0023] In an exemplary embodiment, both the second marker value and the first marker value are real numbers. The second marker value may be less than the first marker value. For example, the second marker value may be 0 and the first marker value may be 1. The pixels in the sample image located between the reduced text box and the original text box can be uniformly marked as transitional real numbers between [0, 1] according to the change in distance.

[0024] In an exemplary embodiment, the method for generating a reduced text box and an enlarged text box can be as follows: the text box is reduced or enlarged using the Vatticlipping algorithm, and the scaling offset D is calculated using the following formula:

[0025]

[0026] Where A is the area of ​​the text box, L is the perimeter of the text box, and r is the scaling factor, set to 0.4.

[0027] Operation S12: Input the labeled sample images into the text detection model to be trained to obtain the trained text detection model.

[0028] In an exemplary embodiment, prior to this step, the method may further include: creating the text detection model to be trained, wherein the text detection model to be trained includes a backbone network and a prediction network, wherein the backbone network is used to extract image features, and the backbone network may be a pyramid network constructed from ResNet-50, wherein the 3×3 convolutional layers in the backbone network may use deformable convolutions.

[0029] In an exemplary embodiment, the prediction network may include a backbone branch and auxiliary branches, wherein the backbone branch is used to output a probability map of the channels corresponding to the reduced text box markers, and the auxiliary branches are used to output a probability map of the channels corresponding to the distance distribution markers and / or the boundary distribution markers.

[0030] In other words, the prediction network of the text detection model to be trained can use multiple branch designs: one is the leaf branch, which outputs a probability map of multiple channels at the end of the network, with each channel corresponding to a label; the other is the trunk branch, which uses the feature map extracted by the backbone network as input to predict the probability map of one channel corresponding to a label.

[0031] like Figure 2 As shown, the prediction network of the text detection model to be trained can include: one 3×3 convolutional layer, two 2×2 transposed convolutional layers, and a batch normalization layer and a ReLU layer immediately following the 3×3 convolutional layer and the first 2×2 transposed convolutional layer. After the second 2×2 transposed convolutional layer, a Sigmoid layer is used to output a probability map that conforms to the standard.

[0032] It is understandable that the trained text detection model is similar to the text detection model to be trained, so it will not be elaborated here.

[0033] In this embodiment, the markers may include the reduced text box markers, the distance distribution markers, and the boundary distribution markers. The total loss of the model training can be a weighted sum of the losses of the reduced text box markers, the distance distribution markers, and the boundary distribution markers; wherein the weighting coefficient of the loss of the reduced text box markers is greater than the weighting coefficient of the loss of the distance distribution markers, and the weighting coefficient of the loss of the reduced text box markers is greater than the weighting coefficient of the loss of the boundary distribution markers. Since the principle of weight setting is based on the target marker, i.e., the reduced text box markers used to predict the text boxes, assigning the largest weight (weighting coefficient) to the loss of the reduced text box markers can yield a more accurate total loss of the model training.

[0034] It is understood that the markers may also include only the reduced text box markers and the distance distribution markers. In this case, the total loss of the model training can be a weighted sum of the losses of the reduced text box markers and the distance distribution markers. Similarly, the markers may also include only the reduced text box markers and the boundary distribution markers. In this case, the total loss of the model training can be a weighted sum of the losses of the reduced text box markers and the boundary distribution markers.

[0035] In an exemplary embodiment, the loss composition of model training, the total training loss can be expressed as a weighted sum of the losses of each label. For example, the total training loss can be:

[0036] L=α×L s +β×L d +γ×L b

[0037] Among them, L s To reduce the loss of the labels, L d For the loss of distance distribution labels, L b The loss is labeled for the boundary distribution. The principle of weight setting is to assign the target label, that is, to the shrinking label used to predict the text box, with the largest weight. The weights of the remaining labels are assigned according to their relevance to the target label. The labels with higher relevance are assigned larger weights, and vice versa. For example, α can be set to 2, and β and Y can be set to 1.

[0038] The losses for the reduced text box markers and the distance distribution markers can be calculated using the binary cross-entropy loss function. Specifically, the loss calculation can employ hard sample mining techniques, with a positive-to-negative sample ratio of 1:3. The specific formula for the loss function is as follows:

[0039]

[0040] Among them, P S For the selected positive and negative sample sets, y i It marks the pixel value, x i It predicts pixel values.

[0041] L b You can choose L1 loss, as follows:

[0042]

[0043] Among them, y i Indicates the marker pixel value, x i R represents the predicted pixel value. d This indicates an enlarged text box area.

[0044] In some embodiments, the dataset may consist of images containing text in various scenarios. The dataset may include a synthetic dataset and real-world detected data. Step S12 may include: sequentially inputting sample images from the labeled synthetic dataset and sample images from the labeled real-world dataset into the text detection model to be trained, thereby obtaining the trained text detection model. The synthetic dataset consists of natural scene text data generated by a computer program, such as the SynthText dataset. By first pre-training the text detection model using a synthetic dataset and then fine-tuning it using a real-world dataset, the model training becomes both fast and accurate.

[0045] In an exemplary embodiment, regarding model training and optimization, the batch size can be 8, the optimization method can be stochastic gradient descent (SGD), the momentum can be 0.9, the weight decay can be 0.0001, and the strategy for changing the learning rate with the number of training iterations can be:

[0046]

[0047] Where r is the current learning rate, r′ is the initial learning rate (which can be set to 0.007), and power can be 0.9.

[0048] Operation S13: Input the image to be detected into the trained text detection model to obtain the text detection result.

[0049] Compared with the prior art, the embodiments of the present invention use distance distribution markers and / or boundary distribution markers for model training. The use of distance distribution markers and / or boundary distribution markers can more accurately describe the ambiguity and sparsity at the text boundaries, enhance the supervision information required for model training, thereby improving the accuracy of text region detection and thus improving the accuracy of text recognition results.

[0050] The second embodiment of the present invention relates to a text detection method. The second embodiment is largely the same as the first embodiment, with the main difference being that in the first embodiment, the image to be detected is input into a trained text detection model to obtain the text detection result. In the second embodiment, only the backbone network and main branches of the trained text detection model are retained to obtain a prediction model, and the image to be detected is input into the prediction model to obtain the text detection result. Furthermore, those skilled in the art will understand that by removing auxiliary branches from the prediction network of the trained text detection model and retaining only the main branches used to generate text boxes, and by using only the probability map corresponding to the reduced markers during model prediction, the text detection speed is improved.

[0051] The text detection method in this embodiment, such as Figure 3 As shown, the following steps may be included:

[0052] S21: Create labels on sample images within the dataset, wherein the labels include at least reduced text box labels, and may also include distance distribution labels and / or boundary distribution labels.

[0053] S22: Input the labeled sample images into the text detection model to be trained to obtain the trained text detection model.

[0054] Steps S21 and S22 in this embodiment are similar to steps S11 and S12 in the first embodiment, and will not be described again here.

[0055] S23: Only retain the backbone network and main branches in the trained text detection model to obtain the prediction model.

[0056] In other words, during the prediction phase, the auxiliary branches of the prediction network of the trained text detection model were removed, and only the main branch used to generate text boxes was retained. When the model predicts, only the probability map corresponding to the reduced marker is used, thereby improving the text detection speed.

[0057] S24: Input the image to be detected into the prediction model to obtain the text detection results.

[0058] Specifically, in the prediction phase, firstly, a fixed threshold is selected to perform threshold segmentation on the probability map to generate a {0, 1} binary map. The threshold is chosen between 0.4 and 0.8 based on the actual segmentation effect. Then, text boxes are determined based on the connected component boundaries of the binary map. Finally, the text boxes are enlarged by a distance D′:

[0059]

[0060] Where A′ is the area of ​​the text box, L′ is the perimeter of the text box, and r′ is the magnification, set to 1.5.

[0061] It should be noted that the pyramid network backbone used for image extraction can be built from other networks or be other types of backbone networks, with the purpose of extracting image features. The prediction network branches can also be other types of network branches, with the purpose of corresponding to multiple supervised learning labels. The text box shrinking method can also be other shrinking methods. The shrinking labels are generated as difference labels between pixels inside and outside the text box, or other values. The distance distribution labels can also be generated in other forms, using a distribution form to transition the difference labels in step five. For training loss calculation, the weights of the loss for each label can also be other values, as long as the target label has the largest weight and the weights of other labels are smaller. The loss calculation for each label can also use other loss functions, including but not limited to binary cross-entropy loss and L1 loss. The model training and optimization methods are not limited to pre-training and fine-tuning, and gradient descent optimization methods, but can also be other optimization methods such as Adam and AdaGrad. The learning rate can also be set using other methods. In model prediction, the method for enlarging the predicted text box can also be other methods.

[0062] As shown in the table below, ResNet-18 and ResNet-50 represent the effects of building the backbone network of the pyramid network and training with only reduced labels, while "Dis" and "Bor" represent the effects of supervised training with added distance distribution labels and boundary distribution labels.

[0063] ResNet-18 84.7 77.0 80.6 ResNet-18+Dis 86.5 80.6 83.5 ResNet-18+Dis+Bor 88.1 79.9 83.8 ResNet-50 90.5 77.9 83.7 ResNet-50+Dis 90.9 80.6 85.4 ResNet-50+Dis+Bor 91.5 80.9 85.9

[0064] The experimental results show that the detection performance of the model is significantly improved after training with distance distribution labels. Further training with boundary distribution labels will further improve the performance of the model. The text detection method in this embodiment is fully verified by ablation experiments on the standard text detection dataset MSRA-TD500, and the effect is significantly improved.

[0065] Compared to existing technologies, the embodiments of this invention utilize distance distribution markers and boundary distribution markers for model training. These markers more accurately describe the ambiguity and sparsity at text boundaries, enhancing the supervisory information required for model training and thus improving the accuracy of text region detection, ultimately leading to improved text recognition accuracy. Furthermore, during the training phase, multiple prediction branches are used to enhance supervised training with multiple markers, improving model training performance. In the prediction phase, only the prediction branch corresponding to the smaller marker is used to predict the final text box, increasing the speed of text detection.

[0066] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0067] According to one embodiment of this application, a terminal is provided, such as... Figure 4 As shown, it includes: at least one processor 201, and a memory 202 communicatively connected to at least one processor 201, wherein the memory 202 stores instructions that can be executed by at least one processor 201, and the instructions are executed by at least one processor 201 to enable at least one processor 201 to perform the text detection method as described above.

[0068] The memory 202 and processor 201 can be connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors 201 and memory 202 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 201 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 201.

[0069] Processor 201 can be used to manage the bus and perform general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 202 can be used to store data used by processor 201 during operation.

[0070] According to one embodiment of this application, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the above-described text detection method embodiment.

[0071] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. Such a computer program can be stored in a device (e.g., a computer) readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus. The computer readable medium includes, but is not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, a readable medium includes any medium by which a device (e.g., a computer) stores or transmits information in a readable form. Those skilled in the art will understand that the steps, measures, and schemes in the various operations, methods, and processes discussed in this invention can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this invention can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0072] Furthermore, the steps, measures, and schemes in the related technologies that are consistent with the various operations, methods, processes, and procedures disclosed in this invention can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0073] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A text detection method, characterized in that, include: Labels are generated on sample images within the dataset, wherein the labels include at least reduced text box labels, and the labels also include distance distribution labels and / or boundary distribution labels; The labeled sample image is input into the text detection model to be trained to obtain the trained text detection model, which is used for text detection. The process of creating labels on sample images within the dataset includes: Generate an original text box and a reduced text box on the sample image; The pixels in the sample image located inside the reduced text box are marked as the first mark value, the pixels in the sample image located outside the original text box are marked as the second mark value, and the pixels in the sample image located between the reduced text box and the original text box are uniformly marked as transitional real numbers between the second mark value and the first mark value according to the change in distance, so as to form the distance distribution mark on the sample image; The process of creating labels on sample images within the dataset also includes: Generate an original text box, a reduced text box, and a larger text box on the sample image; Pixels in the sample image located inside the reduced text box and outside the enlarged text box are marked as second mark values. Pixels in the sample image located on the original text box are marked as first mark values. Pixels in the sample image located between the reduced text box and the original text box, and between the original text box and the enlarged text box, are uniformly marked as transitional real numbers between the second mark value and the first mark value according to the change in distance, so as to form the boundary distribution mark on the sample image.

2. The text detection method according to claim 1, characterized in that, Before inputting the labeled sample image into the text detection model to be trained to obtain the trained text detection model, the process further includes: Create the text detection model to be trained, wherein the text detection model to be trained includes a backbone network and a prediction network, the prediction network includes a main branch and an auxiliary branch, the main branch is used to output the probability map of the channel corresponding to the reduced text box marker, and the auxiliary branch is used to output the probability map of the channel corresponding to the distance distribution marker and / or the boundary distribution marker.

3. The text detection method according to claim 2, characterized in that, After inputting the labeled sample image into the text detection model to be trained to obtain the trained text detection model, the process further includes: Only the backbone network and main branches of the trained text detection model are retained to obtain the prediction model; The image to be detected is input into the prediction model to obtain the text detection result.

4. The text detection method according to claim 1, characterized in that, The markers include the reduced text box markers, the distance distribution markers, and the boundary distribution markers. The total loss of the text detection model to be trained is the weighted sum of the losses of the reduced text box markers, the distance distribution markers, and the boundary distribution markers. Wherein, the weighting coefficient of the loss of the reduced text box marker is greater than the weighting coefficient of the loss of the distance distribution marker, and the weighting coefficient of the loss of the reduced text box marker is greater than the weighting coefficient of the loss of the boundary distribution marker.

5. The text detection method according to claim 4, characterized in that, The loss for the reduced text box marker and the loss for the distance distribution marker are calculated using the binary cross-entropy loss function.

6. The text detection method according to claim 1, characterized in that, The dataset includes a synthetic dataset and real-world detected datasets. The step of inputting the labeled sample images into the text detection model to be trained, to obtain the trained text detection model, includes: The sample images from the labeled synthetic dataset and the sample images from the labeled real dataset are sequentially input into the text detection model to be trained to obtain the trained text detection model.

7. A terminal, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the text detection method as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the text detection method according to any one of claims 1 to 6.

Images