Method and apparatus for detecting dense text
By segmenting dense text images into sub-text boxes and performing feature extraction and anchor point regression, the problem of high computational cost during training of dense text detection models is solved, achieving faster training speed and more accurate detection results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU SHIYUAN ELECTRONICS CO LTD
- Filing Date
- 2021-08-17
- Publication Date
- 2026-05-12
AI Technical Summary
Existing dense text detection models require a large amount of computation during training, resulting in slow training speeds.
Text boxes in dense text images are segmented into multiple sub-text boxes, and the text detection model is trained on a sub-text box basis. This includes feature extraction through convolutional layers, anchor point regression, feature learning through bidirectional long short-term memory network layers, and classification and regression through region candidate network layers.
This reduces the computational load during training, improves the training speed of dense text detection models, and ensures detection performance.
Smart Images
Figure CN115880706B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image detection, and more particularly to a method and apparatus for detecting dense text. Background Technology
[0002] Due to the inherent characteristics of text, text in some images is often quite dense, even overlapping, forming dense text. Because of the mutual interference between characters in dense text, its detection has always been a challenge in text detection.
[0003] In related technologies, two common approaches are used to detect dense text. One approach employs the Progressive Scale Expansion Network (PSENet) algorithm. PSENet incorporates breadth-first search into text segmentation, demonstrating good performance in detecting dense text. The other approach utilizes text detection models (e.g., Textboxes++, Pixel-Anchor) that can detect dense text by replicating anchor points horizontally or vertically.
[0004] However, the PSENet algorithm requires traversing every pixel for training and detection, resulting in a large computational load. Copying anchors horizontally or vertically requires replicating a large number of anchor points to ensure that closely connected lines of text are matched, further increasing the computational load during model training. Therefore, existing text detection models for dense text all have a relatively high computational load during training. Summary of the Invention
[0005] This application provides a method and apparatus for detecting dense text, which can solve the problem of high computational load during training of existing text detection models for dense text. The technical solution is as follows:
[0006] Firstly, embodiments of this application provide a method for training a text detection model, including:
[0007] Obtain a dense text image, wherein the dense text image contains dense text with characters arranged densely, and the dense text is marked with text boxes;
[0008] The text boxes in the dense text image are cut into multiple sub-text boxes;
[0009] The dense text image and the sub-text boxes are used as a sample dataset to train the text detection model;
[0010] The image to be detected is input into the text detection model, and the preliminary predicted text box of the image to be detected is obtained from the output of the text detection model.
[0011] Optionally, the step of dividing the text boxes in the dense text image into multiple sub-text boxes includes:
[0012] Based on the preset height of the sub-text box, the text box is divided into multiple sub-text boxes in a vertical direction from the center of the text box.
[0013] Optionally, after dividing the text box into multiple sub-text boxes, the method further includes:
[0014] If the height of the sub-text boxes cut out from the top and bottom boundaries of the text box is less than the preset height of the sub-text box, then the sub-text boxes cut out from the top and bottom boundaries of the text box will be filled until their height is equal to the preset height of the sub-text box.
[0015] Optionally, training the text detection model using the dense text image and the sub-text boxes as a sample dataset includes:
[0016] The convolutional layer of the text detection model is used to extract features from the dense text image to generate a first feature map of the dense text image;
[0017] Based on the first feature map of the dense text image, candidate reference boxes for the dense text image are generated using an anchor point regression mechanism;
[0018] Match the candidate reference box with the sub-text box to determine the intersection-union ratio parameter of the candidate reference box and the sub-text box;
[0019] Based on the intersection-union ratio parameter, determine whether the candidate reference box should be used as an anchor reference box for regression.
[0020] Optionally, determining whether the candidate reference box should be used as an anchor reference box for regression based on the intersection-union ratio parameter includes:
[0021] If the intersection-union ratio parameter is greater than a threshold, the maximum value of the candidate reference box in the y-axis direction is greater than or equal to the minimum value of the sub-text box cut out from the upper boundary in the y-axis direction, and the minimum value of the candidate reference box in the y-axis direction is less than or equal to the maximum value of the sub-text box cut out from the lower boundary in the y-axis direction, then the candidate reference box is determined as an anchor reference box for regression.
[0022] Optionally, the step of using the dense text image and the sub-text boxes as a sample dataset to train the text detection model further includes:
[0023] The feature vector of the first feature region of the first feature map of the dense text image is converted into a horizontal sequence feature vector to generate a second feature map;
[0024] The transformed second feature image is then input into the bidirectional long short-term memory network layer of the text detection model for feature learning.
[0025] Optionally, the step of using the dense text image and the sub-text boxes as a sample dataset to train the text detection model further includes:
[0026] The anchor reference boxes are classified and regressed using the region candidate network layer of the text detection model.
[0027] Optionally, the category information of the anchor reference box includes text information, non-text information, and boundary information.
[0028] Optionally, after inputting the image to be detected into the text detection model and obtaining the preliminary predicted text box of the image to be detected output by the text detection model, the method further includes:
[0029] The preset algorithm is used to remove redundant boxes from the preliminary predicted text boxes;
[0030] The preliminary predicted text boxes, after removing redundant boxes, are merged to generate the final predicted text boxes.
[0031] Optionally, the preset algorithm includes a nonmaximum suppression algorithm.
[0032] Secondly, embodiments of this application provide a training apparatus for a text detection model, comprising:
[0033] The acquisition module is used to acquire a dense text image, which contains dense text with characters arranged densely and text boxes marked on the dense text;
[0034] A cutting module is used to cut text boxes in the dense text image into multiple sub-text boxes;
[0035] The training module is used to train the text detection model using the dense text image and the sub-text boxes as a sample dataset.
[0036] The detection module is used to input the image to be detected into the text detection model and obtain the preliminary predicted text box of the image to be detected output by the text detection model.
[0037] Optionally, the cutting module is specifically used to cut the text box into multiple sub-text boxes in a vertical direction from the center of the text box, based on the preset height of the sub-text box.
[0038] Optionally, the cutting module is further configured to fill the sub-text boxes cut from the upper and lower boundaries of the text box to a height equal to the preset height of the sub-text box if the height of the sub-text boxes cut from the upper and lower boundaries of the text box is less than the preset height of the sub-text box.
[0039] Optionally, the training module is specifically used to extract features from the dense text image through the convolutional layer of the text detection model to generate a first feature map of the dense text image; generate candidate reference boxes for the dense text image using an anchor point regression mechanism based on the first feature map of the dense text image; match the candidate reference boxes with the sub-text boxes to determine the intersection-union ratio (IU) parameter between the candidate reference boxes and the sub-text boxes; and determine whether the candidate reference boxes are used as anchor point reference boxes for regression based on the IU parameter.
[0040] Optionally, the training module is specifically used to determine the candidate reference box as the anchor reference box for regression if the intersection-union ratio parameter is greater than a threshold, the maximum value of the candidate reference box in the y-axis direction is greater than or equal to the minimum value of the sub-text box cut out from the upper boundary in the y-axis direction, and the minimum value of the candidate reference box in the y-axis direction is less than or equal to the maximum value of the sub-text box cut out from the lower boundary in the y-axis direction.
[0041] Optionally, the training module is specifically used to convert the feature vector of the first feature region of the first feature map of the dense text image into a horizontal sequence feature vector to generate a second feature map; and input the transformed second feature map into the bidirectional long short-term memory network layer of the text detection model for feature learning.
[0042] Optionally, the training module is specifically used to classify and regress the anchor reference boxes through the region candidate network layer of the text detection model.
[0043] Optionally, the category information of the anchor reference box includes text information, non-text information, and boundary information.
[0044] Optionally, the detection module is also used as a clearing module to clear redundant boxes from the preliminary prediction text boxes using a preset algorithm; and a merging module to merge the preliminary prediction text boxes after the redundant boxes have been cleared to generate the final prediction text boxes.
[0045] Optionally, the preset algorithm includes a nonmaximum suppression algorithm.
[0046] Thirdly, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the method described in any of the first aspects.
[0047] Fourthly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.
[0048] Fifthly, embodiments of this application provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.
[0049] In this embodiment, a dense text image is first acquired, containing densely packed text with text boxes marked on it. Next, the text boxes in the dense text image are segmented into multiple sub-text boxes. Then, the dense text image and the sub-text boxes are used as a sample dataset to train a text detection model. Finally, the image to be detected is input into the text detection model, and the preliminary predicted text boxes of the image to be detected are obtained from the output of the text detection model. This method reduces the computational load during training and improves the training speed of the text detection model for dense text images by training the text detection model on a per-segment basis using segmented sub-text boxes. Attached Figure Description
[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0051] Figure 1 This is a schematic diagram of an existing text detection method;
[0052] Figure 2 This is a schematic diagram of another existing text detection method;
[0053] Figure 3 This is a schematic diagram of another existing text detection method;
[0054] Figure 4 A flowchart illustrating a dense text detection method provided in an embodiment of this application;
[0055] Figure 5 A schematic diagram illustrating the principle of training a text detection model provided in an embodiment of this application;
[0056] Figure 6 A schematic diagram of a text box segmentation provided in an embodiment of this application;
[0057] Figure 7This is a schematic diagram of feature map conversion provided in an embodiment of this application;
[0058] Figure 8 A flowchart illustrating a method for detecting dense text provided in an embodiment of this application;
[0059] Figure 9 A schematic diagram of the structure of a dense text detection device provided in an embodiment of this application;
[0060] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0061] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0062] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0063] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in the various drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0064] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0065] Embodiment 1 of this application provides a training method for a text detection model.
[0066] Embodiment 2 of this application provides a method for detecting dense text.
[0067] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.
[0068] The non-contradictory technical features described in multiple different embodiments can also be combined arbitrarily to form optional implementation methods.
[0069] For example, Example 1 describes the training method of the text detection model. To keep the length of this specification short, this feature is not described in Example 2. However, those skilled in the art should be able to deduce after reading this specification that the dense text detection method provided in Example 2 can also include this feature.
[0070] The following section will first introduce existing text detection technologies.
[0071] In the first text detection method, the SegLink++ algorithm can be used for text detection. Figure 1 This is a schematic diagram of an existing text detection method. (For example...) Figure 1 As shown, the SegLink++ algorithm can distinguish dense text by adding attractive links and repulsive links to the image. However, the SegLink++ algorithm is prone to false positives for text blocks with contiguous lines of text, thus failing to distinguish dense text.
[0072] The second text detection method can employ the PSENet algorithm. This algorithm incorporates breadth-first search into text segmentation, demonstrating good performance in detecting dense text. Figure 2 This is a schematic diagram of another existing text detection method. (Example) Figure 2 As shown, the PSENet algorithm requires traversing every pixel, resulting in a large amount of computation and a slow speed.
[0073] In the third type of text detection method, the text detection model can detect dense text by replicating anchor points in the horizontal or vertical direction. Examples of such text detection models include Textboxes++ and Pixel-Anchor. Figure 3 This is a schematic diagram of another existing text detection method. For example... Figure 3 The image shows a schematic of vertically replicating anchor points.
[0074] However, among the text detection methods mentioned above, the text detection models for dense text require a large amount of computation during training, resulting in a slow training speed.
[0075] To address the aforementioned issues, this application provides a method and apparatus for detecting dense text. The method trains the text detection model by dividing it into sub-text boxes, thereby reducing the computational load during training and increasing the training speed of the text detection model for dense text while ensuring the detection performance of dense text.
[0076] It is understood that the above-mentioned dense text detection method can be implemented by the dense text detection device provided in the embodiments of this application. This dense text detection device can be, for example, a terminal device or a server. The technical solutions of the embodiments of this application will be described in detail below with specific examples. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
[0077] The following will be combined with the appendix Figure 4 -Appendix Figure 6 This paper provides a detailed description of the dense text detection method provided in the embodiments of this application.
[0078] Example 1
[0079] This application discloses a method for detecting dense text, which is applied to a dense text detection device.
[0080] Figure 4 This is a flowchart illustrating a method for detecting dense text provided in an embodiment of this application. Figure 5 This is a schematic diagram illustrating the principle of training a text detection model according to an embodiment of this application. The following will be discussed in conjunction with the attached diagram. Figure 4 and Figure 5 This paper provides a detailed description of the dense text detection method provided in the embodiments of this application.
[0081] Please see Figure 4 and Figure 5 The method includes the following steps:
[0082] S201, Obtain dense text image.
[0083] In this application, when it is necessary to train a text detection model, a dense text image can be obtained, and the text boxes of the dense text can be pre-annotated in the dense text image.
[0084] In this application, the text detection model can be a text detection module for dense text, which is used to detect and mark the location of the text.
[0085] The dense text image contains dense text, and text boxes are marked on the dense text. Dense text is text in which the characters are closely arranged. For example, dense text can be text in which the characters are stuck together, or it can be text in which the spacing between the characters is smaller than the identifiable spacing (the identifiable spacing can be, for example, 1 mm).
[0086] It should be understood that a text box on dense text can be interpreted as the ground truth (GT) of the dense text image, that is, the text region in the dense text image as determined by the user.
[0087] It should be understood that the embodiments of this application do not limit how dense text image pages are obtained. In some embodiments, the user can actively input dense text images, while in other embodiments, a dense text detection device can obtain dense text images from the network, and then the user can mark the dense text on the dense text image through a text box.
[0088] S202, cut the text boxes in the dense text image into multiple sub-text boxes.
[0089] In this step, after the dense text detection device acquires the dense text image, it can cut the text boxes in the dense text image into multiple sub-text boxes.
[0090] This application does not limit how the text box is cut. In some embodiments, the dense text detection device can cut the text box into multiple sub-text boxes according to the preset height of the sub-text boxes, with the center of the text box moving up and down.
[0091] For example, Figure 6 This is a schematic diagram illustrating the segmentation of a text box as provided in an embodiment of this application. Figure 6 As shown, when cutting a text box, you can first determine the position of the center line of the text box, and then cut out sub-text boxes in sequence along the center line from the preset height of the text box until all parts of the text box are cut into the sub-text boxes.
[0092] It should be understood that the preset height of the sub-text box is a fixed value that can be set by the user. This application embodiment does not limit the preset height of the sub-text box. A larger preset height results in fewer text boxes being segmented, reducing computational complexity, but also making boundary information less clear. Conversely, a smaller preset height results in more text boxes being segmented, increasing computational complexity, but also making boundary information more clear. It should be noted that the preset height of the sub-text box should not be less than the height of the smallest text box in the sample dataset.
[0093] Continue to refer to Figure 6In some embodiments, if the height of the sub-text boxes cut out from the top and bottom boundaries of the text box is less than the preset height of the sub-text box, then the sub-text boxes cut out from the top and bottom boundaries of the text box are filled to a height equal to the preset height of the sub-text box.
[0094] S203 uses dense text images and sub-text boxes as sample datasets to train the text detection model.
[0095] In this step, after the dense text detection device segments the text boxes in the dense text image into multiple sub-text boxes, the dense text image and the sub-text boxes can be used as a sample dataset to train the text detection model.
[0096] It should be understood that the embodiments of this application do not limit the structure of the text detection model. For example, the text detection model may include convolutional layers, bidirectional long short-term memory network layers, region candidate network layers, etc.
[0097] The following three steps describe the process of detecting dense text using a sample set.
[0098] In the first step, the dense text detection device extracts features from the dense text image using convolutional neural networks (CNNs) in the text detection model, generating a first feature map of the dense text image. Subsequently, based on the first feature map, the device generates candidate reference boxes for the dense text image using an anchor point regression mechanism. Finally, the device matches the candidate reference boxes with the sub-text boxes, determines the intersection-over-union (IoU) parameter between the candidate reference boxes and the sub-text boxes, and uses the IoU parameter to determine whether the candidate reference box should be used as an anchor reference box for regression.
[0099] The intersection-over-union (IoU) parameter is a parameter that measures the accuracy of detecting corresponding objects in a specific dataset. It is used to detect the degree of overlap between candidate reference boxes and sub-text boxes.
[0100] It should be understood that the embodiments of this application do not limit how to determine whether a candidate reference box is used as an anchor reference box for regression. In some embodiments, if the intersection-union ratio parameter is greater than the threshold of the intersection-union ratio parameter, the maximum value of the candidate reference box in the y-axis direction is greater than or equal to the minimum value of the sub-text box cut out from the upper boundary in the y-axis direction, and the minimum value of the candidate reference box in the y-axis direction is less than or equal to the maximum value of the sub-text box cut out from the lower boundary in the y-axis direction, then the candidate reference box is determined to be used as an anchor reference box for regression.
[0101] In this embodiment of the application, the threshold value range of the intersection-union ratio parameter is not limited. It can be set according to the actual situation. For example, the value range of the intersection-union ratio parameter can be greater than or equal to 0.5.
[0102] For example, when the IoU is greater than the threshold, it is not necessarily a positive sample. For sub-text boxes at the top and bottom boundaries, there may be cases where the IoU is greater than the threshold but no text is contained. Therefore, for sub-text boxes at the top and bottom boundaries, a matching condition needs to be added, that is, not only must the IoU be greater than the threshold, but the formula (1) must also be satisfied, as shown below:
[0103]
[0104] in, This represents the maximum value of candidate reference box a along the y-axis. The minimum value of candidate reference box a in the y-axis direction. This represents the minimum value of the sub-text boxes cut out from the upper boundary along the y-axis. This represents the maximum value of the sub-text boxes cut out from the lower boundary along the y-axis.
[0105] This method ensures that the matched anchor reference boxes are positive samples while also facilitating the acquisition of boundary information. The number of anchor reference boxes is set to n, and the width variation coefficient is m, where m < 1. That is, the preset width of the anchor reference boxes is [h' / m, h' / m2, h' / m3, ..., h' / mn]. A smaller m value is more conducive to detecting thin and elongated text. Simultaneously, dilated convolution can be used to increase the receptive field, improve the detection performance of thin and elongated text, and cover text of different widths.
[0106] In the second step, the dense text detection device can first convert the feature vector of the first feature region of the first feature map of the dense text image into a horizontal sequence feature vector to generate a second feature map. Subsequently, the dense text detection device can input the transformed second feature map into the bidirectional long short-term memory network layer of the text detection model for feature learning.
[0107] For example, Figure 7 This is a schematic diagram of feature map conversion provided in an embodiment of this application, such as... Figure 7As shown, a 3x3 sliding window is applied to the first feature map with a scale of N x C x H x W output from a Convolutional Neural Network (CNN) layer. This means that each point combines the features of its surrounding 3x3 region to obtain a 3x3*C feature vector, resulting in a second feature map of N x 9C x H x W output. Subsequently, the second feature map is transformed into (NH) x W x 9C, and then batch-processed with a batch size of NH and a maximum time duration of Tmax = W into a Bidirectional Long Short-Term Memory (BLSTM) layer to learn the sequence features of each row. Before output, the shape is reshaped back to its original form. In this way, the feature map incorporates both spatial and sequence features.
[0108] It should be understood that the characteristics of written characters contain sequential semantic information; that is, there are relationships between the strokes of a character, and there is contextual information between the strokes. CNNs learn spatial information within the receptive field, while Long Short-Term Memory (LSTM) networks are used to learn sequential features. Using bidirectional LSTM, or BLSTM, can learn more contextual information than a typical unidirectional LSTM, and it also helps to distinguish boundaries.
[0109] In the third step, the dense text detection device can classify and regress anchor reference boxes using the Region Proposal Network (RPN) layer of the text detection model. The category information of the anchor reference boxes includes text information, non-text information, and boundary information.
[0110] For example, the anchor reference box containing text information contains text, the anchor reference box containing non-text information does not contain text, and the anchor reference box containing boundary information contains the boundary of the text box.
[0111] For example, if there are n anchor reference boxes, 3n scores represent the category information of the n anchor reference boxes, with three categories: text, non-text, and boundary. 2n horizontal coordinates represent the regression of the left side of the center x and the width of the n anchor reference boxes, and 8n side refinement represents the regression of the upper and lower boundaries, including the center coordinates x, y, and four types of information, plus the upper and lower boundaries, making a total of eight types of information. The formula (2) for the classification loss function is as follows:
[0112]
[0113] In this context, information marked with an asterisk (*) is labeled, while information without an asterisk (*) is predicted. Correspondingly, S i The category labeled for the i-th anchor point reference box. V is the predicted category for the i-th anchor reference box; i Let x be the left side and width of the center x of the reference box mark for the i-th anchor point. The left side and width of the predicted center x of the i-th anchor point reference box; O i The upper and lower boundaries of the reference box for the i-th anchor point are marked. The upper and lower boundaries are predicted for the reference box of the i-th anchor point.
[0114] Furthermore, the classification loss employs a logistic regression (softmax) model, with the regression using a smooth L1 loss function. Correspondingly, L... cls For the regression parameters of the categories, The regression parameters are the left side and the width of the center x. The upper and lower bounds are the regression parameters. λ represents the weights of the regression losses for both tasks, used to balance the loss weights for different tasks. for The corresponding weights for The corresponding weights, due to the differentiation of densely packed text, therefore Greater than
[0115] In this embodiment, the dense text detection device first acquires a dense text image, which contains densely arranged text with text boxes marked on it. Then, the device segments the text boxes in the dense text image into multiple sub-text boxes. Finally, the device uses the dense text image and the sub-text boxes as a sample dataset to train a text detection model. This method reduces the computational load during training and improves the detection speed for dense text within dense text.
[0116] Example 2
[0117] Based on the above embodiments, the following explains how to use a text detection model to detect dense text. Figure 8 For a flowchart illustrating another dense text detection method provided in this application embodiment, please refer to [link / reference]. Figure 8 The method includes the following steps:
[0118] S301, acquire the image to be detected.
[0119] In this embodiment of the application, when a user needs to detect dense text in an image to be detected, the image to be detected can be input into a dense text detection device. After the dense text detection device obtains the image to be detected, it begins to detect the dense text.
[0120] The image to be detected may contain dense text with characters closely arranged. For example, dense text may include text where characters are connected together.
[0121] It should be understood that the embodiments of this application do not limit the type of image to be detected, and it can be a picture, video, etc.
[0122] S302, input the image to be detected into the text detection model, and obtain the preliminary predicted text box of the image to be detected output by the text detection model.
[0123] In this step, after the dense text detection device acquires the image to be detected, it can input the image to be detected into the text detection model and obtain the preliminary predicted text box of the image to be detected output by the text detection model.
[0124] The text detection model is generated by training on a dense text image with pre-annotated text boxes on the dense text as a sample dataset. The text boxes in the dense text image are cut into multiple sub-text boxes.
[0125] It should be noted that this application does not limit the training method of the text detection model, and the training method shown in Embodiment 1 can be used. Furthermore, this application does not limit the structure of the text detection model; for example, the text detection model may include the following layers:
[0126] Convolutional layers are used to extract features from dense text images, generating the first feature map of the dense text image.
[0127] The matching layer is used to generate candidate reference boxes for the dense text image based on the first feature map of the dense text image using the anchor point regression mechanism; the candidate reference boxes are matched with the sub-text boxes to determine the intersection-union ratio (IU) parameter between the candidate reference boxes and the sub-text boxes; and based on the IU parameter, it is determined whether the candidate reference boxes are used as anchor reference boxes for regression.
[0128] Specifically, if the intersection-union ratio parameter is greater than a threshold, the maximum value of the candidate reference box in the y-axis direction is greater than or equal to the minimum value of the sub-text box cut out from the upper boundary in the y-axis direction, and the minimum value of the candidate reference box in the y-axis direction is less than or equal to the maximum value of the sub-text box cut out from the lower boundary in the y-axis direction, then the candidate reference box is determined as the anchor reference box for regression.
[0129] The deformation layer is used to convert the feature vectors of the first feature region of the first feature map of the dense image into horizontal sequence feature vectors, generating the second feature map.
[0130] A bidirectional long short-term memory network layer is used for feature learning using the deformed second feature map.
[0131] The classification and regression layer is used to classify and regress the anchor reference boxes.
[0132] In some embodiments, after obtaining the preliminary predicted text boxes of the image to be detected output by the text detection model, the dense text detection device can also use the Non-Maximum Suppression (NMS) algorithm to remove redundant boxes from the preliminary predicted text boxes, and then merge the preliminary predicted text boxes after removing the redundant boxes to generate the final predicted text boxes.
[0133] For example, a text detection model predicts a series of preliminary predicted text boxes. Using the NMS algorithm, redundant boxes can be removed from these preliminary predicted text boxes. Subsequently, since the preliminary predicted text boxes all contain category information such as text or boundaries, the preliminary predicted text boxes within the upper and lower boundaries after removing redundant boxes are merged to output the final predicted text boxes.
[0134] This method allows for the classification and detection of boundaries in dense text, enabling faster text detection while maintaining accuracy.
[0135] The dense text detection method provided in this application first acquires a target image containing densely packed text. Then, the target image is input into a text detection model, and the model outputs preliminary predicted text boxes. This model is generated after training on a dataset of dense text images pre-annotated with text boxes, where the text boxes are segmented into multiple sub-text boxes. This approach, by training the text detection model on segments of sub-text boxes, reduces computational load while maintaining detection accuracy.
[0136] Example 3
[0137] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.
[0138] Please see Figure 9 , Figure 9This is a schematic diagram of a dense text detection device provided in an embodiment of this application. This dense text detection device can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device includes an acquisition module 401, a segmentation module 402, a training module 403, and a detection module 404.
[0139] The acquisition module 401 is used to acquire a dense text image, which contains dense text with text arranged densely between characters and text boxes marked on the dense text.
[0140] The cutting module 402 is used to cut text boxes in a dense text image into multiple sub-text boxes;
[0141] Training module 403 is used to train the text detection model using dense text images and sub-text boxes as sample datasets.
[0142] The detection module 404 is used to input the image to be detected into the text detection model and obtain the preliminary predicted text box of the image to be detected output by the text detection model.
[0143] Optionally, the cutting module 402 is specifically used to cut the text box into multiple sub-text boxes according to the preset height of the sub-text box, moving vertically from the center of the text box.
[0144] Optionally, the cutting module 402 is also used to fill the sub-text boxes cut out from the upper and lower boundaries of the text box with a height equal to the preset height of the sub-text box if the height of the sub-text boxes cut out from the upper and lower boundaries of the text box is less than the preset height of the sub-text box.
[0145] Optionally, the training module 403 is specifically used to extract features from the dense text image through the convolutional layer of the text detection model to generate a first feature map of the dense text image; based on the first feature map of the dense text image, to generate candidate reference boxes of the dense text image using an anchor point regression mechanism; to match the candidate reference boxes with the sub-text boxes to determine the intersection-union ratio (IU) parameter between the candidate reference boxes and the sub-text boxes; and based on the IU parameter, to determine whether the candidate reference boxes are used as anchor point reference boxes for regression.
[0146] Optionally, the training module 403 is specifically used to determine the candidate reference box as the anchor reference box for regression if the intersection-union ratio parameter is greater than a threshold, the maximum value of the candidate reference box in the y-axis direction is greater than or equal to the minimum value of the sub-text box cut out by the upper boundary in the y-axis direction, and the minimum value of the candidate reference box in the y-axis direction is less than or equal to the maximum value of the sub-text box cut out by the lower boundary in the y-axis direction.
[0147] Optionally, the training module 403 is specifically used to convert the feature vector of the first feature region of the first feature map of the dense text image into a horizontal sequence feature vector to generate a second feature map; and input the transformed second feature map into the bidirectional long short-term memory network layer of the text detection model for feature learning.
[0148] Optionally, training module 403 is specifically used to classify and regress anchor reference boxes through the region candidate network layer of the text detection model.
[0149] Optionally, the category information of the anchor reference box includes text information, non-text information, and boundary information.
[0150] Optionally, the detection module 404 is also used to remove redundant boxes from the preliminary predicted text boxes using a preset algorithm; and to merge the preliminary predicted text boxes after removing the redundant boxes to generate the final predicted text boxes.
[0151] Optionally, the preset algorithm includes a nonmaximum suppression algorithm.
[0152] It should be noted that the dense text detection device provided in the above embodiments is only illustrated by the division of the above functional modules in the dense text detection method. In practical applications, the above functions can be assigned to multiple different functional modules as needed, that is, the internal structure of the device can be divided into multiple different functional modules to complete all or part of the functions described above. In addition, the dense text detection device and the dense text detection method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.
[0153] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0154] By using a dense text detection device, the text detection model is trained on the basis of segmented sub-text boxes. This ensures the detection effect of dense text while reducing the amount of computation during training and improving the training speed of the text detection model for dense text.
[0155] Example
[0156] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.
[0157] The communication bus 1002 is used to realize the connection and communication between these components.
[0158] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.
[0159] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).
[0160] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 1001 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip, without being integrated into the processor 1001.
[0161] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 10 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and operating applications for electronic devices.
[0162] exist Figure 10 In the illustrated electronic device 1000, the user interface 1003 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 1001 can be used to call the electronic device's operating application stored in the memory 1005 and specifically perform the following operations:
[0163] Obtain a dense text image, wherein the dense text image contains dense text with characters closely arranged between them, and the dense text is marked with text boxes;
[0164] The text boxes in the dense text image are cut into multiple sub-text boxes;
[0165] The dense text image and the sub-text boxes are used as a sample dataset to train the text detection model;
[0166] The image to be detected is input into the text detection model, and the preliminary predicted text box of the image to be detected is obtained from the output of the text detection model.
[0167] In one embodiment, the processor 1001 further performs the following steps:
[0168] Based on the preset height of the sub-text box, the text box is divided into multiple sub-text boxes in a vertical direction from the center of the text box.
[0169] In one embodiment, the processor 1001 further performs the following steps:
[0170] If the height of the sub-text boxes cut out from the top and bottom boundaries of the text box is less than the preset height of the sub-text box, then the sub-text boxes cut out from the top and bottom boundaries of the text box will be filled until their height is equal to the preset height of the sub-text box.
[0171] In one embodiment, the processor 1001 further performs the following steps:
[0172] The convolutional layer of the text detection model is used to extract features from the dense text image to generate a first feature map of the dense text image;
[0173] Based on the first feature map of the dense text image, candidate reference boxes for the dense text image are generated using an anchor point regression mechanism;
[0174] Match the candidate reference box with the sub-text box to determine the intersection-union ratio parameter of the candidate reference box and the sub-text box;
[0175] Based on the intersection-union ratio parameter, determine whether the candidate reference box should be used as an anchor reference box for regression.
[0176] In one embodiment, the processor 1001 further performs the following steps:
[0177] If the intersection-union ratio parameter is greater than a threshold, the maximum value of the candidate reference box in the y-axis direction is greater than or equal to the minimum value of the sub-text box cut out from the upper boundary in the y-axis direction, and the minimum value of the candidate reference box in the y-axis direction is less than or equal to the maximum value of the sub-text box cut out from the lower boundary in the y-axis direction, then the candidate reference box is determined as an anchor reference box for regression.
[0178] In one embodiment, the processor 1001 further performs the following steps:
[0179] The feature vector of the first feature region of the first feature map of the dense text image is converted into a horizontal sequence feature vector to generate a second feature map;
[0180] The transformed second feature image is then input into the bidirectional long short-term memory network layer of the text detection model for feature learning.
[0181] In one embodiment, the processor 1001 further performs the following steps:
[0182] The anchor reference boxes are classified and regressed using the region candidate network layer of the text detection model.
[0183] In one embodiment, the category information of the anchor reference box includes text information, non-text information, and boundary information.
[0184] By using this electronic device to train the text detection model on a per-segment basis (cutting sub-text boxes), the computational load during training is reduced while ensuring the detection performance of dense text, thus improving the training speed of the text detection model for dense text.
[0185] In one embodiment, the processor 1001 further performs the following steps:
[0186] The preset algorithm is used to remove redundant boxes from the preliminary predicted text boxes;
[0187] The preliminary predicted text boxes, after removing redundant boxes, are merged to generate the final predicted text boxes.
[0188] In some embodiments, the preset algorithm includes a nonmaximum suppression algorithm.
[0189] This electronic device, by training the text detection model on the basis of segmented sub-text boxes, reduces the amount of computation during training while ensuring the detection effect of dense text, thus improving the training speed of the text detection model for dense text. Correspondingly, it also reduces the amount of computation while ensuring the detection accuracy during the detection process.
[0190] This application also provides a computer storage medium that can store multiple instructions, which are adapted to be loaded and executed by a processor as described above. Figures 4-6 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figures 4-6 The specific details of the illustrated embodiments will not be elaborated here.
[0191] The storage medium can be located in the aforementioned electronic devices.
[0192] This application provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described method.
[0193] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0194] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0195] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0196] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0197] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0198] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0199] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0200] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0201] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A training method for a text detection model, characterized in that, The method includes: Obtain a dense text image, wherein the dense text image contains dense text with characters closely arranged between them, and the dense text is marked with text boxes; wherein the dense text is: text with characters sticking together between them, or text with the interval between characters being smaller than the identifiable interval; The text boxes in the dense text image are cut into multiple sub-text boxes; The dense text image and the sub-text boxes are used as a sample dataset to train the text detection model; The image to be detected is input into the text detection model, and the preliminary predicted text box of the image to be detected is obtained from the output of the text detection model. The step of using the dense text image and the sub-text boxes as a sample dataset to train the text detection model includes: The text detection model extracts features from the dense text image through its convolutional layers to generate a first feature map of the dense text image; based on the first feature map of the dense text image, an anchor point regression mechanism is used to generate candidate reference boxes for the dense text image. The candidate reference box is matched with the sub-text box to determine the intersection-union ratio (IURR) parameter between the candidate reference box and the sub-text box; wherein, the IURR parameter is used to detect the degree of overlap between the candidate reference box and the sub-text box; if the IURR parameter is greater than a threshold, the maximum value of the candidate reference box in the y-axis direction is greater than or equal to the minimum value of the sub-text box cut out from the upper boundary in the y-axis direction, and the minimum value of the candidate reference box in the y-axis direction is less than or equal to the maximum value of the sub-text box cut out from the lower boundary in the y-axis direction, then the candidate reference box is determined as an anchor reference box for regression.
2. The method according to claim 1, characterized in that, The step of cutting the text boxes in the dense text image into multiple sub-text boxes includes: Based on the preset height of the sub-text box, the text box is divided into multiple sub-text boxes in a vertical direction from the center of the text box.
3. The method according to claim 2, characterized in that, After dividing the text box into multiple sub-text boxes, the method further includes: If the height of the sub-text boxes cut out from the top and bottom boundaries of the text box is less than the preset height of the sub-text box, then the sub-text boxes cut out from the top and bottom boundaries of the text box will be filled until their height is equal to the preset height of the sub-text box.
4. The method according to claim 1, characterized in that, The step of using the dense text image and the sub-text boxes as a sample dataset to train the text detection model further includes: The feature vector of the first feature region of the first feature map of the dense text image is converted into a horizontal sequence feature vector to generate a second feature map. The transformed second feature image is then input into the bidirectional long short-term memory network layer of the text detection model for feature learning.
5. The method according to claim 1, characterized in that, The step of using the dense text image and the sub-text boxes as a sample dataset to train the text detection model further includes: The anchor reference boxes are classified and regressed using the region candidate network layer of the text detection model.
6. The method according to claim 5, characterized in that, The category information of the anchor reference box includes text information, non-text information, and boundary information.
7. The method according to claim 1, characterized in that, After inputting the image to be detected into the text detection model and obtaining the preliminary predicted text box of the image to be detected output by the text detection model, the method further includes: The preset algorithm is used to remove redundant boxes from the preliminary predicted text boxes; The preliminary predicted text boxes, after removing redundant boxes, are merged to generate the final predicted text boxes.
8. The method according to claim 7, characterized in that, The preset algorithm includes a nonmaximum suppression algorithm.
9. A training device for a text detection model, characterized in that, The device includes: The acquisition module is used to acquire a dense text image, which contains dense text with characters arranged densely, and the dense text is marked with text boxes; wherein, the dense text is: text with characters sticking together, or text with the interval between characters being smaller than the identifiable interval; A cutting module is used to cut text boxes in the dense text image into multiple sub-text boxes; The training module is used to train the text detection model using the dense text image and the sub-text boxes as a sample dataset. The detection module is used to input the image to be detected into the text detection model and obtain the preliminary predicted text box of the image to be detected output by the text detection model. The training module is specifically used to extract features from the dense text image through the convolutional layer of the text detection model to generate a first feature map of the dense text image; based on the first feature map of the dense text image, to generate candidate reference boxes of the dense text image using an anchor point regression mechanism; to match the candidate reference boxes with the sub-text boxes to determine the intersection-union ratio (IU / R) parameter between the candidate reference boxes and the sub-text boxes; wherein, the IU / R parameter is used to detect the degree of overlap between the candidate reference boxes and the sub-text boxes; if the IU / R parameter is greater than a threshold, the maximum value of the candidate reference box in the y-axis direction is greater than or equal to the minimum value of the sub-text boxes cut out from the upper boundary in the y-axis direction, and the minimum value of the candidate reference box in the y-axis direction is less than or equal to the maximum value of the sub-text boxes cut out from the lower boundary in the y-axis direction, then the candidate reference box is determined as an anchor point reference box for regression.
10. The apparatus according to claim 9, characterized in that, The cutting module is specifically used to cut the text box into multiple sub-text boxes according to the preset height of the sub-text box, moving vertically from the center of the text box.
11. The apparatus according to claim 10, characterized in that, The cutting module is further configured to fill the sub-text boxes cut from the upper and lower boundaries of the text box to a height equal to the preset height of the sub-text box if the height of the sub-text boxes cut from the upper and lower boundaries of the text box is less than the preset height of the sub-text box.
12. The apparatus according to claim 9, characterized in that, The training module is specifically used to convert the feature vector of the first feature region of the first feature map of the dense text image into a horizontal sequence feature vector to generate a second feature map; and then input the transformed second feature map into the bidirectional long short-term memory network layer of the text detection model for feature learning.
13. The apparatus according to claim 9, characterized in that, The training module is specifically used to classify and regress the anchor reference boxes through the region candidate network layer of the text detection model.
14. The apparatus according to claim 13, characterized in that, The category information of the anchor reference box includes text information, non-text information, and boundary information.
15. The apparatus according to claim 9, characterized in that, The detection module is further configured as a clearing module, which uses a preset algorithm to clear redundant boxes from the preliminary prediction text boxes; and a merging module, which merges the preliminary prediction text boxes after the redundant boxes have been cleared to generate the final prediction text boxes.
16. The apparatus according to claim 15, characterized in that, The preset algorithm includes a nonmaximum suppression algorithm.
17. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the method described in any one of claims 1-8.
18. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1-8.
19. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the method steps as claimed in any one of claims 1-8.