Text detection method
By introducing the joint output of the complete segmentation mask, the reduced segmentation mask and the offset detection branch in the text detection model, the problems of low text detection accuracy and insufficient robustness in the existing technology are solved, and more efficient and stable text detection effect.
Patent Information
- Application Number
- CN202411633117.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-02-21
AI Technical Summary
Existing text detection technologies, especially mainstream segmentation genre models such as PANet and DBNet, have problems such as low detection accuracy and long post-processing when processing complex backgrounds, noisy images or irregularly shaped text.
A text detection method is proposed, using a text detection model including backbone network, FPN network and three output branches. The three output branches are the complete segmentation mask branch, the reduced segmentation mask branch and the offset detection branch. Through the joint output of these branches, the precise segmentation and offset detection of text instances are realized, thereby improving detection accuracy and robustness. sex.
By introducing offset branches, simple and efficient aggregation of pixels is achieved, which significantly improves the aggregation speed, and shows higher stability and better fitting effects in practical applications, improving detection accuracy and robustness.
Smart Images
Figure CN119495088A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of text detection and relates to a text detection method. Background Art
[0002] Existing text detection technologies are mainly divided into two schools: segmentation school and non-segmentation school, each with unique advantages and challenges.
[0003] 1. Segmentation school: The text detection algorithm of the segmentation school uses deep learning technology to perform semantic segmentation on the text area in the image, identify the instance region of each line of text, and fit these regions to generate a rotated bounding box. Typical segmentation school algorithms include PSENet, PANet, and DBNet. These algorithms can fit the rotated rectangular box or the rectangular box of any distorted shape more accurately, so they are particularly ideal when dealing with complex text forms and are widely used in practical scenarios. However, although PANet and DBNet perform well in many cases, they still have some limitations. For example, when dealing with complex backgrounds, PANet may have the phenomenon of region mismerging, and the post-processing process takes a long time. DBNet's detection accuracy may be affected when facing images with a lot of noise or dealing with irregularly shaped text.
[0004] 2. Non-segmentation school: Non-segmentation school algorithms usually detect text regions directly without relying on semantic segmentation. Typical algorithms include RRPN, CPTN and TextBoxes. The main difference between RRPN (Rotated Region Proposal Network) and traditional object detection algorithms is that it can output detection boxes with angle information, which requires considering the influence of angle parameters in label preprocessing, candidate box setting, IOU calculation, loss function calculation and post-processing. CPTN (Connectionist Text Proposal Network) is based on the RPN part of Faster R-CNN, and enhances the correlation of the entire row of pixels by introducing the RNN layer. It performs well in detecting horizontal and vertical text, but has poor detection effect on large-angle text. TextBoxes adapts to text detection scenarios by adjusting the anchors in the SSD model to be slender, while TextBoxes++ upgrades it to enhance its adaptability to text at any angle.
[0005] Although the segmentation school performs better in processing rotated rectangular boxes and text regions of arbitrary shapes, the current mainstream segmentation school models such as PANet and DBNet still have certain shortcomings. Summary of the invention
[0006] In order to solve the above problems existing in the existing technology, the present invention provides a text detection method to further improve the detection accuracy and robustness.
[0007] The technical solution adopted by the present invention is:
[0008] A text detection method, the specific steps of which are as follows:
[0009] S1, establishing a text detection model, wherein the text detection model includes a backbone network, an FPN network, and three output branches, wherein the three output branches are a complete segmentation mask branch, a reduced segmentation mask branch, and an offset detection branch;
[0010] S2, obtains the original label, decomposes it to generate label data of three branches, and inputs it into the text detection model for training and supervision;
[0011] S3, input a character image, after the trained text detection model, the three branches respectively output the complete semantic segmentation mask, the reduced version of the semantic segmentation mask and the offset map; by combining the complete semantic segmentation mask and the offset map, the "post-offset coordinates" of each pixel in the complete semantic segmentation mask are calculated; then the reduced version of the semantic segmentation mask is used to determine in which reduced version instance region the "post-offset coordinates" of each pixel in the complete semantic segmentation mask fall, so that each pixel of the complete semantic segmentation mask is attributed to the corresponding text instance. The present invention realizes the re-clustering of pixels in the complete semantic segmentation mask, and can effectively distinguish each line of text and obtain the size of the actual region even when multiple lines of text regions are adhered.
[0012] Furthermore, in step S3, the specific steps of calculating the “offset coordinates” of each pixel in the complete semantic segmentation mask by combining the complete semantic segmentation mask and the offset map are as follows:
[0013] Suppose a pixel point in the original image is P(x,y), and the corresponding offset map has offsets in the x-direction and x-direction dx and dy respectively, then the "post-offset coordinates" P(x_,y_) of the pixel point can be calculated by the following formula:
[0014] x_=x+dx
[0015] y_=y+dy
[0016] Where: x, y are the horizontal and vertical coordinates of the original pixel point P, dx is the x-direction offset output by the offset detection branch, dy is the y-direction offset output by the offset detection branch, and x_, y_ are the horizontal and vertical coordinates of the offset coordinates, respectively;
[0017] Through the above formula, combined with the complete semantic segmentation mask and the offset map, the new position of each pixel after the offset can be calculated, thereby achieving pixel repositioning and aggregation.
[0018] Furthermore, in step S3, the specific steps of using the reduced version of the semantic segmentation mask to determine in which reduced version instance region the “offset coordinates” of each pixel in the complete semantic segmentation mask fall are as follows:
[0019] Step 1, obtain the instance region of the reduced version of the semantic segmentation mask, assuming that each instance region in the reduced version of the mask is denoted as Si, where i is the instance number;
[0020] Step 2, traverse the offset coordinates of each pixel point, and for each pixel point P(x,y) in the complete segmentation mask, calculate its offset coordinates P(x_,y_) according to the offset map;
[0021] Step 3: Match the instance area to which the offset coordinates belong, and determine whether the position of the offset coordinates P(x_, y_) is located in any reduced version instance area Si. The specific method includes:
[0022] 3.1 Spatial position judgment, check whether P(x_,y_) satisfies the geometric boundary conditions of the Si region;
[0023] 3.2 Coverage relationship judgment: Use the binary reduced version of the mask to determine whether the position corresponding to P(x_,y_) is the foreground pixel of Si;
[0024] Step 4: attribute instance labeling. Once it is determined that P(x_,y_) falls within a certain Si, the original pixel point P(x,y) can be attributed to the corresponding text instance i.
[0025] Furthermore, the complete segmentation mask branch outputs a mask aligned with the input image scale at a 4-fold downsampling rate, and the mask contains the original region information of each line of text instance.
[0026] Furthermore, the reduced version segmentation mask branch outputs a reduced version of the mask, which includes reduced version region information of each line of text instance.
[0027] Furthermore, the complete segmentation mask branch and the reduced segmentation mask branch are both responsible for the semantic segmentation task, and focal loss and dice loss are used as loss functions for joint supervision during training.
[0028] Furthermore, in the non-text area, the offset of the offset detection branch is 0; in the text area, the offset is "the shortest distance from each point to the corresponding center line", and the region corresponding to each line of character text has a center line.
[0029] Furthermore, the offset detection branch is responsible for offset detection, and the output map width and height are the same as those of the complete segmentation mask branch and the reduced version of the segmentation mask branch. It is divided into two layers, which are used to regress the offsets in the x direction and the y direction respectively, and use the smooth L1 loss function for supervision.
[0030] Furthermore, the original label in step S2 is the outer envelope of each text instance, which is assumed to be a rotated rectangle.
[0031] Furthermore, the label data of the three branches generated by the original label in step S2 are: a complete segmentation mask, a reduced segmentation mask, and a double-layer mask containing xy offset information.
[0032] Beneficial effects of the present invention:
[0033] 1. The offset branch is introduced to realize pixel aggregation, which is more concise and efficient. It not only significantly improves the aggregation speed, but also shows higher stability and better fitting effect in practical applications.
[0034] 2. Improve detection accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a schematic diagram of the model structure of the present invention.
[0036] Figure 2 Schematic diagram of the original text detection label of the present invention.
[0037] Figure 3 Schematic diagram of a complete segmentation mask of the present invention.
[0038] Figure 4 It is a schematic diagram of a reduced version of the segmentation mask of the present invention.
[0039] Figure 5 A schematic diagram of the offset detection of a text according to the present invention.
[0040] Figure 6 Generate a schematic diagram for the label of the present invention.
[0041] Figure 7 Schematic diagram of model training supervision of the present invention.
[0042] Figure 8 Schematic diagram of the output result of the complete segmentation mask branch of the present invention.
[0043] Fig. 9 It is a schematic diagram of the output result of the reduced version segmentation mask branch of the present invention.
[0044] Fig.10 It is a schematic diagram of the output result of the offset detection branch of the present invention.
[0045] Fig.11 It is a schematic diagram of the effect after the complete segmentation mask branch, the reduced version segmentation mask branch and the offset detection branch of the present invention are combined and processed. DETAILED DESCRIPTION
[0046] The present invention is further described below in conjunction with specific embodiments, but the present invention is not limited to these specific embodiments. Those skilled in the art should recognize that the present invention covers all possible alternatives, improvements and equivalents within the scope of the claims.
[0047] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "clockwise", "counterclockwise" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more, unless otherwise clearly defined.
[0048] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0049] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.
[0050] Terminology explanation:
[0051] Focal Loss
[0052] Definition: Focal Loss is a loss function proposed to solve the problem of class imbalance, which is particularly suitable for the situation where the ratio of foreground and background samples in target detection is unbalanced.
[0053] Formula: F(p) = -α(1-p)γ * log(p)
[0054] Where p: the model's predicted probability for the correct category. α: balancing factor, used to balance the importance of positive and negative samples, usually α∈(0,1). γ: modulation factor, used to reduce the weight of easy-to-classify samples and enhance the influence of difficult-to-classify samples, usually γ≥0.
[0055] Function: By introducing the (1-p)γ term, Focal Loss reduces the focus on correctly classified samples and focuses on those difficult-to-classify samples, thereby improving the performance of the model on unbalanced data sets.
[0056] Dice Loss
[0057] Definition: Dice Loss is based on the Dice coefficient, which is a loss function commonly used in image segmentation tasks to measure the overlap between the predicted results and the true labels.
[0058] Formula: DiceLoss=1-(2*∑ i P i G i +∈) / (∑ i P i +∑ i G i +∈)
[0059] Where P i : The predicted probability of the i-th pixel. G i: The true label of the ith pixel (0 or 1). ∈: A small constant to avoid the denominator being zero.
[0060] Function: DiceLoss maximizes the Dice coefficient to increase the overlap between the predicted mask and the true mask. It is particularly suitable for image segmentation tasks with small foreground areas or complex shapes.
[0061] This embodiment provides a text detection method, and the specific steps are as follows:
[0062] S1, establish a text detection model, which includes a backbone network, an FPN network (feature pyramid network) and three output branches, the three output branches are a complete segmentation mask branch, a reduced version of the segmentation mask branch and an offset detection branch, see Figure 1 As shown;
[0063] The complete segmentation mask branch outputs a mask (segmentation mask) that is aligned with the input image scale at a 4-fold downsampling rate, and the mask contains the original region information of each line of text instance.
[0064] The reduced version segmentation mask branch outputs a reduced version of the mask, which contains the reduced version of the region information of each line of text instance. In this mask, the reduced version of the regions of the text instance are not prone to adhesion, which helps to better distinguish each instance.
[0065] The offset detection branch has an offset of 0 in non-text areas; in text areas, the offset is "the shortest distance from each point to the corresponding center line", and each region corresponding to a line of text has a center line. The output map includes two layers, one for regressing the x-coordinate of the offset and the other for regressing the y-coordinate of the offset.
[0066] S2, obtain the original label, which is the outer envelope of each text instance, assumed to be a rotated rectangle. After decomposition, the label data of three branches are generated and input into the text detection model for training and supervision; this embodiment takes a road sign as an example and marks the text box of each line of characters to more clearly illustrate the processing method of each branch, see Figure 2-5 shown.
[0067] The label data of the three branches generated by the original label are: the complete segmentation mask, the reduced segmentation mask, and the double-layer mask containing xy offset information. The reduced segmentation mask is generated by "reducing the four sides by 0.25h each", and the uncovered part is the background pixel. The offset is calculated by measuring the pixel distance from each point to the center line of the corresponding region, see Figure 6shown.
[0068] The complete segmentation mask branch and the reduced version segmentation mask branch are both responsible for the semantic segmentation task. Focal loss and dice loss are used as loss functions for joint supervision during training. The offset detection branch is responsible for offset detection. The output map width and height are the same as those of the complete segmentation mask branch and the reduced version segmentation mask branch. It is divided into two layers, which are used to regress the offset in the x direction and the y direction respectively, and use the smooth L1 loss function for supervision. See Figure 7 shown.
[0069] S3 inputs a character image. After the trained text detection model, the three branches output the complete semantic segmentation mask, the reduced version of the semantic segmentation mask, and the offset map, respectively. Figure 8-10 As shown in the figure, only the offset diagram in the y direction is provided. The x direction is similar, both of which are offset towards the center line of the target text. The two segmentation labels of the complete semantic segmentation mask are glued together to form a connected domain. The reduced version of the semantic segmentation mask can distinguish instances well, but because it is a reduced version, the region of each line of text is incomplete. The offset of the offset diagram indicates the shortest distance from each pixel in the text area to the corresponding text center line, and the offset of the non-text area is 0.
[0070] By combining the complete semantic segmentation mask and the offset map, the “offset coordinates” of each pixel in the complete semantic segmentation mask are calculated;
[0071] The specific steps are as follows:
[0072] Suppose a pixel point in the original image is P(x,y), and the corresponding offset map has offsets in the x-direction and x-direction dx and dy respectively, then the "post-offset coordinates" P(x_,y_) of the pixel point can be calculated by the following formula:
[0073] x_=x+dx
[0074] y_=y+dy
[0075] Where: x, y are the horizontal and vertical coordinates of the original pixel point P, dx is the x-direction offset output by the offset detection branch, dy is the y-direction offset output by the offset detection branch, and x_, y_ are the horizontal and vertical coordinates of the offset coordinates, respectively;
[0076] Through the above formula, combined with the complete semantic segmentation mask and the offset map, the new position of each pixel after the offset can be calculated, thereby achieving pixel repositioning and aggregation.
[0077] Then, the reduced version of the semantic segmentation mask is used to determine in which reduced version of the instance area the "offset coordinates" of each pixel in the complete semantic segmentation mask fall, so as to attribute each pixel of the complete semantic segmentation mask to the corresponding text instance, see Fig.11 shown.
[0078] The specific steps are as follows:
[0079] Step 1, obtain the instance region of the reduced version of the semantic segmentation mask, assuming that each instance region in the reduced version of the mask is denoted as Si, where i is the instance number;
[0080] Step 2, traverse the offset coordinates of each pixel point, and for each pixel point P(x,y) in the complete segmentation mask, calculate its offset coordinates P(x_,y_) according to the offset map;
[0081] Step 3: Match the instance area to which the offset coordinates belong, and determine whether the position of the offset coordinates P(x_, y_) is located in any reduced version instance area Si. The specific method includes:
[0082] 3.1 Spatial position judgment, check whether P(x_,y_) satisfies the geometric boundary conditions of the Si region;
[0083] 3.2 Coverage relationship judgment: Use the binary reduced version of the mask to determine whether the position corresponding to P(x_,y_) is the foreground pixel of Si;
[0084] Step 4: attribute instance labeling. Once it is determined that P(x_,y_) falls within a certain Si, the original pixel point P(x,y) can be attributed to the corresponding text instance i.
[0085] The present invention realizes the re-clustering of pixels in a complete semantic segmentation mask, and can effectively distinguish each line of text and obtain the actual region size even when multiple lines of text regions are adhered.
[0086] The present invention introduces an offset branch to achieve pixel aggregation, which is more concise and efficient. It not only significantly improves the aggregation speed, but also shows higher stability and better fitting effect in practical applications, and improves detection accuracy and robustness.
Claims
1. A text detection method, the specific steps of which are as follows: S1, establishing a text detection model, wherein the text detection model includes a backbone network, an FPN network, and three output branches, wherein the three output branches are a complete segmentation mask branch, a reduced segmentation mask branch, and an offset detection branch; S2, obtains the original label, decomposes it to generate label data of three branches, and inputs it into the text detection model for training and supervision; S3, input a character image, after the trained text detection model, the three branches output the complete semantic segmentation mask, the reduced version of the semantic segmentation mask and the offset map respectively; by combining the complete semantic segmentation mask and the offset map, the "offset coordinates" of each pixel in the complete semantic segmentation mask are calculated; then the reduced version of the semantic segmentation mask is used to determine in which reduced version instance area the "offset coordinates" of each pixel in the complete semantic segmentation mask fall, so as to attribute each pixel of the complete semantic segmentation mask to the corresponding text instance.
2. A text detection method according to claim 1, characterized in that: In step S3, the specific steps of calculating the "offset coordinates" of each pixel in the complete semantic segmentation mask by combining the complete semantic segmentation mask and the offset map are as follows: Suppose a pixel point in the original image is P(x,y), and the corresponding offset map has offsets in the x-direction and x-direction dx and dy respectively, then the "post-offset coordinate" P(x_,y_) of the pixel point can be calculated by the following formula: x_=x+dx y_=y+dy Where: x, y are the horizontal and vertical coordinates of the original pixel point P, dx is the x-direction offset output by the offset detection branch, dy is the y-direction offset output by the offset detection branch, and x_, y_ are the horizontal and vertical coordinates of the offset coordinates, respectively; Through the above formula, combined with the complete semantic segmentation mask and the offset map, the new position of each pixel after the offset can be calculated, thereby achieving pixel repositioning and aggregation.
3. A text detection method according to claim 2, characterized in that: The specific steps of using the reduced version of the semantic segmentation mask in step S3 to determine in which reduced version instance region the "offset coordinates" of each pixel in the complete semantic segmentation mask fall are as follows: Step 1, obtain the instance region of the reduced version of the semantic segmentation mask, assuming that each instance region in the reduced version of the mask is denoted as Si, where i is the instance number; Step 2, traverse the offset coordinates of each pixel point, and for each pixel point P(x,y) in the complete segmentation mask, calculate its offset coordinates P(x_,y_) according to the offset map; Step 3: Match the instance area to which the offset coordinates belong, and determine whether the position of the offset coordinates P(x_, y_) is located in any reduced version instance area Si. The specific method includes: 3.1 Spatial position judgment, check whether P(x_,y_) satisfies the geometric boundary conditions of the Si region; 3.2 Coverage relationship judgment: Use the binary reduced version of the mask to determine whether the position corresponding to P(x_,y_) is the foreground pixel of Si; Step 4: attribute instance labeling. Once it is determined that P(x_,y_) falls within a certain Si, the original pixel point P(x,y) can be attributed to the corresponding text instance i.
4. A text detection method according to claim 1, characterized in that: The complete segmentation mask branch outputs a mask aligned with the input image scale at a 4-fold downsampling rate, and the mask contains the original region information of each line of text instance.
5. A text detection method according to claim 1, characterized in that: The reduced version segmentation mask branch outputs a reduced version of the mask, which contains reduced version region information of each line of text instance.
6. A text detection method according to any one of claims 1 to 5, characterized in that: The complete segmentation mask branch and the reduced segmentation mask branch are both responsible for the semantic segmentation task, and focal loss and dice loss are used as loss functions for joint supervision during training.
7. A text detection method according to claim 1, characterized in that: The offset detection branch has an offset of 0 in the non-text area; in the text area, the offset is "the shortest distance from each point to the corresponding center line", and the region corresponding to each line of character text has a center line.
8. A text detection method according to claim 1 or 7, characterized in that: The offset detection branch is responsible for offset detection. The output map width and height are the same as those of the complete segmentation mask branch and the reduced version of the segmentation mask branch. It is divided into two layers, which are used to regress the offsets in the x direction and the y direction respectively, and are supervised using the smooth L1 loss function.
9. A text detection method according to claim 1, characterized in that: The original label in step S2 is the outer envelope of each text instance, which is assumed to be a rotated rectangle.
10. A text detection method according to claim 9, characterized in that: The label data of the three branches generated by the original label in step S2 are: a complete segmentation mask, a reduced segmentation mask, and a double-layer mask containing xy offset information.