UNet algorithm-based text-oriented line segment detection method

Abstract

The invention relates to a text-oriented line segment detection method based on a UNet algorithm. The specific process is as follows: obtaining text image data with line segments; performing size normalization processing on the acquired image, performing manual annotation on the image by using an annotation tool label, obtaining a line segment position, expanding the line segment position into a rectangle with the width of two pixel points, obtaining coordinates of four points of the rectangle, performing binarization processing on the image, generating a binarization label image, and obtaining a target detection data set; positions of all line segments in the original image are marked in the binarized label image; constructing a line segment detection network model based on a UNet algorithm, training the line segment detection network model by using the target detection data set, predicting line segments in the text image by using the trained line segment detection network model, and outputting a model prediction map; and post-processing the model prediction map, extracting the positions of the transverse line segments, the longitudinal line segments and the oblique line segments in the text image, realizing detection of the line segments in the text image, and improving the detection precision of the text line segments.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning target detection, and in particular to a text-oriented line segment detection method based on a UNet algorithm. Background Art

[0002] With the continuous development and deepening of text processing technology, text segment detection plays a key role in document digitization, graphic information extraction, printing and typesetting inspection, etc. Professional document scanners or high-resolution image acquisition equipment can obtain a large amount of text image data. These rich data can not only be used for text recognition, but also assist in complex document structure analysis tasks, which makes text segment detection more and more necessary in the document processing process.

[0003] At present, the text line segment detection method based on computer vision technology has significantly outperformed the traditional manual annotation method in terms of efficiency. According to different detection principles, these methods can be divided into text line segment detection based on edge detection and text line segment detection based on deep learning. Although the text line segment detection based on edge detection is slightly lower in accuracy than the text line segment detection based on deep learning, its computational complexity is lower. This feature makes it easier to be integrated into some document processing software with low resource requirements, and is suitable for preliminary line segment detection of large-scale document images. However, the line segments in text images have various shapes, different lengths, and are intertwined with text and other graphic elements (text line segments have various formats, such as underlines in the middle of text, underlines, slashes and table lines below text, of different lengths, and are easily attached to text or other content in the text, resulting in inaccurate detection) and are greatly disturbed by image noise, making it difficult to further improve the detection accuracy. At the same time, document processing often requires accurate extraction of line segment information in a short time, so high requirements are placed on detection accuracy.

[0004] In summary, text line segment detection still has the problems of low detection accuracy, easy missed detection and false detection, and it is difficult to achieve high-precision line segment detection at the same time. Therefore, the present invention proposes a text line segment detection method based on the UNet algorithm to improve the detection performance of the model. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings in the above background and provide a text-oriented line segment detection method based on the UNet algorithm. The method annotates the collected text with line segments and expands the line segments. A binary image is generated based on the expanded line segment position information and combined with the original image as training data, and the UNet algorithm is used for detection. In addition, the present invention introduces the Transformer structure into the Encoder of UNet, replaces the skip connection of UNet with the Res path, makes the loss more sensitive to the size and position of the object, and improves the performance of text line segment detection.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] In a first aspect, the present invention provides a text-oriented line segment detection method based on the UNet algorithm, wherein the process of the detection method is:

[0008] S1, obtaining text image data with line segments;

[0009] S2. Normalize the size of the collected image, use the labelme annotation tool to manually annotate the image, obtain the position of the line segment, and then determine the length direction of the line segment based on the line segment position information. Take the length as the central axis and expand it into a rectangle with a width of two pixels along the length direction perpendicular to the line segment, obtain the coordinates of the four points of the rectangle, and then binarize the image to generate a binary label image to obtain a target detection data set; each data sample in the target detection data set consists of an original image and a corresponding binary label image; the binary label image is annotated with the positions of all line segments in the original image;

[0010] S3. Build a line segment detection network model based on the UNet algorithm, train the line segment detection network model with the target detection dataset, predict the line segments in the text image with the trained line segment detection network model, and output the model prediction graph;

[0011] S4. Post-process the model prediction image to extract the positions of horizontal line segments, vertical line segments, and oblique line segments in the text image to achieve line segment detection in the text image.

[0012] Furthermore, the image resolution is 1024×1024 after size normalization.

[0013] Furthermore, the specific process of step S4 is:

[0014] The model predicts that the pixel value of the background in the image is 0, and the pixel value of the target area is 1;

[0015] S41 uses connected domain detection to mark the area with a pixel value of 1, and the area surrounded by the points with a marked pixel value of 1 is the target connected domain, and the connected domain with less than 32 pixels is eliminated;

[0016] S42 uses the vertical corrosion kernel (2, 20) to perform an corrosion operation on the image processed by step S41, filters out the horizontal line connected domain in the image, obtains the vertical line connected domain, and filters out the connected domain with less than 15 pixels, eliminates the interfering vertical lines, and obtains the real vertical lines; uses expansion to restore the vertical lines corroded by the vertical corrosion kernel, and then performs a closing operation on the vertical line connected domain to optimize the discontinuous line segments. The specific process of the closing operation is: using the expansion kernel (2, 7) to perform an expansion operation on the image, and then using the corrosion kernel (2, 5) to perform an corrosion operation on the image, connecting the discontinuous lines, and obtaining a vertical line segment diagram;

[0017] The collected longitudinal line segment graphs are used with a minimum rectangular frame to collect the minimum rectangle that can surround each longitudinal line segment graph, obtain the center point and height and width of the rectangular frame, and convert them into line segment coordinates, thereby obtaining the longitudinal line segment coordinates;

[0018] S43 uses a horizontal corrosion kernel (20, 2) to perform an corrosion operation on the image processed in step S41, filters out the vertical line connected domain in the image, obtains the horizontal line connected domain, filters out the horizontal line connected domain with less than 15 pixels, eliminates the interfering horizontal lines, and obtains the real horizontal lines;

[0019] The horizontal lines eroded by the horizontal erosion kernel are restored by dilation, and then the connected domains of the horizontal lines are closed to optimize the discontinuous line segments. The specific process of the closing operation is as follows: the image is dilated by using the dilation kernel of (7, 2), and then the image is eroded by using the erosion kernel of (5, 2), the discontinuous lines are connected, and the horizontal line segment graph is obtained;

[0020] The collected horizontal line segment graphs are collected using the minimum rectangular frame to collect the minimum rectangle that can surround each horizontal line segment graph, obtain the center point and height and width of the rectangular frame, and convert them into line segment coordinates, thereby obtaining the horizontal line segment coordinates;

[0021] S44: adding the image of the real vertical line obtained in step S42 and the image of the real horizontal line obtained in step S43 to obtain a combined image of horizontal and vertical lines; and then performing a subtraction operation on the combined image of horizontal and vertical lines using the image processed in S41 to obtain an oblique line segment image;

[0022] S45: filtering out the connected domains with less than 25 pixels in the oblique line segment graph obtained in step S44, and then using Hough transform detection to extract the line segments that may exist in the image. The line segments detected by Hough transform can connect the discontinuous oblique line contours into a whole, which is beneficial to the overall contour detection. However, since the Hough transform detection may generate multiple lines at the same position, the line segments detected by Hough transform are restored to the image before Hough transform detection.

[0023] Then, connected domain extraction is used to mark and extract the oblique line targets in the image, and each connected domain is surrounded by a minimum rectangular frame to obtain the minimum rectangular frame around each connected domain. The center point, height, width and angle of the rectangular frame are used to obtain the coordinates of the oblique line segment.

[0024] S46 combines the vertical line segment coordinates obtained in step S42, the horizontal line segment coordinates obtained in step S43, and the oblique line segment coordinates obtained in step S45, and restores them to the original image, thereby realizing the detection of line segments in the text image with line segments.

[0025] Furthermore, the line segment detection network model is a TR-UNet network, and the TR-UNet network includes an encoder and a decoder composed of a CNN and a Transformer structure; the CNN includes multiple layers of convolution and linear layers connected in sequence, and the hidden feature Hidden Feature output by the multiple layers of convolution is input into the Transformer structure through the linear layer, and the Transformer structure includes several stacked Transformer layers, and the output of the Transformer structure is processed by reshape operation, 3*3 convolution and ReLU function, and upsampling Upsample to obtain the first feature F1;

[0026] The features of the convolution output of each layer of CNN are processed by the residual path Res Path composed of at least one residual block and then concatenated with the input of the corresponding layer in the decoder; the result of each convolution processing of CNN is no longer directly added to the corresponding layer in the decoder, but is combined with the decoder part after a deep residual path. The number of residual blocks in the residual path depends on the current feature depth. The deeper the feature depth, the fewer the number of residual blocks.

[0027] Furthermore, the number of convolutions in the CNN is three, and the size of the convolution kernel is 3*3; the number of Transformer layers in the Transformer structure is 12.

[0028] Furthermore, the residual block is a CNN with three residual blocks connected in series after the first convolution layer, two residual blocks connected in series after the second convolution layer, and one residual block connected in series after the third convolution layer; the residual block includes a first convolution of a 3*3 convolution kernel and a second convolution of a 1*1 convolution kernel in parallel, and the outputs of the first convolution and the second convolution are added as the output of the residual block.

[0029] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which can implement the steps of the detection method when executed by a processor.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The existing technology often uses basic graphics to detect text targets, such as using connected domains combined with corrosion and expansion operations to detect related targets, or using the relationship between pixels to obtain the target position. The present invention uses an expansion method to convert the two-point annotation of the line segment into a four-point annotation, so that its area is used to calculate the IoU, and performs corrosion operations on the detected results to restore the line segment, thereby improving the detection accuracy of text line segments and enabling the UNet network originally used for segmentation to be used for line segment detection.

[0032] The line segment detection network model TR-UNet of the present invention optimizes the baseline network UNet, embeds the Transformer structure into the Encoder structure of UNet, uses Res Path to replace the simple skip connection in the traditional UNet, optimizes global feature extraction, achieves more accurate positioning, improves the convergence speed and optimizes the positioning effect, and further improves the accuracy of line segment detection. Compared with the UNet baseline model, the optimized model TR-UNet has an accuracy increase of 2.2 percentage points on the basis of 96.5%, a recall rate increase of 3.3 percentage points on the basis of 93.1%, and an F value increase of 2.7 percentage points on the basis of 94.8%. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a text-oriented line segment detection method based on the UNet algorithm according to an embodiment of the present invention;

[0034] Figure 2 Schematic diagram of the preprocessing process of the text image with line segments in the present invention;

[0035] Figure 3 A schematic diagram of the structure of a TR-UNet network in an embodiment of the present invention;

[0036] Figure 4A schematic diagram of the structure of an encoder composed of a CNN and a Transformer structure in an embodiment of the present invention;

[0037] Figure 5 The figure is a schematic diagram of the structure of a residual path Res path in an embodiment of the present invention.

[0038] Figure 6 The figure is a flow chart of an embodiment of post-processing the model prediction graph in the present invention.

[0039] Figure 7 The original image of the text image with line segments for detecting the underline under the keywords in the document in the present invention and the effect image obtained by using the method of the present invention.

[0040] Figure 8 The invention discloses an original image of a text image with line segments for detecting underlines under keywords in a text image and line segments in a table, and an effect image obtained by using the method of the invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention is further described in detail below in conjunction with the accompanying drawings. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0042] Example 1

[0043] The present invention provides a text-oriented line segment detection method based on the UNet algorithm, and the method comprises the following steps:

[0044] S1: Get text image data with line segments;

[0045] Obtain document image data with line segments, crawl and search for relevant data on relevant websites such as Baidu and Wensuwang, clean the data, obtain 863 usable data images, and randomly divide the data set into training set, validation set and test set according to a pre-set ratio.

[0046] S2: preprocessing the acquired image with line segments into an image that complies with the UNeT input format;

[0047] First, the size of the collected image is normalized, and the image is manually annotated using the labelme annotation tool to obtain the line segment position. Then, the image is binarized according to the line segment position information to generate a label image and obtain a target detection dataset. Each data sample in the target detection dataset consists of an original image and a corresponding binary label image. The binary label image is annotated with the positions of all line segments in the original image.

[0048] S3: Build a line segment detection network model based on the UNet algorithm, train the line segment detection network model with the target detection dataset, use the trained line segment detection network model to predict the line segments in the text image, and output the model prediction graph;

[0049] S4: Post-process the model prediction image to extract the positions of horizontal line segments, vertical line segments, and oblique line segments in the text image to achieve line segment detection in the text image.

[0050] Example 2

[0051] In this embodiment, the specific process of step S4 is:

[0052] The model predicts that the pixel value of the background in the image is 0, and the pixel value of the target area is 1;

[0053] S41 uses connected domain detection to mark the area with a pixel value of 1, and the area surrounded by the points with a marked pixel value of 1 is the target connected domain, and the connected domain with less than 32 pixels is eliminated;

[0054] Since line segment areas that are too small are often interference areas and not required line segment areas, connected domains with less than 32 pixels are eliminated.

[0055] S42 uses the vertical corrosion kernel (2, 20) to perform an erosion operation on the image processed in step S41, filters out the horizontal line connected domain in the image, obtains the vertical line connected domain, and filters out the too short area again, that is, the connected domain with less than 15 pixels is filtered out to eliminate the interfering vertical lines and obtain the real vertical lines;

[0056] The vertical lines eroded by the vertical erosion kernel are restored by dilation, and then the vertical line connected domain is closed to optimize the discontinuous line segments. The specific process of the closing operation is as follows: the image is dilated by using the dilation kernel (2, 7), and then the image is eroded by using the erosion kernel (2, 5), the discontinuous lines are connected, and the vertical line segment graph is obtained;

[0057] The collected longitudinal line segment graphs are used with a minimum rectangular frame to collect the minimum rectangle that can surround each longitudinal line segment graph, obtain the center point and height and width of the rectangular frame, and convert them into line segment coordinates, thereby obtaining the longitudinal line segment coordinates;

[0058] S43 uses a horizontal corrosion kernel (20, 2) to perform an corrosion operation on the image processed in step S41, filters out the vertical line connected domain in the image, obtains the horizontal line connected domain, filters out the horizontal line connected domain with less than 15 pixels, eliminates the interfering horizontal lines, and obtains the real horizontal lines;

[0059] The horizontal lines eroded by the horizontal erosion kernel are restored by dilation, and then the connected domains of the horizontal lines are closed to optimize the discontinuous line segments. The specific process of the closing operation is as follows: the image is dilated by using the dilation kernel of (7, 2), and then the image is eroded by using the erosion kernel of (5, 2), the discontinuous lines are connected, and the horizontal line segment graph is obtained;

[0060] The collected horizontal line segment graphs are collected using the minimum rectangular frame to collect the minimum rectangle that can surround each horizontal line segment graph, obtain the center point and height and width of the rectangular frame, and convert them into line segment coordinates, thereby obtaining the horizontal line segment coordinates;

[0061] S44: adding the image of the real vertical line obtained in step S42 and the image of the real horizontal line obtained in step S43 to obtain a combined image of horizontal and vertical lines; and then performing a subtraction operation on the combined image of horizontal and vertical lines using the image processed in S41 to obtain an oblique line segment image;

[0062] S45: filtering out the connected domains with less than 25 pixels in the oblique line segment graph obtained in step S44, and then using Hough transform detection to extract the line segments that may exist in the image. The line segments detected by Hough transform can connect the discontinuous oblique line contours into a whole, which is beneficial to the overall contour detection. However, since the Hough transform detection may generate multiple lines at the same position, the line segments detected by Hough transform are restored to the image before Hough transform detection.

[0063] Then, connected domain extraction is used to mark and extract the oblique line targets in the image, and each connected domain is surrounded by a minimum rectangular frame to obtain the minimum rectangular frame around each connected domain. The center point, height, width and angle of the rectangular frame are used to obtain the coordinates of the oblique line segment.

[0064] S46 combines the vertical line segment coordinates obtained in step S42, the horizontal line segment coordinates obtained in step S43, and the oblique line segment coordinates obtained in step S45, and restores them to the original image, thereby realizing the detection of line segments in the text image with line segments.

[0065] Example 3

[0066] In this embodiment, the line segment detection network model is a TR-UNet network, and the TR-UNet network includes an encoder and a decoder composed of a CNN and a Transformer structure; the CNN includes multiple layers of convolution and linear layers connected in sequence, and the hidden feature Hidden Feature output by the multiple layers of convolution is input into the Transformer structure through the linear layer. The Transformer structure includes several stacked Transformer layers, and the output of the Transformer structure is processed by reshape operation, 3*3 convolution and ReLU function, and upsampling Upsample to obtain the first feature F1;

[0067] The features of the convolution output of each layer of CNN are processed by the residual path Res Path composed of at least one residual block and then concatenated with the input of the corresponding layer in the decoder; the result of each convolution processing of CNN is no longer directly added to the corresponding layer in the decoder, but is combined with the decoder part after a deep residual path. The number of residual blocks in the residual path depends on the current feature depth. The deeper the feature depth, the fewer the number of residual blocks.

[0068] The specific process of building a training framework using the TR-UNet network is as follows:

[0069] S3-1: Embed the Transformer structure into the Encoder structure of UNet, and use the hybrid CNN and Transformer structure as the encoder;

[0070] Transformer uses a sub-attention self-attention mechanism to encode tokenized image blocks from convolutional neural network (CNN) feature maps into an input sequence that extracts global context. The decoder upsamples the encoded features and combines them with different high-resolution CNN feature maps skipped from the encoding path to achieve precise positioning. First, the image is serialized, and the input x is converted into a flat two-dimensional patch sequence. Each patch is of size P×P, and the number N=(HW) / P2, that is, the sequence length. Then, using trainable projection, the vectorized patchXp is mapped to the potential d-dimensional embedding space and added to the patch embedding to retain the position information. The calculation formula is as follows:

[0071]

[0072] Where E is the patch embedding projection, E pos Represents position embedding. The Transformer structure is composed of L layers of multi-head attention (MSA) and multi-layer perceptron blocks (MLP). The output of the layer is:

[0073]

[0074] Where LN stands for normalization, represents the image representation of the previous layer, represents the feature representation after multi-head attention, It is the image representation of the current layer after multi-head attention and multi-layer perceptron blocks.

[0075] Using this structure, Transformer encodes image patches from CNN feature maps as input sequences for extracting global features. The decoder upsamples the encoded features and combines them with high-resolution CNN features for precise positioning.

[0076] S3-2: Use Res Path to replace the simple skip connection in traditional UNet;

[0077] Since the number of convolutional layers in the encoder is relatively shallow, the extracted features are often low-level features, and there is a large semantic gap between the two. It is not appropriate to directly concatenate the corresponding layers in the Encoder and Decoder, but should be further processed using convolution, and different numbers of convolutional layers should be used according to the features of different scales to extract deeper semantic features.

[0078] S3-3: The training framework is built using the stochastic gradient descent algorithm (SGD) with momentum, using a large momentum = 0.99. During training, the update of the current optimization step can be determined based on a large number of previous training samples. In order to minimize overhead and maximize the use of GPU memory, a larger batch size is used to reduce the batch to a single image. The final energy function is defined as a pixel-level soft-max of the final feature map, and the EIoU loss function is used for training. Among them, soft-max is defined as:

[0079]

[0080] Among them, a k (x) represents the activation of k on the feature channel, k represents the kth feature on the feature channel, K represents the number of categories, x represents the pixel point, and k' represents the categories from 1 to K. The image is finally at each position The soft-max is:

[0081]

[0082] For each pixel, the label is:

[0083]

[0084] The weight map is:

[0085] ω:Ω→R

[0086] The weight map is obtained by calculation in the pre-experimental stage:

[0087]

[0088] Where ω(x) is used to balance the frequency of occurrence of different categories, d 1 :Ω→R represents the distance between the target and the nearest boundary, d 2 :Ω→R represents the distance between the target and the second closest boundary

[0089] The core idea of ​​the EIOU loss function is to improve the accuracy and efficiency of the bounding box, and to improve the target detection loss calculation by increasing the center point distance loss, considering the size difference and combining the minimum closed size. First, by minimizing the distance between the center point of the predicted box and the true box, the positioning accuracy of the bounding box is improved; secondly, by penalizing the difference in width and height, EIou ensures that the predicted box is closer to the true box in shape; finally, the loss function is combined with the size of the minimum closed box containing the predicted box and the true box, so that the loss is more sensitive to the size and position of the object. The EIoU calculation formula is as follows:

[0090]

[0091] Where ρ represents the Euclidean distance between the target and the actual, b, ω, and h represent the center point, width, and height of the predicted box, respectively, gt represents the actual box, and ω c and h c is the width and height of the smallest bounding box covering the two boxes. In other words, the loss function is divided into three parts, IOU loss, distance loss L dis and direction loss L asp , directly minimizes the difference in width and height between the target box and the real box, improves the convergence speed, and achieves better positioning effect.

[0092] S4: Send the training set and validation set in the collected data set to the TR-UNet target detection model described in right S3 for training and validation. The specific implementation steps are as follows;

[0093] (1) Experimental environment setting: The operating environment of this experiment is Ubuntu 20.04, the GPU is NVIDIA GeForce RTX2080, the memory is 11G, and the compilation environment is python 3.8.10+pytorch 2.3.1+cuda 12.4;

[0094] (2) Setting of model training hyperparameters: The experimental batch size is 8, the number of training rounds is 50, the initial learning rate is 0.01, the input image size is 1024×1024, the momentum coefficient is 0.99, the weight decay coefficient is 0.0005, and the optimizer is SGD;

[0095] (3) Send the training set to the network for training, obtain the trained model, and use the trained model for testing.

[0096] S5: Send the test set in the collected data to the detection model with optimal parameters in S4 for testing, and post-process the results output by the model to obtain line segment position information, output the final detection results, and conduct feasibility evaluation on the detection results.

[0097] Example 4

[0098] In this embodiment, the number of convolutions in CNN is three, and the convolution kernel size is 3*3; the number of Transformer layers in the Transformer structure is 12. The scale feature after each convolution is consistent with the feature size in the decoder connected subsequently. The hidden feature in CNN is the feature after three convolutions of CNN.

[0099] The residual block is a CNN with three residual blocks connected in series after the first convolution layer, two residual blocks connected in series in the second convolution layer, and one residual block connected in series in the third convolution layer; the residual block includes a first convolution of a 3*3 convolution kernel and a second convolution of a 1*1 convolution kernel in parallel, and the outputs of the first convolution and the second convolution are added as the output of the residual block.

[0100] The number of residual blocks in the present invention, such as 3, 2 or 1 residual blocks, mainly depends on the depth of the features. The feature level after the first CNN is shallow, and three residual blocks are used to deepen the feature depth. Subsequently, two or one residual blocks are used to deepen the features of different depths.

[0101] Example 5

[0102] The specific process of post-processing in this embodiment is:

[0103] 1) Mark the points with a pixel value of 1 in the image, delete the outlines of small areas (mostly interference areas), and screen out the connected domains with less than 32 pixels on the basis of the eight connected areas. Using connected domain detection, mark the points with a pixel value of 1 in the image and obtain each line segment area, and screen out the outlines of small areas (connected domains with less than 32 pixels, mostly interference areas)

[0104] 2) Calculate the vertical line segment

[0105] (2.1) For the image after filtering out the small area, use the vertical erosion kernel (2, 20) to filter out the horizontal lines and obtain the vertical lines.

[0106] (2.2) On this basis, the vertically connected domains of small areas with pixel values ​​less than 15 are filtered.

[0107] (3.3) After removing the horizontal lines and filtering the vertical lines in a small area, the image is expanded to restore the vertical lines that were previously eroded.

[0108] (2.4) To prevent some lines from being broken in the above operation, a closing operation is performed on the restored image. The process of the closing operation is: first use a larger kernel for expansion, then use a smaller kernel for erosion (here, the (2, 7) kernel is used for expansion, and the (2, 5) kernel is used for erosion), and then the discontinuous line segments are connected.

[0109] (2.5) Using each connected space in the image and the minimum rectangular frame, obtain the minimum rectangle that can enclose each connected domain, and use the center point and height and width of the rectangle to obtain the line segment that can represent each connected space.

[0110] 3) Calculate the horizontal line segment

[0111] (3.1) For the image after filtering out the small area, use the horizontal erosion kernel (20, 2) to filter out the vertical lines and obtain the horizontal lines.

[0112] (3.2) On this basis, the vertically connected domains of small areas with pixel values ​​less than 15 are filtered.

[0113] (3.3) After removing the vertical lines and filtering the horizontal lines in a small area, the image is expanded to restore the horizontal lines that were previously eroded.

[0114] (3.4) To prevent some lines from being broken in the above operation, a closing operation is performed on the restored image. The process of the closing operation is: first use a larger kernel for expansion, then use a smaller kernel for erosion (here, the (7, 2) kernel is used for expansion, and the (5, 2) kernel is used for erosion), and then the discontinuous line segments are connected.

[0115] (3.5) Using each connected space in the image and the minimum rectangular frame, obtain the minimum rectangle that can enclose each connected domain, and use the center point and height and width of the rectangle to obtain the line segment that can represent each connected space.

[0116] 4) Calculate the oblique line segment

[0117] (4.1) Perform a logical OR operation on the horizontal line image (the connected domain of small horizontal lines with pixel values ​​less than 15 eliminated for the second time) and the vertical line image (the connected domain of small vertical lines with pixel values ​​less than 15 eliminated for the second time) to obtain an image with horizontal and vertical lines.

[0118] (4.2) Subtract the image with horizontal and vertical lines from the image after filtering out small areas (connected domains with pixel values ​​less than 32) to remove the horizontal and vertical lines.

[0119] (4.3) Filter the noise points and remove the connected domains with less than 25 pixels in the connected domains.

[0120] (4.4) Probabilistic Hough line detection is used to detect possible line segments in the image. Hough detection can connect discontinuous oblique line contours into a whole, which is beneficial to overall contour detection.

[0121] (4.5) Since the Hough line detection may miss line segments or generate multiple lines at the same location, the lines are restored here and then the connected domains are obtained again. The minimum rectangular frame is used to obtain the minimum rectangle that can enclose each connected domain, and the center point, height, width and angle of the rectangle are used to obtain the line segments that can represent each connected space.

[0122] 5) Map the acquired line segment coordinates to the original image to obtain the visualization result.

[0123] Example 6

[0124] like Figure 1 As shown, the text-oriented line segment detection method based on the UNet algorithm of the present invention specifically includes:

[0125] S1: Acquire document image data with line segments;

[0126] Obtain document image data with line segments, search for relevant data, clean the data, obtain usable data images, and randomly divide the data set into training set, validation set and test set according to a pre-set ratio.

[0127] In the specific implementation, we crawled and searched for relevant data on relevant websites such as Baidu and Wenshu.com, and cleaned the data to obtain 863 usable data images. The training set has 604 images, the validation set has 173 images, and the test set has 86 images, with a ratio of 7:2:1. The line segments in the image mainly include the line segments used to mark key fields and the line segments contained in the table data in the image, such as the underline or table line that emphasizes the key text in the text image. Figure 7 and Figure 8 shown.

[0128] S2: Preprocess the collected image with line segments into an image that conforms to the UNet input format, such as Figure 2 ;

[0129] First, the size of the captured image is normalized and the image resolution is processed to 1024×1024. The labeling tool labelme is used to manually annotate the image to obtain the position of the line segment. Then, the line segment is expanded into a rectangle based on the line segment position information, and the coordinates of the four points of the rectangle are obtained. The image is then binarized to generate a label image and obtain the target detection dataset. Each data sample consists of an original image and the corresponding binary label image.

[0130] S3: Build the line segment detection network model TR-UNet and training framework;

[0131] The TR-UNet network is based on the UNet network, such as Figure 3 As shown in the figure, after the image undergoes three 3*3 convolutions, the extracted features are integrated into the 12-layer Transformer layer after the linear layer, and the result of each CNN processing is no longer directly added to the corresponding layer in the decoder, but is combined with the decoder part after a deep residual path (Res Path). The number of residual blocks in the residual path mainly depends on the current feature depth.

[0132] The process of building the network model and training framework is as follows:

[0133] S3-1: Embed the Transformer structure into the Encoder structure of UNet, and use the hybrid CNN and Transformer structure as the encoder;

[0134] Transformer uses a self-attention mechanism to encode tokenized image blocks from convolutional neural network (CNN) feature maps into an input sequence that extracts global context. The decoder upsamples the encoded features and combines them with different high-resolution CNN feature maps that are skipped from the encoding path (the features extracted from a certain layer in the encoder are not processed and directly fused with the features in the decoder) to achieve precise positioning. First, the image is serialized, and the input x is converted into a flat two-dimensional patch sequence. Each patch size is P×P, and the number N=(HW) / P2, that is, the sequence length. Then, using trainable projection, the vectorized patch Xp is mapped to the potential d-dimensional embedding space, and added to the patch embedding to retain the position information. The calculation formula is as follows:

[0135]

[0136] Where E is the patch embedding projection, E pos represents position embedding. The Transformer encoder is composed of L layers of multi-head attention (MSA) and multi-layer perceptron blocks (MLP). The output of the layer is:

[0137]

[0138] Where LN stands for normalization, represents the image representation of the previous layer, represents the feature representation after multi-head attention, It is the image representation of the current layer after multi-head attention and multi-layer perceptron blocks.

[0139] Using this structure, the transformer encodes image blocks from the CNN feature map as an input sequence for extracting global features. The decoder upsamples the encoded features and combines them with high-resolution CNN features for precise positioning. The implementation method is shown in Figure 4 .

[0140] S3-2: Use Res Path to replace the simple skip connection in traditional UNet;

[0141] Since the number of convolution layers in the encoder is relatively shallow, the extracted features are often low-level features, and there is a large semantic gap between the two. It is not appropriate to directly splice the corresponding layers in the Encoder and Decoder, but should be further processed using convolution, and different numbers of convolution layers should be used according to the features of different scales to extract deeper features. The implementation method is as follows Figure 5 The feature information in the encoder is further extracted through k times (k depends on the current feature scale) of 3*3 and 1*1 parallel convolutions, and the deeper features finally obtained are integrated into the decoder.

[0142] S3-3: The training framework is built using the stochastic gradient descent algorithm (SGD) with momentum, using a large momentum = 0.99. During training, the update of the current optimization step can be determined based on a large number of previous training samples. In order to minimize overhead and maximize the use of GPU memory, a larger batch size is used to reduce the batch to a single image. The final energy function is defined as pixel-wise soft-max over the final featuremap, while using the EIoU loss function. Among them, soft-max is defined as:

[0143]

[0144] Among them, a k (x) represents the activation of k on the feature channel, k represents the kth feature on the feature channel, K represents the number of categories, x represents the pixel point, and k' represents the categories from 1 to K. The image is finally at each position The soft-max is:

[0145]

[0146] Among them, Ω represents the target category label corresponding to each pixel. For each pixel, the label is:

[0147]

[0148] The weight map is:

[0149] ω:Ω→R

[0150] The weight map is obtained by calculation in the pre-experimental stage:

[0151]

[0152] Among them, ω(x) is used to balance the frequency of occurrence of different categories, ω c (x) represents the weight associated with the category, ω 0 is a constant weight factor, d 1 :Ω→R represents the distance between the target and the nearest boundary, d 2 :Ω→R represents the distance between the target and the second closest boundary, and σ is the standard deviation parameter.

[0153] The EIoU calculation formula is as follows:

[0154]

[0155] Where ρ represents the Euclidean distance between the target and the actual, b, ω, and h represent the center point, width, and height of the predicted box, respectively, gt represents the actual box, and ω c and h c is the width and height of the smallest bounding box covering the two boxes. In other words, the loss function is divided into three parts, IOU loss, distance loss L dis and direction loss L asp , directly minimizes the difference in width and height between the target box and the real box, improves the convergence speed, and achieves better positioning effect.

[0156] S4: Send the training set and validation set in the collected data set to the line segment detection network model TR-UNet described in S3 for training and validation. The specific implementation steps are as follows:

[0157] (1) Experimental environment setup

[0158] The operating environment of the experiment is Ubuntu 20.04, the GPU is NVIDIA GeForce RTX 3090, the memory is 30G, and the compilation environment is python 3.8.10+pytorch 1.11.0+cuda 11.3;

[0159] (2) Setting model training hyperparameters

[0160] The experimental settings are batch size 16, number of training rounds 200, initial learning rate 0.01, input image size 640×640, momentum coefficient 0.937, weight decay coefficient 0.0005, and optimizer SGD;

[0161] (3) Send the training set to the network for training, obtain the trained model, and use the trained model for testing.

[0162] S5: Send the test set data to the detection model with optimal parameters in S4 for testing (optimal parameters mean that as the training progresses, the accuracy, recall rate and F value of the model generated in each round of training on the validation set are optimal), and output the final detection result, and conduct feasibility evaluation on the detection result. The present invention selects three commonly used evaluation indicators, including precision rate P, recall rate R, and F value, and the definitions of these indicators are as follows:

[0163]

[0164] Among them, TP represents the number of correctly identified positive samples, FP and FN represent the number of samples that are incorrectly identified as positive and negative, respectively.

[0165] To verify the effectiveness and feasibility of the improved model, the test set is put into the baseline model UNet and the optimized model TR-UNet to observe the reasoning results. The experimental results are shown in Table 1.

[0166] Table 1 Line segment detection model experiments

[0167] Model P(％) R(％) F-score(%) UNet 96.5 93.1 94.8 TR-UNet 98.7 96.4 97.5

[0168] The experimental results show that the TR-UNet proposed in this embodiment has advantages in detection accuracy compared with the baseline model, and can be fully adapted to achieve usable results.

[0169] Example 7

[0170] In order to make the detection results more consistent with the actual line segments, the present invention performs post-processing on the detection results, and outputs the actual coordinates of each line segment (represented by two points) by horizontal and vertical corrosion of the line segment image originally represented by four points. After marking the position of the line segment, the marked position is firstly detected for the connected domain, and the connected domain with less than 32 pixels is eliminated. The model prediction graph is processed with the horizontal erosion kernel (20,2) to obtain the vertical lines, and the vertical lines of the small area are filtered again, and the filtered vertical lines are expanded and restored, and the discontinuous line segments are optimized in combination with the closed interval to obtain the vertical line segment graph, and the line segment coordinates are obtained in combination with the connected domain; the model prediction graph is processed with the vertical erosion kernel (2,20) to obtain the horizontal lines, and the horizontal lines of the small area are filtered again, and the filtered horizontal lines are expanded and restored, and the discontinuous line segments are optimized in combination with the closed interval to obtain the horizontal line segment graph, and the line segment coordinates are obtained in combination with the connected domain; the horizontal line segment graph and the vertical line segment graph are added, and then the original image and the added image are subtracted to obtain the oblique line segments, and after filtering the noise and Hough transform detection, the oblique line segment coordinates are obtained using the connected domain. The horizontal, vertical and oblique line segments are combined and restored to the original image.

[0171] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A text-oriented line segment detection method based on UNet algorithm, characterized in that: The process of the detection method is: S1, obtaining text image data with line segments; S2. Normalize the size of the collected image, use the labelme annotation tool to manually annotate the image, obtain the position of the line segment, and then determine the length direction of the line segment based on the line segment position information. Take the length as the central axis and expand it into a rectangle with a width of two pixels along the length direction perpendicular to the line segment. Obtain the coordinates of the four points of the rectangle, and then binarize the image to generate a binary label image to obtain the target detection data set; Each data sample in the target detection dataset consists of an original image and a corresponding binary label image; the binary label image marks the locations of all line segments in the original image; S3. Build a line segment detection network model based on the UNet algorithm, train the line segment detection network model with the target detection dataset, predict the line segments in the text image with the trained line segment detection network model, and output the model prediction graph; S4. Post-process the model prediction image to extract the positions of horizontal line segments, vertical line segments, and oblique line segments in the text image to achieve line segment detection in the text image.

2. The detection method according to claim 1, characterized in that: The image resolution after size normalization is 1024×1024.

3. The detection method according to claim 1, characterized in that: The specific process of step S4 is: The model predicts that the pixel value of the background in the image is 0, and the pixel value of the target area is 1; S41 uses connected domain detection to mark the area with a pixel value of 1, and the area surrounded by the points with a marked pixel value of 1 is the target connected domain, and the connected domain with less than 32 pixels is eliminated; S42 uses the vertical corrosion kernel (2, 20) to perform an corrosion operation on the image processed by step S41, filters out the horizontal line connected domain in the image, obtains the vertical line connected domain, and filters out the connected domain with less than 15 pixels, eliminates the interfering vertical lines, and obtains the real vertical lines; uses expansion to restore the vertical lines corroded by the vertical corrosion kernel, and then performs a closing operation on the vertical line connected domain to optimize the discontinuous line segments. The specific process of the closing operation is: using the expansion kernel (2, 7) to perform an expansion operation on the image, and then using the corrosion kernel (2, 5) to perform an corrosion operation on the image, connecting the discontinuous lines, and obtaining a vertical line segment diagram; The collected longitudinal line segment graphs are used with a minimum rectangular frame to collect the minimum rectangle that can surround each longitudinal line segment graph, obtain the center point and height and width of the rectangular frame, and convert them into line segment coordinates, thereby obtaining the longitudinal line segment coordinates; S43 uses a horizontal corrosion kernel (20, 2) to perform an corrosion operation on the image processed in step S41, filters out the vertical line connected domain in the image, obtains the horizontal line connected domain, filters out the horizontal line connected domain with less than 15 pixels, eliminates the interfering horizontal lines, and obtains the real horizontal lines; The horizontal lines eroded by the horizontal erosion kernel are restored by dilation, and then the connected domains of the horizontal lines are closed to optimize the discontinuous line segments. The specific process of the closing operation is as follows: the image is dilated by using the dilation kernel of (7, 2), and then the image is eroded by using the erosion kernel of (5, 2), the discontinuous lines are connected, and the horizontal line segment graph is obtained; The collected horizontal line segment graphs are collected using the minimum rectangular frame to collect the minimum rectangle that can surround each horizontal line segment graph, obtain the center point and height and width of the rectangular frame, and convert them into line segment coordinates, thereby obtaining the horizontal line segment coordinates; S44: adding the image of the real vertical line obtained in step S42 and the image of the real horizontal line obtained in step S43 to obtain a combined image of horizontal and vertical lines; and then performing a subtraction operation on the combined image of horizontal and vertical lines using the image processed in S41 to obtain an oblique line segment image; S45: filtering out the connected domains with less than 25 pixels in the oblique line segment graph obtained in step S44, and then using Hough transform detection to extract the line segments that may exist in the image. The line segments detected by Hough transform can connect the discontinuous oblique line contours into a whole, which is beneficial to the overall contour detection. However, since the Hough transform detection may generate multiple lines at the same position, the line segments detected by Hough transform are restored to the image before Hough transform detection. Then, connected domain extraction is used to mark and extract the oblique line targets in the image, and each connected domain is surrounded by a minimum rectangular frame to obtain the minimum rectangular frame around each connected domain. The center point, height, width and angle of the rectangular frame are used to obtain the coordinates of the oblique line segment. S46 combines the vertical line segment coordinates obtained in step S42, the horizontal line segment coordinates obtained in step S43, and the oblique line segment coordinates obtained in step S45, and restores them to the original image, thereby realizing the detection of line segments in the text image with line segments.

4. The detection method according to claim 1, characterized in that: The line segment detection network model is a TR-UNet network, which includes an encoder and a decoder composed of CNN and Transformer structures; the CNN includes multiple layers of convolution and linear layers connected in sequence, and the hidden feature Hidden Feature output by the multiple layers of convolution is input into the Transformer structure through the linear layer. The Transformer structure includes several stacked Transformer layers, and the output of the Transformer structure is processed by reshape operation, 3*3 convolution and ReLU function, and upsampling Upsample to obtain the first feature F1; The features of each convolution output of CNN are processed by the residual path Res Path composed of at least one residual block and then concatenated with the input of the corresponding layer in the decoder; The result of each convolution processing of CNN is no longer directly added to the corresponding layer in the decoder, but is combined with the decoder part after a deep residual path. The number of residual blocks in the residual path depends on the current feature depth. The deeper the feature depth, the fewer the number of residual blocks.

5. The detection method according to claim 4, characterized in that: The number of convolutions in the CNN is three, and the size of the convolution kernel is 3*3; the number of Transformer layers in the Transformer structure is 12.

6. The detection method according to claim 5, characterized in that: The residual block is a CNN with three residual blocks connected in series after the first convolution layer, two residual blocks connected in series in the second convolution layer, and one residual block connected in series in the third convolution layer; the residual block includes a first convolution of a 3*3 convolution kernel and a second convolution of a 1*1 convolution kernel in parallel, and the outputs of the first convolution and the second convolution are added as the output of the residual block.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the detection method described in any one of claims 1 to 6 can be implemented.

Images