A scanning file layout analysis method based on Mask R-cnn algorithm and type segmentation

Abstract

The application provides a scanning file layout analysis method based on a Mask R-cnn algorithm and type segmentation, and belongs to the field of deep learning.The method mainly adopts a type segmentation technical scheme, is implemented based on the Mask R-cnn algorithm, and aims at improving the accuracy of layout analysis.The method specifically comprises the following steps: taking a scanned file color image as input, firstly performing table identification and segmentation, taking the image after table segmentation as input, performing illustration identification and segmentation, then removing a red seal, and finally performing title identification on the image after removing the table, the illustration and the red seal, and outputting the un-identified part as a text part.In a complex layout scanning file, the method solves the problem that the accuracy of the top-down, bottom-up and comprehensive methods is not high, so that the image classification, text processing and OCR and other technologies can be further optimized, and the scanning file layout analysis accuracy is finally improved.

Description

Technical Field

[0001] This invention belongs to the field of deep learning, specifically relating to a method for analyzing the layout of scanned documents based on the Mask R-CNN algorithm and type segmentation. Background Technology

[0002] Analyzing the layout of scanned documents is of great significance. Firstly, scanned documents offer advantages such as portability, ease of storage, and readability without easy modification, hence their widespread use. Secondly, the digitization of scanned documents effectively preserves information, and layout analysis is a crucial step in this process; accurate and effective layout analysis significantly contributes to ensuring precision. Furthermore, with the emergence of technologies such as image classification and text processing, layout analysis has become increasingly important. Therefore, if one wishes to intelligently process text and illustrations within scanned documents, layout analysis is an indispensable step in solving this problem.

[0003] There are three types of layout analysis: top-down, bottom-up, and combined. Top-down analysis starts with the entire document image and uses relevant algorithms to segment different regions from the whole page. This algorithm is suitable for documents with relatively standardized and simple layouts. Bottom-up analysis starts with local information and gradually merges different document regions through methods such as connected components. This algorithm is suitable for more complex layout analysis, but its efficiency is often low, and it is difficult to establish a unified merging rule.

[0004] Based on the advantages and disadvantages of the two algorithms mentioned above, a comprehensive algorithm combines top-down and bottom-up approaches, which is also the most commonly used method in layout analysis. Yang proposed a texture-based analysis method, analyzing document images based on the obvious line spacing and line texture features of image tables. Tian et al. proposed a hybrid approach combining bottom-up and top-down methods, combining features such as connected region distance, connected region size, horizontal and vertical arrangement, and reference rows. However, for scanned documents with complex page layouts, it is difficult to extract elements effectively, such as those containing red stamps or irregular tables. These areas are difficult to segment, resulting in low accuracy. Therefore, a more effective method for identifying and classifying scanned documents is needed.

[0005] Mask R-CNN is an algorithm used to detect objects in images while generating a high-quality segmentation mask for each instance. This method, called a masked region convolutional neural network, scales up the region convolutional neural network faster by adding a branch to an existing branch to predict the target mask in parallel. However, when using the Mask R-CNN algorithm to handle multi-class classification problems in layout analysis, the accuracy in identifying all tables, illustrations, titles, and text is relatively low. This is because the Mask R-CNN algorithm has certain limitations in multi-object detection. Summary of the Invention

[0006] This invention addresses the problems of low accuracy and high loss rate in the layout analysis of scanned documents by proposing a layout analysis method for scanned documents based on the Mask R-CNN algorithm and type segmentation. The core of this method is to propose a new approach: type segmentation, and to analyze the layout of scanned documents based on the Mask R-CNN algorithm and this type segmentation method, accurately identifying tables, illustrations, titles, text, and red seals respectively.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for analyzing the layout of scanned documents based on the Mask R-CNN algorithm and type segmentation is proposed. The original scanned color image is used as input. First, tables in the image are identified and segmented. The image after table segmentation is used as input for identifying and segmenting illustrations. Then, red stamps are removed from the image. Finally, the image after removing tables, illustrations, and red stamps is used for title recognition. Unrecognized portions are output as text. The specific steps include:

[0009] Step 1: Define the loading interface and Mask R-CNN model, and train the model;

[0010] Step 2: Acquire the original color image of the scanned document in real time and perform preprocessing;

[0011] Step 3: Perform classification and recognition based on the trained model. First, identify the table region in the color image based on the table recognition model, extract the table, and output the table and the image after table extraction 1.

[0012] Step 4: Then identify the illustration area in image 1, extract the illustration, and output the illustration and image 2 after extracting the illustration;

[0013] Step 5: Remove the red stamp from image 2 and output image 3;

[0014] Step 6: Finally, identify and label the title in image 3. The unlabeled part is text. Output the image after labeling the title.

[0015] Furthermore, the specific process of step 1 is as follows:

[0016] Step 1.1: For defining the loading interface, firstly, the obtained historical dataset is rewritten, the dataset is shuffled and sorted, and the training and test sets are divided. Secondly, the mapping relationship between images and masks is obtained, the mask is converted into a Tensor, and the coordinates and labels of the mask are obtained for image segmentation. Finally, the images in the dataset are augmented, and the original images are converted into PyTorch tensors.

[0017] Step 1.2: Defining the Mask R-CNN model includes obtaining the model architecture and the number of input features, and modifying the parameters of the feature category and mask category of the output model to 2; it also includes defining the training and validation data loader and using auxiliary functions to obtain the Mask R-CNN model.

[0018] Step 1.3: Use different datasets to train models for recognizing tables, illustrations, titles, and text respectively. Finally, save the trained model parameters. At this point, three models are obtained: table recognition model, illustration recognition model, and title recognition model. Each recognition model is trained using Mask R-CNN, and ResNet101 residual network is selected as the backbone feature extraction network of Mask R-CNN.

[0019] The Adam optimizer is used during model training to adjust the model's update weights and bias parameter θ. t θ t From formula (1),

[0020]

[0021] Where, θ t-1 The parameter represents the bias obtained in the previous iteration, α represents the default learning rate, and m represents the bias parameter obtained in the previous iteration. i V represents the gradient mean. t Represents the exponential moving average of the squared gradient, ∈=10 -8 ;

[0022] Among them, the learning rate plan program reduces the learning rate by a factor of 10 every 3 epochs;

[0023] During model training, a multi-task loss function L is defined for each RoI interface, and the learning rate is continuously updated by calling the optimizer; after training, the trained model parameters are saved.

[0024] L is obtained from formula (2).

[0025] L = L CLS +L BOX +L mask (2)

[0026] Among them, L CLS L represents the classification loss. BOX L represents the regression box loss. mask This indicates the loss of the newly added mask.

[0027] Furthermore, in step 2, the original color image of the scanned document is enhanced, and the enhanced image is converted to RGB three-channel format.

[0028] Furthermore, the specific process of step 3 is as follows:

[0029] Step 3.1: Input the preprocessed scanned file image into a pre-trained residual network ResNet101 and FPN to obtain the corresponding feature map. That is, input the scanned file image into ResNet101 to extract the table. ResNet101 is divided into 5 stages, and C1-C5 are the outputs of each stage. At this time, the output corresponds to all possible table regions in the scanned file.

[0030] Step 3.2: Set candidate ROI coordinates for each point in all possible table areas to obtain multiple candidate ROI coordinates;

[0031] Step 3.3: Feed multiple candidate ROIs into the RPN network for binary classification and BB regression to filter out some candidate ROIs. The specific process is as follows: first, convolve and pool the candidate ROIs to continuously reduce the size of their feature maps; then, perform deconvolution interpolation to continuously increase their feature maps; finally, classify each pixel value to achieve accurate segmentation of the input image.

[0032] Step 3.4: Perform ROIAlign operation on the remaining candidate ROI coordinates. First, map the pixels of the original image and the feature map together. Then, map the feature map together with the fixed features. Obtain the pixel values ​​of the four fixed point coordinates through bilinear interpolation. Bilinear interpolation is a linear interpolation performed in two directions. First, perform linear interpolation in the x-direction, obtained from formulas (3) and (4); then perform interpolation in the y-direction, obtained from formula (5).

[0033]

[0034]

[0035]

[0036] Where f(P) represents finding the coordinates of the function f at point P = (x, y), R1 = (x, y1), R2 = (x, y2), and it is assumed that the function is known at point Q. 11 = (x1, y1), Q 21 = (x2, y1)Q 12 = (x1, y2)Q 22 =The values ​​of the four points (x2, y2);

[0037] Step 3.5: Classify, perform bounding box regression and mask prediction on the candidate coordinate ROIs. The results return the prediction score, coordinates and labels. Based on the score, determine the recognition accuracy. Output the coordinates and labels of the scores greater than 85. The output coordinates determine the location of the true table area.

[0038] Step 3.6: Segment and save the real table area; after obtaining the coordinates of the real table, crop it using the crop() method in PIL and save it, outputting the saved table and image 1.

[0039] Furthermore, the specific process of step 4 is as follows:

[0040] Step 4.1: Set the cropped portion in Image 1 to the background color, loop through all pixels of the cropped portion, and set each pixel to the background color as input for illustration recognition;

[0041] Step 4.2: Input the image into the illustration recognition model. First, input the image into the pre-trained ResNet101 residual network to obtain the corresponding feature map. Then, set candidate coordinate ROIs for each point in the illustration region to obtain multiple candidate coordinate ROIs. Then, send these candidate coordinate ROIs into the RPN network to filter out some candidate coordinate ROIs. Finally, perform the ROIAlign operation on the remaining ROIs and obtain the pixel values ​​of four fixed point coordinates through bilinear interpolation. At this time, the location of the real illustration region is determined by the coordinates of the four fixed points.

[0042] Step 4.3: Segment and save the illustration area; after obtaining the coordinate position of the illustration area, crop it using the crop() method in PIL and save it, outputting the saved illustration and image 2.

[0043] Furthermore, the specific process of step 5 is as follows:

[0044] Step 5.1: Set the threshold: Pass a parameter cv2.THRESH_OTSU to the cv2.threshold function and set the initial value of the threshold threshold to 0. The cv2.threshold function will automatically find the optimal threshold for threshold.

[0045] Step 5.2: Remove red stamps. First, separate the three channels of the RGB image to obtain grayscale images of the blue, green, and red channels. Then, use the cv2.threshold() method in OpenCV to perform binarization. The red stamp parts of a grayscale image with pixel grayscale values ​​greater than the optimal threshold thresh are processed into white, and those with grayscale values ​​less than the optimal threshold are processed into black. After processing, output the image 3 with red stamps removed.

[0046] Furthermore, the specific process of step 6 is as follows:

[0047] Step 6.1: Set the cropped portion of image 3 to the background color, loop through all pixels of the cropped portion, and set each pixel to the background color as input for title recognition;

[0048] Step 6.2: Input the image into the title recognition model; First, input the image into the pre-trained ResNet101 residual network to obtain the corresponding feature map. Second, set candidate coordinate ROIs for each point in the title region to obtain multiple candidate coordinate ROIs. Then, send these candidate coordinate ROIs into the RPN network to filter out some candidate coordinate ROIs. Finally, perform the ROIAlign operation on the remaining ROIs and obtain the pixel values ​​of the four fixed point coordinates through bilinear interpolation. At this time, the coordinates of the four fixed points are the real title region position.

[0049] Step 6.3: Recognize and select the title, label the title area with "title", and output the unrecognized part as text; finally, output the image after labeling the title.

[0050] The beneficial technical effects of this invention are as follows:

[0051] 1. To address the issue of complex layouts in scanned documents, a method using masked region convolutional neural networks for layout analysis is proposed.

[0052] 2. To address the issue of low accuracy of Mask R-CNN in simultaneously recognizing tables, illustrations, titles, and text, a type segmentation method is proposed. The scanned image is used as input, and the table is first recognized and segmented. The image after table segmentation is used as input to recognize and segment illustrations. Finally, the image after removing the table and illustrations is used to recognize the title, and the unrecognized part is output as the text part, which improves the accuracy by 10% to 15%.

[0053] 3. To address the issue of red stamps on some scanned documents, which resulted in lower accuracy when recognizing titles and text, the red stamps were removed, further improving recognition accuracy. Attached Figure Description

[0054] Figure 1The flowchart shows the scanning document layout analysis method based on the Mask R-CNN algorithm and type segmentation of this invention.

[0055] Figure 2 This is a schematic diagram illustrating the training process of the three models in this invention. Detailed Implementation

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0057] like Figure 1 As shown, this invention proposes a method for analyzing the layout of scanned documents based on the Mask R-CNN algorithm and type segmentation. The method involves scanning the document to obtain the original color image, first identifying and segmenting tables in the image, using the segmented image as input, then identifying and segmenting illustrations, removing red stamps from the image, and finally performing title recognition on the image after removing tables, illustrations, and red stamps. The unrecognized portion is output as the text portion. Specifically, the method includes the following steps:

[0058] Step 1: Define the loading interface and the Mask R-CNN model, and train the model. The specific process is as follows:

[0059] Step 1.1: For defining the loading interface, firstly, the acquired historical dataset is rewritten by shuffling and sorting the dataset, and dividing it into training and testing sets. Secondly, the mapping relationship between images and masks is obtained, the masks are converted into Tensors, and the coordinates and labels of the mask are obtained for image segmentation. Finally, the images in the dataset are augmented, and the original images are converted into PyTorch tensors.

[0060] Step 1.2: Defining the Mask R-CNN model includes obtaining the model architecture and the number of input features, and modifying the parameters of the feature category and mask category of the output model to 2; it also includes defining the training and validation data loader and using auxiliary functions to obtain the Mask R-CNN model.

[0061] Step 1.3: Use different datasets to train models for recognizing tables, illustrations, titles, and text respectively. Finally, save the trained model parameters, such as... Figure 2 At this point, three models were trained: a table recognition model, an illustration recognition model, and a title recognition model. Each recognition model was trained using Mask R-CNN, and the residual network ResNet101 was selected as the backbone feature extraction network of Mask R-CNN.

[0062] The Adam optimizer is used during model training to adjust the model's update weights and bias parameter θ.t θ t It can be obtained from formula (1).

[0063]

[0064] Where, θ t-1 The parameter represents the bias obtained in the previous iteration, α represents the default learning rate, and m represents the bias parameter obtained in the previous iteration. i V represents the gradient mean. t Represents the exponential moving average of the squared gradient, ∈=10 -8 Avoid dividing by 0.

[0065] The learning rate plan is set to reduce the learning rate by a factor of 10 every 3 epochs.

[0066] During model training, a multi-task loss function L is defined for each RoI interface, and the learning rate is continuously updated by calling the optimizer; after training, the trained model parameters are saved.

[0067] L can be obtained from formula (2).

[0068] L = L CLS +L BOX +L mask (2)

[0069] Among them, L CLS L represents the classification loss. BOX L represents the regression box loss. mask This indicates the loss of the newly added mask.

[0070] Step 2: Acquire the original color image of the scanned document in real time and perform preprocessing, specifically including: enhancing the original color image of the scanned document and converting the enhanced image to RGB three-channel format.

[0071] Step 3: Perform classification and recognition based on the trained model. First, use the table recognition model to identify table regions in the color image, extract the table, and output the table and the image after table extraction (Image 1). The specific process is as follows:

[0072] Step 3.1: Input the preprocessed scanned file image into a pre-trained residual network ResNet101 and FPN to obtain the corresponding feature maps. That is, input the scanned file image into ResNet101 for table extraction. ResNet101 is divided into 5 stages, with C1-C5 being the output of each stage. At this time, the output corresponds to all possible table regions in the scanned file (at this time, the output table regions contain regions with incorrect recognition, which need to be further removed).

[0073] Step 3.2: Set candidate ROI coordinates for each point in all possible table areas to obtain multiple candidate ROI coordinates;

[0074] Step 3.3: Feed multiple candidate ROIs into the RPN network for binary classification (foreground or background) and BB regression to filter out some candidate ROIs. The specific process is as follows: first, convolve and pool the candidate ROIs to continuously reduce the size of their feature maps; then, perform deconvolution, i.e., interpolation, to continuously increase their feature maps; finally, classify each pixel value to achieve accurate segmentation of the input image.

[0075] Step 3.4: Perform ROIAlign operation on the remaining candidate ROI coordinates. That is, first match the pixels of the original image with the feature map, and then match the feature map with the fixed features. Obtain the pixel values ​​of the four fixed point coordinates through bilinear interpolation. Bilinear interpolation is essentially linear interpolation in two directions. First, perform linear interpolation in the x direction, which can be obtained by formulas (3) and (4); then perform interpolation in the y direction, which can be obtained by formula (5). This makes the discontinuous operation continuous, and the error is smaller when returning to the original image.

[0076]

[0077]

[0078]

[0079] Where f(P) represents finding the coordinates of the function f at point P = (x, y), R1 = (x, y1), R2 = (x, y2), and it is assumed that the function is known at point Q. 11 = (x1, y1), Q 21 = (x2, y1)Q 12 = (x1, y2)Q 22 =The values ​​of the four points (x2, y2).

[0080] Step 3.5: Classify, perform bounding box regression and mask prediction on the candidate ROI coordinates. The results return the prediction score, coordinates and labels. Based on the score, determine the recognition accuracy. Output the coordinates and labels of the scores greater than 85. The output coordinates determine the location of the real table area.

[0081] Step 3.6: Segment and save the real table area; after obtaining the coordinates of the real table, crop it using the crop() method in PIL and save it, outputting the saved table and image 1.

[0082] Step 4: Then, identify the inset area in image 1, extract the inset, and output the inset and image 2 after inset extraction. The specific process is as follows:

[0083] Step 4.1: Set the cropped portion in Image 1 to the background color, that is, loop through all the pixels of the cropped portion and set each pixel to the background color as the input for recognizing the illustration;

[0084] Step 4.2: Input the image into the illustration recognition model. First, the image is input into the pre-trained ResNet101 residual network to obtain the corresponding feature map. Then, candidate coordinate ROIs are set for each point in the illustration region to obtain multiple candidate coordinate ROIs. These candidate coordinate ROIs are then fed into the RPN network to filter out some candidate coordinate ROIs. Finally, the ROIAlign operation is performed on the remaining ROIs. The pixel values ​​of four fixed point coordinates are obtained through bilinear interpolation. At this time, the location of the real illustration region is determined by the coordinates of the four fixed points.

[0085] Step 4.3: Segment and save the illustration area; after obtaining the coordinate position of the illustration area, crop it using the crop() method in PIL and save it, outputting the saved illustration and image 2.

[0086] Step 5: Remove the red stamp from image 2 and output image 3; the specific process is as follows:

[0087] Step 5.1: Set the threshold: Pass a parameter cv2.THRESH_OTSU to the cv2.threshold function and set the initial value of the threshold threshold to 0. The cv2.threshold function will automatically find the optimal threshold for threshold.

[0088] Step 5.2: Remove the red stamp. First, separate the three channels of the RGB image to obtain grayscale images of the blue, green, and red channels. Then, use the cv2.threshold() method in OpenCV for binarization. That is, the part of the grayscale image whose pixel grayscale value is greater than the optimal threshold (i.e., the red stamp part) is processed into white, and the part that is less than the optimal threshold is processed into black. In other words, the corresponding red stamp part is processed into white, which can be regarded as removing the red stamp. After processing, output the image 3 with the red stamp removed.

[0089] Step 6: Finally, identify and label the title in image 3; the unlabeled portion is output as text. The specific process is as follows:

[0090] Step 6.1: Set the cropped portion of image 3 to the background color, that is, loop through all pixels of the cropped portion and set each pixel to the background color as input for title recognition.

[0091] Step 6.2: Input the image into the title recognition model. First, the image with the illustration removed is input into the pre-trained ResNet101 residual network to obtain the corresponding feature map. Then, candidate coordinate ROIs are set for each point in the title region to obtain multiple candidate coordinate ROIs. These candidate coordinate ROIs are then fed into the RPN network to filter out some candidate coordinate ROIs. Finally, the ROIAlign operation is performed on the remaining ROIs. The pixel values ​​of the four fixed points are obtained through bilinear interpolation. At this time, the coordinates of the four fixed points are the real title region location.

[0092] Step 6.3: Identify and select the title; label the title area with the tag "title", and the unrecognized part is the text; output the image after labeling the title.

[0093] Mask R-CNN can perform multi-class object detection, but its recognition performance is poor when performing layout analysis on scanned documents. Therefore, the multi-class task is converted into multiple binary classification tasks and performed sequentially. That is, tables are identified first, illustrations are identified second, red seals are removed, and titles are identified last. Unrecognized parts are output as text, with the aim of improving the accuracy of layout analysis.

[0094] This invention demonstrates the feasibility and superiority of its method through verification and comparative experiments. To prevent randomness, a total of twenty-one images were selected for this experiment. Nine images were scanned documents containing only tables, illustrations, or titles per page; nine images were scanned documents containing tables and illustrations, tables and titles, or illustrations and titles per page; and three images were scanned documents containing tables, illustrations, and titles per page. The twenty-one images were input sequentially. The specific steps of the verification experiment are as follows:

[0095] Experimental environment configuration: Windows 10 operating system, AMD Ryzen 3600X CPU@4.4GHz, 16GB RAM, python3.8, PyTorch1.1.1

[0096] The experiment selected different datasets for identifying different types of data, as detailed in Table 1:

[0097] Table 1 Dataset Information

[0098]

[0099] Input: Image X.

[0100] Output 1: The result of extracting the table from image X, and X' is the result after removing the table from X.

[0101] Output 2: The result after extracting the inset from image X', where X” is the result after removing the inset from X.

[0102] Output 3: The result after selecting the title in the image "X".

[0103] Step 1: Input the scanned file into a pre-trained ResNet101 to obtain the corresponding table regions;

[0104] Step 2: Set a predetermined number of ROIs for each point in this table area to obtain multiple candidate ROIs;

[0105] Step 3: Feed these candidate ROIs into the RPN network for binary classification (foreground or background) and BB regression to filter out some candidate ROIs;

[0106] Step 4: Perform the ROIAlign operation on these remaining ROIs, that is, first match the pixels of the original image with the pixels of the feature map, and then match the feature map with the fixed features;

[0107] Step 5: Classify, perform bounding box regression and mask prediction on the candidate ROI coordinates. The results return the prediction score, coordinates and labels. Based on the score, determine the recognition accuracy. The results output the coordinates and labels of those with a score greater than 85.

[0108] Step 6: Split the table and save it;

[0109] Step 7: Set the cropped portion of the scanned image to the background color and use it as input for illustration recognition;

[0110] Step 8: Repeat steps 1-8 to identify the illustrations and use the scanned file with the illustrations removed as input for the title recognition;

[0111] Step 9: Repeat steps 1 to 7 to identify the title, select the title, label it "title", and output the unrecognized part as text.

[0112] Output 1: The table in image X;

[0113] Output 2: Inset in image X;

[0114] Output 3: Image X with tables, illustrations, and red stamps removed, followed by the title of the image Y.

[0115] To evaluate the merits of this invention, it was compared with the pp-structure method proposed by PaddlePaddle, the AlexNet method proposed by Wang et al., and the multi-class MASK R-CNN method. The results are shown in Table 2.

[0116] Table 2 Comparison results of different methods

[0117]

[0118] As shown in Table 2, on the same dataset, the method proposed in this invention improves the accuracy of scanned documents by 10%-15% compared to the multi-class MASK R-CNN method. It also improves by 4.7% compared to the AlexNet method. Since pp-structure's table recognition technology mainly uses the attention-based image description model RARE, its table recognition rate is high, but its recognition rate for illustrations and titles is low. The overall recognition rate of the method proposed in this invention is approximately 1.2% higher than that of pp-structure. Therefore, the method proposed in this invention achieves good results in layout analysis and classification accuracy of scanned documents.

[0119] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A scanning file layout analysis method based on Mask R-cnn algorithm and type segmentation, characterized in that, The scanned original color image is taken as input, and first, the table in the image is identified and segmented, the image after removing the table is taken as input, the illustration is identified and segmented, the illustration is removed, then the red seal in the image is removed, and finally the image after removing the table, illustration and red seal is subjected to title identification, and the part not identified is output as a text part, which specifically comprises the following steps: Step 1, define the loading interface and Mask R-cnn model, and train the model; Different data sets are used for identifying tables, illustrations, titles and texts, and models for identifying tables, illustrations and titles are trained, and finally the trained model parameters are saved. At this time, three models are obtained by training: a table identification model, an illustration identification model and a title identification model. Each identification model is trained using Mask R-cnn, and a residual network ResNet101 is selected as the backbone feature extraction network of Mask R-cnn; Step 2, real-time acquisition of scanned file original color image and pre-processing; Step 3, classification and identification based on the trained model, first, identify the table area in the color image based on the table identification model, extract the table, and output the table and the image 1 after removing the table; Step 4, then identify the illustration area in the image 1, extract the illustration, and output the illustration and the image 2 after removing the illustration; Step 5, remove the red seal in the image 2, and output the image 3; Step 6, finally, identify and label the title in the image 3, and the part not labeled is the text, and output the image after labeling the title.

2. The method of claim 1, wherein the Mask R-cnn algorithm and type segmentation-based layout analysis of a scanned document is characterized by, The specific process of step 1 is as follows: Step 1.1, for defining the loading interface, first, rewrite the obtained historical data set, shuffle the data set, divide the training set and the test set, second, obtain the mapping relationship between the image and the mask, convert the mask to Tensor, obtain the coordinates and labels of the mask mask, which are used for image segmentation; finally, enhance the image in the data set, and convert the original image to a PyTorch tensor; Step 1.2, for defining the Mask R-cnn model, including obtaining the model architecture and the input feature number of the model, modifying the feature categories and mask categories of the output model to 2; also including defining the training and validation data loader, and using the auxiliary function to obtain the Mask R-cnn model; Step 1.

3. The Adam optimizer is used in the model training to optimize the model update weights and bias parameters , From equation (1), (1) wherein, denotes the bias parameter from the last iteration, denotes the default learning rate, denotes the mean of the gradients, denotes the exponentially moving average of the squared gradients, ; Wherein, the learning rate plan program is set to reduce the learning rate by 10 times every 3 epochs; During the model training, a multi-task loss function L is defined for each RoI interface, and the learning rate is constantly updated by calling the optimizer; after the training is completed, the trained model parameters are saved; L is obtained from formula (2), (2) where, represents the classification loss, represents the regression bounding box loss, represents the newly added mask loss.

3. The method of claim 1, wherein the Mask R-cnn algorithm and type segmentation are used for layout analysis of a scanned document. In step 2, the scanned original color image is subjected to image enhancement, and the enhanced image is converted to an RGB three-channel format.

4. The method of claim 1, wherein the Mask R-cnn algorithm and type segmentation-based layout analysis of a scanned document is characterized by, The specific process of step 3 is as follows: Step 3.1, input the pre-processed scanned file image into a pre-trained residual network ResNet101 and FPN to obtain the corresponding feature map, that is, input the scanned file image into ResNet101 for table extraction, ResNet101 is divided into 5 stages, C1-C5 is the output of each stage, at this time the output corresponds to all possible table regions in the scanned file; Step 3.2, set a candidate coordinate ROI for each point in all possible table regions to obtain a plurality of candidate coordinate ROIs; Step 3.3, send the plurality of candidate coordinate ROIs into the RPN network for binary classification and BB regression to filter out a part of the candidate coordinate ROIs; the specific process is: first, the candidate coordinate ROIs are convolved and pooled to continuously reduce the size of the feature map; then, the interpolation operation of deconvolution is performed to continuously increase the feature map, and finally, each pixel value is classified to accurately segment the input image; Step 3.4, perform ROIAlign operation on the remaining candidate coordinate ROIs, first correspond the pixels of the original image and the feature map, then correspond the fixed features and the feature map, and obtain the pixel values of the fixed four point coordinates through bilinear interpolation, bilinear interpolation is linear interpolation in two directions, first linear interpolation in x direction, obtained by formula (3) (4); then interpolation in y direction, obtained by formula (5), (3) (4) (5) wherein, denotes the derivative of the function f at the point with respect to the coordinate , , assuming that the values of the function at , , , four points are known; Step 3.5, classify, frame regress and mask predict the candidate coordinate ROIs, and return the prediction score, coordinate and label, according to the score, judge the accuracy of identification, output the coordinates and labels with a score greater than 85, and the output coordinates determine the position of the real table region; Step 3.6, segment and save the real table region; after obtaining the coordinates of the real table, crop and save through the crop() method in PIL, output the saved table and image 1.

5. The method of claim 1, wherein the method is based on Mask R-cnn algorithm and type segmentation. The specific process of step 4 is as follows: Step 4.1, set the cropped part in image 1 to the background color, loop through all the pixel points of the cropped part, and set each pixel point to the background color as the input for identifying the illustration; Step 4.2, input the image into the illustration recognition model; first, input the image into the pre-trained residual network ResNet101 to obtain the corresponding feature map, second, set a candidate coordinate ROI for each point in the illustration region to obtain a plurality of candidate coordinate ROIs, then send these candidate coordinate ROIs into the RPN network to filter out a part of the candidate coordinate ROIs, and finally, perform ROIAlign operation on the remaining ROIs to obtain the pixel values of the fixed four point coordinates, at this time, the real illustration region position is determined through the fixed four point coordinates; Step 4.3, segment and save the illustration region; after obtaining the coordinate position of the illustration region, crop and save through the crop() method in PIL, and output the saved illustration and image 2.

6. The method of claim 1, wherein the method is based on Mask R-cnn algorithm and type segmentation. The specific process of step 5 is as follows: Step 5.1, set threshold: pass in a parameter cv2.THRESH_OTSU in the cv2.threshold function, and set the initial value of the threshold thresh to 0, and the cv2.threshold function will automatically find the optimal threshold thresh; Step 5.2, remove red seal, first split the three channels of the RGB image to get the blue, green and red channel gray images, secondly, use the cv2.threshold() method in OpenCV to binarize the image, and the red seal part of the image with pixel gray value greater than the optimal threshold thresh is processed to white, and the part with pixel gray value less than the optimal threshold is processed to black; After processing, output the image 3 without red seal.

7. The method of claim 1, wherein the method is based on Mask R-cnn algorithm and type segmentation. The specific process of step 6 is as follows: Step 6.1, set the cropped part in image 3 to the background color, loop through all the pixel points in the cropped part, and set each pixel point to the background color as the input for title recognition; Step 6.2, input the image into the title recognition model; first input the image into the pre-trained residual network ResNet101 to obtain the corresponding feature map, secondly, set the candidate coordinates ROI for each point in the title area to obtain multiple candidate coordinates ROI, then input these candidate coordinates ROI into the RPN network to filter out a part of the candidate coordinates ROI, finally, perform ROIAlign operation on the remaining ROI to obtain the pixel value of the fixed four point coordinates, at this time the coordinates of the fixed four points are the real title area position; Step 6.3, recognize and frame the title, label the title area as title, and output the text part as text; finally, output the image with labeled title.

Images