A document layout analysis method based on mask constraint

Abstract

The application discloses a document layout analysis method based on mask constraint, and uses a detection method to process semantic gap defects existing in document layout analysis, and belongs to the technical field of digital image processing.The application proposes a novel MASK constraint aggregation algorithm to ensure that global semantic information in input can be further mined and the problem of overlapping of bounding boxes after MASK R-CNN processing is solved.Finally, the application uses a document benchmark data set with complex layout as a verification index (DSSE-200, FPD), and the result shows that the method proposed by the application can achieve obvious performance improvement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a document layout analysis method based on mask constraint. BACKGROUND

[0002] Document layout analysis (DLA) plays a very important role in the field of computer vision. The goal of document layout analysis is to divide the high-level semantic regions (i.e. figures, tables, text, background) in a document image. Traditional DLA can be divided into two categories, namely top-down and bottom-up strategies. With the vigorous development of deep neural networks, deep learning algorithms have become very common in solving pattern recognition and computer vision problems. Therefore, deep learning-based document layout analysis techniques have received more attention in solving complex layout analysis. For example, fully convolutional neural network FCNN, multi-scale deep neural network. Recently, a lot of progress has been made in treating the DLA task as a semantic segmentation task, including multi-task fully convolutional network, deep generative model for page layout design, cross-domain document object detection algorithm, FCNN framework including only one convolutional layer. The experimental results obtained are comparable to a series of complex deep neural networks.

[0003] At present, the method based on semantic segmentation and the method based on object detection are two mainstream methods of document layout analysis. Compared with the method based on semantic segmentation, the method based on object detection has a unique advantage in ensuring the integrity of the segmentation result, and the most prominent work is based on Mask R-CNN. However, since the detection frame of Mask R-CNN generally exists in the form of a rectangle, the method using Mask R-CNN has certain challenges in detecting complex documents with non-Manhattan layout. When using the Mask R-CNN framework (Mask R-CNN retrained on the document layout analysis dataset) to process document layout analysis, the semantic gap will mislead the model, resulting in incomplete detection. Because the pictures in the document are special, many pictures contain a large amount of text, but the text is considered as a separate category. Therefore, a semantic gap in classification is caused, which leads to the problem of some semantic objects being truncated. At the same time, using Mask R-CNN will cause the overlap of the detection area. Therefore, how to solve the problems of semantic object being truncated and detection area overlapping caused by Mask R-CNN and improve the accuracy of document layout analysis is a key problem to be solved. SUMMARY

[0004] In view of this, the purpose of the present invention is to provide a document layout analysis method based on mask constraints. In order to solve the problem of semantic errors, the present invention embeds a structure containing global semantic information. In order to process the fused multi-channel data, the present invention establishes a lightweight structure similar to Spatial Pyramid Pooling. In order to solve the problem of overlapping detection areas, the present invention proposes a post-processing algorithm, which is similar to the clustering algorithm. The present invention integrates the object detection framework of R-CNN into the segmentation inside the image, and then superimposes the MASK of the input image with the original image to form an input object with 4 channels and puts it into the MASK constraint. Finally, the constraint aggregation algorithm is used to obtain the prediction result.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A document layout analysis method based on mask constraints includes the following steps:

[0007] Step 1: Integrate Mask R-CNN’s object detection framework into the image segmentation.

[0008] Step 2: Superimpose the input image’s mask with the original image, and then superimpose the image’s RGB channel information onto the output of Mask R-CNN to form an input object with 4 channels.

[0009] Step 3: Put the input object into the MASK constraint, and finally get an output result with 6 channels;

[0010] Step 4: Use the image background estimation method based on depth map segmentation, namely the constrained aggregation algorithm to obtain the prediction result.

[0011] The step 3 is specifically as follows:

[0012] 3.1: Build a pyramid pooling model to make more comprehensive use of the 4-channel input;

[0013] 3.2: In order to extract information from different receptive fields, each convolution upgrades the input dimension to 8 channels, resulting in a total of 32 channels of output;

[0014] 3.3: Adaptive average pooling is used to extract the global information of the entire input to achieve global information more effectively;

[0015] 3.4: The feature information of the 32 channels is connected, and then they are subjected to 1×1 convolution operation to reduce the dimension, reducing the number of channels to 8;

[0016] 3.5: The result of the operation is subjected to a 3×3 convolution operation again, and the number of channels remains unchanged at 8;

[0017] 3.6: Finally, use 1×1 convolution operation and 6 convolution kernels to reduce the number of channels to 6 as the output result.

[0018] Step 4 uses the image background estimation method based on depth map segmentation, that is, the constrained aggregation algorithm to obtain the prediction result, specifically:

[0019] 4.1: Put all the coordinate points in the output of step 3 that are not classified as background into a list;

[0020] =; mode rgb_b to get the RGB value of mask1 in img as 0;

[0021] 4.2: Starting from any point in the list, use RGB digital operations to search for similar points in its four neighborhoods. If there are similar points, continue searching and remove the searched points from the list; enumerate all points x in img whose mask1 is not 0 and whose rgb_b distance is greater than t; expand the surrounding rgb distance to t, and the mask2 value of the points where mask1 is 0;

[0022] 4.3: If not, return; then continue searching for other points in the list until the list is empty;

[0023] will return;

[0024] 4.4: Merge the found regions and output the predicted results; the specific form is:

[0025] Input: h××3: original image, h×: original image mask, t: threshold;

[0026] Output: h×: final prediction;

[0027] The loss function is as follows:

[0028]

[0029] in:

[0030] y is the target category label of pixel i;

[0031] , is the predicted posterior probability of pixel i and category y;

[0032] The threshold t is set to select only the top K pixels with the largest loss;

[0033] Set K = 0.2 × N.

[0034] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0035] The accuracy of the DSSE processed by the present invention is 92%, which is close to the actual value. In particular, the method is superior to the general semantic segmentation method in terms of preserving the integrity of the region.

[0036] The regional integrity and the accuracy of the recognition results of the method of the present invention are similar to those of the actual image. Especially after using the mask constraint, the results of the present invention are more complete and the segmentation problem caused by semantic gap rarely occurs.

[0037] The model of the present invention has more significant performance in the category with total constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the present invention;

[0039] Figure 2 Schematic diagram of a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below with reference to specific examples and accompanying drawings.

[0041] Such as process Figure 1 and Figure 2 As shown, the document layout analysis method based on mask constraints of the present invention includes the following steps:

[0042] (1) Design the DLA framework MCNET with MaskR-CNN as the backbone.

[0043] (1.1) Pass the input image through ResNet101 to obtain the feature map

[0044] (1.2) Use RPN to select the sub-network in the generated feature map input area

[0045] (1.3) In the bounding box fine-tuning step in RPN, ROI pooling is used to process multiple input sizes. Obtain ROIMaps

[0046] (1.4) Output the processed image of ROIMaps through the fully connected layer.

[0047] (2) Integrate the original Mask R-CNN object detection framework into the segmentation within the image.

[0048] (3) After obtaining the mask of the input image, the mask is superimposed on the original image to form an input object with 4 channels, and adaptive average pooling is used to extract the global information of the entire object to more effectively realize global information.

[0049] (4) Put the input object into the MASK constraint and finally get the prediction result.

[0050] in, Figure Two The C in it is the concat operation, which is used for feature fusion; AVP is Adaptive average pooling, that is, adaptive average pooling.

[0051] Example

[0052] See Figure 2 The newspaper image on the far left of the image is used as input. Using Mask R-CNN as the backbone, the input image is passed through ResNet101 to generate feature maps. The resulting feature maps are then fed into the RPN subnetwork for region selection. Within the RPN, bounding box fine-tuning is performed, and ROI pooling is used to handle various input sizes. ROIMaps are obtained and processed through a fully connected layer to produce the processed image output. This is the original segmentation on the left of the first Concat. The original Mask R-CNN object detection framework is then integrated into the internal segmentation of the image. After obtaining the input image mask, the mask is overlaid with the original image to form an input object with four channels. Adaptive average pooling is then used to extract global information about the entire object for more efficient global information. The input object is then constrained by the mask, resulting in the predicted result, which is the new image segmentation on the right. Comparing the two, the latter shows significantly more accurate and detailed segmentation.

[0053] The above description is only a preferred embodiment of the present invention, and certain modifications may be made thereto within the scope defined by the claims of the present invention, but all modifications will fall within the scope of protection of the present invention.

Claims

1. A document layout analysis method based on mask constraints, characterized in that: The following steps are involved: Step 1: Integrate the Mask R-CNN object detection framework into the image internal segmentation; Step 2: Superimpose the input image’s mask with the original image, and then superimpose the image’s RGB channel information onto the output of Mask R-CNN to form an input object with 4 channels. Step 3: Put the input object into the MASK constraint, and finally get an output result with 6 channels; Step 4: Use the image background estimation method based on depth map segmentation, namely the constrained aggregation algorithm to obtain the prediction result; where: The step 3 is specifically as follows: 3.1: Build a pyramid pooling model to make more comprehensive use of the 4-channel input; 3.2: In order to extract information from different receptive fields, each convolution upgrades the input dimension to 8 channels, resulting in a total of 32 channels of output; 3.3: Adaptive average pooling is used to extract the global information of the entire input to achieve global information more effectively; 3.4: The feature information of the 32 channels is connected, and then they are subjected to 1×1 convolution operation to reduce the dimension, reducing the number of channels to 8; 3.5: The result of the operation is subjected to a 3×3 convolution operation again, and the number of channels remains unchanged at 8; 3.6: Finally, use 1×1 convolution operation and 6 convolution kernels to reduce the number of channels to 6 as the output result; Step 4 uses the image background estimation method based on depth map segmentation, that is, the constrained aggregation algorithm to obtain the prediction result, specifically: 4.1: Put all the coordinate points in the output of step 3 that are not classified as background into a list; ; Get the rgb value mode rgb_b when mask1 in img is 0; 4.2: Start from any point in the list and use RGB digital operations to search for similar points in its four neighborhoods. If there are similar points, continue searching and remove the searched points from the list; enumerate all points x in img whose mask1 is not 0 and whose rgb_b distance is greater than t; expand the surrounding rgb distance to t, and the mask2 value of the points where mask1 is 0 is ; 4.3: If not, return; then continue searching for other points in the list until the list is empty; Will return; 4.4: Merge the found regions and output the predicted results; the specific form is: enter: : original image, : original image mask, t: threshold; Output: :Final prediction; The loss function is as follows: ； in: is the target category label of pixel i; For pixel i and category The predicted posterior probability of ; Threshold The setting is to select only the first K pixels with the largest loss; Set K = 0.2 × N.

Images