Background document layout analysis method based on visual model and graph model

Abstract

The invention discloses a background document layout analysis method based on a visual model and a graph model. The method comprises the steps that natural scene document pictures are collected, and data set construction is completed; obtaining a preliminary layout analysis result by using a visual network based on large kernel convolution; and correcting by using the graph neural network. The method has the beneficial effects that a natural scene layout analysis data set is established, a scene of a document layout analysis task is expanded, a visual model is constructed to carry out layout analysis, a large-kernel convolutional neural network with higher feature extraction capability is used, the detection capability on distorted and wrinkled document image components is improved, and the detection efficiency is improved. And the model is lightened while the accuracy of layout analysis is improved. And finally, constructing a position relationship between graph neural network analysis components to eliminate the interference of illumination and shadow on layout analysis tasks, and correcting a visual model result to further improve the accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of document intelligent analysis, and in particular to a method for background document layout analysis based on a visual model and a graph model. Background Art

[0002] In recent years, document analysis and recognition technology has been widely used in people's daily life and work, such as automatic recognition of postal addresses, recognition and retrieval of ancient manuscripts, form or document processing, digital archives, digital libraries, etc., which greatly improves the retrieval efficiency and utilization rate of information. At the same time, with the continuous improvement of barrier-free policies in recent years, barrier-free paper reading has received more attention. In addition to the field of digital information management, more humanized intelligent applications have also emerged with the support of document analysis and recognition technology, such as blind assistive devices, online education systems, content-based image or video retrieval systems, etc., which subtly affect people's quality of life. In short, the emergence and development of document analysis and recognition technology has brought great convenience to all walks of life, and actively promoted the development of society towards fully automated office and digital information management.

[0003] Evaluating the effectiveness of different methods in real-world scenarios (pictures) is a huge challenge; most datasets only include scientific articles, which are typeset using a unified template and lack variability. The document language is limited, and the English documents in the existing datasets account for a large proportion, with only a small number of Chinese datasets; therefore, the lack of natural scene Chinese document image datasets has become a major obstacle to improving layout analysis tasks. At the same time, natural scene document images have complex background interference factors such as distortion, wrinkles, lighting and shadows, which makes layout analysis tasks more difficult.

[0004] Therefore, it is necessary to propose a method for background document layout analysis based on visual model and graphic model in view of the above technical solution. Summary of the invention

[0005] The purpose of the present invention is to provide a method for background document layout analysis based on visual model and graphic model to solve the above problems.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for background document layout analysis based on a visual model and a graph model, the method steps are as follows:

[0008] S1, collect natural scene document images and complete the dataset construction;

[0009] S2. Use a large kernel convolution-based visual network to obtain preliminary layout analysis results;

[0010] S3. Use graph neural network for correction.

[0011] In step S1, crawler technology is used to crawl document images, and then manual filtering and labeling are performed to complete the construction of the natural scene data set.

[0012] The document types include scientific articles, books, announcements, government documents and examination papers.

[0013] Document annotations include text, title, picture, picture title, table, table title, header, footer, comments, formula, stamp, and QR code.

[0014] In step S2, a large convolution kernel layer is used to establish a sufficient effective receptive field (ERF), thereby saving computational budget for other effective structures.

[0015] In step S2, the visual network introduces the occlusion attention mechanism SEAM into the visual model to enable it to effectively handle occlusion scenes.

[0016] The visual model introduces a separation and enhancement attention module to enhance the responsiveness of the occluded components after the output of the Neck layer, thereby improving the accuracy of model recognition.

[0017] Step S3 uses the graph neural network's ability to analyze and extract the spatial location information of document components to correct the visual network.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention establishes a natural scene layout analysis data set to expand the scenarios of document layout analysis tasks, and secondly constructs a visual model to perform layout analysis, using a large-core convolutional neural network with stronger feature extraction capabilities to improve the detection capabilities of twisted and wrinkled document image components, while improving the accuracy of layout analysis and making the model lightweight, constructing a graph neural network to analyze the positional relationship between components to eliminate the interference of light and shadow on the layout analysis task, and correcting the visual model results to further improve the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the process flow of the layout analysis method provided by the present invention;

[0020] Figure 2 Schematic diagram of the SEAM attention mechanism provided by the present invention;

[0021] Figure 3 A schematic diagram of the overall flow of the model provided by the present invention;

[0022] Figure 4 A schematic diagram of the overall architecture provided by the present invention. DETAILED DESCRIPTION

[0023] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0024] The following is combined with Figures 1 to 4 The present invention is described in further detail.

[0025] like Figure 1 As shown, an embodiment of the present invention provides: a method for background document layout analysis based on a visual model and a graph model, wherein the method steps are as follows:

[0026] S1, collect natural scene document images and complete the dataset construction;

[0027] S2. Use a large kernel convolution-based visual network to obtain preliminary layout analysis results;

[0028] S3. Use graph neural network for correction.

[0029] In step S1, crawler technology is used to crawl document images, and then manual filtering and labeling are performed to complete the construction of the natural scene data set.

[0030] The document types include scientific articles, books, announcements, government documents and examination papers.

[0031] Document annotations include text, title, picture, picture title, table, table title, header, footer, comments, formula, stamp, and QR code.

[0032] In step S2, a large convolution kernel layer is used to establish a sufficient effective receptive field (ERF), thereby saving computational budget for other effective structures.

[0033] In step S2, the visual network introduces the occlusion attention mechanism SEAM into the visual model to enable it to effectively handle occlusion scenes.

[0034] The visual model introduces a separation and enhancement attention module to enhance the responsiveness of the occluded components after the output of the Neck layer, thereby improving the accuracy of model recognition.

[0035] Step S3 uses the graph neural network's ability to analyze and extract the spatial location information of document components to correct the visual network.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention establishes a natural scene layout analysis data set to expand the scenario of document layout analysis tasks, and then constructs a visual model to perform layout analysis, using a large-core convolutional neural network with stronger feature extraction capabilities to improve the detection capabilities of twisted and wrinkled document image components, while improving the accuracy of layout analysis and making the model lightweight. Finally, the positional relationship between the graph neural network analysis components is constructed to eliminate the interference of light and shadow on the layout analysis task, and to correct the visual model results to further improve the accuracy.

[0037] Graph neural networks have unique advantages in document space layout modeling. In many structural analysis tasks, graph neural networks have achieved quite good results, such as excellent performance in heterogeneous and complex structural relationships such as table structure recognition. Document writers will typeset documents according to a certain layout logic when writing documents. Many components have extremely obvious positional features, so many components can be used to mine rich logical relationships through positional relationships.

[0038] In the layout analysis tasks in actual application scenarios, on the one hand, the granularity of data annotation is reduced. Image documents do not contain rich metadata like PDF files, and it is impossible to extract fine-grained or even single-character positional relationships from PDF files. It is extremely difficult to produce many paragraph-level or character-level annotation data sets. On the other hand, there is a disconnect between visual models and graph models. Many studies use existing visual models to generate component candidate boxes as the input of graph model training tasks. However, since the visual model is only used as a pre-module for information extraction during the entire model construction process, the synergy between the two models cannot be maximized.

[0039] In natural scenes, on the one hand, factors such as lighting and shadows will cause great interference to the visual model, causing the visual model to form incorrect candidate boxes on the document image, but such misjudged components are very easy to judge using positional relationships; on the other hand, document images in natural scenes are distorted and wrinkled, and the changes in the position information of the component candidate boxes will be more obvious and frequent compared to the previous neat PDF files, which will increase the interference to the graph model. The previous model of simply combining the visual model and the graph model will not be able to combine the advantages of the two models, resulting in difficulty in jointly leveraging the greatest advantages of both models on natural scene data sets. At the same time, in the layout analysis task, the scholars who conducted the research used traditional algorithms in the definition of node-edge relationships in graph neural networks, and did not analyze the rich information obtained from the visual model much, but simply integrated it into the node information.

[0040] Step S1, completing the dataset construction for the natural scene layout analysis dataset.

[0041] Most public datasets are currently images converted from PDF documents, lacking actual document images. Models trained on these datasets cannot be well generalized to real-world scenarios. In order to obtain more datasets, document images are crawled using crawler technology, and then manually filtered and annotated to complete the construction of natural scene datasets. The Albumentations algorithm increases the size and diversity of the labeled training set by utilizing input transformations that retain corresponding output labels, thereby obtaining enough data to pre-train the model and improve the convergence, generalization, and robustness of non-distributed samples. Combined with the needs of production practice, the document annotation categories are divided into 11: text, title, image, image title, table, table title, header, footer, note, formula, seal, and QR code. The collected document types include scientific articles, books, announcements, government documents, test papers, etc.

[0042] The background of document images in natural scenes is complex. Document images in natural scenes are interfered by many factors such as lighting, shadows, distortions and wrinkles. Complex scenes and diverse layout formats make natural scene layout analysis more difficult and less accurate.

[0043] Step S2, constructing a visual model to perform layout analysis, using a large-core convolutional neural network with stronger feature extraction capabilities to improve the ability to detect twisted and wrinkled document image components, thereby improving the accuracy of layout analysis while making the model lightweight.

[0044] In most layout analysis visual model tasks, the extraction of image features uses a convolutional neural network architecture. The convolutional neural network loses a lot of information in the pooling layer, which leads to a reduction in spatial resolution. Therefore, when the input changes slightly, the output of the network remains basically unchanged. The internal data representation of the convolutional neural network does not take into account the important spatial hierarchy between simple and complex objects.

[0045] The background of document images in natural scenes is complex. Document images in natural scenes are interfered by many factors such as lighting, shadows, distortions and wrinkles. Complex scenes and diverse layout formats make natural scene layout analysis difficult and less accurate. Therefore, the traditional convolutional neural network architecture cannot be used to extract features, and the ability of large-core convolutional neural networks to retain image spatial information has great potential.

[0046] Large kernel convolutional neural networks have received extensive research attention. Recent studies have shown that the excellent performance of Vision transformers is due to large receptive fields. Therefore, large convolutional kernel design has become an ideal solution to make convolutional neural networks excellent again. It adopts an efficient structure that can communicate between channels and perform spatial aggregation, increase depth, and improve the ability to extract features. Large kernel convolutional networks can see more things without going deep. Since increasing the size of the convolution kernel is much more effective than stacking more layers when expanding the receptive field, a small number of large convolutional kernel layers can be used to establish a sufficient effective receptive field (ERF), thereby saving computational budget for other effective structures. These structures are more effective in increasing the level of abstraction of spatial patterns or increasing depth. The essential advantage of large convolutional kernels is that they do not rely on large receptive fields stacked in depth, which also makes the entire model more lightweight and is conducive to the deployment of production practices.

[0047] Convolution can be expressed by formula 1:

[0048]

[0049] Where (i, j) represents the position of the sliding window on the image or feature map, kw and kh are the dimensions of the convolution kernel, p represents the position information, w and × are the weights and features at the corresponding positions, let kw>kh, the large convolution kernel can be decomposed by the following transformation (decomposition of the kernel with large structure), as shown in Formula 2:

[0050]

[0051] in Is the rounding up function. From Formula 2, it is easy to see that the decomposed large convolution kernel, that is, the case of kw>kh, can be replaced by multiple regular small convolution kernels. Of course, this requires adjusting hyperparameters such as offset control and padding settings, where a 15×3 convolution can be equivalently represented as 5 3×3 convolutions, and a displacement operation is required. Unlike the previous case, the displacement needs to move farther in a certain dimension and corresponds to a grid of size kh.

[0052] The occlusion attention mechanism SEAM is introduced into the visual network to effectively handle occlusion scenes. Its structural diagram is shown in the figure Figure 2 As shown in Figure 2, the architecture of SEAM is on the left, and the architecture of CSMM (Channel and Spatial Mixed Module) is on the right. CSMM uses multi-scale and feature-depth separable convolutions to learn the correlation between spatial dimensions and channels.

[0053] The first part of SEAM is the depthwise separable convolution with residual connection. Although the depthwise separable convolution can learn the importance of different channels and reduce the number of parameters, it ignores the information relationship between channels. To make up for this loss, in the second part, the outputs of convolutions of different depths are combined after pointwise (1x1) convolution. In the third part, a two-layer fully connected network is used to fuse the information of each channel, so that the network strengthens the connection between channels.

[0054] Step S3, construct the positional relationship between the graph neural network analysis components to eliminate the interference of lighting and shadows on the layout analysis task, correct the visual model results, and further improve the accuracy.

[0055] Graph neural networks have unique advantages in modeling document spatial layout patterns. In many structural analysis tasks, graph neural networks have achieved very good results. For example, they perform well in heterogeneous and complex structural relationships such as table structure recognition. Document writers will layout the documents according to a certain layout logic when writing them. Many components have extremely obvious positional characteristics, so many components can mine rich logical relationships through positional relationships.

[0056] However, in the layout analysis tasks in actual application scenarios, on the one hand, the granularity of data annotation is reduced. Image documents do not contain rich metadata like PDF files, and it is impossible to extract fine-grained or even single-character positional relationships from PDF files. It is extremely difficult to produce many paragraph-level or character-level annotation data sets. On the other hand, there is a disconnect between visual models and graph models. Many studies use existing visual models to generate component candidate boxes as the input of graph model training tasks. However, since the visual model is only used as a pre-module for information extraction during the entire model construction process, the synergy between the two models cannot be maximized.

[0057] like Figure 3 As shown, in the first stage of training, the visual model and the graph model are pre-trained separately to achieve the best effect of the two models. In the second stage of training, the two models are jointly trained. After the document image obtains information such as position and category through the visual model, it is sent to the pre-trained graph network for secondary training, and the common category judgment results of the models are combined to make the final trade-off, so that the overall model can enhance the resistance to interference from factors such as lighting changes.

[0058] In natural scenes, on the one hand, factors such as lighting and shadows will cause great interference to the visual model, causing the visual model to form incorrect candidate boxes on the document image, but such misjudged components are very easy to judge using positional relationships; on the other hand, document images in natural scenes are distorted and wrinkled, and the changes in the position information of the component candidate boxes will be more obvious and frequent compared to the previous neat PDF files, which will increase the interference to the graph model. The previous model of simply combining the visual model and the graph model will not be able to combine the advantages of the two models, resulting in difficulty in jointly leveraging the greatest advantages of both models on natural scene data sets. At the same time, in the layout analysis task, the scholars who conducted the research used traditional algorithms in the definition of node-edge relationships in graph neural networks, and did not analyze the rich information obtained from the visual model much, but only integrated it into the node information.

[0059] After referring to human reading habits, we found that it is easy to judge the connection between components through their semantic information, visual features, and positional features during the reading process. For example, the text styles of the title and the text are obviously different visually, and people can quickly distinguish the categories of the two; similarly, semantic features and positional features are also important auxiliary judgment factors. The edge weight generator (ERG) is proposed. Through pre-training, ERG not only relies on traditional KNN, β-skeleton and other traditional relational algorithms to generate graph structures with the help of position information, but also uses image information (Image Embedding) and text information (Content Embedding) to jointly construct the adjacency matrix. In the process of making the layout analysis data set, the graph structure is pre-generated through the KNN and β-skeleton relational algorithms, and then the edge weight data set is constructed based on the existing annotation relationship using the algorithm to complete the pre-training of the ERG module.

[0060] The integration of visual model and graph model is shown in the figure Figure 4As shown in the figure, after the visual model under the large kernel neural network is used to convolve (Conv) the input image to generate visual features, the component candidate box (ComponentCandidates) is generated through the detection head (Detect), and then the bounding box image features and edit box position information obtained by RoIAlign are encoded as position information encoding (Box Info Embedding) and image feature encoding (Image Embedding) respectively. Then, the OCR text recognition module is used to recognize the text in the area, and word2vec is used to encode the corresponding text of each recognition box into the distribution space. Then, a convolution layer with a convolution kernel size of 7×1×d and a step size of 1 is used to obtain text features (Content Embedding) from the text sequence features. The above three features are respectively input into the node information and the ERG module for node feature fusion and edge weight information generation. After the graph structure is constructed, it is input into the graph neural network (GNN) for training, and the global position relationship is integrated into the node information. Finally, the node category information is obtained through the fully connected layer (FC), and the recognition result of the previous visual model is combined with its confidence to output the comprehensive result.

[0061] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for background document layout analysis based on visual model and graph model, characterized by: The method steps are: S1, collect natural scene document images and complete the dataset construction; S2. Use a large kernel convolution-based visual network to obtain preliminary layout analysis results; S3. Use graph neural network for correction.

2. The method for background document layout analysis based on visual model and graphic model as claimed in claim 1, characterized in that: In step S1, crawler technology is used to crawl document images, and then manual filtering and labeling are performed to complete the construction of the natural scene data set.

3. The method for background document layout analysis based on visual model and graphic model as claimed in claim 1, characterized in that: The document types include scientific articles, books, announcements, government documents and examination papers.

4. The method for background document layout analysis based on visual model and graphic model as claimed in claim 1, characterized in that: Document annotations include text, title, picture, picture title, table, table title, header, footer, comments, formula, stamp, and QR code.

5. The method for background document layout analysis based on visual model and graphic model as claimed in claim 1, characterized in that: In step S2, a large convolution kernel layer is used to establish a sufficient effective receptive field, thereby saving computational budget for other effective structures.

6. The method for background document layout analysis based on visual model and graphic model as claimed in claim 1, characterized in that: In step S2, the visual network introduces the occlusion attention mechanism SEAM into the visual model to enable it to effectively handle occlusion scenes.

7. The method for background document layout analysis based on visual model and graphic model as claimed in claim 1, characterized in that: The visual model introduces a separation and enhancement attention module to enhance the responsiveness of the occluded components after the output of the Neck layer, thereby improving the accuracy of model recognition.

8. The method for background document layout analysis based on visual model and graphic model as claimed in claim 1, characterized in that: Step S3 uses the graph neural network's ability to analyze and extract the spatial location information of document components to correct the visual network.

Images