A document image layout analysis method based on edge information fusion of deformable attention mechanism

Abstract

The present invention discloses a document image layout analysis method based on edge information fusion of a deformable attention mechanism, which aims to improve the accuracy of model layout analysis by fusing document image edge information. First, in the document image preprocessing stage: the method of the present invention proposes a boundary information extraction module containing three different edge information extraction operators to fully extract the edge information of the document image. In order to more effectively utilize the key edge information of the document image, the method of the present invention proposes an edge information fusion module based on a deformable attention mechanism, which can adaptively adjust the receptive field and attention weight of its attention mechanism according to the document image edge information to adapt to different document elements, enhance the model's perception of the document image edge information, and make the model pay more attention to the edge features in the document image, thereby further improving the accuracy of the model layout analysis during document image layout analysis.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and document image recognition, and in particular relates to a layout analysis method based on document image target detection and instance segmentation. Background Art

[0002] With the advent of the digital age, more and more traditional paper documents are being replaced by electronic documents, making document storage, transmission, and processing more convenient. How to quickly and accurately identify, classify, and locate document content has become a major technical challenge. The rise of document image layout analysis technology has brought new opportunities for document digitization. This technology helps to more accurately convert image documents into digital documents and improve work efficiency. Document image layout analysis has become an important task in document intelligent processing, such as document retrieval, content classification, and text recognition. This is because any task related to document understanding requires a structured representation to help locate key information stored in the document.

[0003] Initially, classic convolution-based algorithms made significant progress in document layout analysis, such as Faster RCNN for document object detection, Mask RCNN for instance segmentation, and other specialized architectures. These architectures performed well in detecting some specific types of objects, such as table detection in scientific document images, but they lacked the ability to generalize to other similar tasks.

[0004] Transformers have recently demonstrated superior performance by leveraging positional embeddings and attention mechanisms. Models such as DocFormer and LayoutLmv3 have outperformed CNNs in intelligent document processing. However, among Transformer-based models, the best detection and segmentation models remain heterogeneous, hindering task and data collaboration between object detection and instance segmentation tasks. This hinders mutual guidance between the detection and segmentation modules, impacting their performance. MaksDINO addresses this issue, but most existing models and methods focus on extracting features directly from the raw color channels of document images without considering the distinct edge information characteristics of various document categories. For example, text and tables exhibit significant differences in edge information characteristics. Text and graphics also exhibit distinct edge information characteristics: text regions have denser edge textures, graphics have relatively smoother edges, and tables have more straight lines. Specifically, edge information typically combines the classification properties of image regions with the boundary properties of identified object regions. Therefore, leveraging edge features in document images can provide skeletal information that helps models better understand document structure. Summary of the Invention

[0005] In order to solve the above problems and make up for the shortcomings of the existing technology, the present invention proposes an edge information fusion method based on a deformable attention mechanism to implement an image document layout analysis method. It mainly includes the following two aspects: 1. Introducing an edge information extraction module, which contains three different edge information extraction operators to fully extract the edge information of the document image and emphasize the contours and boundary details of each element of the document image; 2. In order to more effectively utilize the key edge information of the document image and enhance the model's perception of the edge information of the document image, the present invention proposes an edge information fusion module based on a deformable attention mechanism, which deeply fuses the edge information features and the original image features. When calculating the deformable attention, the model can more effectively focus on the important edge information areas in the document image by dynamically adjusting the receptive field and sampling weights, making the structural features of the document layout more prominent, and ultimately making the prediction results of the document image layout analysis output by the model more accurate.

[0006] The present invention proposes a document image layout analysis method based on edge information fusion of a deformable attention mechanism, comprising the following steps:

[0007] Step (1): Construct an edge information extraction module, input the image to be detected as the original image into the edge information extraction module, and use three different edge extraction operators to obtain a three-channel edge information map; the original image, that is, the image of the original color channel and the edge information map are preprocessed in the same way and then input into the Swin Transformer backbone network respectively, and the sliding window mechanism is used to extract image features to obtain multi-scale feature maps Image Feat ∈[res2,res3,res4,res5] and multi-scale image edge information map Image_edge Feat ∈[res2 e ,res3 e ,res4 e ,res5 e ]; Among them, res2~res5 represent 4 picture feature maps of different scales, res2 e ~res5 e Represents 4 image edge information feature maps of different scales;

[0008] Step (2): Multi-scale feature map Image obtained in step (1) Feat and multi-scale image edge information graph Image_edge Feat , perform channel number processing, downsampling and flattening operations on multi-scale feature Image Feat Perform position encoding; flatten the image feature map Image Feat, edge information feature map Image_edge Feat They are input into the Transformer encoder respectively with the position code Pos to obtain the enhanced image feature sequence memory and edge information feature sequence memory edge ;

[0009] Step (3): Re-segment the enhanced image feature sequence memory according to the size of each feature map before the original input Transformer encoder to obtain the feature maps of each scale after the encoder Image_encoder Feat ∈[res2′,res3′,res4′,res5′,res6′]; At the same time, res2 and res2′ are fused to obtain the pixel embedding map mask_features for instance segmentation;

[0010] Step (4): Encode the multi-scale feature map Image_encoder obtained in step (3) Feat Re-flatten the image; and together with the edge information feature sequence memory obtained in step (2) edge The unified query screening model is input together. The unified query screening model includes the proposal generation process and three prediction heads: classification head, detection head and segmentation head. Then, the top K features are selected according to the classification scores generated by the classification head and used as content query and edge information query respectively. edge , use the generated prediction mask to further initialize the reference point coordinates refpoint_embed;

[0011] Step 5: The image content query Query and edge information query Query obtained in step f4) are edge , reference point coordinates refpoint_embed, input is a Transformer decoder composed of 9 layers of decoder layers; each decoder layer includes an edge information fusion module based on a deformable attention mechanism, and the edge information fusion module based on a deformable attention mechanism includes a linear layer Linear offset , Linear layer weig,t ; Use the edge information fusion module based on the deformable attention mechanism to query image content and edge information in the embedding dimension edge Perform fusion to obtain the fused image query Query Fusion ;Query the fused image Fusion Input linear layer Linear offset and Linear layer weig,t, respectively obtain the sampling offset after paying attention to the edge information and the attention weight matrix after paying attention to the edge information; according to the multi-scale image feature sequence memory obtained in step (2), perform weighted feature value sampling and addition, and output the feature sequence array hs and the reference point coordinate reference;

[0012] Step 6: Input the feature sequence array hs and reference point coordinates reference generated by the Transformer decoder in step (5), as well as the pixel embedding map mask_features for instance segmentation obtained in step (3), into the unified classification prediction head and instance segmentation head to obtain the category prediction result and the preliminary mask prediction result;

[0013] Step 7: The feature sequence array hs and the reference point coordinate reference generated by the Transformer decoder in step (5) are sent to the bounding box prediction iteration module to perform bounding box iteration operations based on each layer of feature sequence to obtain the final bounding box prediction result for the target detection task; the category prediction results and mask prediction results obtained in step (6) are concatenated to obtain the final overall prediction result for the document image layout analysis.

[0014] Preferably, in step (1), the three-channel edge information map is obtained by using three different edge extraction operators, specifically including: using the following three edge extraction operators, Canny operator, Sobal operator and Laplacian operator, to locate the object boundary in the image, obtaining three single-channel image edge information grayscale maps, and then performing black and white conversion on the pixel values ​​thereof to keep them consistent with the pixel values ​​of the original image; finally, merging them in the channel dimension to obtain a three-channel edge information map of the same size as the original image.

[0015] Preferably, in step (1), the preprocessing is specifically pixel value standardization, defining a mean and a standard deviation, subtracting the mean from the pixel value, and dividing the result by the standard deviation to obtain a standardized pixel value; the sliding window mechanism includes a non-overlapping local window and an overlapping cross-corss window.

[0016] Preferably, in step (2), performing position coding specifically includes performing position coding according to the following formula:

[0017]

[0018] Where pos represents the pixel position, d model represents the dimension of position embedding, and PE(·) represents the position encoding calculation formula.

[0019] Preferably, in step (2), the downsampling operation specifically includes: converting the multi-scale feature map Image Feat The res5 feature map with the smallest scale is downsampled by an additional convolutional layer with a kernel size of 3x3 and a stride of 2, and a 1-pixel padding is added to keep the size consistent to obtain the feature map res6, which is added to the multi-scale feature map Image in step (1). Feat middle:

[0020] Imag Feat ∈[res2,res3,res4,res5,res6]

[0021] Multi-scale image edge information feature map Image_edge Feat After and multi-scale feature map Image Feat After the same operation, the downsampled feature map res6 is obtained e , then add the downsampled feature map res6 based on the multi-scale image edge information feature map obtained in step (1) e , forming a complete multi-scale image edge information feature Image_edge Feat :

[0022] Image_edge Feat ∈[res2 e ,res3 e ,res4 e ,res5 e ,res6 e ].

[0023] Preferably, in step (2), the flattening operation specifically includes:

[0024] The multi-scale feature map Image Feat and multi-scale image edge information features Image_edge Feat Expand into a feature sequence containing all feature information, and obtain image feature sequence tokens respectively Feat , edge information feature sequence token_edge Feat

[0025]

[0026] Where lvl represents the number of feature layers, H and W represent the height and width of the feature map respectively, and D represents the feature dimension.

[0027] Preferably, in step (2), the enhanced image feature sequence memory and edge information feature sequence memory are edge , specifically including:

[0028] The Transformer encoder consists of 6 encoder layers based on deformable attention. Each layer includes a multi-scale deformable attention mechanism, a feed-forward network, a dropout layer, and a layer normalization LayerNorm operation. Finally, the enhanced image feature sequence memory and edge information feature sequence memory are obtained respectively. edge :

[0029]

[0030] Where lvl represents the number of feature layers, H and W represent the height and width of the feature map respectively, and D represents the feature dimension.

[0031] Preferably, in step (5), the multi-scale image feature sequence memory obtained in step (2) is used to perform weighted feature value sampling and addition, and output a feature sequence array hs and a reference point coordinate reference, including: performing a deformable attention mechanism calculation that integrates edge information according to the multi-scale image feature sequence memory, and then extracting the feature value of the corresponding position from the enhanced image feature sequence memory for each sampling point and multiplying it by its corresponding weight in the attention weight attention_weight, and then summing the results of all sampling points; outputting a feature sequence Query that has been fused with edge information final ; Query based on feature sequence final And the reference point of the previous decoder layer output, get the reference point of the current decoder layer, after all the decoder layers are processed, the feature sequence Query of each layer output final The result of the reference point coordinate ref_point is combined into an array to obtain the feature sequence array hs and the reference point coordinate reference in array form.

[0032] Beneficial effects of the method of the present invention:

[0033] The present invention proposes a document image layout analysis method based on edge information fusion of a deformable attention mechanism. The method introduces edge information in the document image into the model by inputting the original image into a boundary information extraction module containing three different edge information extraction operators. At the same time, in terms of edge feature fusion, an edge information fusion module based on a deformable attention mechanism is proposed. After deeply fusing the image content query with the edge information query, the model can adaptively adjust the receptive field and attention weight of its attention mechanism according to the edge information of the document image to adapt to different document elements, more effectively capture the document structure details and improve the model's ability to capture key edge features. The present invention helps the model better understand the document structure and identify the category and boundary attributes of each object in the document by additionally introducing edge information in the document image and deeply fusing it with the original image information through a deformable attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of the steps of a document image layout analysis method based on edge information fusion of a deformable attention mechanism of the present invention;

[0035] Figure 2 This is a structural diagram of the Transformer decoder part in a document image layout analysis method based on edge information fusion of a deformable attention mechanism in the present invention;

[0036] Figure 3 This is a framework diagram of the edge information fusion module based on the deformable attention mechanism of the present invention; DETAILED DESCRIPTION

[0037] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, a document image layout analysis method based on an edge information fusion module with a deformable attention mechanism is proposed. The method proposes an image document layout analysis model called SwinDocEdgeSegment. The SwinDocEdgeSegment model is an improvement on the SwinDocSegment model, which mainly includes an image feature extraction module, a Transformer-based encoder, a unified query screening model, a Transformer-based decoder, multiple prediction heads, and a denoising training strategy. After the original image undergoes image feature extraction through the backbone network, it is input into the Transformer encoder for feature enhancement. A unified query selection module is then used to obtain the query that needs to be input into the Transformer decoder. Finally, sub-task predictions are performed based on the multiple prediction heads to achieve document image layout analysis.

[0038] In the SwinDocEdgeSegment model proposed in this method, we added an edge information extraction module before the image feature extraction module to generate a three-channel document image edge information map from the original image. This method also improves the Transformer-based decoder by introducing a deformable attention module based on edge information fusion at each decoder layer, achieving a deep fusion of original image features and edge information features.

[0039] The method comprises the following steps:

[0040] Step (1): Document image edge information extraction: First, the original image is input into the edge information extraction module, and three different edge extraction operators are used to obtain a three-channel image edge information map. The original image and edge information map are then preprocessed in the same way and input into the same Swin Transformer backbone network to perform image feature extraction on the preprocessed image and edge information map, further obtaining two multi-scale feature maps.

[0041] The specific implementation process of step (1) is as follows:

[0042] First, the original image is input into the edge information extraction module for edge extraction. Considering that the edge information extracted by one operator cannot well represent the edge features of each element of the entire image, and in order to balance the number of channels of the input model. Before preprocessing and transforming the image, this method first uses three different edge extraction algorithms on the original document image: Canny operator, Sobal operator and Laplacian operator to locate sharp intensity changes and find the object boundaries in the image, and obtain three single-channel image edge information grayscale maps. Then, the pixel values ​​are converted to black and white to keep them consistent with the pixel values ​​of the original image. Finally, they are merged in the channel dimension to obtain a three-channel edge information map with the same size as the original image.

[0043] The original image and edge graph undergo the same preprocessing transformation to adapt to the model input format, facilitating subsequent batch processing and model training or inference. Image pixel value normalization involves performing pixel value normalization on each image in the input image set. The pixel values ​​in each image tensor are subtracted from the corresponding mean and divided by the standard deviation, where the mean and standard deviation are typically predefined vectors. This can accelerate model learning and improve convergence.

[0044] The pre-processed original image and edge information map are respectively input into the same Swin Transformer backbone network model for image feature extraction. The Swin Transformer backbone network feature extraction model uses a sliding window mechanism, including non-overlapping local windows and overlapping cross-corss windows. Finally, a series of multi-scale image feature tensors Image are obtained. Feat Image_edge and image edge information feature tensor Feat :

[0045] Image Feat ∈[res2,res3,res4,res5]

[0046] Image_edge Feat ∈[res2 e ,res3 e ,res4 e ,res5 e ]

[0047] Among them, res2~res5 represent the original image feature maps of 4 different scales, res2 e ~res5 e It represents 4 image edge information feature maps of different scales.

[0048] Step (2): Perform channel number processing, downsampling and flattening operations on the multi-scale feature map and the multi-scale image edge information map obtained in step (1), and perform position encoding on the input multi-scale feature map; the flattened image feature map Image Feat , edge information feature map Image_edge Feat They are input into the Transformer encoder respectively with the position code Pos to obtain the enhanced image feature sequence memory and edge information feature sequence memory edge ;

[0049] The specific implementation process of step (2) is as follows:

[0050] Step (2.1) first converts the multi-scale feature map Image obtained in step (1) Feat Input a convolution layer with a kernel size of 1x1 and a stride of 1 to map the number of channels of feature maps of different scales and project them to the same dimension of 256. Then perform group normalization GroupNorm and downsampling. The downsampling operation is as follows: FeatThe res5 feature map with the smallest scale is input into a convolution layer with a kernel size of 3x3 and a stride of 2 for downsampling (equivalent to halving the height and width), and 1 pixel of padding is added to keep the size consistent, and the res5 feature map after downsampling is obtained. The multi-scale feature map Image obtained in step (1) is Feat On the basis of the downsampled feature map res6, a complete multi-scale feature map Image is formed. Feat :

[0051] Image Feat ∈[res2,res3,res4,res5,res6]

[0052] Similarly, the multi-scale image edge information feature map Image_edge Feat After and multi-scale feature map Image Feat After the same operation, we finally get the downsampled feature map res6 e , then add the downsampled feature map res6 based on the multi-scale image edge information feature map obtained in step (1) e , forming a complete multi-scale image edge information feature Image_edge Feat :

[0053] Image_edge Feat ∈[res2 e ,res3 e ,res4 e ,res5 e ,res6 e ]

[0054] The multi-scale feature map Image Feat and multi-scale image edge information features Image_edge Feat Perform a flattening operation, that is, expand the multi-scale feature map into a feature sequence containing all feature information, and obtain the image feature sequence token respectively Feat , edge information feature sequence token_edge Feat

[0055]

[0056] Where lvl represents the number of feature layers, H and W represent the height and width of the feature map respectively, and D represents the feature dimension.

[0057] Step (2.2) simultaneously performs multi-scale feature map Image FeatEach size feature in the image is position-encoded using sine and cosine waves of different frequencies. The embedding vector for each position is composed of a series of sine and cosine values, the frequencies of which increase or decrease according to a certain rule. Finally, the position code Pos is output. The position code calculation formula is as follows:

[0058]

[0059]

[0060] Where pos represents the pixel position, d model represents the dimension of the position embedding, and PE(·) represents the position encoding calculation formula. This approach ensures that different positions have different encodings, and that the same position has different encodings in different dimensions. This not only addresses the model's sensitivity to sequence position, but also maintains computational efficiency and the model's generalization capability.

[0061] Step (2.3) finally flattens the image feature sequence token obtained in step (2.1) Feat , edge information feature sequence token_edge Feat The position code Pos obtained in step (2.2) is input into the Transformer-based encoder for feature enhancement. The Transformer encoder consists of 6 encoder layers based on deformable attention. Each layer includes a multi-scale deformable attention mechanism, a feed-forward network, a dropout layer, and a layer normalization LayerNorm operation. Finally, the enhanced image feature sequence memory and edge information feature sequence memory are obtained respectively. edge :

[0062]

[0063] Where lvl represents the number of feature layers, H and W represent the height and width of the feature map respectively, and D represents the feature dimension.

[0064] Step (3): Unflatten the image feature sequence memory after the Transformer encoder calculation in step (2), that is, re-segment it according to the size of each feature map before the original input Transformer encoder to obtain the feature maps of each scale after the encoder Image_encoder Feat ; At the same time, obtain the multi-scale feature map Image_encoder obtained in step (1) and this step FeatThe maximum scale feature maps res2 and res2′ in are fused to obtain the pixel embedding map mask_features for instance segmentation.

[0065] The specific implementation process of step (3) is as follows:

[0066] Step (3.1) first calculates the number of sequences contained in each scale based on the position index of each scale calculated before flattening, and generates a segmentation array. Then, based on this array, the sequence memory of all pixel features contained is segmented to obtain a multi-scale feature map Image_encoder of the same scale as before flattening. Feat , as shown below:

[0067] Image_encoder Feat ∈[res2′,res3′,res4′,res5′,res6′]

[0068] Step (3.2) simultaneously obtains res2 in step (1) and multi-scale image features Image_encoder in step (3.1) Feat The maximum scale feature map res2′ in the image is obtained; res2 is input into the convolution layer to process the number of channels, and the original channels are projected to a unified dimension of 256. After that, the two feature maps are unified to the same tensor scale and added together; finally, a convolution layer with a kernel size of 3x3 and a stride of 1 is input to obtain the final pixel embedding map mask_features for instance segmentation. This pixel embedding map is mainly used in the subsequent mask prediction head to generate a mask representation of the prediction result.

[0069]

[0070] Among them, N represents the number of batch images, H max ,W max They represent the height and width of the largest feature map in the multi-scale feature map.

[0071] Step (4): The multi-scale feature map Image_encoder obtained in step (3.1) Feat After re-flattening, together with the edge information feature sequence memory obtained in step (2.3) edge The query filtering model includes a proposal generation process and three prediction heads: classification head, detection head, and segmentation head. The top K features are selected based on the classification scores generated by the classification head and used as content query and edge information query respectively. edgeProvided to the Transformer decoder. At the same time, the generated prediction mask is used to further initialize the reference point coordinates refpoint_embed.

[0072] The specific implementation process of step (4) is as follows:

[0073] Step (4.1) converts the multi-scale feature map Image_encoder in step (3.1) Feat ∈[res2′,res3′,res4′,res5′,res6′] is flattened again into the same enhanced image feature sequence memory containing all pixels as in step (2.3).

[0074] Because this model method is a two-stage detection and segmentation method, it is necessary to first use the enhanced image feature sequence memory as a shared feature map for ROIProposal, which is responsible for generating a preliminary proposal for the target detection task for each feature map level and using the proposal for initializing the subsequent anchor box coordinate reference points. Furthermore, in the proposal generation method, a two-dimensional grid coordinate system is created, covering each pixel position of the current level feature map, and these coordinate axes are merged into a tensor of shape (H, W, 2), representing the (x, y) coordinates of each pixel, where H and W represent the height and width of the feature map. At the same time, a scaling factor is created to normalize the grid coordinates to the valid area. Its value is scaled based on the level index lvl to adapt to target detection of different scales. Then the normalized coordinates and size information are concatenated to form a tensor of shape (N, H*W, 4). Each four elements represent the (cx, cy, w, h) of a proposal, where (cx, cy) is the center coordinate and (w, h) is the width and height. Valid prediction proposals are screened based on whether each proposal value is between (0.01, 0.99). Finally, a logarithmic transformation is performed on the filtered proposals, which is usually used to convert probabilities or confidences into a form that is easier to optimize. For all proposals marked as False (i.e. invalid), they will be set to float(0) in the output feature sequence output_memory, and the initial proposal result output_proposals will be output.

[0075] Similarly, according to memory edge After the same operation as above, we can get output_memory edge .

[0076] Among them output_memory, output_memory edge The dimensions of the initial proposal results output_proposals are as follows:

[0077]

[0078]

[0079] Where lvl represents the number of feature layers, H and W represent the height and width of the feature map respectively, and D represents the feature dimension.

[0080] Step (4.2) The method of the present invention selects features that have passed through the Transformer encoder as input decoder content query Query and edge information query Query edge , thereby improving the performance of the model in the decoder stage. In order to obtain the top K content queries, edge information queries, and reference points of the initial screening, the output_memory is first input into a unified prediction classification head to obtain the score of each pixel in all classification categories. The classification score of each tag is regarded as the confidence of selecting the top-ranked features and providing them to the decoder as content queries. The top K predicted pixel position indices with the highest classification scores are selected. At the same time, the output_memory is input into the initial object detection detection head and combined with the output_proposals initial proposal results obtained in step (4.1) to further generate the initial reference point coordinates refpoint_embed.

[0081] According to the topK predicted pixel position indexes with the highest classification scores obtained above, the feature sequence output_memory and edge information feature sequence output_memory are respectively edge Collect pixel features at the specified location and reference point coordinates to obtain content query and edge information query edge And the reference point coordinates refpoint_embed, the dimensions are as follows:

[0082] Query, Query edge ∈R (N,topK,D)

[0083] refpoint_embed∈R (N,topK,4)

[0084] Where topK represents the number of selected features and D represents the feature dimension.

[0085] Step (4.3) is different from the traditional DETR-based target detection model method. In this model method, the content query Query obtained in step (4.2) and the pixel embedding map mask_features generated in step (3) are first input into the instance segmentation head. The specific implementation operation is as follows: First, the selected content query Query is input into a mask prediction head composed of three linear layers to generate an embedding vector mask_embed for mask prediction. The dot product operation is performed on each pixel of the mask embedding mask_embed and the pixel embedding map mask_features, and then all masks are summed to generate the initial mask prediction result output_mask. The predicted mask further initializes the reference point coordinate refpoint_embed. Specifically, the BitMasks class is used to convert the generated initial mask prediction output_mask result into the initial anchor box coordinate information. This method is slower but more accurate because it directly extracts the precise bounding box from the binary mask and only needs to compare the semantic similarity of each pixel. Therefore, this mask-enhanced anchor box initialization method can better initialize the reference point coordinates refpoint_embed that are subsequently input into the Transformer decoder. Through this effective task collaboration between object detection and instance segmentation, the enhanced box initialization can greatly improve the overall detection performance of the model.

[0086] Step (5): The image content query Query and edge information query Query obtained in step (4) are edge At the same time, the input is a Transformer decoder composed of 9 layers of decoder layers. Each decoder layer includes the edge information fusion module based on the deformable attention mechanism proposed by the method of the present invention, which performs deep fusion on the embedding dimension. The fused image query Query Fusion Input the corresponding linear layer and activation function to get the value of each Query Fusion The sampling offset and attention weight matrix after paying attention to the edge information. Finally, according to the multi-scale image feature sequence memory obtained in step (2.3), the feature value sampling and addition combined with the weight are performed to output the feature sequence array hs enhanced by edge information and the reference point coordinate array reference, as shown in the following example: Figure 2 and Figure 3 shown.

[0087] The specific implementation process of step (5) is as follows:

[0088] The top K image content query Query and edge information query Query filtered in step (4) are edgeThe reference point coordinates refpoint_embed are fed into a Transformer decoder consisting of 9 decoder layers. Each decoder layer includes a multi-head self-attention module, an edge information fusion module based on a deformable attention mechanism, a feed-forward network, and an activation function. The enhanced image feature sequence memory obtained in step (2) is also fed into the Transformer decoder.

[0089] In order to better fuse the original color image features with the edge information features to improve the model's ability to capture key edge features, the present invention introduces an edge information fusion module based on a deformable attention mechanism in each layer of the Transformer decoder.

[0090] The specific implementation process in each decoder layer is as follows: The top K image content queries filtered in step (4) are first input into a multi-head self-attention module to obtain an image content query containing more context information. self ;

[0091] Query the image content self The input is based on an edge information fusion module based on a deformable attention mechanism, and a periodic sinusoidal position embedding Pos is generated according to the input reference point coordinate refpoint_embed Query , and then query the content self and position embed Pos Query Add the eigenvalues ​​to get the image content query with location information pos ; Then compare it with the edge information query in step (4) edge , perform splicing on the embedding dimension, and query the image content pos The embedding dimension is changed from 256 to 512, and the fused image content query Query is obtained. Fusion ;

[0092] Then query the fused image content Fusion Input the linear layer Linear for generating sampling offset and attention weight matrix respectively offset and Linear weig,t , get information about each Query Fusion The sampling offset and attention weight matrix after paying attention to the edge information.

[0093] The specific implementation process of sampling offset and attention weight matrix generation is as follows: First, each Query FusionInput an offset calculation network as a linear layer Linear offset , calculate each Query Fusion The sampling offset sampling_offset in each attention head and each feature layer. Fusion Four feature points are sampled in each attention head and each feature layer. The sampling points are obtained as follows: the reference point coordinate refpoint_embed is added to the sampling offset sampling_offset to obtain the sampling point sampling_point;

[0094] Then each Query Fusion Input an attention weight calculation network as a linear layer Linear weig,t , calculate each Query Fusion Attention weights in each attention head and each feature layer attention_weight;

[0095] Construct a sampling offset matrix sampling_offsets based on each sampling offset sampling_offset; construct a deformable attention weight matrix attention_weights based on each attention weight attention_weight;

[0096] The dimensions of the sampling offset matrix sampling_offsets and the deformable attention weight matrix attention_weights are as follows:

[0097] sampling_offsets∈R (N,topK,n_heads,n_levels,n_points,2)

[0098] attention_weights∈R (N,topK,n_heads,n_levels,n_points)

[0099] Where topK represents the number of selected features, n_heads represents the number of multiple heads, n_levels represents the number of multi-scale feature layers, n_points represents the number of sampling points, and 2 represents the offset on the x and y axes.

[0100] Finally, according to the multi-scale image feature sequence memory obtained in step (2.3), the deformable attention mechanism that integrates the edge information is calculated, and then the Aggregate operation is performed: according to the coordinate position of each sampling point, the feature value of the corresponding position is extracted from the enhanced image feature sequence memory and multiplied by the corresponding weight in the attention weight attention_weight, and then the results of all sampling points are summed; the feature sequence Query after edge information fusion is output final .

[0101] Feature Sequence Query final The reference point of the current decoder layer is obtained by combining the reference point output by the previous decoder layer. The specific process is as follows;

[0102] The coordinates of the reference point are updated iteratively, that is, in each layer of the decoder, the output feature sequence Query final To predict the relative offset (△x,△y,△w,△h) and to update the reference point coordinates of the layer, (x',y',w',h')=(x,y,w,h)+(△x,△y,△w,△h), and then obtain a more accurate reference point coordinate ref_point.

[0103] After passing through all decoder layers, the feature sequence Query output of each layer final The results of the feature sequence array hs and the reference point coordinates ref_point are combined into an array and the feature sequence array hs and the reference point coordinate array reference are returned respectively, which are used as the input of the subsequent prediction head to predict each subtask in the image document layout analysis.

[0104] Step (6): The feature sequence array hs and reference point coordinate array reference generated by the Transformer decoder in step (5), as well as the pixel embedding map mask_features for instance segmentation obtained in step (3), are input into the unified classification prediction head and instance segmentation head to obtain the category prediction results and preliminary mask prediction results.

[0105] The specific implementation process of step (6) is as follows:

[0106] The feature sequence array hs and reference point coordinate array reference returned in step (5), as well as the pixel embedding map mask_features for instance segmentation obtained in step (3), are input into the unified classification prediction head and instance segmentation head to generate the predicted classification result prediction_class and the predicted mask result prediction_mask respectively; the specific operation of generating the predicted mask result is: performing a dot product between the mask embedding and each pixel feature map mask_featujres, and then summing all masks to generate the output result of each pixel position, which represents the response or contribution of each mask to the mask feature map.

[0107]

[0108] Among them, N represents the number of batch images, topK represents the number of queries, and H max ,W max They represent the height and width of the largest feature map in the multi-scale feature map.

[0109] Step (7): The feature sequence array hs and the reference point coordinate array reference generated by the Transformer decoder in step (5) are fed into the bounding box prediction iteration module to perform bounding box iteration based on each layer of feature sequence to obtain the final bounding box prediction result for the target detection task. Finally, the category prediction result and mask prediction result obtained in step (6) are concatenated to obtain the final overall prediction result for the document image layout analysis.

[0110] The specific implementation process of step (7) is as follows:

[0111] The feature sequence hs in each layer returned in step (5) and the reference point coordinate information reference result are input into the bounding box prediction iteration module, where the bounding box prediction iteration module contains 9 layers of MLP with the same weight, each MLP contains 3 linear layers, and the MLP is applied to process the feature sequence Query output by each layer decoder final , get the bounding box offset, add the calculated bounding box offset to the reference point coordinates of the previous layer, and get a series of consecutive predicted decoder layer bounding box coordinates. The generated bounding box coordinates are applied with the sigmoid() function to ensure that the bounding box coordinate value range is between (0,1).

[0112] This step is to iterate the feature sequence Query after the edge information of each layer output by the decoder is fused. final , based on the bounding box offset generated by the reference point coordinates of the previous layer and the feature sequence of the current layer, the bounding box coordinates are gradually adjusted and updated to make the bounding box coordinates of the target detection more accurate.

[0113] Finally, the category prediction result prediction_class, mask prediction result prediction_mask generated in step (6) and the bounding box prediction result prediction_boxes completed in this step are combined to obtain the final overall prediction result of the document image layout analysis task.

[0114] Experimental implementation details:

[0115] The document image layout analysis experiments using this method were implemented in Python 3.8 (Ubuntu 20.04) and Cuda 11.8. The hardware environment consisted of an RTX 4090 and a 15vCPU Intel(R) Xeon(R) Platinum 8474C. The Adam optimizer was used, with a BASE_LR of 0.0001, a batch size of 1, and a maximum number of iterations of 15,000. This experiment used the weights of a Swin-B model pre-trained on ImageNet-22K, a 224x224 dataset, as the basis for training. The topK query selection in the unified query filtering model was set to 300.

[0116] Dataset: This experiment randomly selects 2000 images from the PubLayNet dataset for training and 500 for testing. PubLayNet is a large-scale document image dataset whose layout is annotated using bounding boxes and polygon segmentation and is divided into five categories: text, title, list, table, and figure.

[0117] As can be seen from Table 1, compared to the original SwinDocSegmenter model, the edge information enhancement method in this invention achieves better overall AP performance and improved detection accuracy in both object detection and instance segmentation tasks. As can be seen from Table 2, the recognition performance of each category in the document is also more accurate in both object detection and instance segmentation tasks.

[0118] Table 1 Comparison of the overall AP performance of the proposed method and the original model in target detection and instance segmentation tasks

[0119]

[0120] Table 2 Comparison of the performance of the proposed method and the original model in each category on target detection and instance segmentation tasks

[0121]

[0122] Experimental results show that the document layout analysis method using edge information enhancement in the present invention can not only improve the overall accuracy of model detection, but also improve the recognition accuracy of the model for each category.

Claims

1. A document image layout analysis method based on edge information fusion of deformable attention mechanism, characterized by: The following steps are involved: Step (1): Construct an edge information extraction module, input the image to be detected as the original image into the edge information extraction module, and use three different edge extraction operators to obtain a three-channel edge information map; perform the same preprocessing on the original image and the edge information map, and input them into the Swin Transformer backbone network respectively, and use the sliding window mechanism to extract image features to obtain multi-scale feature maps Image Feat ∈[res2,res3,res4,res5] and multi-scale image edge information map Image_edge Feat ∈[res2 e , res3 e , res4 e ,res5 e ]; Step (2): Multi-scale feature map Image Feat and multi-scale image edge information graph Image_edge Feat , perform channel number processing, downsampling and flattening operations on multi-scale feature Image Feat Perform position encoding; flatten the image feature map Image Feat , edge information feature map Image_edge Feat They are input into the Transformer encoder respectively with the position code Pos to obtain the enhanced image feature sequence memory and edge information feature sequence memory edge ; Step (3): Re-segment the enhanced image feature sequence memory according to the size of each feature map before the original input Transformer encoder to obtain the feature maps of each scale after the encoder Image_encoder Feat ∈[res2′,res3′,res4′,res5′,res6′]; At the same time, res2 and res2′ are fused to obtain the pixel embedding map mask_features for instance segmentation; Step (4): Multi-scale feature map Image_encoder Feat Re-flatten the operation; together with the edge information feature sequence memory edge The unified query screening model is input together. The unified query screening model includes the proposal generation process and three prediction heads: classification head, detection head and segmentation head. Then, the top K features are selected according to the classification scores generated by the classification head and used as content query and edge information query respectively. edge , use the generated prediction mask to further initialize the reference point coordinates refpoint_embed; Step 5: The image content query Query and edge information query Query obtained in step (4) edge , reference point coordinates refpoint_embed, input is a Transformer decoder composed of 9 layers of decoder layers; each decoder layer includes an edge information fusion module based on a deformable attention mechanism, and the edge information fusion module based on a deformable attention mechanism includes a linear layer Linear offset , Linear layer weight ; Use the edge information fusion module based on the deformable attention mechanism to query image content and edge information in the embedding dimension edge Perform fusion to obtain the fused image query Query Fusion ;Query the fused image Fusion Input linear layer Linear offset and Linear layer weight , respectively obtain the sampling offset after paying attention to the edge information and the attention weight matrix after paying attention to the edge information; according to the multi-scale image feature sequence memory obtained in step (2), perform weighted feature value sampling and addition, and output the feature sequence array hs and the reference point coordinate reference; Step 6: Input the feature sequence array hs and reference point coordinates reference generated by the Transformer decoder in step (5), as well as the pixel embedding map mask_features for instance segmentation obtained in step (3), into the unified classification prediction head and instance segmentation head to obtain the category prediction result and the preliminary mask prediction result; Step 7: The feature sequence array hs and the reference point coordinate reference generated by the Transformer decoder in step (5) are sent to the bounding box prediction iteration module to perform bounding box iteration operations based on each layer of feature sequence to obtain the final bounding box prediction result for the target detection task; the category prediction results and mask prediction results obtained in step (6) are concatenated to obtain the final overall prediction result for the document image layout analysis.

2. The document image layout analysis method based on edge information fusion of a deformable attention mechanism according to claim 1, characterized in that: In step (1), the three-channel edge information map is obtained by using three different edge extraction operators, specifically including: using the following three edge extraction operators, Canny operator, Sobal operator and Laplacian operator, to locate the object boundary in the image, obtaining three single-channel image edge information grayscale maps, and then performing black and white conversion on the pixel values ​​to keep them consistent with the pixel values ​​of the original image; finally, merging them in the channel dimension to obtain a three-channel edge information map of the same size as the original image.

3. The document image layout analysis method based on edge information fusion of deformable attention mechanism as claimed in claim 1, characterized in that: In step (1), the preprocessing is specifically pixel value standardization, defining the mean and standard deviation, subtracting the mean from the pixel value, and dividing it by the standard deviation to obtain the standardized pixel value; the sliding window mechanism includes a non-overlapping local window and an overlapping cross-corss window.

4. The document image layout analysis method based on edge information fusion of a deformable attention mechanism according to claim 1, characterized in that: In step (2), the position encoding specifically includes performing position encoding according to the following formula: Where pos represents the pixel position, d model represents the dimension of position embedding, and PE(·) represents the position encoding calculation formula.

5. The document image layout analysis method based on edge information fusion of deformable attention mechanism as claimed in claim 1, characterized in that: In step (2), the downsampling operation specifically includes: converting the multi-scale feature map Image Feat The res5 feature map with the smallest scale is downsampled by an additional convolutional layer with a kernel size of 3x3 and a stride of 2, and a 1-pixel padding is added to keep the size consistent to obtain the feature map res6, which is added to the multi-scale feature map Image in step (1). Feat middle: Image Feat ∈[res2,res3,res4,res5,res6] Multi-scale image edge information feature map Image_edge Feat After and multi-scale feature map Image Feat After the same operation, the downsampled feature map res6 is obtained e , then add the downsampled feature map res6 based on the multi-scale image edge information feature map obtained in step (1) e , forming a complete multi-scale image edge information feature Image_edge Feat : Image_edge Feat ∈[res2 e ,res3 e ,res4 e ,res5 e ,res6 e ]。 6. The document image layout analysis method based on edge information fusion of a deformable attention mechanism as claimed in claim 1, characterized in that: In step (2), the flattening operation specifically includes: The multi-scale feature map Image Feat and multi-scale image edge information features Image_edge Feat Expand into a feature sequence containing all feature information, and obtain image feature sequence tokens respectively Feat , edge information feature sequence token_edge Feat Where lvl represents the number of feature layers, H and W represent the height and width of the feature map respectively, and D represents the feature dimension.

7. The document image layout analysis method based on edge information fusion of a deformable attention mechanism as claimed in claim 1, characterized in that: In step (2), the enhanced image feature sequence memory and edge information feature sequence memoryedge are specifically obtained, including: The Transformer encoder consists of 6 encoder layers based on deformable attention. Each layer includes a multi-scale deformable attention mechanism, a feed-forward network, a dropout layer, and a layer normalization LayerNorm operation. Finally, the enhanced image feature sequence memory and edge information feature sequence memoryedge are obtained respectively: Where lvl represents the number of feature layers, H and W represent the height and width of the feature map respectively, and D represents the feature dimension.

8. The document image layout analysis method based on edge information fusion of a deformable attention mechanism as claimed in claim 1, characterized in that: In step (5), the weighted feature value sampling and addition are performed based on the multi-scale image feature sequence memory obtained in step (2), and the feature sequence array hs and the reference point coordinate reference are output, including: performing a deformable attention mechanism calculation that integrates edge information based on the multi-scale image feature sequence memory, and then extracting the feature value of the corresponding position from the enhanced image feature sequence memory for each sampling point and multiplying it by its corresponding weight in the attention weight attention_weight, and then summing the results of all sampling points; outputting the feature sequence Query that has been fused with edge information final ; Query based on feature sequence final The reference point of the current decoder layer is obtained by combining the output of the previous decoder layer with the reference point. After processing by all decoder layers, the output feature sequence Queryfinal of each layer and the reference point coordinate ref_point are arrayed to obtain the feature sequence array hs and the reference point coordinate reference in array form.

Images