Newspaper image layout analysis method based on panoramic segmentation

Abstract

The invention discloses a newspaper image layout analysis method based on panoramic segmentation, which solves two classical problems in layout analysis, namely a high foreground pixel omission ratio and an adjacent region error merging problem, by constructing a general model Panoptic-DLA for solving a complex newspaper image layout segmentation and classification problem. The method is suitable for document layout analysis of historical newspapers, and the newspaper images are used as input and output to obtain various elements and positions thereof. The model framework is composed of a semantic segmentation branch and an instance segmentation branch. Semantic segmentation takes a background as stuff and a foreground region as tracking, and predicts a pixel category. An example segmentation branch adopts a proposal-free design, an area center-boundary probability graph and an area center directional diagram are used for replacing a traditional center point regression method, the shape and position information of an object is coded, and closely adjacent areas in newspaper are separated. In the post-processing, a panoramic segmentation image is generated through instance grouping, semantic category distribution and Regroup operation, and the accuracy of newspaper image layout analysis is improved.

Description

Technical Field

[0001] The present invention relates to the field of document image analysis and processing, and in particular to a newspaper image layout analysis method based on panoptic segmentation. Background Art

[0002] Document Layout Analysis (DLA) is a key step in document understanding systems, and its goal is to segment document images into different regions and classify them. Existing layout analysis methods are mainly divided into segmentation-based methods and regression-based methods. Segmentation-based methods classify pixels through semantic segmentation models (such as DeepLab), but have difficulty processing closely adjacent regions; regression-based methods (such as Faster R-CNN and Mask R-CNN) segment regions by predicting bounding boxes, but they rely heavily on the prediction accuracy of bounding boxes and are prone to overlap or missed detection problems. In addition, most existing methods ignore the shape and position information of objects in document pages, resulting in limited performance under complex layouts and facing many challenges in the layout analysis of historical document images, such as high missed detection rates of foreground pixels and mis-merging of adjacent regions. Summary of the Invention

[0003] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a newspaper image layout analysis method based on panoptic segmentation, which is not limited by bounding box prediction and can well segment adjacent areas for layout analysis of historical newspaper documents.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A newspaper image layout analysis method based on panoptic segmentation includes the following steps:

[0006] Step 1: Input a newspaper image and use the encoder backbone network to extract features from the newspaper image to provide deep image features for the semantic segmentation branch and the instance segmentation branch.

[0007] Step 2: For each branch, a decoupled atrous spatial pyramid pooling module (ASPP) and an upsampling decoding module are used to generate a multi-scale feature map and a high-resolution feature map; wherein the atrous spatial pyramid pooling module performs multi-scale feature extraction on the output feature map of the backbone network; the upsampling decoding module outputs a feature map with a lower resolution than the original image through upsampling, and uses a skip connection to fuse the high-resolution low-level feature map in the backbone network with the upsampled high-level feature map to obtain more refined features;

[0008] Step 3. In the prediction stage, the semantic segmentation branch outputs a semantic probability map to predict the semantic category of each pixel, including the background and various foreground areas. The instance segmentation branch generates a Region Center-Border Probability Map (RCBP) to identify the boundary of each layout area, and a Region Center Direction Map (RCD) to identify the direction of each layout area, and assigns an instance ID to each foreground pixel. The instance segmentation prediction head only predicts class-independent instances, whose semantic information is provided by the semantic segmentation head.

[0009] Step 4 combines the probability map and the region center direction map obtained in step 3 to generate a panoramic segmentation map of the input image. This includes: grouping instances based on the RCBP map or RCD map to determine each page area; combining the semantic probability map to determine the semantic category of each page area; and regrouping ungrouped foreground pixels to generate a panoramic segmentation map of the input newspaper image.

[0010] In one embodiment, the dilated spatial pyramid pooling module is used to extract and fuse multi-scale information for segmentation, with convolution ratios of 6, 12, and 18 respectively;

[0011] Based on the decoder of DeepLab V3+, the upsampling decoding module introduces the 1 / 8 resolution low-level feature map extracted from the middle layer of the backbone network into the upsampling process of the decoder, and fuses it with the high-level feature map upsampled to 1 / 8 resolution output by the void spatial pyramid pooling module, and preferably uses 5×5 depthwise separable convolution instead of 3×3 convolution kernel.

[0012] In one embodiment, the semantic segmentation branch adopts the typical semantic segmentation model DeepLabV3+, and is trained using a weighted bootstrap cross entropy loss function to predict the semantic category of each pixel; the semantic probability map includes prediction results of background and foreground areas, wherein the foreground area includes a dividing line, a text area, and a chart area; the dividing line includes a horizontal frame, a vertical frame, a horizontal dividing line, and a vertical dividing line; the text area includes a title area and a paragraph text area; the chart area includes a picture area and a table area.

[0013] In one embodiment, the region center-edge probability map is a mountain-shaped three-dimensional score map, where the region center-edge probability value of each pixel is between 0 and 1, representing the height of the point in the mountain region; the pixel point at the center of the region is the peak point and is set to a high value, while the pixel point at the region edge is the foot point and is set to a low value;

[0014] The shape of the foreground area is defined as the minimum bounding rectangle of the instance mask area. The score of each point is the ratio of the shortest distance to the height of the rectangle. The region center-boundary probability value of each pixel is determined by calculating the ratio of the shortest distance from each pixel to the four sides of the rectangle to the height of the rectangle. The higher the value, the closer the pixel is to the region center, and the lower the value, the closer the pixel is to the region boundary.

[0015] In one embodiment, for a pixel point x in the foreground area, the region center-edge probability value formula is as follows:

[0016]

[0017] Among them, a i is the perpendicular vector from the ith edge to the pixel x, ‖a i ‖ is the length of the vector, i=1,2,3,4, h x is the height of the minimum circumscribed rectangle, which is obtained by calculating the minimum distance from the pixel point x to the two opposite sides, that is, min(‖a1‖,‖a2‖,‖a3‖,‖a4‖); the closer the region center-boundary probability value is to 1, the closer the pixel is to the region center, and the closer it is to 0, the closer the pixel is to the region boundary.

[0018] In one embodiment, the region center direction map represents the thrust direction of each pixel in the foreground region, the thrust direction points from the edge of the foreground region to the center of the region, and the thrust value is proportional to the distance from the pixel to the edge of the region, and the closer the distance, the greater the thrust value.

[0019] In one embodiment, the direction vector formula of the region center direction pattern is as follows:

[0020]

[0021] where h x is the height of the minimum bounding rectangle, express Each side of the minimum circumscribed rectangle is considered to have a thrust effect, pushing point x towards the center, and the thrust direction is from the edge to the point; the closer to the edge, the greater the thrust; if the distance exceeds half the height of the rectangle, the thrust disappears; the direction map of the center of the region uses the unit vector u x , expressed as Region separation is achieved by assigning completely different region center direction values to pixels on the intersection edge of two adjacent regions.

[0022] In one embodiment, the instance grouping is implemented as follows:

[0023] First, the pixels in the region center direction map that are larger than the set threshold are regarded as peaks, and each peak is assigned an independent class ID;

[0024] Then, each pixel at the foot of the mountain is assigned to the corresponding peak by the probability growth direction in the region center-boundary probability map or the vector direction in the region center direction map. For the region center-boundary probability map, the maximum value in the 8-neighborhood of each pixel is selected as the next point. For the region center direction map, a vector with an angle less than 3 / 8π is selected as the next vector. The next point value of each point is recursively assigned to all points until their values no longer change, thereby effectively solving the problem in parallel.

[0025] In one embodiment, the Regroup operation specifically includes: for the foreground pixels that are not grouped in the instance grouping in step 4, further completing the labeling by recursively assigning the instance IDs of neighboring pixels with the same semantic category in the 8-neighborhood to the pixel.

[0026] Compared to existing technologies, the present invention is a proposal-free panoptic segmentation-based document layout analysis method (Panoptic-DLA). It can segment foreground and background pixels, then label all pixels in the foreground region with their respective semantic categories and instance identification numbers, completing region shape modeling. This method combines semantic segmentation with instance segmentation to generate semantic labels and instance identification numbers for each pixel in the document image, without being restricted by region bounding box prediction.

[0027] To address the problem of mismerging adjacent regions, the instance segmentation branch of this invention models the region shape and simultaneously predicts the vectors pointing from edge pixels to the region center, helping edge pixels better locate the region center. This invention can solve the problem of analyzing the layout of complex newspaper images and demonstrates excellent functionality and advantages in analyzing the layout of historical document images. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the Panoptic-DLA model architecture of the present invention.

[0029] Figure 2 These are the input image (left image) and output result (right image) of an embodiment of the present invention. The specific content of the Mongolian text in the image is irrelevant to the present invention and has been blurred.

[0030] Figure 3 It is the vertical distance from the four edges to the point in the segmented page area of the example of the present invention.

[0031] Figure 4 It is a segmentation result diagram of the semantic segmentation method of the present invention.

[0032] Figure 5It is the example grouping algorithm of the present invention.

[0033] Figure 6 It is the recall rate of the present invention's analysis of foreign historical newspaper layouts. DETAILED DESCRIPTION

[0034] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0035] like Figure 1 As shown, the present invention is a newspaper image layout analysis method based on panoptic segmentation, the main steps are as follows:

[0036] Step 1: Input a newspaper image and use the encoder backbone network to extract features of the newspaper image to provide deep image features for the semantic segmentation and instance segmentation branches.

[0037] For example, the encoder backbone network of the present invention adopts the Xception network architecture improved based on DeepLabv3+, and the backbone network outputs a feature map shared by the semantic segmentation branch and the instance segmentation branch of the architecture. In the existing Xception network architecture, the data first passes through the inlet stream, which contains multiple convolutional layers and separable convolutional layers, then passes through the intermediate stream, which is repeated 16 times, and finally passes through the outlet stream. All convolutional and separable layers are followed by batch normalization. Since the feature map is generated using standard convolution, this reduces the resolution of the output map to 1 / 32 of the resolution of the input image, which is too coarse for accurate image segmentation. In order to retain detailed information on the object boundary, the present invention replaces the original standard convolution with a hole convolution to generate a feature map with a resolution of 1 / 16 of the original resolution, significantly improving the ability to retain detailed features such as edges.

[0038] In addition, during training, this step can perform data enhancement operations on newspaper images, including rotation, horizontal flipping, and random segmentation, to improve data diversity.

[0039] In step 2, a decoupled atrous spatial pyramid pooling module (ASPP) and an upsampling decoding module are used for each branch to generate multi-scale feature maps and high-resolution feature maps, respectively.

[0040] The ASPP module extracts multi-scale features based on the output feature map of the backbone network and generates multiple feature maps with different receptive fields. The upsampling decoding module outputs feature maps with lower resolution than the original image through upsampling, and uses skip connections to fuse the high-resolution low-level feature maps in the backbone network with the upsampled high-level feature maps to obtain more refined features.

[0041] For example, the ASPP module of the present invention is used to extract and fuse multi-scale information for segmentation. The convolution ratios are 6, 12, and 18, respectively. Dilated convolution can capture feature maps with a larger field of view, which is advantageous for newspaper pages with larger areas.

[0042] For example, the upsampling decoding module of the present invention follows the design of Panoptic-DeepLab and makes two modifications based on the decoder of DeepLab V3+. First, an additional low-level feature map (1 / 8 scale) is reintroduced into the decoder, and the 1 / 8 resolution low-level feature map extracted by the middle layer of the backbone network is introduced into the upsampling process of the decoder, and fused with the high-level feature map output by the dilated spatial pyramid pooling module upsampled to 1 / 8 resolution, which helps to preserve the spatial information in the original image (for example, object boundaries). Second, instead of using the typical 3×3 convolution kernel, the decoder uses a 5×5 depthwise deparable conv, which can reduce parameters and speed up the operation, thereby producing better performance at minimal cost.

[0043] In step 3, during the prediction phase, the semantic segmentation branch predicts the semantic category of each pixel, including background (stuff) and various foreground regions (things), and outputs a semantic probability map. The instance segmentation branch generates a Region Center-Border Probability Map (RCBP) and a Region Center Direction Map (RCD), which are used to identify the boundaries and directions of each layout region, respectively, and assigns an instance ID to each foreground pixel.

[0044] Specifically, in the semantic segmentation branch, the H×W×128 feature map is convolved through a 5×5 convolution to obtain a H×W×32 feature map, which is then convolved through a 1×1 convolution to obtain a H×W×1 feature map. Each point is predicted to have an integer value between 0 and 10, representing the semantic category of the point. In the instance segmentation branch, similarly, 5×5 and 1×1 convolutions are used to output feature maps of H×W×1 and H×W×2, respectively.

[0045] For example, the semantic segmentation branch of the present invention uses the typical semantic segmentation model DeepLabV3+ to integrate the encoded position and shape information into the data, and uses the weighted bootstrap cross entropy loss function for training to predict the semantic category of each pixel. The resulting semantic probability map contains the prediction results of the background and foreground areas, where the foreground area includes dividing lines such as horizontal frames, vertical frames, horizontal dividing lines, and vertical dividing lines, text areas such as title areas and paragraph text areas, and chart areas such as picture areas and table areas.

[0046] Using the method of the present invention, Figure 2 The newspaper page on the left is the input image, and its output is as follows Figure 2 As shown in the image on the right, it can be seen that the newspaper layout area is divided into title (title), text (text), horizontal dividing line (hline), vertical dividing line (vline), horizontal border line (hframe), vertical border line (vframe), table (table), picture (pic), bgr (background), etc.

[0047] The comparative experiments of semantic segmentation methods are shown in Table 1:

[0048] Table 1 Comparative experiments of semantic segmentation methods

[0049]

[0050] Table 1 shows the performance of different post-processing methods for the semantic segmentation model DeepLab V3+, the panoramic segmentation method Panoptic_DeepLab, and Panoptic-DLA on the MMT-LS, ENP, and HJ Datasets. This includes the average segmentation precision (P), average recall (R), and the geometric mean (F) of P and R. The performance is also compared. Panoptic-DLA (a) uses only RCBP values for grouping in the instance segmentation post-processing stage; Panoptic-DLA (b) uses RCBP and RCD for grouping in the instance segmentation post-processing stage; and Panoptic-DLA (c) uses RCBP in the instance segmentation post-processing stage and performs a reassembly operation on unsegmented pixels.

[0051] On the MMT-LS dataset, only the RCBP post-processing method achieved the best performance, and regional reorganization brought a 0.92% improvement on the ENP dataset. Panoptic-DLA(c) achieved the highest F-value on both the ENP and HJ datasets.

[0052] Specifically, the RCBP map is a three-dimensional score map of the mountain shape, and the RCBP value of each pixel is a probability value between 0 and 1, representing the height of the pixel in the mountain area. The pixel point at the center of the area is the peak point, which is set to a higher value, while the pixel point at the boundary of the area is the foot point, which is set to a lower value. The shape of the foreground area is defined as the minimum circumscribed rectangle of the instance mask area. The score of each point is calculated as the ratio of the shortest distance to the height of the rectangle. The RCBP value of each pixel is determined by calculating the distance relationship between each pixel and the center and boundary of the area (that is, the ratio of the shortest distance to the four sides of the minimum circumscribed rectangle to the height of the rectangle). The higher the value, the closer the pixel is to the center of the area, and the lower the value, the closer the pixel is to the boundary of the area.

[0053] For a pixel point x in the foreground area, the calculation formula for its RCBP value is as follows:

[0054]

[0055] refer to Figure 3 , in the figure, a i is the vertical vector from the i-th edge to the pixel x, i = 1, 2, 3, 4, ‖a i ‖ is the length of the vector, h x is the height of the rectangle, which can be obtained by calculating the minimum distance from the point to the two opposite sides, that is, min(‖a1‖,‖a2‖,‖a3‖,‖a4‖). The closer the RCBP value is to 1, the closer the pixel is to the center of the region, and the closer it is to 0, the closer the pixel is to the region boundary.

[0056] The instance segmentation prediction head only predicts the class, not the instance. Semantic information is provided by the semantic segmentation head. Specifically, the RCD map represents the thrust direction of each pixel in the foreground region. The thrust direction points from the region edge to the region center, and the thrust value is proportional to the distance from the pixel to the region edge. The closer the distance, the greater the thrust value.

[0057] The direction vector formula of RCD is as follows:

[0058]

[0059] where h x is the height of the minimum bounding rectangle, express Each side of the minimum circumscribed rectangle is considered to have a thrust that pushes point x towards the center, and the thrust direction is from the edge to the point. The closer to the edge, the greater the thrust. If the distance exceeds half the height of the rectangle, the thrust disappears. RCD uses the unit vector u x , expressed as It helps to separate regions by assigning completely different RCD values to pixels on the intersection edge of two adjacent regions.

[0060] like Figure 4 As shown in the figure, a comparison of the segmentation results of semantic segmentation methods of different models in the layout segmentation task is shown, including Ground Truth, Panoptic-DeepLab, DeepLab V3+ and Panoptic-DLA, highlighting the significant effect of the instance segmentation branch in dealing with the classification problem of adjacent regions. DeepLab V3+ is a pure semantic segmentation method, and instance segmentation is completed by connected domain analysis in the post-processing stage. It performs well in predicting non-adjacent or distant areas, but has certain limitations in dealing with adjacent areas with the same semantic labels. The merging of the same semantic regions is tolerable without affecting the reading order, for example Figure 4 The text area after the first line of the picture title and the text area after the third line of the picture are merged. Since Mongolian newspapers are read from left to right and there are no dividing lines between the areas, this does not affect the understanding of the layout. However, the mistaken merging of the upper and lower areas will bring serious consequences. Figure 4 The second row of DeepLab V3+ predictions shows that it incorrectly merges the upper and lower regions and the left and right regions, combining the three regions into a single large region. This directly undermines the readability of the newspaper content and disrupts the logical relationship between the pages. In the ENP dataset, due to the separator between the left and right regions, merging errors often occur between the upper and lower paragraphs. Japanese historical newspaper pages are also prone to mismerging in the context region. By introducing the instance segmentation module, these merging errors are significantly reduced.

[0061] Step 5: Group instances according to the RCBP or RCD map. First, determine each instance area (page area), and then determine the semantic category of each instance in combination with the semantic probability map; perform a Regroup operation on the ungrouped foreground pixels, and finally generate a panoramic segmentation map of the input newspaper image.

[0062] Specifically, the post-processing algorithm combines the probability maps output by the instance segmentation prediction head and the semantic segmentation prediction head to generate a panoptic segmentation map of the input image. The post-processing algorithm is completed in two steps. The first step is to group instances according to the RCBP or RCD map and determine the region (layout area) of each instance. The second step is to determine the semantic category of each instance by combining the semantic probability map.

[0063] refer to Figure 5As shown, the example grouping process of the present invention includes the following steps: First, peaks are generated through RCBP>γ, that is, the pixels in the region center direction map that are greater than the set threshold are regarded as peaks, and each peak is assigned an independent class ID. Then, each pixel at the foot of the mountain is assigned to the corresponding peak through the probability growth direction in RCBP or the vector direction in RCD. For RCBP, the maximum value in the 8-neighborhood of each pixel is selected as the next point; for RCD, the vector with an angle less than 3 / 8π is selected as the next vector. The direction of each point is determined, so this task can be effectively solved in parallel by recursively assigning the next point value of each point to all points until their values no longer change. Unlabeled pixels are regrouped according to their semantic categories and 8-neighborhood pixels.

[0064] After instance grouping is complete, the semantic labels of the predicted instance masks are inferred by majority voting on the corresponding predicted semantic labels, given the semantic segmentation results and the class-independent instance segmentation results. Foreground pixels that are not detected and classified in the instance grouping are further labeled by recursively assigning them the instance IDs of neighboring pixels with the same semantic category in the 8-neighborhood.

[0065] Table 2 shows the comparison results of the regression-based object detection models Faster R-CNN, Mask R-CNN, RetinaNet and Panoptic-DLA.

[0066] Table 2 Comparative experiments of regression methods

[0067]

[0068] The analysis results are as follows: Panoptic-DLA achieved the highest recall on all three datasets. On the ENP and HJ datasets, the state-of-the-art detectors, Faster R-CNN and Mask R-CNN, achieved comparable performance to Panoptic-DLA, with F-scores of 65.52% and 99.64%, respectively. However, due to region overlap and missed detection, which are key challenges in regression-based object detection models, the overall recall of these detectors was lower than that of Panoptic-DLA. Figure 6The paper presents a comparison of recall rates for various regions of a foreign newspaper dataset using different segmentation methods. For European historical newspapers, missed detection rates are often concentrated on the boundary pixels of page objects. Because the proposed method ensures that as many foreground pixels as possible are segmented from background pixels, Panoptic-DLA's "paragraph" recall on the ENP dataset is 1.53% higher than that of the best "paragraph" detector, Faster R-CNN, and 0.31% higher than that of Mask R-CNN. In Mongolian lead type newspaper images, Panoptic-DLA achieves the highest recall in the title region. Its segmentation performance in the paragraph region is second only to Panoptic-DeepLab.

[0069] In summary, the present invention takes a newspaper image as input and outputs various elements and their positions in the newspaper image. The model framework follows the typical natural scene panoramic segmentation method and consists of two branches: a semantic segmentation branch and an instance segmentation branch. Semantic segmentation treats the background as stuff and the foreground area as things, without the need for binarization, and effectively classifies pixels of different categories in the background area and the foreground area. The instance segmentation branch is designed using a proposal-free instance segmentation method, so it is not restricted by regional bounding box prediction. In addition, layout area information is added to perform 2D or 3D shape modeling on the layout area to further improve the classification accuracy of pixels in adjacent areas. The present invention can effectively improve the accuracy of newspaper image layout analysis.

Claims

1. A newspaper image layout analysis method based on panoptic segmentation, characterized in that: The steps include: Step 1: Input a newspaper image and use the encoder backbone network to extract features from the newspaper image to provide deep image features for the semantic segmentation branch and the instance segmentation branch; Step 2: In each branch, a decoupled dilute spatial pyramid pooling module is used to generate a multi-scale feature map, and an upsampling decoding module is used to generate a high-resolution feature map. The dilute spatial pyramid pooling module performs multi-scale feature extraction on the output feature map of the backbone network, and the upsampling decoding module outputs a feature map with a lower resolution than the original image through upsampling. A skip connection is used to fuse the high-resolution low-level feature map in the backbone network with the upsampled high-level feature map to obtain finer features. Step 3: In the prediction phase, the semantic segmentation branch outputs a semantic probability map to predict the semantic category of each pixel, including background and various foreground regions. The instance segmentation branch generates a region center-boundary probability map to identify the boundary of each layout region and a region center direction map to identify the direction of each layout region, and assigns an instance ID to each foreground pixel. In step 4, combining the probability map and region center direction map obtained in step 3, each page area is determined by instance grouping, the semantic category of each page area is determined according to the semantic probability map, and the ungrouped foreground pixels are regrouped to finally generate a panoramic segmentation map of the input newspaper image.

2. The newspaper image layout analysis method based on panoptic segmentation according to claim 1 is characterized in that: The encoder backbone network adopts the improved Xception network architecture based on DeepLabv3+. The improvement refers to replacing the original standard convolution with a dilated convolution, which increases the output feature map resolution from 1 / 32 of the input image to 1 / 16.

3. The newspaper image layout analysis method based on panoptic segmentation according to claim 1, characterized in that: The dilated spatial pyramid pooling module is used to extract and fuse multi-scale information for segmentation, with convolution ratios of 6, 12, and 18 respectively; Based on the DeepLab V3+ decoder, the upsampling decoding module introduces the 1 / 8 resolution low-level feature map extracted from the middle layer of the backbone network into the decoder's upsampling process, fuses it with the high-level feature map output by the atrous spatial pyramid pooling module upsampled to 1 / 8 resolution, and uses 5×5 depthwise separable convolution instead of 3×3 convolution kernel.

4. The newspaper image layout analysis method based on panoptic segmentation according to claim 1, characterized in that: The semantic segmentation branch adopts the typical semantic segmentation model DeepLabV3+, and is trained using the weighted bootstrap cross entropy loss function to predict the semantic category of each pixel; the semantic probability map contains the prediction results of the background and foreground areas, wherein the foreground area includes the dividing line, the text area and the chart area; the dividing line includes the horizontal frame, the vertical frame, the horizontal dividing line, and the vertical dividing line; the text area includes the title area and the paragraph text area; the chart area includes the picture area and the table area.

5. The newspaper image layout analysis method based on panoptic segmentation according to claim 1, characterized in that: The region center-edge probability map is a three-dimensional score map of a mountain shape. The region center-edge probability value of each pixel is between 0 and 1, representing the height of the point in the mountain region. The pixel point at the center of the region is the peak point and is set to a high value, while the pixel point at the region edge is the foot point and is set to a low value. The shape of the foreground area is defined as the minimum bounding rectangle of the instance mask area. The score of each point is the ratio of the shortest distance to the height of the rectangle. The region center-boundary probability value of each pixel is determined by calculating the ratio of the shortest distance from each pixel to the four sides of the rectangle to the height of the rectangle. The higher the value, the closer the pixel is to the region center, and the lower the value, the closer the pixel is to the region boundary.

6. The newspaper image layout analysis method based on panoptic segmentation according to claim 5, characterized in that: For a pixel point x in the foreground area, the region center-edge probability value formula is as follows: Among them, a i is the perpendicular vector from the ith edge to the pixel x, ‖a i ‖ is the length of the vector, i=1,2,3,4, h x is the height of the minimum circumscribed rectangle, which is obtained by calculating the minimum distance from the pixel point x to the two opposite sides, that is, min(‖a1‖,‖a2‖,‖a3‖,‖a4‖); the closer the region center-boundary probability value is to 1, the closer the pixel is to the region center, and the closer it is to 0, the closer the pixel is to the region boundary.

7. The newspaper image layout analysis method based on panoptic segmentation according to claim 1, characterized in that: The region center direction map represents the thrust direction of each pixel in the foreground region. The thrust direction points from the edge of the foreground region to the region center, and the thrust value is proportional to the distance from the pixel to the region edge. The closer the distance, the greater the thrust value.

8. The newspaper image layout analysis method based on panoptic segmentation according to claim 7, characterized in that: The direction vector formula of the regional center direction pattern is as follows: where h x is the height of the minimum bounding rectangle, express Each side of the minimum circumscribed rectangle is considered to have a thrust effect, pushing point x towards the center, and the thrust direction is from the edge to the point; the closer to the edge, the greater the thrust; if the distance exceeds half the height of the rectangle, the thrust disappears; the direction map of the center of the region uses the unit vector u x , expressed as Region separation is achieved by assigning completely different region center direction values to pixels on the intersection edge of two adjacent regions.

9. The newspaper image layout analysis method based on panoptic segmentation according to claim 1, characterized in that: The example grouping is implemented as follows: First, the pixels in the region center direction map that are larger than the set threshold are regarded as peaks, and each peak is assigned an independent class ID; Then, each pixel at the foot of the mountain is assigned to the corresponding peak by the probability growth direction in the region center-boundary probability map or the vector direction in the region center direction map. For the region center-boundary probability map, the maximum value in the 8-neighborhood of each pixel is selected as the next point. For the region center direction map, a vector with an angle less than 3 / 8π is selected as the next vector. The next point value of each point is recursively assigned to all points until their values no longer change, thereby effectively solving the problem in parallel.

10. The newspaper image layout analysis method based on panoptic segmentation according to claim 1, characterized in that: The Regroup operation specifically includes: for the foreground pixels that are not grouped in the instance grouping in step 4, further labeling is completed by recursively assigning the instance IDs of the neighboring pixels with the same semantic category in the 8-neighborhood.

Images