Printing periodical typesetting intelligent identification method based on deep learning

Abstract

The invention discloses a printing publication typesetting intelligent identification method based on deep learning. According to the method, a typesetting intelligent identification mode is constructed and optimized from the perspective of layout character layout design. Firstly, a text detection model is trained to identify a text position, then the text areas are combined and independent according to the distribution relation and distance between the text areas, then a to-be-identified printed periodical tree and a printed matter template tree are constructed, and a tree structure is used for storing combined and independent text area coordinates. And finally, calculating a structure matching degree and a leaf node matching degree between the printed periodical tree to be identified and the printed matter template tree, and determining a typesetting style according to the structure matching degree and the leaf node matching degree. The method comprises three main modules: 1) training a DBNet model to detect the position of a text string; 2) merging and independence are carried out on the plurality of text areas; and 3) calculating global and local similarities between the periodical template and the document.

Description

Technical Field

[0001] The present invention relates to a method for intelligently identifying typesetting of printed publications based on deep learning, and belongs to the technical field of the combination of computer vision and artificial intelligence. Background Art

[0002] As humans enter the information and digital age, the amount of various documents and materials is growing exponentially, and users often need to find the documents they need from the massive amount of materials. If you first classify the document template type from the document, you can narrow the scope of the user's search for documents and push the required and interesting materials to the user more accurately. Document template classification makes it easy for users to quickly use the appropriate document format, which helps users focus on their own content.

[0003] 1. Traditional optical character recognition (OCR) methods have been applied to handwritten character and text recognition, image-text conversion, etc. The OCR method integrated with deep learning shows the advantages of high accuracy and good robustness. Text detection is an important part of optical character recognition technology (OCR). Traditional text detection uses binarization, rotational affine transformation and other algorithms to locate the position of the text string. At present, text detection methods based on deep learning have been applied to OCR. According to the different text objects detected, text detection methods based on deep learning can be divided into regression-based text detection methods and segmentation-based text detection methods. Segmentation-based text detection methods usually perform pixel-level, paragraph-level, and contour-level segmentation. Regression-based text detection methods regard the text string as a whole object and directly regress the bounding box of the text instance.

[0004] 2. Use the Hungarian algorithm to calculate the best match, that is, to minimize the cost of matching based on the maximum match. The main steps are as follows:

[0005] 1) The smallest element value in each row is subtracted from each element in each row of the matrix.

[0006] 2) The minimum element value of each column is subtracted from each element of each column.

[0007] 3) Cover all zero elements of the matrix with the minimum number of horizontal or vertical lines.

[0008] 4) If the number of lines is equal to min(n, m), it means the best allocation is found and the algorithm ends. Otherwise, go to step 5.

[0009] 5) Find the smallest element that is not covered by any line, subtract this element from each row that is not covered by a line, add this element to each column that is not covered by a line, and return to step 3. Summary of the invention

[0010] The present invention provides a method for intelligent recognition of printed publication typesetting based on deep learning, and the technical solutions adopted are mainly as follows:

[0011] Step 1: Train a text detection model to detect the location of text strings in printed publications;

[0012] Step 2: Determine the positional relationship between text regions, calculate the distance between text regions, and merge and separate multiple text regions;

[0013] Step 3: construct the representation tree of the printed publication to be identified and the representation tree of the printed publication template, calculate the matching degree between the trees, and determine the global and local similarity between the publication template and the document;

[0014] Step 4: Determine the layout style of the printed publication to be identified.

[0015] Further, the specific steps of step 1 are as follows:

[0016] Step 1.1: Train the DBNet text detection model, classify each pixel in the image into text or non-text, and find the coordinates of the text string;

[0017] Step 1.2: Preprocess the printed publication images to generate a dataset of uniform size, input it into the trained DBNet model, and output the coordinates of the four corners of the rectangular text string.

[0018] Further, the specific steps of step 2 are as follows:

[0019] Step 2.1: Use a text region bounding box to describe a rectangular region block generated by merging text regions. The text region bounding box can be generated by merging multiple text strings.

[0020] Step 2.2: Determine the distribution relationship between the bounding boxes of the text regions. The distribution relationship between the bounding boxes of two text regions on the coordinate axis can be simplified into four forms: intersection, horizontal intersection, vertical intersection, and distance;

[0021] Step 2.3: Calculate the distance between text regions based on the four relationship forms of the text region bounding box. Suppose the coordinates of the upper left corner of the first text region bounding box are (x1, y1) and the coordinates of the lower right corner are (x 1t ,y 1t ), the coordinates of the upper left corner of the second text area bounding box are (x2, y2), and the coordinates of the lower right corner are (x 2t ,y 2t ), the distance between text regions is calculated as

[0022] Step 2.4: If the distance between the bounding boxes of the two text regions is less than the set threshold, the two text regions are merged, otherwise they are retained as independent text regions. The merging method is Where (x t ,y t)、(x u ,y u ) represent the coordinates of the upper left corner and lower right corner of the merged region bounding box.

[0023] Further, the specific steps of step 3 are as follows:

[0024] Step 3.1: Construct the representation tree of the printed publication to be identified and the representation tree of the publication template, use the root node to represent the document, the first-level child nodes of the root node represent the main functional text area, the second-level nodes represent the regional bounding boxes that make up the functional text area, and the merged regional bounding boxes are located at the leaf nodes of the document representation tree;

[0025] Step 3.2: Calculate the matching degree of leaf nodes. The smaller the cost of matching, the higher the matching degree.

[0026] Step 3.3: Calculate the matching degree between tree structures to supplement the explanation of the overall distribution difference of the region bounding box;

[0027] Step 3.4: Calculate the similarity of the regional distribution of the publication page text using the method S = α*M node +β*M stru , where α and β represent the proportion of the matching degree between leaf nodes and the matching degree between tree structures in calculating the similarity, M node Indicates the leaf node matching degree, M stru Represents the tree structure matching degree; the algorithm is designed to find the template with the highest matching degree with the printed publication to be identified, and the calculation method is: Where n is the number of printed publication templates, i is the i-th template, S i Indicates the matching degree between the printed publication to be identified T1 and the i-th printed publication template T2.

[0028] Furthermore, the specific steps of step 3.2 are as follows:

[0029] (3.21) Calculate the matching loss C based on the non-overlapping area ratio of the two text area bounding boxes ij , method C ij =1-(X i ∩Y j ) / (X i +Y j -(X i ∩Y j )), where X represents all leaf nodes in the tree structure of the printed publication to be identified, X i represents the i-th leaf node in the tree structure of the printed publication to be identified, Y represents all the leaf nodes in the tree structure of the publication template, and Y j Represents the jth leaf node in the publication template tree structure;

[0030] (3.22) By the matching loss C ij Get the cost matrix C of matching X to Y;

[0031] (3.23) The Hungarian algorithm is used to calculate the best match. The minimum cost calculation method for the optimal allocation of X to Y is: Where S node represents the minimum cost consumed by matching leaf nodes of two trees, W ij represents a Boolean matrix, if the best match is to match row i with column j, then W ij =1, otherwise W ij =0;

[0032] (3.24) S node Normalization is then performed to calculate the matching degree between the tree leaf node representing the printed publication to be identified and each tree leaf node representing the template in the printed publication template library. The calculation method is: Where n represents the number of printed publication templates, M node (i) represents the matching degree between the tree node representing the tree of the printed publication to be identified and the tree node representing the tree of the i-th template.

[0033] Furthermore, the specific steps of step 3.3 are as follows:

[0034] (3.31) The tree structure similarity is calculated by comparing the differences in the nodes at each layer of the two trees. The calculation method is: Where T1 is the tree representing the printed publication to be identified, T2 is the tree representing the publication template, |T1| and |T2| represent the number of nodes in trees T1 and T2 respectively, TreeMatch(T1, T2) represents the number of node matches between the two trees, S stru Indicates the structural similarity of trees T1 and T2;

[0035] (3.32) S stru After normalization, the matching degree of T1 and T2 can be calculated as follows: Where n represents the number of printed publication templates, M stru (i) represents the matching degree with the i-th template tree structure.

[0036] Furthermore, the specific steps of step 4 are as follows: find the template with the highest matching degree according to the subscript of the maximum value in the matching degree vector S, obtain the publication layout with the highest text area distribution similarity, and thus determine the layout style of the printed publication to be identified, wherein the printed publication template can be a paper layout, a poetry layout, a letter layout, etc.

[0037] The beneficial effects of the present invention are as follows: the layout of printed publications is mainly composed of pictures, tables, texts, hyperlinks, and hypermedia, and the texts occupy most of the layout. The DBNet network is used to give the position coordinates of the text string of the publication, and the matching of the text area between the printed publication to be identified and the printed publication template is compared to improve the efficiency of identifying the layout style of the printed publication and reduce the amount of calculation; the position of each functional area in the layout is relatively fixed, and the printed publication will be typeset according to its specific layout style, and the generation of the text area bounding box can improve the prediction performance; the tree structure is used to represent the publication layout, and the text is represented with coarse granularity; combined with the tree leaf node matching method and the tree structure matching algorithm, the local and global layout between the publication template and the document can be fully considered, and the accuracy of identifying the layout style of the printed publication is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The present invention is a flow chart for identifying the typesetting style of printed publications.

[0039] Figure 2 This is the FPN network structure diagram of the present invention.

[0040] Figure 3 2 is an example diagram of the text detection result of the present invention.

[0041] Figure 4 It is a text area distribution relationship diagram of the present invention.

[0042] Figure 5 This is an example diagram of generating a regional bounding box according to the present invention.

[0043] Figure 6 It is a template diagram of a printed publication of the present invention.

[0044] Figure 7 It is the editorial layout tree structure diagram of the present invention. DETAILED DESCRIPTION

[0045] The present invention provides a method for intelligent recognition of printed publication typesetting based on deep learning. The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0046] Figure 1 The flow chart of the printed publication typesetting style recognition of the present invention is shown. First, the DBNet network is used to give the position coordinates of the text string of the publication; then the set area merging algorithm is used to generate the text area bounding box; finally, the tree structure is used to represent the publication layout, and the publication layout is determined by combining the tree structure matching algorithm and the tree leaf node matching algorithm.

[0047] 1. Text Detection

[0048] The DBNet network classifies each pixel in the image into text and non-text, and finds the coordinates of the text string. The DBNet network uses probability maps, threshold maps, and approximate binary maps as supervisory information for training. Its structure mainly consists of three modules: FPN network, FCN network, and DB operation. The FPN network adopts a multi-scale feature fusion method, which can significantly improve the scale robustness of feature expression without significantly increasing the amount of calculation. The present invention uses ResNet50 as the main network of the FPN network. Figure 2 The structure diagram of the FPN network is shown. First, the input image is convolved, and the size of the output feature map is reduced to half of the original size. Then, after four downsamplings, four scale feature maps are obtained respectively. Then, upsampling is performed from top to bottom and fused with the feature map of the same size generated from bottom to top. The convolution operation is used to eliminate the aliasing effect in the fused features. Finally, the output results of each layer are resampled and unified into four 1 / 4 original size feature maps, and the four feature maps are merged to generate a feature map of the same size as the original size. The FCN network generates a probability map and a threshold map from the feature map output by the FPN network. The DB operation obtains an approximate binary map by performing differentiable binarization on the probability map and the threshold map. The calculation method of the differentiable binarization is in The value range is between 0 and 1, (i, j) represents the coordinate point, P i，j represents the probability of image pixels, T i，j Represents the pixel threshold.

[0049] The DBNet network training data set can use ICDAR2015. Before algorithm processing, the image size is adjusted to 640×640. Figure 3 An example of the text detection result of the present invention is shown, highlighting the text string recognized by the text detection model.

[0050] 2. Text area bounding box generation

[0051] The text region bounding box is used to describe the region block generated by merging the text string bounding boxes. The text string bounding box can be regarded as the component unit of the text region bounding box. The distribution relationship between the two text region bounding boxes on the coordinate axis can be simplified into four forms: intersection, horizontal intersection, vertical intersection and distance, such as Figure 4 As shown. By judging the positional relationship between text areas, calculating the distance between text areas, and comparing the distance between text areas with the set threshold, the coordinates of the upper left and lower right corners of the bounding box of the merged rectangular area are determined. Suppose the coordinates of the upper left corner of the bounding box of the first text area are (x1, y1), and the coordinates of the lower right corner are (x 1t ,y 1t ). The coordinates of the upper left corner of the second text area bounding box are (x2, y2), and the coordinates of the lower right corner are (x 2t ,y 2t), the coordinate of the upper left corner of the bounding box of the area generated by merging the two text areas (x t ,y t ), the lower right corner coordinate (x u ,y u ). The distance between text regions is calculated as The merging method is

[0052] The present invention can set the region bounding box distance threshold to 30 on an image of size (640, 640). If the distance between two text region bounding boxes is less than the threshold, the two text regions are merged, otherwise they are still retained as independent text regions. Figure 5 An example of a region bounding box result generated by merging in the present invention is shown, and the region bounding core is drawn with a wireframe.

[0053] 3. Similarity calculation

[0054] Printed publications with different layout styles have different typesetting styles. The present invention uses 6 common printed publication layouts as publication templates, including reports, editorials, notices, letters, poems and papers, such as Figure 6 As shown. According to the position and proportion of the area bounding box in the entire page, the publication template tree structure is constructed, and the text area coordinate position is set for each leaf node. Figure 7 A diagram of the editorial layout tree structure is shown.

[0055] The leaf node matching algorithm first calculates the matching loss C based on the non-overlapping area ratio of the bounding boxes of the two text regions. ij , method C ij =1-(X i ∩Y j ) / (X i +Y j -(X i ∩Y j )), where X i represents the i-th leaf node in the tree structure of the printed publication to be identified, Y j represents the jth leaf node in the publication template tree structure; then the Hungarian algorithm is used to calculate the best match, and the calculation method is: Where S node represents the minimum cost consumed by matching leaf nodes of two trees, W ij represents a Boolean matrix, if the best match is to match row i with column j, then W ij =1, otherwise W ij =0; Finally, S node Normalization is then performed to calculate the matching degree between the tree leaf node representing the printed publication to be identified and each tree leaf node representing the template in the printed publication template library. The calculation method is: Where n represents the number of printed publication templates, M node (i) represents the matching degree between the tree node representing the tree of the printed publication to be identified and the tree node representing the tree of the i-th template.

[0056] The tree structure matching algorithm first calculates the tree structure similarity by comparing the differences in the nodes at each layer of the two trees. The calculation method is Where T1 is the tree representing the printed publication to be identified, T2 is the tree representing the publication template, |T1| and |T2| represent the number of nodes in trees T1 and T2 respectively, TreeMatch(T1, T2) represents the number of node matches between the two trees, S stru Represents the structural similarity of trees T1 and T2; then S stru After normalization, the matching degree of T1 and T2 can be calculated as follows: Where n represents the number of printed publication templates, M stru (i) represents the matching degree with the i-th template tree structure.

[0057] The present invention combines the tree structure matching algorithm and the leaf node matching algorithm to calculate the matching degree between the printed publication to be identified and the printed publication template. The calculation method is as follows: Where α and β represent the proportion of the matching degree between leaf nodes and the matching degree between tree structures in calculating the similarity, n represents the number of printed publication templates, i represents the i-th template, and M node Indicates the leaf node matching degree, M stru Indicates the tree structure matching degree, S i Indicates the matching degree between the printed publication to be identified and the i-th printed publication template. The present invention determines the template with the highest matching degree as the typesetting style of the printed publication.

[0058] In summary, the present invention discloses a method for intelligent recognition of printed publication typesetting based on deep learning. The method constructs and optimizes the typesetting intelligent recognition mode from the perspective of page text layout design. First, the text position is identified using a text detection model, and then the region bounding box is generated based on the distance between the text regions. Finally, the final typesetting style is determined based on the matching degree of the region bounding box distribution of the printed publication to be identified and all printed publication templates. The proposed recognition method is a novel typesetting recognition mode for printed publications.

[0059] The above describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.

[0060] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

Claims

1. A method for intelligent recognition of printed publication typesetting based on deep learning, characterized in that The following steps are involved: Step 1: Train a text detection model to detect the location of text strings in printed publications; Step 2: Determine the positional relationship between text regions, calculate the distance between text regions, and merge and separate multiple text regions; Step 3: construct the representation tree of the printed publication to be identified and the representation tree of the printed publication template, calculate the matching degree between the trees, and determine the global and local similarity between the publication template and the document; Step 4: Determine the layout style of the printed publication to be identified.

2. The method for intelligent recognition of printed publication typesetting based on deep learning according to claim 1, characterized in that: The specific steps of step 1 are as follows: Step 1.1: Train the DBNet text detection model, classify each pixel in the image into text or non-text, and find the coordinates of the text string; Step 1.2: Preprocess the printed publication images to generate a dataset of uniform size, input it into the trained DBNet model, and output the coordinates of the four corners of the rectangular text string.

3. The method for intelligent recognition of printed publication typesetting based on deep learning according to claim 1, characterized in that: The specific steps of step 2 are as follows: Step 2.1: Use a text region bounding box to describe a rectangular region block generated by merging text regions. The text region bounding box can be generated by merging multiple text strings. Step 2.2: Determine the distribution relationship between the bounding boxes of the text regions. The distribution relationship between the bounding boxes of two text regions on the coordinate axis can be simplified into four forms: intersection, horizontal intersection, vertical intersection, and distance; Step 2.3: Calculate the distance between text regions based on the four relationship forms of the text region bounding box. Suppose the coordinates of the upper left corner of the first text region bounding box are (x1, y1) and the coordinates of the lower right corner are (x 1t ,y 1t ), the coordinates of the upper left corner of the second text area bounding box are (x2, y2), and the coordinates of the lower right corner are (x 2t ,y 2t ), the distance between text regions is calculated as Step 2.4: If the distance between the bounding boxes of the two text regions is less than the set threshold, the two text regions are merged, otherwise they are retained as independent text regions. The merging method is Where (x t ,y t )、(x u ,y u ) represent the coordinates of the upper left corner and lower right corner of the merged region bounding box.

4. The method for intelligent recognition of printed publication typesetting based on deep learning according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3.1: Construct the representation tree of the printed publication to be identified and the representation tree of the publication template, use the root node to represent the document, the first-level child nodes of the root node represent the main functional text area, the second-level nodes represent the regional bounding boxes that make up the functional text area, and the merged regional bounding boxes are located at the leaf nodes of the document representation tree; Step 3.2: Calculate the matching degree of leaf nodes. The smaller the cost of matching, the higher the matching degree. Step 3.3: Calculate the matching degree between tree structures to supplement the explanation of the overall distribution difference of the region bounding box; Step 3.4: Calculate the similarity of the regional distribution of the publication page text using the method S = α*M node +β*M stru , where α and β represent the proportion of the matching degree between leaf nodes and the matching degree between tree structures in calculating the similarity, M node Indicates the leaf node matching degree, M stru Represents the tree structure matching degree; the algorithm is designed to find the template with the highest matching degree with the printed publication to be identified, and the calculation method is: Where n is the number of printed publication templates, i is the i-th template, S i Indicates the matching degree between the printed publication to be identified T1 and the i-th printed publication template T2.

5. The method for intelligent recognition of printed publication typesetting based on deep learning according to claim 3, characterized in that: In step 3.2, the reduction process is as follows: (3.21) Calculate the matching loss C based on the non-overlapping area ratio of the two text area bounding boxes ij , calculated as C ij =1-(X i ∩Y j ) / (X i +Y j -(X i ∩Y j )), where X represents all leaf nodes in the tree structure of the printed publication to be identified, X i represents the i-th leaf node in the tree structure of the printed publication to be identified, Y represents all the leaf nodes in the tree structure of the publication template, and Y j Represents the jth leaf node in the publication template tree structure; (3.22) By the matching loss C ij Get the cost matrix C of matching X to Y; (3.23) The Hungarian algorithm is used to calculate the best match. The minimum cost calculation method for the optimal allocation of X to Y is: Where S node represents the minimum cost consumed by matching leaf nodes of two trees, W ij represents a Boolean matrix, if the best match is to match row i with column j, then W ij =1, otherwise W ij =0; (3.24) S node Normalization is then performed to calculate the matching degree between the tree leaf node representing the printed publication to be identified and each tree leaf node representing the template in the printed publication template library. The calculation method is: Where n represents the number of printed publication templates, M node (i) represents the matching degree between the tree node representing the tree of the printed publication to be identified and the tree node representing the tree of the i-th template.

6. The method for intelligent recognition of printed publication typesetting based on deep learning according to claim 3, characterized in that: In step 3.3, the reduction process is as follows: (3.31) The tree structure similarity is calculated by comparing the differences in the nodes at each layer of the two trees. The calculation method is: Where T1 is the tree representing the printed publication to be identified, T2 is the tree representing the publication template, |T1| and |T2| represent the number of nodes in trees T1 and T2 respectively, TreeMatch(T1, T2) represents the number of node matches between the two trees, S stru Indicates the structural similarity of trees T1 and T2; (3.32) S stru After normalization, the matching degree of T1 and T2 can be calculated as follows: Where n represents the number of printed publication templates, M stru (i) represents the matching degree with the i-th template tree structure.

7. The method for intelligent recognition of printed publication typesetting based on deep learning according to claim 1, characterized in that: The step 4 finds the template with the highest matching degree according to the subscript of the maximum value in the matching degree vector S, obtains the publication layout with the highest text area distribution similarity, and thus determines the layout style of the printed publication to be identified, wherein the printed publication template can be a paper layout, a poetry layout, a letter layout, etc.

Images