Document image layout analysis method and device, computer device and storage medium

Abstract

The application relates to the technical field of computer vision, and provides a document image layout analysis method and device, computer equipment and a storage medium, the method comprises the following steps: performing multi-size high-dimensional semantic feature extraction on a document image to obtain multi-size high-dimensional semantic features; inputting the multi-size high-dimensional semantic features into a trained text subnetwork to mine text line context semantic information and perform multi-size semantic merging to obtain a text detection result; inputting the multi-size high-dimensional semantic features into a trained picture subnetwork and a table subnetwork respectively, outputting a picture detection result by the picture subnetwork and outputting a table detection result by the table subnetwork; and combining the text detection result, the picture detection result and the table detection result to obtain a layout analysis result. The method can improve the element recognition accuracy and the positioning precision of the layout analysis.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method, apparatus, computer device, and storage medium for document image layout analysis. Background Technology

[0002] Documents are the most common information carriers in people's daily lives. They organize text, tables, images, and other document elements in diverse forms, vividly and efficiently presenting information to users. Existing documents are mainly divided into paper documents and electronic documents. Electronic documents can be converted into paper documents using printing equipment; however, converting paper documents into electronic documents requires a more complex process: first, the paper document needs to be scanned into a document image; then, document image layout analysis technology is used to obtain a series of document elements such as text, tables, and images; next, OCR-related technologies are used to convert the document elements into editable text, tables, and vector graphics; finally, the information elements are organized into an electronic document. Therefore, document image layout analysis is an indispensable core step in the process of converting paper documents into electronic documents.

[0003] Currently, the mainstream document image layout analysis methods in academia and industry are based on deep learning strategies, which offer higher accuracy and stronger generalization compared to traditional rule-based and region-based methods. However, deep learning-based document layout analysis methods mainly borrow from object detection methods, treating all text, images, and tables in the document as objects for layout analysis. This ignores the differences and inherent characteristics between text, images, and tables, leading to problems such as inaccurate element identification and insufficient accuracy in element location during text layout analysis. Summary of the Invention

[0004] Therefore, it is necessary to provide a document image layout analysis method, apparatus, computer equipment, and storage medium that can improve the accuracy of element recognition and position positioning in layout analysis, thereby addressing the aforementioned technical problems.

[0005] This invention provides a document image layout analysis method, comprising:

[0006] Multi-size high-dimensional semantic features are extracted from document images to obtain multi-size high-dimensional semantic features;

[0007] The multi-size high-dimensional semantic features are input into the trained text sub-network to mine the semantic information of the text line context and perform multi-size semantic merging to obtain the text detection result;

[0008] The multi-size high-dimensional semantic features are respectively input into the trained image sub-network and table sub-network, and the image sub-network outputs the image detection result and the table sub-network outputs the table detection result.

[0009] The layout analysis results are obtained by combining the text detection results, the image detection results, and the table detection results.

[0010] In one embodiment, the step of inputting the multi-size high-dimensional semantic features into a trained text sub-network to mine text line context semantic information and performing multi-size semantic merging to obtain text detection results includes:

[0011] By merging features of the same size among the multi-size high-dimensional semantic features, a multi-size merged feature is obtained;

[0012] The transform network is used to mine text line context semantic information from the multi-size merged features and the merged features are separated to obtain multi-size context semantic features.

[0013] By fusing features of different sizes from the multi-size contextual semantic features, a multi-size fused feature is obtained;

[0014] Based on the multi-size fusion features, category and border detection are performed to obtain text detection results.

[0015] In one embodiment, fusing features of different sizes from the multi-size contextual semantic features to obtain multi-size fused features includes:

[0016] Traverse each size feature in the multi-size context semantic features, and fuse the current size feature and the features that are adjacent to the current size feature and have a smaller size than the current size feature.

[0017] The features obtained by fusing the largest feature among the multi-size contextual semantic features with the features are used as multi-size fusion features.

[0018] In one embodiment, the multi-size high-dimensional semantic features are respectively input into a trained image sub-network, and the image sub-network outputs image detection results, including:

[0019] The image sub-network performs convolution operations on features of different sizes in the multi-size high-dimensional semantic features based on weight sharing to obtain convolutional features corresponding to each size.

[0020] Based on the convolutional features corresponding to each size, category and bounding box predictions are performed, and image detection results are output.

[0021] In one embodiment, the multi-size high-dimensional semantic features are respectively input into a trained table sub-network, and the table sub-network outputs table detection results, including:

[0022] The table subnetwork performs convolution operations on features of different sizes in the multi-size high-dimensional semantic features based on weight sharing to obtain convolutional features corresponding to each size.

[0023] The convolutional features corresponding to each size are fused, and category and bounding box detection are performed based on the fused features to obtain the table detection results.

[0024] In one embodiment, the training steps for the text subnetwork, the image subnetwork, and the table subnetwork include:

[0025] Obtain the labeled element dataset and divide the labeled element dataset into a training set and a test set;

[0026] Select the text hyperparameters, image hyperparameters, and table hyperparameters corresponding to the text subnetwork, the image subnetwork, and the table subnetwork from the labeled element dataset, respectively.

[0027] Non-text elements, non-image elements, and non-table elements are removed from the training set to obtain a text training set, an image training set, and a table training set.

[0028] A text subnetwork is trained separately based on the text hyperparameters and the text training set; an image subnetwork is trained separately based on the image hyperparameters and the image training set; and a table subnetwork is trained separately based on the table hyperparameters and the table training set.

[0029] The text subnetwork, the image subnetwork, and the table subnetwork, which have been trained individually, are fully trained using the training set and the test set until the test results converge.

[0030] In one embodiment, the step of selecting the text hyperparameters, image hyperparameters, and table hyperparameters corresponding to the text subnetwork, the image subnetwork, and the table subnetwork respectively from the labeled element dataset includes:

[0031] From the annotation results corresponding to the labeled element dataset, select the annotation borders for text, image and table elements respectively to obtain text annotation borders, image annotation borders and table annotation borders;

[0032] The border height and border width of the text annotation border, the image annotation border, and the table annotation border are respectively used as samples to form a size sample set, resulting in a text border size sample set, an image border size sample set, and a table border size sample set;

[0033] Based on a preset number of clusters, the text border size sample set, the image border size sample set, and the table border size sample set are clustered respectively, and the number of cluster centers equal to the number of clusters is used as the corresponding text hyperparameter, image hyperparameter, and table hyperparameter.

[0034] A document image layout analysis device, comprising:

[0035] The shallow semantic extraction module is used to extract multi-size high-dimensional semantic features from document images to obtain multi-size high-dimensional semantic features.

[0036] The text detection module is used to input the multi-size high-dimensional semantic features into the trained text sub-network to mine the contextual semantic information of the text lines and perform multi-size semantic merging to obtain the text detection result;

[0037] The image and table detection module is used to input the multi-size high-dimensional semantic features into the trained image sub-network and table sub-network respectively, and the image sub-network outputs the image detection result and the table sub-network outputs the table detection result.

[0038] The result combination module is used to combine the text detection results, the image detection results, and the table detection results to obtain the layout analysis results.

[0039] The present invention also provides a computer device, the computer device including a processor and a memory, the memory storing a computer program, the processor executing the computer program to implement the steps of the document image layout analysis method described in any of the above claims.

[0040] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the document image layout analysis method described in any of the preceding claims.

[0041] The aforementioned document image layout analysis method, apparatus, computer equipment, and storage medium, by considering the inherent features and differences among text, images, and tables in document images under real-world scenarios, design separate text sub-networks, image sub-networks, and table sub-networks to detect corresponding text, images, and tables. This enhances the ability to distinguish elements in document images and improves the accuracy of element recognition and location positioning in layout analysis. Furthermore, by extracting basic multi-size high-dimensional semantic features for use by all sub-networks, the computational load within the model can be greatly reduced, improving processing efficiency. Attached Figure Description

[0042] Figure 1 This is a diagram illustrating the application environment of a document image layout analysis method in one embodiment.

[0043] Figure 2 This is a flowchart illustrating a document image layout analysis method in one embodiment.

[0044] Figure 3 This is a flowchart illustrating the training steps for the text subnetwork, image subnetwork, and table subnetwork in one embodiment.

[0045] Figure 4 This is a schematic diagram of the structure of a document image layout analysis model in one embodiment.

[0046] Figure 5 This is a structural block diagram of a document image layout analysis method apparatus in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] The document image layout analysis method provided in this application can be applied to, for example... Figure 1 In the application environment shown, the application environment involves a terminal 102 and a server 104. The terminal 102 communicates with the server 104 via a network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and the server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0049] When terminal 102 receives a document image layout analysis command, it can implement the document image layout analysis method independently. Alternatively, terminal 102 can send the document image layout analysis command to the communicating server 104, and server 104 can implement the document image layout analysis method. Taking server 104 as an example, specifically, server 104 extracts multi-size high-dimensional semantic features from the document image to obtain multi-size high-dimensional semantic features; server 104 inputs the multi-size high-dimensional semantic features into a trained text sub-network to mine the semantic information of the text line context and performs multi-size semantic merging to obtain text detection results; server 104 inputs the multi-size high-dimensional semantic features into a trained image sub-network and table sub-network respectively, and the image sub-network outputs image detection results and the table sub-network outputs table detection results; server 104 combines the text detection results, image detection results, and table detection results to obtain the layout analysis results.

[0050] In one embodiment, such as Figure 2 As shown, a document image layout analysis method is provided. Taking the application of this method to a server as an example, the method includes the following steps:

[0051] Step S201: Extract multi-size high-dimensional semantic features from the document image to obtain multi-size high-dimensional semantic features.

[0052] Among them, document images are electronic images of documents that need to be analyzed for layout, and multi-size high-dimensional semantic features are features of multiple different sizes extracted through a shallow image semantic feature model.

[0053] Specifically, after the server obtains the document image, it inputs the document image into the front-end shallow semantic feature model to extract multi-size high-dimensional semantic features, thus obtaining multi-size high-dimensional semantic features. In this embodiment, the front-end shallow image semantic feature model preferably uses a ResNet18 network, and the output is three high-dimensional features of different sizes. These three high-dimensional semantic features of different sizes can be referred to as large, medium, and small size features, or as the first size feature, the second size feature, and the third size feature. For example, the three large, medium, and small size features can be (b, h / 32, w / 32, 512), (b, h / 16, w / 16, 512), and (b, h / 8, w / 8, 512).

[0054] Step S202: Input the multi-size high-dimensional semantic features into the trained text sub-network to mine the contextual semantic information of the text lines and perform multi-size semantic merging to obtain the text detection result.

[0055] In this context, text elements in document images vary greatly in size; some text consists of only one line (width much larger than height), while others have multiple lines (width much smaller than height). Furthermore, text elements can be further subdivided into eight subcategories: titles, body text, headers, footers, footers, notes, formulas, image titles, and table titles. These subcategories share commonalities but also exhibit differences, requiring a network with high discriminative power. Therefore, based on these properties, this embodiment defines a separate text sub-network specifically for detecting text elements. Thus, the text sub-network in this embodiment can be understood as a pre-trained network model dedicated to detecting text elements in document images.

[0056] Specifically, after the shallow image semantic feature model at the front end extracts high-dimensional semantic features of multiple sizes, the server inputs these high-dimensional semantic features of multiple sizes into a pre-trained text sub-network. The text sub-network then mines the semantic information of the text line context based on the high-dimensional semantic features of each size, and after mining, the features of each size are merged for detection, thereby obtaining the text detection result.

[0057] In one embodiment, step S202 includes: merging features of the same size in multi-size high-dimensional semantic features to obtain multi-size merged features; using a transform network to mine text line context semantic information in the multi-size merged features and separating the merged features to obtain multi-size context semantic features; fusing features of different sizes in the multi-size context semantic features, and performing category and bounding box detection based on the fused features to obtain text detection results.

[0058] Specifically, the process of detecting text elements in the text subnetwork mainly includes three steps: mining the semantic context of text lines, multi-size semantic merging, and text element detection. The detailed processing procedures for each step are as follows:

[0059] The process of mining text line context semantics is as follows: First, high-dimensional semantic features of the same size are merged to obtain multi-size merged features. Then, the text line context semantic information is mined from the multi-size features obtained by the transform network. After mining, the merged features are separated again to obtain context semantic features of different sizes. For example, taking the small-size feature (b, h / 8, w / 8, 512) as an example, all small-size features (b, h / 8, w / 8, 512) in the multi-size high-dimensional semantic features are first merged, that is, the dimension is transformed into (b*h / 8, w / 8, 512). Then, the transform network is used to treat the dimension of w / 8 as individual word information and the dimension of 512 as the high-dimensional feature of the word, to learn the context semantic information of the same line, and output a feature with the dimension of (b*h / 8, w / 8, 256). Finally, this feature is separated to obtain the small-size context semantic features, that is, the feature (b*h / 8, w / 8, 256) is transformed again into (b, h / 8, w / 8, 256). Similarly, the medium-sized and large-sized features in the multi-size high-dimensional semantic features are processed in the same way as the small-sized features described above. After processing, the corresponding feature dimensions are (b, h / 16, w / 16, 256) and (b, h / 32, w / 32, 256), respectively. At this point, the three contextual semantic features of different sizes (b, h / 8, w / 8, 256), (b, h / 16, w / 16, 256), and (b, h / 32, w / 32, 256) are the multi-size contextual semantic features in this embodiment.

[0060] The process of multi-size semantic merging is the process of fusing to obtain multi-size fused features. In one embodiment, fusing features of different sizes in multi-size context semantic features to obtain multi-size fused features includes: traversing each size feature in multi-size context semantic features, fusing the current size feature and features adjacent to the current size feature that are smaller than the current size feature; and using the fused features obtained by fusing the largest size feature in multi-size context semantic features with the features as multi-size fused features.

[0061] Specifically, taking multi-size contextual semantic features, including features of three sizes (large, medium, and small), as an example, firstly, the large-size contextual semantic feature (b, h / 32, w / 32, 256) is upsampled, resulting in an output dimension of (b, h / 16, w / 16, 256). Then, the sampled feature is fused with the medium-size contextual semantic feature to obtain the medium-size fused feature, with an output dimension of (b, h / 16, w / 16, 512). Similarly, following the same method of fusing large and medium-size features, the medium-size feature is fused with the small-size feature to output the small-size fused feature, with a corresponding dimension of (b, h / 8, w / 8, 512). After all sizes of features have been fused in this way, the largest-size contextual semantic feature in the multi-size contextual semantic features is fused with each of the fused features obtained from the feature fusion. This means that the large-size contextual semantic feature is combined with the medium-size and small-size fused features to form the multi-size fused feature. That is, the feature fusion process in this embodiment can be understood as follows: the context semantic features of adjacent sizes are fused sequentially in descending order to obtain each fused feature, and then the context semantic feature of the largest size is retained and combined with each fused feature to obtain a multi-size fused feature.

[0062] The text element detection process is as follows: For the multi-size fusion features obtained through multi-size semantic merging, the output results are obtained using the category prediction and bounding box regression methods in a general object detection strategy. In other words, based on the fused multi-size features, a general object detection method is used to detect the text element categories and corresponding bounding boxes in the document image, thus obtaining the text detection results. The anchor points used in bounding box regression are the text hyperparameters determined when training the text sub-network.

[0063] Step S203: Input the multi-size high-dimensional semantic features into the trained image sub-network and table sub-network respectively, and output the image detection results from the image sub-network and the table detection results from the table sub-network.

[0064] Among these, image elements in document images exhibit less size variation compared to text elements and are highly distinguishable. Furthermore, table elements in document images show relatively little size variation and often involve multiple rows to multiple columns, lacking strict semantic information about text lines. Therefore, while defining a text sub-network, separate image and table sub-networks are defined to improve the detection and localization capabilities of image and table elements.

[0065] Specifically, while the text sub-network performs text detection, the image sub-network and table sub-network can be invoked to detect images and tables in the document image. That is, multi-size high-dimensional semantic features are input into the trained image sub-network and table sub-network respectively, and the image sub-network and table sub-network output the image detection results and table detection results respectively.

[0066] In one embodiment, multi-size high-dimensional semantic features are input into a trained image sub-network, and the image sub-network outputs image detection results, including: the image sub-network performing convolution operations on features of different sizes in the multi-size high-dimensional semantic features based on weight sharing to obtain convolutional features corresponding to each size; and performing category and bounding box prediction based on the convolutional features corresponding to each size to output image detection results.

[0067] Specifically, the image sub-network performs image detection primarily through two steps: image semantic learning and object detection. The image semantic learning process is as follows: For features of different sizes within a multi-size high-dimensional semantic feature set, a weight-sharing convolution operation is first performed. In this embodiment, the preferred convolution kernel size is 5, the preferred number of filters is 256, and the preferred stride is 1, thus outputting convolutional features corresponding to different sizes. Taking three sizes of features (large, medium, and small) as an example, the output feature dimensions after the convolution operation are (b, h / 32, w / 32, 256), (b, h / 16, w / 16, 256), and (b, h / 8, w / 8, 256), respectively. Then, for the convolutional features of different sizes output after convolution, the class prediction and bounding box regression methods from the general object detection strategy are used to obtain the image detection results. The image detection results include image elements and corresponding bounding boxes. The anchor points used in the bounding box regression are the image hyperparameters determined during the training of the image sub-network.

[0068] In one embodiment, multi-size high-dimensional semantic features are input into a trained table sub-network, and the table sub-network outputs table detection results. This includes: the table sub-network performing convolution operations on features of different sizes in the multi-size high-dimensional semantic features based on weight sharing to obtain convolutional features corresponding to each size; fusing the convolutional features corresponding to each size; and performing category and bounding box detection based on the fused features to obtain text detection results.

[0069] Specifically, the table sub-network's table detection mainly includes three steps: table semantic learning, multi-size semantic merging, and object detection. The table semantic learning process is similar to that of the image sub-network. After inputting multi-size high-dimensional semantic features, it first performs convolution operations using weight sharing to obtain convolutional features corresponding to different sizes. Before performing table object detection, the table sub-network also needs to perform multi-size semantic merging. The process of multi-size semantic merging is the same as that of the text sub-network, that is, the features of different sizes obtained from convolution are fused using the same processing method as the text sub-network to obtain multi-size fused features. Finally, based on the fused multi-size features, the class prediction and bounding box regression methods in the general object detection strategy are used to obtain the table detection results. The table detection results include table elements and corresponding bounding boxes. The anchor points used in the bounding box regression are the table hyperparameters determined during the training of the table sub-network.

[0070] Step S204: Combine the text detection results, image detection results, and table detection results to obtain the layout analysis results.

[0071] Specifically, after detecting the text, images, and tables in the document image, the combined results of each detection can yield a complete layout analysis.

[0072] The aforementioned document image layout analysis method considers the inherent features and differences among text, images, and tables in real-world document images. It designs separate text, image, and table sub-networks to detect these elements, thereby enhancing the ability to distinguish elements within the document image and improving the accuracy of element recognition and location. Furthermore, by extracting basic multi-size, high-dimensional semantic features for use across all sub-networks, the computational load within the model is significantly reduced, improving processing efficiency.

[0073] In one embodiment, such as Figure 3 As shown, the training steps for the text subnetwork, image subnetwork, and table subnetwork include: Step S301, obtaining the labeled element dataset and dividing the labeled element dataset into a training set and a test set.

[0074] Specifically, a certain number of document images are collected according to requirements, and preprocessed such as image alignment to obtain a document image set. Next, annotation tools are used to add borders to the elements in the document images within the document image set. The annotated element types are divided into three main categories: text, images, and tables. Text elements are further subdivided into eight subcategories: titles, body text, headers, footers, notes, formulas, image titles, and table titles. A total of 10 subcategories of document elements are annotated, as detailed in Table 1 below. Then, based on the annotation results, an annotated element dataset S = {(I1, ...} is constructed for document image layout analysis.

[0075] L1),(I2,L2),…,(I n L n The dataset is defined as follows: I represents the document image, L represents the corresponding annotation result, and each annotation result contains several document elements, i.e., L = {cx1, cy1, bw1, bh1, t1), ...}, where cx, cy, bw, bh, and t represent the center point coordinates (x, y), border width (w), border height (h), and document element category (t) of a document element's bounding box, respectively. The annotated element dataset is then divided into a training set S1 and a test set S2 in an 8:2 ratio.

[0076] Table 1

[0077]

[0078] Step S302: Select the text hyperparameters, image hyperparameters, and table hyperparameters corresponding to the text subnetwork, image subnetwork, and table subnetwork respectively from the labeled element dataset.

[0079] Specifically, hyperparameters are the parameters used to train a network model. Therefore, different hyperparameters are selected from the labeled element dataset for different networks.

[0080] In one embodiment, step S302 includes: selecting annotation borders for text, image, and table elements respectively from the annotation results corresponding to the annotation element dataset, to obtain text annotation borders, image annotation borders, and table annotation borders; using the border height and border width of the text annotation borders, image annotation borders, and table annotation borders as samples to form size sample sets, to obtain text border size sample sets, image border size sample sets, and table border size sample sets; and clustering the text border size sample sets, image border size sample sets, and table border size sample sets based on a preset number of clusters, to obtain a number of cluster centers equal to the number of clusters as the corresponding text hyperparameters, image hyperparameters, and table hyperparameters.

[0081] Specifically, the text hyperparameter selection process is as follows: From the annotation results corresponding to the annotated element dataset S, select document annotation borders whose major category is text (i.e., document element category t is [0,7]). Combine the border width bw and border height bh of the document annotation borders into a sample (bw, bh). Then, use the KMeans algorithm to cluster the text border size samples based on a preset number of clusters. In this embodiment, the preferred number of clusters for the text border size samples is 5, and preferably, 5 cluster centers are selected as text hyperparameters, recorded as Text_BA = {(aw1, ah1), ..., (aw5, ah5)}.

[0082] The image hyperparameter selection process is as follows: Similar to the selection of text hyperparameters, document annotation borders with the major category of "image" are selected from the annotation results corresponding to the annotation element dataset S. The border width bw and border height bh of the document annotation borders with the major category of "image" are combined into a sample (bw, bh). All samples are combined into an image border size sample set. Then, the KMeans algorithm is used to cluster the image border size samples based on a preset number of clusters. In this embodiment, the preferred number of clusters for the image border size samples is 2, and two cluster centers are preferably selected as image hyperparameters, recorded as Fig_BA={(aw1, ah1),(aw2, ah2)}.

[0083] The table hyperparameter selection process is as follows: Similar to the selection of text hyperparameters, document annotation borders with the major category of "table" are selected from the annotation results corresponding to the annotation element dataset S. The border width bw and border height bh of the document annotation borders with the major category of "table" are combined into a sample (bw, bh). All samples are then combined into a table border size sample set. Then, the KMeans algorithm is used to cluster the table border size samples based on a preset number of clusters. The preferred number of clusters for the table border size samples is also 2, and two cluster centers are preferably selected as table hyperparameters, recorded as Table_BA = {(aw1, ah1), (aw2, ah2)}.

[0084] Step S303: Remove non-text elements, non-image elements, and non-table elements from the training set to obtain text training set, image training set, and table training set.

[0085] Specifically, since the text subnetwork, image subnetwork, and table subnetwork are dedicated to detecting text, images, and tables respectively, during network training, to avoid the influence of other elements, the training sets used by each subnetwork need to be modified by removing other elements to form new training sets. That is, the training set for the text subnetwork should be a text training set removed from the training set S. Similarly, the training set for the image subnetwork should be an image training set removed from the training set S, and the training set for the table subnetwork should be a table training set removed from the training set S.

[0086] Step S304: Train the text subnetwork separately based on text hyperparameters and text training set, train the image subnetwork separately based on image hyperparameters and image training set, and train the table subnetwork separately based on table hyperparameters and table training set.

[0087] Specifically, after selecting the appropriate hyperparameters and training set, each sub-network is trained independently using its corresponding hyperparameters and training set. Independent training means that when training the current sub-network, the parameters of other networks and sub-networks should be frozen, and only the current sub-network should be trained. For example, when training the text sub-network, the preceding shallow image semantic feature model (ResNet18 network), image sub-network, and table sub-network need to be frozen. The training strategies for the image and table sub-networks are the same as those for the text sub-network.

[0088] Step S305: Use the training set and test set to perform full training on the text subnetwork, image subnetwork and table subnetwork after individual training, until the test results converge.

[0089] Specifically, after training each sub-network individually, all parameters are unfrozen, and the entire network is trained using the training set S1. After each epoch, the network is tested using the test set S2 until the test results on S2 converge. The model parameters with the best test results are then fixed, resulting in a well-trained document image layout analysis model. Subsequently, for any input document image, the trained model is used to predict document element categories and border positions, completing the layout analysis.

[0090] Furthermore, the network loss function defined in this embodiment is a weighted multi-task loss function, and its definition formula is as follows:

[0091] cost=Loss_text+α*Loss_fig+β*Loss_table

[0092] Where Loss_text, Loss_fig, and Loss_table represent the total loss for category prediction and bounding box regression for the text subnetwork, image subnetwork, and table subnetwork, respectively, and α and β are weight adjustment factors with values ​​of 0.4 and 0.6, respectively.

[0093] The defined network optimization strategy is an adaptive network optimizer, and its definition formula is shown below:

[0094]

[0095]

[0096] in, Represents θ t-1 The time-based loss value, where α is the learning rate.

[0097] In one embodiment, such as Figure 4 As shown, a schematic diagram of a document image layout analysis model is provided. The model includes a pre-construction shallow image semantic feature model (ResNet18 network), a text sub-network, an image sub-network, and a table sub-network. The document image layout analysis model is explained based on this model.

[0098] For details, please refer to Figure 4 After acquiring the document image, it is first input into a ResNet18 network for multi-dimensional semantic feature extraction. Then, these multi-dimensional semantic features are input into a multi-task heterogeneous network, specifically into a text sub-network, an image sub-network, and a table sub-network, for text detection, image detection, and table detection, respectively. Finally, the layout analysis results are obtained based on the text detection results, image detection results, and table detection results.

[0099] It should be understood that, although Figure 2-3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2-3 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0100] In one embodiment, such as Figure 5 As shown, a document image layout analysis device is provided, comprising:

[0101] The shallow semantic extraction module 501 is used to extract multi-size high-dimensional semantic features from the document image to obtain multi-size high-dimensional semantic features.

[0102] The text detection module 502 is used to input multi-size high-dimensional semantic features into the trained text sub-network to mine the contextual semantic information of text lines and perform multi-size semantic merging to obtain text detection results.

[0103] The image and table detection module 503 is used to input multi-size high-dimensional semantic features into the trained image sub-network and table sub-network respectively, and the image sub-network outputs the image detection result and the table sub-network outputs the table detection result.

[0104] The result combination module 504 is used to combine text detection results, image detection results, and table detection results to obtain layout analysis results.

[0105] In one embodiment, the text detection module 502 is further configured to merge features of the same size in multi-size high-dimensional semantic features to obtain multi-size merged features; use a transform network to mine text line context semantic information in the multi-size merged features and separate the merged features to obtain multi-size context semantic features; fuse features of different sizes in the multi-size context semantic features, and perform category and bounding box detection based on the fused features to obtain text detection results.

[0106] In one embodiment, the text detection module 502 is further configured to traverse each size feature in the multi-size context semantic features, fuse the current size feature and the features adjacent to the current size feature and smaller in size than the current size feature, and use the fused features obtained by fusing the largest feature in the multi-size context semantic features with the features as multi-size fused features.

[0107] In one embodiment, the image table detection module 503 is further used to perform convolution operations on features of different sizes in multi-size high-dimensional semantic features in the image sub-network based on weight sharing, to obtain convolution features corresponding to each size; and to perform category and bounding box prediction based on the convolution features corresponding to each size, and output image detection results.

[0108] In one embodiment, the image table detection module 503 is further configured to perform convolution operations on features of different sizes in multi-size high-dimensional semantic features using a weight-sharing method to obtain convolution features corresponding to each size; fuse the convolution features corresponding to each size; and perform category and border detection based on the fused features to obtain text detection results.

[0109] In one embodiment, a training module is further included, used to obtain a labeled element dataset, divide the labeled element dataset into a training set and a test set; select text hyperparameters, image hyperparameters, and table hyperparameters corresponding to the text subnetwork, image subnetwork, and table subnetwork respectively from the labeled element dataset; remove non-text elements, non-image elements, and non-table elements from the training set respectively to obtain a text training set, an image training set, and a table training set; train the text subnetwork separately based on the text hyperparameters and the text training set, train the image subnetwork separately based on the image hyperparameters and the image training set, and train the table subnetwork separately based on the table hyperparameters and the table training set; and perform full training on the text subnetwork, image subnetwork, and table subnetwork after individual training using the training set and the test set until the test results converge.

[0110] In one embodiment, the training module is further configured to select text, image, and table annotation borders respectively from the annotation results corresponding to the annotation element dataset, to obtain text annotation borders, image annotation borders, and table annotation borders; use the border height and border width of the text annotation borders, image annotation borders, and table annotation borders as samples to form size sample sets, to obtain text border size sample sets, image border size sample sets, and table border size sample sets; and cluster the text border size sample sets, image border size sample sets, and table border size sample sets respectively based on a preset number of clusters, to obtain a number of cluster centers equal to the number of clusters as the corresponding text hyperparameters, image hyperparameters, and table hyperparameters.

[0111] Specific limitations regarding the document image layout analysis device can be found in the limitations of the document image layout analysis method described above, and will not be repeated here. Each module in the aforementioned document image layout analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various document image layout analysis method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0112] In one embodiment, a computer device is provided, which may be a server, including a processor, memory, and a network interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a document image layout analysis method. Exemplarily, the computer program may be divided into one or more modules, one or more of which are stored in memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device.

[0113] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0114] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0115] Those skilled in the art will understand that the computer device structure shown in this embodiment is only a partial structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. The specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0116] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0117] Multi-size high-dimensional semantic features are extracted from document images to obtain multi-size high-dimensional semantic features;

[0118] Multi-size high-dimensional semantic features are input into a trained text sub-network to mine text line context semantic information and perform multi-size semantic merging to obtain text detection results;

[0119] Multi-size high-dimensional semantic features are input into the trained image sub-network and table sub-network respectively, and the image sub-network outputs the image detection results and the table sub-network outputs the table detection results.

[0120] By combining the text detection results, image detection results, and table detection results, the layout analysis results are obtained.

[0121] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0122] By merging features of the same size in multi-size high-dimensional semantic features, multi-size merged features are obtained. The transform network is used to mine text line context semantic information in the multi-size merged features and the merged features are separated to obtain multi-size context semantic features. Features of different sizes in multi-size context semantic features are fused, and category and bounding box detection are performed based on the fused features to obtain text detection results.

[0123] In one embodiment, when the processor executes the computer program, it further performs the following steps: traversing each size feature in the multi-size context semantic features, fusing the current size feature and the features adjacent to the current size feature that are smaller than the current size feature; and using each fused feature obtained by fusing the largest feature in the multi-size context semantic features with the features as the multi-size fused features.

[0124] In one embodiment, when the processor executes the computer program, it further implements the following steps: the image subnetwork performs convolution operations on features of different sizes in multi-size high-dimensional semantic features based on weight sharing to obtain convolution features corresponding to each size; and performs category and bounding box prediction based on the convolution features corresponding to each size to output image detection results.

[0125] In one embodiment, when the processor executes the computer program, it further implements the following steps: the table subnetwork performs convolution operations on features of different sizes in multi-size high-dimensional semantic features based on weight sharing to obtain convolution features corresponding to each size; the convolution features corresponding to each size are fused, and category and bounding box detection are performed based on the fused features to obtain text detection results.

[0126] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a labeled element dataset and dividing the labeled element dataset into a training set and a test set; selecting text hyperparameters, image hyperparameters, and table hyperparameters corresponding to the text subnetwork, image subnetwork, and table subnetwork respectively from the labeled element dataset; removing non-text elements, non-image elements, and non-table elements from the training set to obtain a text training set, an image training set, and a table training set; training the text subnetwork separately based on the text hyperparameters and the text training set, training the image subnetwork separately based on the image hyperparameters and the image training set, and training the table subnetwork separately based on the table hyperparameters and the table training set; and performing full training on the text subnetwork, image subnetwork, and table subnetwork after individual training using the training set and the test set until the test results converge.

[0127] In one embodiment, when the processor executes the computer program, it further performs the following steps: selecting annotation borders for text, image, and table elements respectively from the annotation results corresponding to the annotation element dataset, to obtain text annotation borders, image annotation borders, and table annotation borders; using the border height and border width of the text annotation borders, image annotation borders, and table annotation borders as samples to form size sample sets, to obtain text border size sample sets, image border size sample sets, and table border size sample sets; and clustering the text border size sample sets, image border size sample sets, and table border size sample sets based on a preset number of clusters, to obtain a number of cluster centers equal to the number of clusters as the corresponding text hyperparameters, image hyperparameters, and table hyperparameters.

[0128] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0129] Multi-size high-dimensional semantic features are extracted from document images to obtain multi-size high-dimensional semantic features;

[0130] Multi-size high-dimensional semantic features are input into a trained text sub-network to mine text line context semantic information and perform multi-size semantic merging to obtain text detection results;

[0131] Multi-size high-dimensional semantic features are input into the trained image sub-network and table sub-network respectively, and the image sub-network outputs the image detection results and the table sub-network outputs the table detection results.

[0132] By combining the text detection results, image detection results, and table detection results, the layout analysis results are obtained.

[0133] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: merging features of the same size in multi-size high-dimensional semantic features to obtain multi-size merged features; using a transform network to mine text line context semantic information in the multi-size merged features and separating the merged features to obtain multi-size context semantic features; fusing features of different sizes in the multi-size context semantic features, and performing category and bounding box detection based on the fused features to obtain text detection results.

[0134] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: traversing each size feature in the multi-size context semantic features, fusing the current size feature and the features adjacent to the current size feature that are smaller than the current size feature; and using the fused features obtained by fusing the largest feature in the multi-size context semantic features with the features as multi-size fused features.

[0135] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: the image subnetwork performs convolution operations on features of different sizes in the multi-size high-dimensional semantic features based on weight sharing to obtain convolution features corresponding to each size; and performs category and bounding box prediction based on the convolution features corresponding to each size to output the image detection results.

[0136] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: the table subnetwork performs convolution operations on features of different sizes in multi-size high-dimensional semantic features based on weight sharing to obtain convolution features corresponding to each size; the convolution features corresponding to each size are fused, and category and bounding box detection are performed based on the fused features to obtain text detection results.

[0137] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a labeled element dataset and dividing the labeled element dataset into a training set and a test set; selecting text hyperparameters, image hyperparameters, and table hyperparameters corresponding to the text subnetwork, image subnetwork, and table subnetwork respectively from the labeled element dataset; removing non-text elements, non-image elements, and non-table elements from the training set to obtain a text training set, an image training set, and a table training set; training the text subnetwork separately based on the text hyperparameters and the text training set, training the image subnetwork separately based on the image hyperparameters and the image training set, and training the table subnetwork separately based on the table hyperparameters and the table training set; and performing full training on the text subnetwork, image subnetwork, and table subnetwork after individual training using the training set and the test set until the test results converge.

[0138] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: selecting annotation borders for text, image, and table elements respectively from the annotation results corresponding to the annotation element dataset, to obtain text annotation borders, image annotation borders, and table annotation borders; using the border height and border width of the text annotation borders, image annotation borders, and table annotation borders as samples to form size sample sets, to obtain text border size sample sets, image border size sample sets, and table border size sample sets; and clustering the text border size sample sets, image border size sample sets, and table border size sample sets based on a preset number of clusters, to obtain a number of cluster centers equal to the number of clusters as the corresponding text hyperparameters, image hyperparameters, and table hyperparameters.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A document image layout analysis method, characterized in that, include: Multi-size high-dimensional semantic features are extracted from document images to obtain multi-size high-dimensional semantic features; The multi-size high-dimensional semantic features are input into a trained text sub-network to mine text line context semantic information and perform multi-size semantic merging to obtain text detection results. This includes: merging features of the same size in the multi-size high-dimensional semantic features to obtain multi-size merged features; using a transform network to mine text line context semantic information in the multi-size merged features and separating the merged features to obtain multi-size context semantic features. Specifically: traversing each size feature in the multi-size context semantic features, fusing the current size feature and features adjacent to the current size feature but smaller in size, and using the largest feature in the multi-size context semantic features as the fused features; fusing features of different sizes in the multi-size context semantic features to obtain multi-size fused features; and performing category and bounding box detection based on the multi-size fused features to obtain text detection results. The multi-size high-dimensional semantic features are respectively input into the trained image sub-network and table sub-network. The text sub-network, image sub-network, and table sub-network are heterogeneous networks. The text sub-network uses a Transformer network to mine the semantic information of text line context. The image sub-network and table sub-network use weight-sharing convolution operations. The image sub-network outputs the image detection result based on the weight-sharing convolution operation, and the table sub-network outputs the table detection result based on the weight-sharing convolution operation and the fusion of multi-size features. The layout analysis results are obtained by combining the text detection results, the image detection results, and the table detection results.

2. The method according to claim 1, characterized in that, The output of the image detection result by the convolution operation based on weight sharing of the image sub-network includes: The image sub-network performs convolution operations on features of different sizes in the multi-size high-dimensional semantic features based on weight sharing to obtain convolutional features corresponding to each size. Based on the convolutional features corresponding to each size, category and bounding box predictions are performed, and image detection results are output.

3. The method according to claim 1, characterized in that, The table detection result output by the table sub-network based on weight-sharing convolutional operations and multi-size feature fusion includes: The table subnetwork performs convolution operations on features of different sizes in the multi-size high-dimensional semantic features based on weight sharing to obtain convolutional features corresponding to each size. The convolutional features corresponding to each size are fused, and category and bounding box detection are performed based on the fused features to obtain the table detection results.

4. The method according to claim 1, characterized in that, The training steps for the text subnetwork, the image subnetwork, and the table subnetwork include: Obtain the labeled element dataset and divide the labeled element dataset into a training set and a test set; Select the text hyperparameters, image hyperparameters, and table hyperparameters corresponding to the text subnetwork, the image subnetwork, and the table subnetwork from the labeled element dataset, respectively. Non-text elements, non-image elements, and non-table elements are removed from the training set to obtain a text training set, an image training set, and a table training set. A text subnetwork is trained separately based on the text hyperparameters and the text training set; an image subnetwork is trained separately based on the image hyperparameters and the image training set; and a table subnetwork is trained separately based on the table hyperparameters and the table training set. The text subnetwork, the image subnetwork, and the table subnetwork, which have been trained individually, are fully trained using the training set and the test set until the test results converge.

5. The method according to claim 4, characterized in that, The step of selecting the text hyperparameters, image hyperparameters, and table hyperparameters corresponding to the text subnetwork, the image subnetwork, and the table subnetwork respectively from the labeled element dataset includes: From the annotation results corresponding to the labeled element dataset, select the annotation borders for text, image and table elements respectively to obtain text annotation borders, image annotation borders and table annotation borders; The border height and border width of the text annotation border, the image annotation border, and the table annotation border are respectively used as samples to form a size sample set, resulting in a text border size sample set, an image border size sample set, and a table border size sample set; Based on a preset number of clusters, the text border size sample set, the image border size sample set, and the table border size sample set are clustered respectively, and the number of cluster centers equal to the number of clusters is used as the corresponding text hyperparameter, image hyperparameter, and table hyperparameter.

6. A document image layout analysis device, characterized in that, include: The shallow semantic extraction module is used to extract multi-size high-dimensional semantic features from document images to obtain multi-size high-dimensional semantic features. The text detection module is used to input the multi-size high-dimensional semantic features into a trained text sub-network to mine text line context semantic information and perform multi-size semantic merging to obtain text detection results. Specifically, it is used to merge features of the same size in the multi-size high-dimensional semantic features to obtain multi-size merged features; and to use a transform network to mine text line context semantic information in the multi-size merged features and to separate the merged features to obtain multi-size context semantic features. Specifically, it traverses each size feature in the multi-size context semantic features, fuses the current size feature and features adjacent to the current size feature but smaller in size, and uses the largest size feature in the multi-size context semantic features as the fused features obtained by fusing the features. By fusing features of different sizes from the multi-size contextual semantic features, a multi-size fused feature is obtained; Based on the multi-size fusion features, category and border detection are performed to obtain text detection results; The image and table detection module is used to input the multi-size high-dimensional semantic features into the trained image sub-network and table sub-network respectively. The text sub-network, image sub-network, and table sub-network are heterogeneous networks. The text sub-network uses a Transformer network to mine the semantic information of the text line context. The image sub-network and table sub-network use weight-sharing convolution operations. The image sub-network outputs the image detection result based on the weight-sharing convolution operation, and the table sub-network outputs the table detection result based on the weight-sharing convolution operation and the fusion of multi-size features. The result combination module is used to combine the text detection results, the image detection results, and the table detection results to obtain the layout analysis results.

7. A computer device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that, The processor is used to execute the computer program to implement the document image layout analysis method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the document image layout analysis method according to any one of claims 1-5.

Images