A document layout restoration method and device, electronic equipment and storage equipment

Abstract

The present disclosure provides a document layout restoration method and device, electronic equipment and storage equipment, and belongs to the technical field of computer, and specifically relates to the technical field of image recognition, deep learning and computer vision. The specific implementation scheme is: obtaining image data of a document to be restored; performing coarse-grained layout detection, fine-grained layout detection and text line detection on the image data to obtain coarse-grained layout information, fine-grained layout information and text line information of the image data; and restoring the document layout of the document to be restored based on the coarse-grained layout information, the fine-grained layout information and the text line information of the image data to obtain a layout restoration result. The content of the layout document can be accurately and completely restored to a flow document, and the conversion from the layout document to the flow document is realized.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the fields of image recognition, deep learning, and computer vision. Background Technology

[0002] Optical Character Recognition (OCR) technology has matured significantly and is widely used in document structuring scenarios. Among related technologies, the mainstream solution for restoring formatted documents to streaming documents is to use open-source software like LibreOffice to convert Portable Document Format (PDF) documents to Word documents. For converting non-scanned PDF documents, methods such as parsing the PDF source code and using tools like pdfminder (PDF information extraction) and pymupdf (an application for processing PDF documents) can be employed to obtain information such as text style, content, and layout from the document. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for restoring document layout.

[0004] According to one aspect of this disclosure, a method for restoring document layout is provided, comprising:

[0005] Obtain the image data of the document whose layout needs to be restored;

[0006] The image data is subjected to coarse-grained layout detection, fine-grained layout detection, and text line detection to obtain the coarse-grained layout information, fine-grained layout information, and text line information of the image data.

[0007] Based on the coarse-grained layout information, fine-grained layout information, and text line information of the image data, the document layout of the document to be restored is restored to obtain the layout restoration result.

[0008] According to a second aspect of this disclosure, a document format restoration apparatus is provided, comprising:

[0009] The acquisition module is used to acquire image data of the document whose layout needs to be restored;

[0010] The detection module is used to perform coarse-grained layout detection, fine-grained layout detection, and text line detection on the image data to obtain coarse-grained layout information, fine-grained layout information, and text line information of the image data.

[0011] The restoration module is used to restore the document layout of the document to be restored based on the coarse-grained layout information, fine-grained layout information and text line information of the image data, and obtain the layout restoration result.

[0012] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in any of the first aspects.

[0016] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform any of the steps of the method described in the first aspect.

[0017] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0018] This disclosure provides a document layout restoration method, apparatus, electronic device, and storage medium, which can convert a layout document into a streaming document.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a flowchart of a document layout restoration method provided in an embodiment of this disclosure;

[0022] Figure 2 This is a schematic diagram illustrating the different granularity format detection categories provided in the embodiments of this disclosure;

[0023] Figure 3 This is a flowchart of obtaining image data layout information at different granularities provided in an embodiment of this disclosure;

[0024] Figure 4 This is a model structure diagram of the multi-granularity version detection model provided in the embodiments of this disclosure;

[0025] Figure 5 This is a flowchart illustrating the process of obtaining fusion features provided in this embodiment of the disclosure;

[0026] Figure 6 This is a schematic diagram illustrating the effect of different granularity format detection provided in the embodiments of this disclosure;

[0027] Figure 7 This is a flowchart illustrating the process of obtaining the layout restoration result provided in this embodiment of the disclosure;

[0028] Figure 8 This is another flowchart of the document layout restoration method provided in this embodiment of the disclosure;

[0029] Figure 9 This is a flowchart of the end-to-end document layout restoration method provided in this embodiment.

[0030] Figure 10 This is a schematic diagram of a document layout restoration device provided in an embodiment of this disclosure;

[0031] Figure 11 This is another schematic diagram of the document layout restoration device provided in this embodiment of the disclosure;

[0032] Figure 12 This is a block diagram of an electronic device used to implement the document layout restoration method of the embodiments of this disclosure. Detailed Implementation

[0033] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0034] For ease of understanding, the following explanations are provided regarding fixed-layout documents and streaming documents. Fixed-layout documents are electronic document formats with a fixed page layout. Their presentation is device-independent; the layout remains consistent across various devices for reading, printing, or distributing. Examples include PDF documents and Open Fixed-layout Documents (OFD). Fixed-layout documents are primarily used for the publication, dissemination, and archiving of completed documents. Streaming documents, on the other hand, are based on characters as their basic unit. They are ordered collections of characters and support free editing, such as Word documents and text documents (TXT).

[0035] With the continuous advancement of mobile internet, scanning and snapshots are ubiquitous. Images, PDFs, and other formatted documents have gradually become common data carriers. Enterprise digitization is no longer limited to archiving formatted documents such as books, files, and forms. How to convert formatted documents into streaming documents for parsing, secondary editing, modification, and even using the parsing results for Natural Language Processing (NLP) tasks is a crucial problem that urgently needs to be solved in current enterprise digitization.

[0036] Among related technologies, the solution of converting PDF documents to Word documents based on LibreOffice often results in Word documents where each line of text has anchor points. Deleting or adding text often leads to content stacking issues, failing to truly meet the user's editing needs. Furthermore, tools like pdfminder and pymupdf can only obtain the original information of the document; they cannot recognize and extract text from scanned document images, nor can they restore the layout document to an editable form.

[0037] To address at least one of the aforementioned problems, this disclosure provides a document layout restoration method, apparatus, electronic device, and storage medium.

[0038] Figure 1 This is a flowchart of a document layout restoration method provided in an embodiment of this disclosure. Figure 1 As shown, it includes the following steps:

[0039] Step S101: Obtain the image data of the document to be restored.

[0040] The format of the document to be restored is not fixed; it can be a PDF document, OFD document, etc. The document to be restored can be converted into an image format. For example, open the document and use a screenshot tool to convert it into an image format, or open the document and save it as an image to obtain the image data. If the document to be restored is already in image format, the image data can be obtained directly.

[0041] Step S102: Perform coarse-grained layout detection, fine-grained layout detection, and text line detection on the image data to obtain coarse-grained layout information, fine-grained layout information, and text line information of the image data.

[0042] The image data obtained in step S101 is subjected to layout detection based on different granularities to obtain image data information of different granularities. For example... Figure 2As shown, there are three different granularities: coarse-grained layout detection, fine-grained layout detection, and text line detection. Coarse-grained layout detection detects layouts at a coarse level in the image, while fine-grained layout detection detects layouts at a fine level. In one example, coarse-grained layout detection categories include headers, footers, page numbers, columns, and footnotes; fine-grained layout detection categories include images, titles, tables, stamps, tables of contents, and text; and text line detection detects text line by line. The information obtained from these three different granularities can be learned through multi-task neural network learning.

[0043] Step S103: Based on the coarse-grained layout information, fine-grained layout information, and text line information of the image data, the document layout of the document to be restored is restored to obtain the layout restoration result.

[0044] Based on the coarse-grained layout information, fine-grained layout information, and text line information of the image data, the content of the image data is restored to the streaming document to obtain the layout restoration result.

[0045] In this embodiment, image data of a document whose layout needs to be restored is acquired; coarse-grained layout detection, fine-grained layout detection, and text line detection are performed on the image data to obtain coarse-grained layout information, fine-grained layout information, and text line information of the image data; based on the coarse-grained layout information, fine-grained layout information, and text line information of the image data, the document layout of the document to be restored is restored to obtain the layout restoration result. Because the image data is detected at different granularities, the information of the image data can be detected completely and accurately. Based on the different granularity layout information obtained from the detection, the content of the image data can be completely and accurately restored to the streaming document.

[0046] In one possible implementation, when the document to be restored is a PDF document, the image data of the document to be restored is acquired, including:

[0047] Step A: Obtain the PDF version of the document to be restored.

[0048] Step B involves decrypting the document to be restored to obtain the document information.

[0049] Decrypting a restored document is primarily used to automatically remove read and edit restrictions from PDF documents, enabling subsequent operations. Decrypting a restored document reveals detailed information such as page numbers and text styles.

[0050] Step C: Based on the document information, determine whether the document to be restored is a scanned PDF.

[0051] PDF recognition software can be used to determine whether the document to be restored is a scanned PDF.

[0052] Step D: If the document to be restored is a scanned PDF, convert the document to be restored into image data to obtain the image data of the document to be restored.

[0053] The method of converting a document whose layout to be restored into image data Figure 1 As illustrated in the examples shown, they will not be repeated here.

[0054] In real-world scenarios, it's more common for the document to be restored to be a PDF document. In this embodiment, the PDF document is decrypted to obtain document information. Based on this information, it's determined whether the document to be restored is a scanned PDF. If so, the scanned PDF document is converted into image data. This process of processing the PDF document to obtain image data facilitates subsequent steps in restoring the PDF document's layout.

[0055] In one possible implementation, the document to be restored is not a scanned PDF document. Instead, PDF structured parsing is performed on the document to be restored to obtain the document layout restoration result.

[0056] PDF documents are essentially structured documents. Text, images, and lines within these documents are organized in a specific structure. Open-source software like pymupdf and pdfminder can be used to perform structured parsing on non-scanned PDF documents, extracting text information, images, lines, etc. The text information includes text content, font, font size, color, and position. Then, strategies are used to extract paragraphs and tables. For example, text alignment, font, and line spacing are used to determine if adjacent text belongs to the same line or paragraph, thus extracting paragraphs. Tables are identified based on the relationships between lines; the areas enclosed by lines on all sides are the table cells. Finally, the extracted paragraphs, tables, and other content are written back to the streaming document to obtain the restored document layout.

[0057] In this embodiment of the disclosure, for non-scanned PDF documents, PDF structured parsing can be used to write the document content back to a streaming document based on the parsing results, thereby obtaining the document layout restoration result. This makes the document layout restoration solution provided in this disclosure more complete and allows for wider application in real-world scenarios.

[0058] In one possible implementation, such as Figure 3 As shown, coarse-grained layout detection, fine-grained layout detection, and text line detection are performed on image data to obtain coarse-grained layout information, fine-grained layout information, and text line information of the image data, including the following steps:

[0059] Step S301: Input the image data into a pre-trained deep learning model, and extract image features at multiple levels from the image data through the feature extraction network of the deep learning model.

[0060] The pre-trained deep learning model can be a Connectionist Text Proposal Network (CTPN), an Efficient and Accurate SceneText Detector (EAST), or similar; the feature extraction network can be a Residual Network (RestNet) or a PVANet (a lightweight feature extraction network). Image data is input into the feature extraction network of the pre-trained deep learning model, and depending on the depth of the feature extraction network, multiple levels of image features can be extracted from the image data.

[0061] Step S302: Through the feature fusion network of the deep learning model, feature fusion is performed on image features at multiple levels to obtain fused features.

[0062] Feature extraction networks extract image features at different levels. For example, when a feature extraction network contains multiple convolutional layers, each convolutional layer outputs image features at a corresponding level, including shallow image features and deep image features. The image features at each level are different. For example, shallow image features have a smaller receptive field and focus more on image details, such as text information; deep image features have a larger receptive field and can extract a wider range of features, such as layout and column information. Therefore, it is necessary to fuse image features at different levels to obtain fused features, which can represent all the content of the image data.

[0063] Step S303: The fused features are analyzed through the coarse-grained layout detection network of the deep learning model to obtain the coarse-grained layout information of the image data.

[0064] By inputting the fused features into the coarse-grained layout detection network, coarse-grained layout information of the image data can be obtained, such as the position and category information of headers, footers, page numbers, columns, and footnotes in the image. The position information can be displayed by coarse-grained prediction detection boxes, and the category can be identified by the attribute information of the prediction detection boxes. For example, different colored prediction detection boxes can be used to identify different categories, such as using green prediction detection boxes to identify columns and red prediction detection boxes to identify page numbers.

[0065] Step S304: The fused features are analyzed through the fine-grained layout detection network of the deep learning model to obtain the fine-grained layout information of the image data.

[0066] By inputting the fused features into the fine-grained layout detection network, fine-grained layout information of the image data can be obtained, such as the location and category information of images, titles, tables, seals, catalogs, and text in the image. The location information can be displayed by fine-grained predicted detection boxes, and the category can be identified by the attribute information of the predicted detection boxes.

[0067] Step S305: The fused features are analyzed through the text line detection network of the deep learning model to obtain the text line information of the image data.

[0068] By inputting the fused features into the text line detection network, we can obtain the text line information of the image data, and the text line prediction detection box can identify each line of text.

[0069] Figure 3 The execution process described is for illustrative purposes only. This disclosure does not limit the execution order of steps S303, S304, and S305. The three steps can be executed in any order, or at least two steps can be executed in parallel, all of which are within the protection scope of this disclosure.

[0070] In this embodiment, a deep learning model is used to extract features from image data to obtain image features at multiple levels. These multiple levels of image features are then fused to obtain fused features. The fused features can represent all the content of the image data. The fused features are then input into three detection networks of different granularities to obtain layout information of different granularities. Because the fused features are analyzed based on different granularities, the content of the image data can be detected completely and accurately.

[0071] Figure 4 This is a model structure diagram of the multi-granularity layout detection model provided in this embodiment, which can input image data. Figure 4 The model shown is used for detection to obtain layout information at different granularities. In one possible implementation, the feature extraction network includes multiple cascaded convolutional modules, each convolutional module including at least one cascaded convolutional layer;

[0072] Image data is input into a pre-trained deep learning model. The feature extraction network of the deep learning model extracts multiple levels of image features from the image data, including:

[0073] Image data is input into the feature extraction network of a deep learning model. The outputs of each convolutional module of the feature extraction network are obtained, resulting in multiple levels of image features from the image data. The output of each convolutional module is the image feature corresponding to that layer. For example... Figure 4As shown, the feature extraction network uses PVANet, which includes four cascaded convolutional modules, each containing at least one cascaded convolutional layer. Before performing the convolution operation, the image data can be resized to a preset size, which can be customized according to actual needs. Figure 4 In the model shown, the image data is resized to a fixed 7×7 size with 16 channels. " / 2" indicates that the image data's length and width are reduced to half their original size. The image data resized to the preset size is input into convolution module 1, resulting in a feature map with 64 channels and reduced length and width to half their original size (equivalent to the image features mentioned above), denoted as f4. f4 is input into convolution module 2, resulting in a feature map with 128 channels and reduced length and width to half their original size, denoted as f3. f3 is input into convolution module 3, resulting in a feature map with 256 channels and reduced length and width to half their original size, denoted as f2. f2 is input into convolution module 4, resulting in a feature map with 384 channels and reduced length and width to half their original size, denoted as f1. During feature extraction, as the network depth increases, the number of channels in the feature map increases, while the size of the feature map decreases. Feature extraction networks can extract image features at multiple levels from image data. Different levels of image features can reflect different details of the image data, thus enabling the complete extraction of image data features.

[0074] In one possible implementation, the feature extraction network includes N convolutional modules, where the i-th image feature is the image feature output by the i-th convolutional module, i is an integer from 1 to N, and N is a positive integer greater than 1.

[0075] Figure 5 This is a flowchart of obtaining the fusion feature provided in the embodiments of this disclosure, such as... Figure 5 As shown, a feature fusion network based on a deep learning model is used to fuse image features at multiple levels to obtain fused features. This process includes the following steps:

[0076] Step S501: The Nth image feature is upsampled through the feature fusion network of the deep learning model. The upsampled Nth image feature is then concatenated with the (N-1)th image feature and convolved to obtain the first concatenated image feature.

[0077] In feature extraction networks, the size of feature maps decreases as the convolutional module depth increases. Therefore, in feature fusion networks, image features need to be upsampled to increase their size. The size of the Nth image feature is increased to be the same as that of the (N-1)th image feature to facilitate image feature concatenation. Image feature concatenation can be performed using concat (concat is used to join two strings to form a single string). The concatenated image features are then convolved to obtain the first concatenated image feature.

[0078] exist Figure 4 In the model structure shown, f1 and h1 both refer to the image features output by convolutional module 4. "×2" indicates that the length and width of the image features are increased to twice their original size. This is because the size of the image features output by each convolutional module in the feature extraction network is half the size of the image features output by the previous convolutional module. Therefore, the feature fusion module needs to increase the size of the image features to twice their original size to ensure that the Nth image feature has the same size as the (N-1)th image feature during stitching. h1 is upsampled, and the upsampled h1 is stitched with f2. The stitched image features are then subjected to 1×1 convolution and 3×3 convolution to obtain the first stitched image feature with 128 channels, denoted as h2.

[0079] Step S502: Upsample the j-th stitched image feature, and stitch the upsampled j-th image feature with the (N-1)-j-th image feature and perform convolution processing to obtain the (j+1)-th stitched image feature, where j is an integer from 1 to N-2, and the (N-2)-th stitched image feature is a fusion feature.

[0080] exist Figure 4 In the model structure shown, h2 is upsampled and then concatenated with f3, followed by convolution to obtain a concatenated image feature with 64 channels, denoted as h3; h3 is upsampled and then concatenated with f4, followed by convolution to obtain a concatenated image feature with 32 channels, denoted as h4, thus obtaining a fusion feature with 32 channels.

[0081] In this embodiment of the disclosure, image features at multiple levels are fused to obtain fused features. The fused features can fully reflect the content of the image data. Analyzing the fused features at different granularities can ensure the detection effect at different granularities.

[0082] In one possible implementation, the coarse-grained layout detection network includes a coarse-grained category detection module, a coarse-grained rotated rectangle detection module, and a coarse-grained quadrilateral prediction module.

[0083] By analyzing the fused features through a coarse-grained layout detection network of a deep learning model, coarse-grained layout information of the image data is obtained, including:

[0084] The fused features are input into the coarse-grained layout detection network of the deep learning model. Coarse-grained rectangle detection boxes are obtained from the coarse-grained rotated rectangle detection module, coarse-grained quadrilateral detection boxes are obtained from the coarse-grained quadrilateral prediction module, and coarse-grained layout categories are obtained from the coarse-grained category detection module. The coarse-grained layout information includes coarse-grained rectangle detection boxes, coarse-grained quadrilateral detection boxes, and coarse-grained layout categories.

[0085] Among them, coarse-grained rectangular detection boxes and coarse-grained quadrilateral detection boxes can predict the coarse-grained pattern category and region of the current pixel. Figure 4 In the model structure shown, 'c' in the first layer of the coarse-grained layout detection network represents the number of categories. Here, 'c' is 6, representing five coarse-grained layout categories: header, footer, page number, column, and footnote, and one background category (coarse-grained layout detection). The coarse-grained rotating rectangle detection module is used to display rotating rectangles containing the current pixel. It outputs a total of 5 channels: 4 channels for the text box (4c in the figure represents 4 channels) generated by 1×1 convolution, representing the four distances from the pixel position to the top, right, bottom, and left sides of the rectangle, respectively; and 1 channel for the bounding box rotation angle generated by 1×1 convolution. This allows it to predict rotating rectangles containing the current pixel. The coarse-grained rotating rectangle detection module is used to predict the text within rotating rectangles. The coarse-grained quadrilateral prediction module is used to predict text in irregular quadrilaterals. It contains 8 channels (8c in the figure represents 8 channels), representing the coordinate offsets in the x and y directions from the current pixel to the four corner vertices of the target quadrilateral. The quadrilateral prediction module can predict text in both rectangular and regular quadrilateral shapes.

[0086] By inputting the fused features into the coarse-grained layout detection network, coarse-grained rectangular detection boxes, coarse-grained quadrilateral detection boxes, and coarse-grained layout categories can be obtained. The detection boxes accurately identify the categories and regions of the coarse-grained layouts, which is beneficial for subsequent steps in document layout reconstruction. For specific detection results, please refer to [reference needed]. Figure 6 The first image in the series uses a detection box to identify columns, headers, and page numbers in the image.

[0087] In one possible implementation, the coarse-grained layout categories include at least one of header, footer, page number, column, and footnote. In practical applications, the coarse-grained layout categories can be selected as needed. For example, if there is no header in the image, the header category can be omitted, or categories can be added as needed. This provides flexibility for practical applications and makes coarse-grained layout detection more convenient.

[0088] In one possible implementation, the fine-grained layout detection network includes a fine-grained category detection module, a fine-grained rotated rectangle detection module, and a fine-grained quadrilateral prediction module.

[0089] By using a fine-grained layout detection network based on a deep learning model, the fused features are analyzed to obtain fine-grained layout information of the image data, including:

[0090] The fused features are input into the fine-grained layout detection network of the deep learning model. Fine-grained rectangle detection boxes are obtained from the fine-grained rotated rectangle detection module, fine-grained quadrilateral detection boxes are obtained from the fine-grained quadrilateral prediction module, and fine-grained layout categories are obtained from the fine-grained category detection module. The fine-grained layout information includes fine-grained rectangle detection boxes, fine-grained quadrilateral detection boxes, and fine-grained layout categories.

[0091] Among them, fine-grained rectangular detection boxes and fine-grained quadrilateral detection boxes can predict the fine-grained pattern category and region of the current pixel. Figure 4 In the model structure shown, C in the first layer of the fine-grained layout detection network represents the number of categories. Here, C is 7, representing 6 fine-grained layout categories: image, title, table, seal, table of contents, and text, and 1 background category (fine-grained layout detection). The fine-grained rotating rectangle detection module is used to display the rotating rectangle containing the current pixel, with a total output of 5 channels. The fine-grained quadrilateral prediction module is used to predict text in irregular quadrilaterals, containing 8 channels. The meaning of the channels is the same as in the coarse-grained layout detection network, and will not be repeated here.

[0092] By inputting the fused features into the fine-grained layout detection network, fine-grained rectangular detection boxes, fine-grained quadrilateral detection boxes, and fine-grained layout categories can be obtained. These detection boxes accurately identify the categories and regions of fine-grained layouts, which is beneficial for subsequent steps in document layout reconstruction. For specific detection results, please refer to [reference needed]. Figure 6 The second image shows the detection boxes marking text, images, tables, and titles within the image.

[0093] In one possible implementation, the fine-grained layout categories include at least one of image, title, table, stamp, target, and text. In practical applications, the fine-grained layout categories can be selected as needed. For example, if there is no table in the image, the table category can be omitted, or categories can be added as needed. This provides flexibility for practical applications and makes fine-grained layout detection more convenient.

[0094] In one possible implementation, the text line detection network includes a text line category detection module, a text line rotated rectangle detection module, and a text line quadrilateral prediction module.

[0095] By analyzing the fused features through a deep learning model's text line detection network, the text line information of the image data is obtained, including:

[0096] The fused features are input into the text line detection network of the deep learning model. Text line rectangle detection boxes are obtained from the text line rotation rectangle detection module, text line quadrilateral detection boxes are obtained from the text line quadrilateral prediction module, and text line layout categories are obtained from the text line category detection module. The text line information includes text line rectangle detection boxes, text line quadrilateral detection boxes, and text line layout categories.

[0097] Among them, the text line rectangular detection box and the text line quadrilateral detection box can predict the text line layout category and region of the current pixel. Figure 4 In the model structure shown, the first layer of the text line layout detection network outputs the number of categories, which is 2 here, representing 1 text line class and 1 background class (text line layout detection); the text line rotating rectangle detection module is used to display the rotating rectangle containing the current pixel, and outputs a total of 5 channels; the text line quadrilateral prediction module is used to predict irregular quadrilateral text, and contains 8 channels. The meaning of the channels is the same as in the coarse-grained layout detection network, and will not be repeated here.

[0098] By inputting the fused features into the text line layout detection network, we can obtain rectangular and quadrilateral text line detection boxes, as well as text line layout categories. These detection boxes accurately identify text lines and regions in the image, which is beneficial for subsequent steps in document layout reconstruction. For specific detection results, please refer to [link / reference needed]. Figure 6 The third image in the sequence shows the detection boxes marking the text lines in the image.

[0099] After obtaining layout information at different granularities from the image data, document layout can be restored based on this information, such as... Figure 7 As shown, based on the coarse-grained layout information, fine-grained layout information, and text line information of the image data, the document layout of the document to be restored is restored to obtain the layout restoration result, including the following steps:

[0100] Step S701: Based on the coarse-grained layout information of the image data, restore the coarse-grained document layout of the document to be restored, and obtain the coarse-grained document layout restoration result.

[0101] Based on the coarse-grained layout category and the area marked by the detection box, the coarse-grained layout is restored, such as restoring the content of columns and headers to the corresponding areas of the flowing document. It can also calculate the page settings of image data based on the page information, such as determining whether the page orientation is landscape or portrait by calculating the page aspect ratio, and the page type, such as A4 (297×210 mm) or B5 (176 mm×250 mm).

[0102] Step S702: Based on the fine-grained layout information of the image data, restore the fine-grained document layout of the document to be restored, and obtain the fine-grained document layout restoration result.

[0103] Based on the fine-grained layout category and the area marked by the detection box, restore the fine-grained layout, such as restoring the content of the title and seal to the corresponding area of ​​the flowing document.

[0104] Step S703: Based on the text line information of the image data, restore the text line document layout of the document to be restored to obtain the text line document layout restoration result.

[0105] Based on the text line layout category and the area identified by the detection box, the text line layout is restored. For example, the spacing between each detected text line area can be used to calculate the paragraph before and after margins, and the page margins can be calculated, then mapped to the point values ​​in the flowing document. The color of the character is obtained by sampling the pixels of the area where the character is located; the font size is obtained by calculating the height of the character.

[0106] The layout restoration results include coarse-grained document layout restoration results, fine-grained document layout restoration results, and text line document layout restoration results.

[0107] Figure 7 The execution process described is for illustrative purposes only. This disclosure does not limit the execution order of steps S701, S702, and S703. The three steps can be executed in any order, or at least two steps can be executed in parallel, all of which are within the protection scope of this disclosure.

[0108] In this embodiment of the disclosure, the document layout of the document to be restored is restored based on the coarse-grained layout information, fine-grained layout information, and text line information of the image data, and the layout restoration result is obtained. Because the image data is detected based on three different granularities, the layout information of the image data can be detected accurately and completely. Therefore, when the layout of the document to be restored is performed based on the layout information of different granularities, an accurate and complete restoration result can be obtained, which can improve the user experience.

[0109] Figure 8 This is another flowchart of the document layout restoration method provided in this embodiment of the disclosure, such as... Figure 8 As shown, document layout restoration methods also include:

[0110] Step S801: Perform text recognition and table structure recognition on the image data to obtain text recognition results and table structure recognition results.

[0111] OCR text detection algorithms can be used to recognize text in image data, obtaining text recognition results. This can be done directly on image data or on text content within layout information of different granularities. Taking the table region output by a fine-grained layout detection network as input, a table structure parsing model is used to identify the table structure, yielding table structure recognition results.

[0112] Step S802: Write the text recognition results and table structure recognition results into a document with a preset format according to the layout restoration results.

[0113] The preset format document can be a streaming document, and the specific format can be customized according to actual needs.

[0114] In this embodiment of the disclosure, the text table structure of the image data is recognized, and the recognition result is written back to the streaming document according to the layout restoration result. This realizes the conversion from layout document to streaming document, and can accurately write back the content and corresponding layout of the layout document to the streaming document, which is convenient for secondary editing.

[0115] Figure 9 This is a flowchart of the end-to-end document layout restoration method provided in this embodiment of the disclosure, such as... Figure 9 As shown, the document to be restored is a PDF document. At the beginning of the process, the PDF document is acquired, decrypted, and its information is obtained. Based on this information, it is determined whether the PDF document is a scanned version. If not, structured parsing is performed. Based on the parsing results, the document's layout is restored using a strategy to obtain a streaming document, and the process ends. Alternatively, the PDF document can be input into a multi-granularity layout detection model to obtain paragraphs, tables, headers, etc., and then layout restoration is performed based on the structured parsing results. Because structured parsing has already been performed, text line detection and character recognition are not required.

[0116] If the current PDF document is a scanned version, it is converted to an image format. The image layout is then parsed, including table structure parsing, style analysis, page analysis, and global content analysis. Based on the image layout analysis results, inferences are made about font styles, paragraph styles, and page styles to reconstruct global layout information. This includes reconstructing paragraphs, tables, titles, images, indexes, headers, footers, page numbers, section breaks, footnotes, columns, fonts, and a table of contents. The reconstructed content is then restored to the streaming document, and the process ends. If the PDF document contains multiple pages, semantic inference is performed on the content of different pages to merge cross-page paragraphs, ensuring semantic integrity between paragraphs on different pages. In this embodiment, the user only needs to input the PDF document to be restored, or an image, form, or other formatted document, to obtain the corresponding streaming document. The document layout restoration process is convenient and improves the user experience.

[0117] Figure 10 This is a schematic diagram of a document format restoration device provided in an embodiment of this disclosure, with reference to... Figure 10 The device includes:

[0118] Module 1001 is used to acquire image data of the document to be restored.

[0119] The detection module 1002 is used to perform coarse-grained layout detection, fine-grained layout detection, and text line detection on the image data to obtain coarse-grained layout information, fine-grained layout information, and text line information of the image data.

[0120] The restoration module 1003 is used to restore the document layout of the document to be restored based on the coarse-grained layout information, fine-grained layout information and text line information of the image data, and obtain the layout restoration result.

[0121] In this embodiment, image data of a document whose layout needs to be restored is acquired; coarse-grained layout detection, fine-grained layout detection, and text line detection are performed on the image data to obtain coarse-grained layout information, fine-grained layout information, and text line information of the image data; based on the coarse-grained layout information, fine-grained layout information, and text line information of the image data, the document layout of the document to be restored is restored to obtain the layout restoration result. Because the image data is detected at different granularities, the information of the image data can be detected completely and accurately. Based on the different granularity layout information obtained from the detection, the content of the image data can be completely and accurately restored to the streaming document.

[0122] In one possible implementation, the acquisition module is specifically used to acquire a PDF document to be restored; decrypt the document to be restored to obtain document information; determine, based on the document information, whether the document to be restored is a scanned PDF; and if the document to be restored is a scanned PDF, convert the document to be restored into image data to obtain the image data of the document to be restored.

[0123] In one possible implementation, the device further includes:

[0124] The structured parsing module is used to perform PDF structured parsing on the document to be restored when the document to be restored is not a scanned PDF, so as to obtain the document layout restoration result of the document to be restored.

[0125] In one possible implementation, the detection module includes:

[0126] The feature extraction submodule is used to input the image data into a pre-trained deep learning model, and extract multiple levels of image features from the image data through the feature extraction network of the deep learning model.

[0127] The feature fusion submodule is used to perform feature fusion on the image features at multiple levels through the feature fusion network of the deep learning model to obtain fused features;

[0128] The coarse-grained layout detection submodule is used to analyze the fused features through the coarse-grained layout detection network of the deep learning model to obtain the coarse-grained layout information of the image data.

[0129] The fine-grained layout detection submodule is used to analyze the fused features through the fine-grained layout detection network of the deep learning model to obtain the fine-grained layout information of the image data.

[0130] The text line layout detection submodule is used to analyze the fused features through the text line detection network of the deep learning model to obtain the text line information of the image data.

[0131] In one possible implementation, the feature extraction network includes a plurality of convolutional modules in series, each of the convolutional modules including at least one convolutional layer in series.

[0132] The feature extraction submodule is specifically used to input the image data into the feature extraction network of the deep learning model, obtain the output of each convolutional module of the feature extraction network, and obtain image features of the image data at multiple levels.

[0133] In one possible implementation, the feature extraction network includes N convolutional modules, where the i-th image feature is the image feature output by the i-th convolutional module, and i is an integer belonging to 1 to N;

[0134] The feature fusion submodule is specifically used to upsample the Nth image feature through the feature fusion network of the deep learning model, and then concatenate and convolve the upsampled Nth image feature with the (N-1)th image feature to obtain the first concatenated image feature.

[0135] The j-th stitched image feature is upsampled, and the upsampled j-th image feature is stitched together with the (N-1)-j-th image feature and then convolved to obtain the (j+1)-th stitched image feature, where j is an integer from 1 to N-2, and the (N-2)-th stitched image feature is the fused feature.

[0136] In one possible implementation, the coarse-grained layout detection network includes a coarse-grained category detection module, a coarse-grained rotated rectangle detection module, and a coarse-grained quadrilateral prediction module.

[0137] The coarse-grained layout detection submodule is specifically used to input the fused features into the coarse-grained layout detection network of the deep learning model, obtain coarse-grained rectangle detection boxes from the coarse-grained rotated rectangle detection module, obtain coarse-grained quadrilateral detection boxes from the coarse-grained quadrilateral prediction module, and obtain coarse-grained layout categories from the coarse-grained category detection module; wherein, the coarse-grained layout information includes coarse-grained rectangle detection boxes, coarse-grained quadrilateral detection boxes, and coarse-grained layout categories.

[0138] In one possible implementation, the coarse-grained layout category includes at least one of header, footer, page number, column, and footnote.

[0139] In one possible implementation, the fine-grained layout detection network includes a fine-grained category detection module, a fine-grained rotated rectangle detection module, and a fine-grained quadrilateral prediction module.

[0140] The fine-grained layout detection submodule is specifically used to input the fused features into the fine-grained layout detection network of the deep learning model, obtain fine-grained rectangle detection boxes from the fine-grained rotated rectangle detection module, obtain fine-grained quadrilateral detection boxes from the fine-grained quadrilateral prediction module, and obtain fine-grained layout categories from the fine-grained category detection module; wherein, the fine-grained layout information includes fine-grained rectangle detection boxes, fine-grained quadrilateral detection boxes, and fine-grained layout categories.

[0141] In one possible implementation, the fine-grained layout categories include at least one of images, titles, tables, stamps, targets, and text.

[0142] In one possible implementation, the text line detection network includes a text line category detection module, a text line rotated rectangle detection module, and a text line quadrilateral prediction module.

[0143] The text line layout detection submodule is specifically used to input the fused features into the text line detection network of the deep learning model, obtain text line rectangle detection boxes from the text line rotation rectangle detection module, obtain text line quadrilateral detection boxes from the text line quadrilateral prediction module, and obtain text line layout categories from the text line category detection module; wherein, the text line information includes text line rectangle detection boxes, text line quadrilateral detection boxes, and text line layout categories.

[0144] In one possible implementation, the restoration module is specifically configured to restore the coarse-grained document layout of the document to be restored based on the coarse-grained layout information of the image data, to obtain a coarse-grained document layout restoration result; restore the fine-grained document layout of the document to be restored based on the fine-grained layout information of the image data, to obtain a fine-grained document layout restoration result; and restore the text line document layout of the document to be restored based on the text line information of the image data, to obtain a text line document layout restoration result; wherein, the layout restoration result includes the coarse-grained document layout restoration result, the fine-grained document layout restoration result, and the text line document layout restoration result.

[0145] In one possible implementation, such as Figure 11 As shown, the device further includes:

[0146] The recognition module 1101 is used to perform text recognition and table structure recognition on the image data to obtain text recognition results and table structure recognition results;

[0147] The writing module 1102 is used to write the text recognition result and the table structure recognition result into a document with a preset format according to the layout restoration result.

[0148] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0149] Figure 12A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] like Figure 12 As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0151] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the document layout restoration method. For example, in some embodiments, the document layout restoration method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the document layout restoration method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform a document layout restoration method by any other suitable means (e.g., by means of firmware).

[0153] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0155] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0157] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0158] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0159] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0160] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A document layout restoration method, comprising: obtaining image data of a document to be restored; performing coarse-grained layout detection, fine-grained layout detection and text line detection on the image data to obtain coarse-grained layout information, fine-grained layout information and text line information of the image data; restoring the document layout of the document to be restored based on the coarse-grained layout information, fine-grained layout information and text line information of the image data to obtain a layout restoration result; wherein the coarse-grained layout detection, fine-grained layout detection and text line detection on the image data to obtain the coarse-grained layout information, fine-grained layout information and text line information of the image data comprises: inputting the image data into a pre-trained deep learning model, extracting multiple levels of image features of the image data through a feature extraction network of the deep learning model; performing feature fusion on the multiple levels of image features through a feature fusion network of the deep learning model to obtain fused features; analyzing the fused features through a coarse-grained layout detection network of the deep learning model to obtain coarse-grained layout information of the image data; the coarse-grained layout information comprises coarse-grained rectangular detection boxes, coarse-grained quadrilateral detection boxes and coarse-grained layout categories; analyzing the fused features through a fine-grained layout detection network of the deep learning model to obtain fine-grained layout information of the image data; the fine-grained layout information comprises fine-grained rectangular detection boxes, fine-grained quadrilateral detection boxes and fine-grained layout categories; analyzing the fused features through a text line detection network of the deep learning model to obtain text line information of the image data; the text line information comprises text line rectangular detection boxes, text line quadrilateral detection boxes and text line layout categories.

2. The method of claim 1, wherein, The obtaining of the image data of the document to be restored comprises: obtaining a PDF format document to be restored; decrypting the document to be restored to obtain document information; judging whether the document to be restored is a scanned PDF based on the document information; in the case that the document to be restored is a scanned PDF, converting the document to be restored into image data to obtain the image data of the document to be restored.

3. The method of claim 2, further comprising: in the case that the document to be restored is not a scanned PDF, performing PDF structured parsing on the document to be restored to obtain a document layout restoration result of the document to be restored.

4. The method of claim 1, wherein, The feature extraction network comprises a plurality of convolution modules connected in series, and each convolution module comprises at least one convolution layer connected in series; The inputting of the image data into the pre-trained deep learning model and the extraction of the multiple levels of image features of the image data through the feature extraction network of the deep learning model comprises: inputting the image data into the feature extraction network of the deep learning model, obtaining the output of each convolution module of the feature extraction network, and obtaining the multiple levels of image features of the image data.

5. The method of claim 4, wherein, The feature extraction network comprises N convolution modules, an i-th image feature is an image feature output by an i-th convolution module, i is an integer belonging to 1 to N; The feature fusion network of the deep learning model is used to perform feature fusion on the multiple levels of image features to obtain a fusion feature, comprising: The N-th image feature is up-sampled by the feature fusion network of the deep learning model, and the up-sampled N-th image feature is spliced with the N-1-th image feature and subjected to convolution processing to obtain a first spliced image feature; The j-th spliced image feature is up-sampled, and the up-sampled j-th image feature is spliced with the N-1-j-th image feature and subjected to convolution processing to obtain a j+1-th spliced image feature, wherein j is an integer belonging to 1 to N-2, and the N-2-th spliced image feature is the fusion feature.

6. The method of claim 1, wherein, The coarse-grained layout detection network comprises a coarse-grained category detection module, a coarse-grained rotating rectangular frame detection module and a coarse-grained quadrilateral prediction module; The coarse-grained layout detection network of the deep learning model is used to analyze the fusion feature to obtain coarse-grained layout information of the image data, comprising: The fusion feature is input into the coarse-grained layout detection network of the deep learning model, a coarse-grained rectangular detection frame is obtained from the coarse-grained rotating rectangular frame detection module, a coarse-grained quadrilateral detection frame is obtained from the coarse-grained quadrilateral prediction module, and a coarse-grained layout category is obtained from the coarse-grained category detection module.

7. The method of claim 1, wherein, The coarse-grained layout category comprises at least one of a header, a footer, a page number, a column, and a footnote.

8. The method of claim 1, wherein, The fine-grained layout detection network comprises a fine-grained category detection module, a fine-grained rotating rectangular frame detection module and a fine-grained quadrilateral prediction module; The fine-grained layout detection network of the deep learning model is used to analyze the fusion feature to obtain fine-grained layout information of the image data, comprising: The fusion feature is input into the fine-grained layout detection network of the deep learning model, a fine-grained rectangular detection frame is obtained from the fine-grained rotating rectangular frame detection module, a fine-grained quadrilateral detection frame is obtained from the fine-grained quadrilateral prediction module, and a fine-grained layout category is obtained from the fine-grained category detection module.

9. The method of claim 1, wherein, The fine-grained layout category comprises at least one of a picture, a title, a table, a seal, an object and text.

10. The method of claim 1, wherein, The text line detection network comprises a text line category detection module, a text line rotating rectangular frame detection module and a text line quadrilateral prediction module; The text line detection network of the deep learning model is used to analyze the fusion feature to obtain text line information of the image data, comprising: The fusion feature is input into the text line detection network of the deep learning model, a text line rectangular detection frame is obtained from the text line rotating rectangular frame detection module, a text line quadrilateral detection frame is obtained from the text line quadrilateral prediction module, and a text line layout category is obtained from the text line category detection module.

11. The method of claim 1, wherein, The coarse-grained layout information, the fine-grained layout information and the text line information based on the image data are used to restore the document layout of the document to be restored, so as to obtain a layout restoration result, which comprises: The coarse-grained layout information based on the image data is used to restore the coarse-grained document layout of the document to be restored, so as to obtain a coarse-grained document layout restoration result; The fine-grained layout information based on the image data is used to restore the fine-grained document layout of the document to be restored, so as to obtain a fine-grained document layout restoration result; The text line information based on the image data is used to restore the text line document layout of the document to be restored, so as to obtain a text line document layout restoration result; The layout restoration result comprises the coarse-grained document layout restoration result, the fine-grained document layout restoration result and the text line document layout restoration result.

12. The method of claim 1, further comprising: performing character recognition and table structure recognition on the image data to obtain a character recognition result and a table structure recognition result; writing the character recognition result and the table structure recognition result into a document in a preset format according to the layout restoration result.

13. A document layout restoration apparatus, comprising: an acquisition module configured to acquire image data of a document to be restored; a detection module configured to perform coarse-grained layout detection, fine-grained layout detection and text line detection on the image data to obtain coarse-grained layout information, fine-grained layout information and text line information of the image data; a restoration module configured to restore the document layout of the document to be restored based on the coarse-grained layout information, the fine-grained layout information and the text line information of the image data to obtain a layout restoration result; The detection module comprises: a feature extraction submodule configured to input the image data into a pre-trained deep learning model, and extract multiple levels of image features of the image data through a feature extraction network of the deep learning model; a feature fusion submodule configured to perform feature fusion on the multiple levels of image features through a feature fusion network of the deep learning model to obtain fused features; a coarse-grained layout detection submodule configured to analyze the fused features through a coarse-grained layout detection network of the deep learning model to obtain coarse-grained layout information of the image data; the coarse-grained layout information comprises coarse-grained rectangular detection boxes, coarse-grained quadrilateral detection boxes and coarse-grained layout categories; a fine-grained layout detection submodule configured to analyze the fused features through a fine-grained layout detection network of the deep learning model to obtain fine-grained layout information of the image data; the fine-grained layout information comprises fine-grained rectangular detection boxes, fine-grained quadrilateral detection boxes and fine-grained layout categories; a text line layout detection submodule configured to analyze the fused features through a text line detection network of the deep learning model to obtain text line information of the image data; the text line information comprises text line rectangular detection boxes, text line quadrilateral detection boxes and text line layout categories.

14. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, the computer instructions are for causing the computer to perform the method of any one of claims 1-12.

16. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-12.

Images