Document restoration method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a document restoration method and device, electronic equipment, a storage medium and a program product. The invention provides a document restoration method. The document restoration method comprises the following steps: acquiring a target document image; extracting positions of document elements in the target document image and text contents corresponding to the document elements; determining a document element block in the target document image and the position of the document element block; determining an arrangement sequence of the document element blocks based on the positions of the document element blocks; performing preliminary sorting on the positions of the document elements and the text content based on the sorting sequence of the document element blocks to obtain the positions of the document elements subjected to preliminary sorting and the text content subjected to preliminary sorting; performing feature processing on the target document image, the positions of the document elements subjected to the preliminary sorting and the text contents subjected to the preliminary sorting to obtain multi-modal features; and performing reduction based on the multi-modal features to obtain target document content corresponding to the target document image. The invention further provides a document restoration device, electronic equipment, a computer readable storage medium and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a document restoration method, device, electronic device, storage medium, and program product. Background Art

[0002] Currently, document image restoration typically involves first extracting text, tables, illustrations, and other information from the document image through layout analysis and text detection. This information is then sorted using heuristic methods to restore the document image's content. However, these methods suffer from low accuracy when applied to complex documents like those in papers and magazines, which involve double or multiple columns, tables, and illustrations. Summary of the Invention

[0003] In view of this, embodiments of the present disclosure provide a document restoration method, apparatus, electronic device, storage medium, and program product, which can solve or partially solve the above-mentioned problems to a certain extent.

[0004] In some embodiments of the present disclosure, the document restoration method described in the embodiments of the present disclosure may include: obtaining a target document image; extracting the positions of document elements in the target document image and the text content corresponding to the document elements; determining the document element blocks in the target document image and the positions of the document element blocks; determining the arrangement order of the document element blocks based on the positions of the document element blocks; preliminarily sorting the positions of the document elements and the text content based on the arrangement order of the document element blocks to obtain the preliminarily sorted positions of the document elements and the preliminarily sorted text content; performing feature processing on the target document image, the preliminarily sorted positions of the document elements and the preliminarily sorted text content to obtain multimodal features; restoring based on the multimodal features to obtain the target document content corresponding to the target document image; wherein the target document content includes the text content corresponding to the document elements sorted in reading order.

[0005] In some embodiments of the present disclosure, document elements include text lines and non-text; and extracting the positions of document elements in a target document image and the text content corresponding to the document elements includes: performing text line recognition on the target document image using a text line recognition model to obtain the positions of multiple text lines and the text content corresponding to the multiple text lines; performing non-text detection on the target document image using a layout analysis model to obtain the positions of the non-text and the text content corresponding to the non-text.

[0006] In some embodiments of the present disclosure, document element blocks include text blocks and non-text blocks; determining the document element blocks in the target document image and the positions of the document element blocks includes: sorting multiple text lines from top to bottom based on the positions of the text lines, and determining the vertical distance between any two adjacent text lines after sorting; dividing the multiple text lines into one or more candidate text blocks based on the vertical distance between any two adjacent text lines; for the text lines in each candidate text block, determining the horizontal distance between any two left and right adjacent text lines based on the positions of the text lines, and dividing the candidate text block into one or more candidate text sub-blocks based on the horizontal distance between any two left and right adjacent text lines; setting each candidate text sub-block as a text block, and determining the position of the text block based on the position of the candidate text sub-block; setting each non-text as a non-text block, and determining the position of the non-text block based on the position of the non-text.

[0007] In some embodiments of the present disclosure, determining the document element blocks in the target document image and the positions of the document element blocks includes: performing image binarization processing on the target document image to obtain a binarized image corresponding to the target document image; segmenting the target document image in the vertical and horizontal directions based on the binarized image to obtain one or more image blocks and the positions of the image blocks; treating each image block as a document element block, and determining the position of the document element block based on the position of the image block.

[0008] In some embodiments of the present disclosure, a target document image includes a plurality of document element blocks; determining an arrangement order of the document element blocks based on positions of the document element blocks includes: reading a document element block as a target document element block in a top-to-bottom order based on the positions of the plurality of document element blocks, and setting an arrangement order of the target document element blocks according to the reading order; determining whether there are other document element blocks to the right of the target document element block based on the positions of the plurality of document element blocks; in response to determining that there are other document element blocks to the right of the target document element block, reading the document element block to the right of the target document element block as the target document element block, and setting an arrangement order of the target document element block according to the reading order. In response to determining that there are no other document element blocks on the right side of the target document element block, the method further comprises the steps of: determining whether there are other document element blocks below the target document element block based on the positions of the multiple document element blocks; in response to determining that there are other document element blocks below the target document element block, the method further comprises the steps of reading a document element block from top to bottom based on the positions of the multiple document element blocks as the target document element block, and setting the arrangement order of the target document element blocks according to the reading order; and ending in response to determining that there are no other document element blocks below the target document element block.

[0009] In some embodiments of the present disclosure, the positions and textual contents of document elements are preliminarily sorted based on the arrangement order of document element blocks to obtain the preliminarily sorted positions of document elements and the preliminarily sorted textual contents, including: grouping document elements according to the document elements included in the document element blocks to obtain one or more document element groups; sorting the document element groups according to the arrangement order of the document element blocks to obtain the arrangement order of the document element groups; for each sorted document element group, sorting the document elements included therein from top to bottom to obtain the arrangement order of the document elements included in each document element group; determining the arrangement order of the document elements based on the arrangement order of the document element groups and the arrangement order of the document elements included in each document element group; sorting the positions of the document elements and the textual contents corresponding to the document elements based on the arrangement order of the document elements to obtain the preliminarily sorted positions of the document elements and the preliminarily sorted textual contents.

[0010] In some embodiments of the present disclosure, feature processing is performed on the target document image, the positions of the preliminarily sorted document elements, and the preliminarily sorted text content to obtain multimodal features, including: segmenting and encoding the target document image to obtain visual features; determining the layout features and semantic features of the document elements based on the positions of the preliminarily sorted document elements and the preliminarily sorted text content, and adding the layout features and semantic features of the document elements to obtain element features; and splicing the element features and the visual features to obtain multimodal features.

[0011] In some embodiments of the present disclosure, restoring the target document content corresponding to the target document image based on multimodal features includes: inputting the multimodal features into the decoder of the first reading order model, obtaining the text content corresponding to the document elements sorted in the reading order by an autoregressive method, and adding formatting characters to the text content corresponding to the document elements sorted in the reading order to obtain the target document content.

[0012] In some embodiments of the present disclosure, restoring the target document content corresponding to the target document image based on multimodal features includes: inputting the multimodal features into the decoder of the second reading order model to obtain a document element index sequence corresponding to the document elements sorted in the reading order; replacing the document element index in the document element index sequence corresponding to the document elements sorted in the reading order with the text content corresponding to the document element to obtain the target document content.

[0013] In some embodiments of the present disclosure, restoring the target document content corresponding to the target document image based on the multimodal features includes: inputting the multimodal features into the decoder of the third reading order model to obtain a document element index sequence corresponding to the document elements sorted in the reading order after run-length encoding; run-length decoding the document element index sequence corresponding to the document elements sorted in the reading order after run-length encoding to obtain a document element index sequence corresponding to the document elements sorted in the reading order; and replacing the document element index in the document element index sequence corresponding to the document elements sorted in the reading order with the text content corresponding to the document elements to obtain the target document content.

[0014] In some embodiments of the present disclosure, a document element index sequence corresponding to a document element that has been run-length encoded and sorted in a reading order includes: delimiters and a run-length encoding subsequence arranged alternately; wherein the run-length encoding subsequence includes: a document element index and the number of document elements; run-length decoding is performed on the document element index sequence corresponding to the document element that has been run-length encoded and sorted in a reading order to obtain a document element index sequence corresponding to the document element that has been run-length encoded and sorted in a reading order, comprising: sequentially reading the run-length encoding subsequence from the document element index sequence corresponding to the document element that has been run-length encoded and sorted in a reading order based on the delimiter; and respectively for each run-length encoding A subsequence is generated based on the document element index and the number of document elements in the run-length encoding subsequence, and a document element index subsequence corresponding to the run-length encoding subsequence is generated; wherein the document element index subsequence includes one or more document element indexes that increase in sequence; the first document element index of the document element index subsequence is the document element index in the run-length encoding subsequence; the number of document element indexes included in the document element index subsequence is equal to the number of document elements in the run-length encoding subsequence; the document element index subsequence is used to replace the run-length encoding subsequence corresponding to the document element index subsequence, and a document element index sequence corresponding to the document elements sorted in reading order is obtained.

[0015] Corresponding to the above method, an embodiment of the present disclosure further discloses a document restoration device, comprising:

[0016] An image acquisition module, used for acquiring a target document image;

[0017] A document element extraction module is used to extract the position of the document element in the target document image and the text content corresponding to the document element;

[0018] A document element block extraction module is used to determine the document element blocks in the target document image and the positions of the document element blocks;

[0019] A document element block sorting module, used to determine the order of arrangement of the document element blocks based on the positions of the document element blocks;

[0020] A document element sorting module, configured to preliminarily sort the positions and text contents of the document elements based on the arrangement order of the document element blocks, and obtain the preliminarily sorted positions of the document elements and the preliminarily sorted text contents;

[0021] A feature processing module is used to perform feature processing on the target document image, the positions of the preliminarily sorted document elements, and the preliminarily sorted text content to obtain multimodal features;

[0022] The restoration module is used to restore the target document content corresponding to the target document image based on the multimodal features; wherein the target document content includes text content corresponding to the document elements sorted in reading order.

[0023] In addition, an embodiment of the present disclosure further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned document restoration method when executing the computer program.

[0024] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions for causing a computer to execute the above-mentioned document restoration method.

[0025] An embodiment of the present disclosure further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the above-mentioned document restoration method.

[0026] It can be seen from this that the above-mentioned document restoration method, device, electronic device, storage medium and program product can extract the position and text content of the document elements in the target document image; and determine the position and arrangement order of the document element blocks in the target document image; then, preliminarily sort the text content based on the arrangement order of the document element blocks; then, perform feature processing on the target document image, the position of the document elements and the preliminarily sorted text content to obtain multimodal features; finally, restore the multimodal features to obtain the document content. The above-mentioned document restoration method realizes the use of multimodal features to restore the document content corresponding to the image in reading order, especially when complex scenes are involved, which can greatly improve the accuracy of document restoration. In addition, by first preliminarily sorting the extracted text content based on the arrangement order of the document element blocks, and then generating multimodal features based on the preliminarily sorted text content, the complexity of using multimodal features to restore the document content corresponding to the image in reading order can be reduced, further effectively improving the accuracy and efficiency of document restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.

[0029] Figure 2 A schematic diagram of the implementation flow of the document restoration method provided in some embodiments of the present disclosure.

[0030] Figure 3 A schematic diagram of a target document image provided for some embodiments of the present disclosure.

[0031] Figure 4 A schematic diagram of another target document image provided in some embodiments of the present disclosure.

[0032] Figure 5 A schematic diagram of yet another target document image provided for some embodiments of the present disclosure.

[0033] Figure 6 A schematic diagram of a target document image provided in some embodiments of the present disclosure.

[0034] Figure 7 The present invention is a schematic diagram of an implementation flow of determining a document element block and a position of the document element block in a target document image in some embodiments of the present disclosure.

[0035] Figure 8A The present invention is a schematic diagram of an implementation flow of determining a document element block and a position of the document element block in a target document image in some other embodiments of the present disclosure.

[0036] Figure 8B A schematic diagram showing an exemplary binarized image is shown.

[0037] Figure 8C Shows the Figure 8B Schematic diagram of the vertical projection histogram obtained by projecting the binary image in the vertical direction.

[0038] Figure 8D Shows the Figure 8B Schematic diagram of the horizontal projection histogram obtained by projecting the binary image in the horizontal direction.

[0039] Figure 9 A schematic diagram of a target document image provided in some embodiments of the present disclosure.

[0040] Figure 10 This is a schematic diagram of an implementation flow of determining the arrangement order of document element blocks based on the positions of the document element blocks according to an embodiment of the present disclosure.

[0041] Figure 11 A schematic diagram of a target document image provided in some embodiments of the present disclosure.

[0042] Figure 12 This is a schematic diagram of an implementation flow of preliminarily sorting the positions and text contents of document elements based on the arrangement order of document element blocks in an embodiment of the present disclosure to obtain the preliminarily sorted positions of document elements and preliminarily sorted text contents.

[0043] Figure 13 The present invention provides an implementation process of a method for performing feature processing on a target document image, positions of preliminarily sorted document elements, and preliminarily sorted text content to obtain multimodal features, as provided in some embodiments of the present invention.

[0044] Figure 14 A schematic diagram of target document content provided in some embodiments of the present disclosure.

[0045] Figure 15 The internal structure of the document restoration device described in some embodiments of the present disclosure is shown.

[0046] Figure 16 A more specific schematic diagram of the hardware structure of an electronic device described in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0048] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0049] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0050] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0051] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0052] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0053] For the sake of clarity in description, before describing the specific technical solutions of the embodiments of the present disclosure, the technical terms involved in the embodiments of the present disclosure are first explained.

[0054] Optical Character Recognition (OCR) refers to the process by which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shapes by detecting dark and light patterns, and then uses character recognition methods to translate the shapes into computer text.

[0055] Deep learning specifically refers to machine learning based on deep neural network models and methods. Deep learning is developed based on statistical machine learning, artificial neural networks, and other algorithmic models, combined with the development of modern big data and massive computing power.

[0056] Image binarization can refer to the process of converting a color or grayscale image into an image with only two colors (usually black and white). After the image is binarized, the grayscale value of each pixel in the image is only 0 and 255, representing black and white respectively.

[0057] Image projection may refer to projecting an image in a certain direction. A projection histogram may refer to a histogram obtained by projecting an image in a predetermined direction (e.g., vertically or horizontally). A vertical projection histogram may refer to a histogram obtained by projecting an image in a vertical direction (or vertical direction), and a horizontal projection histogram may refer to a histogram obtained by projecting an image in a horizontal direction. These projection histograms represent the number of pixels belonging to an object in each column or row of the image.

[0058] Run-Length Encoding (RLE) is a basic data compression algorithm. Its core concept is to achieve compression by reducing the storage of duplicate data. Run-length encoding is particularly well-suited for processing data containing a large number of consecutive repeating elements, such as solid color areas in image files and repeated characters in text files. The basic principle of run-length encoding is to group consecutive identical elements in a data sequence into segments, called runs. Each run consists of two parts: a run value and a run length. For example, in the string "AAAABBBCCDAA", "AAAA" is a run with a run value of "A" and a length of 4; while "BB" is another run with a run value of "B" and a length of 2. Run-length encoding can represent these runs as a run value + run length. For example, the string "AAAABBBCCDAA" can be encoded as "A4B3C2D1A2", thereby reducing the storage of duplicate data.

[0059] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.

[0060] like Figure 1 As shown, system 100 may include terminal device 102, terminal device 104, and server 106. A medium (e.g., a network) providing a communication link may be included between terminal device 102, terminal device 104, and server 106. The network may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0061] Exemplarily, an application (APP) or software that can implement document restoration can be installed on the terminal device 102 and the terminal device 104. The terminal device 102 and the terminal device 104 here can be hardware or software. When the terminal device 102 and the terminal device 104 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, laptop computers (Laptops) and desktop computers (PCs), etc. When the terminal device 102 and the terminal device 104 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0062] The server 106 may be a server that provides document restoration services, such as a background server that provides support for applications or software displayed on the terminal devices 102 and 104. The server 106 here may also be hardware or software. When the server 106 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 106 is software, it may be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is given here.

[0063] It should be understood that Figure 1 The numbers of terminal devices, users and servers in the embodiment are merely illustrative. Any number of terminal devices, users and servers may be used as required.

[0064] As an example scenario, server 106 can provide a document restoration service. User 112 can use a document restoration application on terminal device 102 to send a target document image to server 106. Server 106 can then perform document restoration based on the target document image and obtain the target document content corresponding to the target document image. Server 106 can then provide the obtained target document content to user 112 via terminal device 102. It should be noted that the target document content includes the text content in the target document image sorted in reading order.

[0065] In another exemplary scenario, the document restoration application downloaded and installed by the terminal device 104 from the server 106 may support offline document restoration services. In this case, the user 114 can directly use the document restoration application on the terminal device 104 to restore the submitted target document image into the target document content. In other words, the specific document restoration process can be completed independently and offline by the terminal device 104 without the real-time participation of the server 106.

[0066] Based on the above system 100, in order to solve the problem of low accuracy of document restoration in related technologies, the present disclosure provides a document restoration method. The document restoration method provided by the present disclosure is described below with reference to specific embodiments and drawings.

[0067] Figure 2 The document restoration method provided in some embodiments of the present disclosure is provided as a flowchart. The document restoration method can be executed by a document restoration device, wherein the document restoration device can be implemented in software and / or hardware, for example, by the server 106 or the terminal device 102. The document restoration device can generally be integrated into an electronic device. Figure 2 As shown, the above document restoration method may specifically include the following steps.

[0068] In step 210, a target document image is acquired.

[0069] In step 220 , the positions of the document elements in the target document image and the text contents corresponding to the document elements are extracted.

[0070] In step 230 , the document element blocks and the positions of the document element blocks in the target document image are determined.

[0071] In step 240 , the arrangement order of the document element blocks is determined based on the positions of the document element blocks.

[0072] In step 250 , the positions and text contents of the document elements are preliminarily sorted based on the arrangement order of the document element blocks to obtain the preliminarily sorted positions of the document elements and the preliminarily sorted text contents.

[0073] In step 260 , feature processing is performed on the target document image, the positions of the preliminarily sorted document elements, and the preliminarily sorted text content to obtain multimodal features.

[0074] In step 270 , the target document content corresponding to the target document image is restored based on the multimodal features; wherein the target document content includes text content corresponding to the document elements sorted in reading order.

[0075] Thus, the document restoration method can extract the positions of document elements and the textual content corresponding to the document elements from a target document image, and further determine the positions and arrangement order of the document element blocks therein; then, preliminarily sort the positions of the document elements and the textual content corresponding to the document elements based on the arrangement order of the document element blocks; then, feature processing is performed on the target document image, the preliminarily sorted positions of the document elements, and the preliminarily sorted textual content to obtain multimodal features; and finally, the multimodal features are restored to obtain the document content. The document restoration method achieves the goal of restoring the document content corresponding to the image in reading order using multimodal features, which can greatly improve the accuracy of document restoration, especially when complex scenarios such as multi-column layouts and charts are involved. In addition, by preliminarily sorting the extracted positions of the document elements and the textual content corresponding to the document elements based on the arrangement order of the document element blocks, and then generating multimodal features based on the preliminarily sorted positions of the document elements and the textual content corresponding to the document elements, the complexity of restoring the document content corresponding to the image in reading order using multimodal features can be reduced, further effectively improving the accuracy and processing efficiency of document restoration.

[0076] The specific implementation of each step of the above document restoration method will be further described in detail below with reference to the accompanying drawings and specific examples.

[0077] With respect to step 210 above, the target document image may be any image including a document. In the embodiments of the present disclosure, the document may include: text, images, tables, and other document contents. For example, the target document image may be an image obtained by photographing a paper. It should be noted that in the embodiments of the present disclosure, the number of the target document images may be one or more, and the specific number may be set according to actual circumstances. In other words, the embodiments of the present disclosure do not limit the number of target document images.

[0078] Furthermore, typically, the document restoration device can obtain a target document image input by a user. However, the embodiments of the present disclosure do not limit the source of the target document image. For example, the target document image can be an image captured by the user in real time, or an image obtained by the user from the Internet.

[0079] For example, Figure 3 A schematic diagram of a target document image provided in some embodiments of the present disclosure. Figure 3 As shown in FIG, an exemplary target document image 300 is shown. As an example only, the target document image includes the first page of a paper.

[0080] Regarding step 220, the document elements may be a general term for the document content included in the target document image. In embodiments of the present disclosure, the document elements may include text lines and non-text. A text line may be a unit consisting of one or more consecutive characters. Characters may include at least one of Chinese characters, letters, numbers, punctuation marks, and spaces. Generally, a text line is a single line of text. Non-text may include images and / or tables. It is understood that non-text is optional within the document elements of the target document image, meaning that the document elements of the target document image may or may not include non-text. The location of the document element may include the coordinates of the four vertices of the region where the document element resides in the target document image, for example, the coordinates of the top left corner of the region where the document element resides, the coordinates of the bottom left corner of the region where the document element resides, the coordinates of the top right corner of the region where the document element resides, and the coordinates of the bottom right corner of the region where the document element resides. The coordinates are typically a two-dimensional array consisting of a horizontal coordinate and a vertical coordinate. In some embodiments, the coordinates may be pixel coordinates. That is, a rectangular coordinate system uv is established with the top left corner of the target document image as the origin, using pixels as the unit. The horizontal coordinate u and vertical coordinate v of a pixel represent the column number and row number of the pixel in the image array, respectively. The text content of the document element can be one or more texts included in the document element in the target document image. In embodiments of the present disclosure, the number of document elements in the target document image can be one or more.

[0081] Specifically, in some embodiments of the present disclosure, after acquiring a target document image, the document restoration device may utilize a recognition algorithm to identify the acquired target document image, thereby determining the position of text lines and the textual content corresponding to the text lines, as well as the position of non-text and the textual content corresponding to the non-text. The above-mentioned recognition algorithm can be configured based on actual circumstances and typically includes an algorithm for recognizing text lines and an algorithm for recognizing non-text. Optionally, the specific form of the position of document elements and textual content is not limited.

[0082] In some embodiments of the present disclosure, the method of extracting the positions of document elements in the target document image and the text content corresponding to the document elements as described in the above step 220 may include: using a text line recognition model to perform text line recognition on the target document image to obtain the positions of multiple text lines and the text content corresponding to the multiple text lines; and using a layout analysis model to perform non-text detection on the target document image to obtain the positions of non-text and the text content corresponding to the non-text.

[0083] In some embodiments, the text line recognition model can be a model for identifying the position and content of text lines in an image. For example, the text line recognition model can include an OCR or real-time scene text detection deep learning model. The layout analysis model can be a model for detecting non-text such as tables and / or images in an image and extracting the textual content contained therein.

[0084] In some embodiments of the present disclosure, the recognition result output by the above-mentioned text line recognition model may include multiple records, wherein each record corresponds to a text line, and is used to record the position of the text line and the text content corresponding to the text line. Specifically, a record output by the above-mentioned text line recognition model may adopt the following format: text line number, x1, y1, x2, y2, x3, y3, x4, y4, text content. Among them, the text line number can be the number of the text line recognized by the text line recognition model; (x1, y1) can be the coordinates of the upper left corner vertex of the text line; (x2, y2) can be the coordinates of the upper right corner vertex of the text line; (x3, y3) can be the coordinates of the lower right corner vertex of the text line; (x4, y4) can be the coordinates of the lower left corner vertex of the text line; and the text content can be the recognized text content corresponding to the text line. It can be understood that, under normal circumstances, among the coordinates of the above four vertices, the horizontal coordinates of the upper left vertex and the lower left vertex can be the same; the horizontal coordinates of the upper right vertex and the lower right vertex can be the same; the horizontal coordinates of the upper left vertex and the upper right vertex can be the same; and the horizontal coordinates of the lower left vertex and the lower right vertex can be the same. It should be noted that the format of the recognition result output by the text line recognition model is only an example, and other formats can also be used. The embodiments of the present disclosure do not limit this. As mentioned above, the above coordinates can be pixel coordinates.

[0085] In some embodiments of the present disclosure, the recognition result output by the layout analysis model may also include one or more records, wherein each record corresponds to a non-text and is used to record the location of the non-text and the text content corresponding to the non-text. In particular, it can be understood that when the target document image does not include non-text, the recognition result output by the layout analysis model may also be empty. Specifically, a record output by the above-mentioned layout analysis model may adopt the following format: non-text serial number, x5, y5, x6, y6, x7, y7, x8, y8, text content. Among them, the non-text serial number can be the serial number of the non-text recognized by the layout analysis model; (x5, y5) can be the coordinates of the upper left corner vertex of the non-text; (x6, y6) can be the coordinates of the upper right corner vertex of the non-text; (x7, y7) can be the coordinates of the lower right corner vertex of the non-text; (x8, y8) can be the coordinates of the lower left corner vertex of the non-text; and the text content can be the type of the recognized non-text (for example, image or table) and / or the text content corresponding to the recognized non-text. It is also understood that, under normal circumstances, among the coordinates of the above four vertices, the horizontal coordinates of the upper left vertex and the lower left vertex can be the same; the horizontal coordinates of the upper right vertex and the lower right vertex can be the same; the horizontal coordinates of the upper left vertex and the upper right vertex can be the same; and the horizontal coordinates of the lower left vertex and the lower right vertex can be the same. It should be noted that the format of the recognition result output by the layout analysis model is only an example, and other formats can also be used. The embodiments of the present disclosure do not limit this. As mentioned above, the above coordinates can also be pixel coordinates.

[0086] For example, Figure 4 Schematic diagram of another target document image provided by some embodiments of the present disclosure. Figure 4 As shown in the figure, a target document image 400 including only document elements of text lines is shown. In the figure, each text line is selected by a rectangular frame, indicating that its position and text content can be identified.

[0087] For example, Figure 5 Schematic diagram of another target document image provided by some embodiments of the present disclosure. Figure 5 As shown, the figure shows a target document image 500 including a text line and two images. In the figure, each text line and each image are framed by a rectangular frame, indicating that their position and text content can be recognized.

[0088] It should be noted that the positions of the document elements and the textual content corresponding to the document elements in the target document image determined and outputted by the above method are typically sorted directly based on the positions of the document elements in a top-to-bottom and left-to-right order. Thus, when the target document image is arranged in columns (e.g., a double-column or multi-column layout), the arrangement order of the document elements is typically different from the reading order, and the difference is significant.

[0089] For example, Figure 6 Schematic diagram of a target document image provided by some embodiments of the present disclosure. Figure 6 As shown, the figure shows a target document image 600 consisting solely of text lines. Each text line is framed by a rectangular box, indicating that its position and text content can be identified. Furthermore, the numbers listed in the upper left corner of each text line in the figure represent the order of the text lines output by the text line recognition model (e.g., the text line sequence numbers described above). As can be seen, since the main text of the article in target document image 600 is in a double-column layout, the order of the text lines described above is different from the order in which they are read.

[0090] Regarding step 230, the document element block may be a collection of adjacent document elements of the same type, wherein the spacing between different document element blocks is typically greater than a preset document element block spacing threshold. For example, for a target document image containing text, different portions of the text or different columns typically have large spacing between them, and thus are generally divided into different document element blocks. For a target document image containing both text and non-text, the spacing between the text and non-text is also typically large, and thus are generally divided into different document element blocks. In embodiments of the present disclosure, the document element blocks may include text blocks and non-text blocks. Specifically, the text block may be a collection of vertically adjacent text lines. The text lines within the same text block typically have approximately the same horizontal position, and the spacing between vertically adjacent text lines is typically less than or equal to the preset document element block spacing threshold. Typically, for documents whose page setup involves columnar typesetting, text lines located in different columns on the left and right sides of the document typically belong to different text blocks. For non-text, in some embodiments of the present disclosure, each non-text block can typically be directly designated as a separate non-text block. That is, a non-text block may include only one non-text item, such as an image or table. Of course, as an alternative to the above solution, a non-text block may also be a collection of adjacent non-text items, wherein the non-text items within the same non-text block typically have the same horizontal position and the spacing between adjacent non-text items is typically less than or equal to the aforementioned pre-set document element block spacing threshold. However, in actual documents, at least textual content describing the title or content of the non-text items is typically included between non-text items. Therefore, multiple non-text items typically do not appear consecutively within a document. Therefore, in the subsequent embodiments of this disclosure, the implementation of individually defining a non-text item as a non-text block is used as a representative example. Furthermore, it should be noted that in some embodiments of this disclosure, the aforementioned document element block spacing threshold may include: a document element block horizontal spacing threshold and a document element block vertical spacing threshold. Specifically, the document element block horizontal spacing threshold may be used to measure the horizontal spacing of document elements, while the document element block vertical spacing threshold may be used to measure the vertical spacing of document elements. The document element block horizontal spacing threshold and the document element block vertical spacing threshold may be the same or different.

[0091] Figure 7 FIG. 1 is a schematic diagram of an implementation flow of determining a document element block and a position of the document element block in a target document image in some embodiments of the present disclosure. Figure 7 As shown, in some embodiments of the present disclosure, a method for determining a document element block and a position of the document element block in a target document image may include the following steps.

[0092] In step 710 , a plurality of text lines are sorted from top to bottom based on the positions of the text lines, and the vertical distance between any two text lines adjacent to each other after sorting is determined.

[0093] In the embodiment of the present disclosure, the text lines can be sorted from top to bottom according to the vertical coordinate of any one of the four vertices of the text line. Generally, if pixel coordinates are used, a smaller vertical coordinate represents a higher vertical position. In other words, in the above steps, the text lines can be sorted in ascending order according to the vertical coordinate of any one of the four vertices of the text line.

[0094] In an embodiment of the present disclosure, the vertical distance between the two adjacent text lines can be the difference between the vertical coordinates of the upper left corner vertex or the upper right corner vertex of the next text line and the vertical coordinates of the lower left corner vertex or the lower right corner vertex of the previous text line.

[0095] In step 720 , the plurality of text lines are divided into one or more candidate text blocks based on the vertical distance between any two upper and lower adjacent text lines.

[0096] In an embodiment of the present disclosure, the upper and lower consecutive text lines (that is, no non-text is included in between) can be preliminarily divided into a candidate text block according to the position of the text lines. For each candidate text block, in response to determining that the vertical distance between two upper and lower adjacent text lines in the candidate text block is greater than a preset document element block spacing threshold, the candidate text block is divided into two upper and lower candidate text blocks with these two text lines as the boundary. The above steps are repeated until the vertical distance between any two adjacent text lines in all candidate text blocks is less than or equal to the preset document element block spacing threshold. It should be noted that, assuming that the preset document element block spacing threshold includes: a document element block horizontal spacing threshold and a document element block vertical spacing threshold, the preset document element block spacing threshold described in the above steps can actually be a document element block vertical spacing threshold.

[0097] In step 730, for each text line in a candidate text block, the horizontal distance between any two text lines adjacent to the left and right is determined based on the position of the text line, and the candidate text block is divided into one or more candidate text sub-blocks based on the horizontal distance between any two text lines adjacent to the left and right.

[0098] In an embodiment of the present disclosure, the horizontal distance between the two adjacent text lines on the left and right can be the difference between the horizontal coordinates of the upper left corner vertex or the lower left corner vertex of the right text line and the horizontal coordinates of the upper right corner vertex or the lower right corner vertex of the left text line.

[0099] For each candidate text block, in response to determining that the horizontal distance between two left and right adjacent text lines in the candidate text block is greater than a pre-set document element block spacing threshold, the candidate text block is divided into two left and right candidate text sub-blocks with the two text lines as the boundary. The above steps are repeated until the vertical distance between any two left and right adjacent text lines in all candidate text blocks and candidate text sub-blocks is less than or equal to the pre-set document element block spacing threshold. Moreover, in response to determining that the candidate text block does not include left and right adjacent text lines or the horizontal distance between two left and right adjacent text lines is less than or equal to the pre-set document element block spacing threshold, the candidate text block can be directly regarded as a candidate text sub-block. It should be noted that, assuming that the pre-set document element block spacing threshold includes: a document element block horizontal spacing threshold and a document element block vertical spacing threshold, the pre-set document element block spacing threshold described in the above steps can actually be a document element block horizontal spacing threshold.

[0100] In step 740 , each candidate text sub-block is set as a text block, and the position of the text block is determined based on the position of the candidate text sub-block.

[0101] In an embodiment of the present disclosure, the position of the candidate text sub-block may be represented by the coordinates of the four vertices of the candidate text sub-block; the position of the text block may be represented by the coordinates of the four vertices of the text block. Among them, as mentioned above, the above coordinates can be pixel coordinates. At this time, among the four vertices of the above-mentioned candidate text sub-blocks, the horizontal coordinate of the upper left vertex can be the minimum value of the horizontal coordinate among the four vertex coordinates of all text lines contained in the candidate text sub-block; the vertical coordinate of the upper left vertex can be the minimum value of the vertical coordinate among the four vertex coordinates of all text lines contained in the candidate text sub-block; the horizontal coordinate of the upper right vertex can be the maximum value of the horizontal coordinate among the four vertex coordinates of all text lines contained in the candidate text sub-block; the vertical coordinate of the upper right vertex can be the minimum value of the vertical coordinate among the four vertex coordinates of all text lines contained in the candidate text sub-block; the horizontal coordinate of the lower left vertex can be the minimum value of the horizontal coordinate among the four vertex coordinates of all text lines contained in the candidate text sub-block; the vertical coordinate of the lower left vertex can be the maximum value of the vertical coordinate among the four vertex coordinates of all text lines contained in the candidate text sub-block; the horizontal coordinate of the lower right vertex can be the maximum value of the horizontal coordinate among the four vertex coordinates of all text lines contained in the candidate text sub-block; and the vertical coordinate of the lower right vertex can be the maximum value of the vertical coordinate among the four vertex coordinates of all text lines contained in the candidate text sub-block. Thus, in the above steps, the coordinates of the four vertices of each candidate text sub-block can be respectively set as the coordinates of the corresponding vertices of its corresponding text block, thereby determining the position of the text block.

[0102] In step 750 , each non-text is set as a non-text block, and the position of the non-text block is determined based on the position of the non-text.

[0103] In the above step 750 , the position of each non-text may be used as the position of its corresponding text block.

[0104] As can be seen, the above method can utilize the positions of the document elements output in the previous step to aggregate the document elements identified from the target document image into one or more document element blocks based on their arrangement in the target document image and the spacing between the document elements. It will be appreciated that for documents using columnar layout, text lines in different columns will belong to different document element blocks. Therefore, the arrangement order of the text lines within the above document element blocks can generally be the same as the reading order.

[0105] In addition to the above-mentioned method of determining the document element blocks and the positions of the document element blocks in the target document image using the positions of the document elements output in the previous step, other image processing methods can also be used to determine the document element blocks and the positions of the document element blocks in the target document image.

[0106] Figure 8A FIG. 1 is a schematic diagram of an implementation flow of determining a document element block and a position of the document element block in another embodiment of the present disclosure. Figure 8A As shown, in some embodiments of the present disclosure, a method for determining a document element block and a position of the document element block in a target document image may include the following steps.

[0107] In step 810 , the target document image is subjected to image binarization processing to obtain a binarized image corresponding to the target document image.

[0108] It will be appreciated that, in the embodiments of the present disclosure, prior to performing image binarization processing on the target document image, preprocessing such as denoising may also be performed on the target document image. Furthermore, for color target document images, preprocessing may also typically include converting the color target document image into a grayscale image. The embodiments of the present disclosure do not limit the specific procedures for the aforementioned preprocessing operations or the specific procedures for implementing the image binarization processing.

[0109] In step 820 , the target document image is segmented in the vertical direction and the horizontal direction based on the binarized image to obtain one or more image blocks and positions of the image blocks.

[0110] Specifically, in some embodiments of the present disclosure, segmenting the target document image in the vertical direction and the horizontal direction based on the binarized image described in step 820 may specifically include the following processes.

[0111] First, the binary image is projected in the horizontal direction to obtain a horizontal projection histogram of the binary image.

[0112] For example, Figure 8B A binary image is shown. By projecting the binary image in the horizontal direction, the horizontal projection histogram of the binary image can be obtained, as shown in FIG. Figure 8C As shown. Figure 8B and Figure 8C The provided image projection method can help us understand how to obtain the horizontal projection histogram of this step.

[0113] Then, the binary image is segmented based on the horizontal projection histogram to obtain one or more candidate image blocks.

[0114] In an embodiment of the present disclosure, the above-mentioned segmentation of the binary image based on the horizontal projection histogram may include: determining, based on the horizontal projection histogram, a segmentation position where the width of the continuous blank area of the target document image in the vertical direction is greater than a preset document element block spacing threshold; for example, a position between different parts of the document; and segmenting the binary image at the determined segmentation position. It can be seen that such segmentation is to segment the target document image from top to bottom along the vertical direction, and multiple candidate image blocks adjacent to each other can be obtained. Reference Figure 8B and Figure 8C As shown, when the value of a certain row of the target document image in the horizontal projection histogram is less than a quantity threshold (for example, 1, 2, 3, etc.), the row can be considered to belong to a blank row in the image. And when the values of multiple consecutive rows corresponding to the target document image in the horizontal projection histogram are all less than the quantity threshold, the areas in the target document image corresponding to these rows can be considered to belong to continuous blank areas. When the width of the continuous blank area is greater than the document element block spacing threshold, any position of the continuous blank area can be used as the segmentation position (involving the vertical coordinate of the segmentation position). It should be noted that, assuming that the pre-set document element block spacing threshold includes: a document element block horizontal spacing threshold and a document element block vertical spacing threshold, the pre-set document element block spacing threshold described in the above steps can actually be a document element block vertical spacing threshold.

[0115] Next, each candidate image block is projected in the vertical direction to obtain a vertical projection histogram of each candidate image block.

[0116] For example, Figure 8B A binary image is shown. By projecting the binary image in the vertical direction (or perpendicular direction), a vertical projection histogram of the binary image can be obtained, as shown in FIG. Figure 8D As shown. Figure 8B and Figure 8DThe image projection method provided can help us understand how to obtain the vertical projection histogram of this step.

[0117] Finally, each candidate image block is segmented based on the vertical projection histogram to obtain one or more image blocks.

[0118] In an embodiment of the present disclosure, segmenting each candidate image block based on a vertical projection histogram may include: performing the following operations on each candidate image block: determining a segmentation position where the width of a continuous blank area in the horizontal direction of the image block is greater than a preset document element block spacing threshold based on the vertical projection histogram; for example, the position of a column when the document adopts a multi-column layout; and segmenting the image block at the determined segmentation position. It can be seen that such segmentation is to segment the candidate image block from left to right along the horizontal direction, and multiple image blocks adjacent to each other on the left and right can be obtained. Reference Figure 8B and Figure 8D As shown, when the value of a column corresponding to the candidate image block in the vertical projection histogram is less than the quantity threshold (for example, 1, 2, 3, etc.), the column can be considered to belong to the blank column in the candidate image block. And when the values of multiple consecutive columns corresponding to the candidate image block in the vertical projection histogram are all less than the quantity threshold, the areas in the candidate image blocks corresponding to these columns can be considered to belong to continuous blank areas. When the width of the continuous blank area is greater than the document element block spacing threshold, any position of the continuous blank area can be used as the segmentation position (involving the horizontal coordinate of the segmentation position). It should be noted that, assuming that the pre-set document element block spacing threshold includes: a document element block horizontal spacing threshold and a document element block vertical spacing threshold, the pre-set document element block spacing threshold described in the above steps can actually be a document element block horizontal spacing threshold.

[0119] Furthermore, after respectively determining the vertical coordinates and horizontal coordinates of the segmentation positions of each segmentation, the coordinates of the four vertices of each image block can be determined, that is, the position of each image block can be determined.

[0120] In step 830, each image block is treated as a document element block, and the position of the document element block is determined based on the position of the image block.

[0121] After each image block is set as a document element block, the position of the document element block can be represented by the coordinates of the four vertices of the document element block, which are the coordinates of the four vertices of the corresponding image block.

[0122] For example, Figure 9 Schematic diagram of a target document image provided by some embodiments of the present disclosure. Figure 9As shown in FIG, a target document image 900 including only text lines is shown. In the figure, each text block is selected by a rectangular frame, indicating that its position and text content can be identified. Figure 9 As shown, the target document image 900 may include multiple text blocks or document element blocks. As can be seen, the arrangement order of text lines within each text block is usually the same as the reading order.

[0123] Regarding the above step 240, in an embodiment of the present disclosure, the arrangement order of the above document element blocks may be substantially the same as the reading order. Specifically, Figure 10 This is a schematic diagram of the implementation process of determining the arrangement order of document element blocks based on the positions of document element blocks according to an embodiment of the present disclosure. Figure 10 As shown, determining the arrangement order of document element blocks based on the positions of the document element blocks may include the following steps.

[0124] In step 1010 , based on the positions of the plurality of document element blocks, a document element block is read in order from top to bottom as a target document element block, and an arrangement order of the target document element block is set according to the reading order.

[0125] In the embodiments of the present disclosure, the reading order can be directly set to the arrangement order of the target document element blocks. For example, for the first document element block read, the arrangement order of the document element block can be 1; and for the nth document element block read, the arrangement order of the document element block can be n, where n is a positive integer.

[0126] In step 1020, based on the positions of multiple document element blocks, it is determined whether there are other document element blocks to the right of the target document element block. In response to determining that there are other document element blocks to the right of the target document element block, step 1030 is executed; in response to determining that there are no other document element blocks to the right of the target document element block, step 1040 is executed.

[0127] In step 1030 , the document element block to the right of the target document element block is read as the target document element block, and the arrangement order of the target document element block is set according to the reading order; then, the process returns to step 1020 .

[0128] In step 1040, based on the positions of multiple document element blocks, determine whether there are other document element blocks below the target document element block. In response to determining that there are other document element blocks below the target document element block, execute step 1010; in response to determining that there are no other document element blocks below the target document element block, end.

[0129] From this, it can be seen that according to the above method, the document element blocks will be sorted in order from top to bottom, and when encountering a multi-column layout, the document layout can be further sorted in the order of left first and then right in this column part. In this way, the document element blocks can be roughly initially sorted in the reading order, so that the order of the document element blocks after sorting is basically consistent with the reading order.

[0130] For example, Figure 11 Schematic diagram of a target document image provided by some embodiments of the present disclosure. Figure 11 As shown, the figure shows a target document image 1100 consisting only of text lines. Each text block is enclosed in a rectangular box, indicating that its position and text content can be identified. The number in the upper left corner of each text block represents the order in which the text blocks are arranged. As can be seen, after the above sorting process, the resulting order of the text blocks is essentially consistent with the reading order.

[0131] Regarding step 250, in an embodiment of the present disclosure, Figure 12 This is a schematic diagram of the implementation process of performing a preliminary sorting of the positions and text contents of document elements based on the arrangement order of document element blocks according to the embodiment of the present disclosure, and obtaining the positions of the document elements and the text contents that have been preliminarily sorted. Figure 12 As shown, the method of preliminarily sorting the positions and text contents of document elements based on the arrangement order of document element blocks to obtain the preliminarily sorted positions of document elements and preliminarily sorted text contents includes the following steps.

[0132] In step 1210, document elements are grouped according to the document elements included in the document element block to obtain one or more document element groups.

[0133] For the solution of determining document element blocks by using the positions of document elements, document element blocks are composed of combinations of document elements. Therefore, the document elements included in each document element block can be directly determined, thereby obtaining one or more document element groups. For the solution of determining document element blocks by image processing, the document elements included in the document element block can be determined based on the positions of the document element blocks and the positions of the document elements. Specifically, for each document element, the document element block to which it belongs can be determined by determining which document element block's position range the document element falls within. After determining the document element block to which each document element belongs, the document elements can be directly grouped according to the document elements included in the document element block, thereby obtaining one or more document element groups.

[0134] In step 1220, the document element groups are sorted according to the arrangement order of the document element blocks to obtain the arrangement order of the document element groups.

[0135] In step 1230, for each sorted document element group, the document elements included therein are sorted in order from top to bottom to obtain the arrangement order of the document elements included in each document element group.

[0136] At step 1240 , the arrangement order of the document elements is determined based on the arrangement order of the document element groups and the arrangement order of the document elements included in each document element group.

[0137] In step 1250 , the positions of the document elements and the text contents corresponding to the document elements are sorted based on the arrangement order of the document elements to obtain the positions of the document elements and the text contents that have been preliminarily sorted.

[0138] In some embodiments of the present disclosure, the positions of the preliminarily sorted document elements and the preliminarily sorted text contents can be identified using a document element index. The document element index can be represented by several characters such as letters, numbers, and symbols. For example, the document element index can usually be represented by<idx_n> The format is . Among them, idx represents the meaning of index, and n is the serial number, which can be counted from 0. The serial number size usually reflects the position of the document element and the arrangement order of the text content. For example, "<idx_1> " can represent the second document element after preliminary sorting. Based on this, the position of the preliminary sorted document elements and the preliminary sorted text content obtained by the above method can be recorded in the following format, which is only an example:

[0139] <idx_0> , the position of the document element 1, the text content corresponding to the document element 1

[0140] <idx_1> , the position of the document element 2, the text content corresponding to the document element 2

[0141] <idx_2> , the position of the document element 3, the text content corresponding to the document element 3

[0142] …

[0143] As can be seen from this, the above method can be used to perform a preliminary sorting of document elements based on the order of the document element blocks, thereby obtaining the positions of the preliminary sorted document elements and the preliminary sorted text content. Furthermore, the positions and text content of document elements at the same sort position should correspond to the same document element and be identified by the same document element index. Furthermore, as can be seen from the above example, the document element indexes output after preliminary sorting are incremented.

[0144] For step 260, the multimodal features may refer to global features extracted from the target document image, including visual features, semantic features, layout features, and other features. Document restoration using the multimodal features can improve accuracy.

[0145] In an embodiment of the present disclosure, a document restoration device may input a target document image, the positions of preliminarily sorted document elements, and the preliminarily sorted text content into an encoder of a reading order model (wherein the encoder is configured to encode the input data and extract features), and then utilize the encoder of the reading order model to perform feature extraction to obtain the aforementioned multimodal features. The encoder may include a visual branch that uses a convolutional neural network or a visual transformer to extract spatial features such as the layout structure and text area positions of the target document image, and a text branch that uses a pre-trained language model to semantically encode text content lines and simultaneously incorporates position embedding to mark document element positions. It should be noted that, in addition to the encoder, the reading order model may also include a decoder that decodes the input features using an autoregressive approach to obtain content sorted according to the reading order. The decoder may be implemented as a transformer decoder. It is understood that the reading order model may be a model that restores document content from an image using an autoregressive approach. In some embodiments, the reading order model may also be implemented using a large multimodal model, for example, using a pre-trained visual-language large model as the encoder basis and fine-tuning to adapt it to the document reading order task. It should also be noted that, in the specific embodiments below, for different specific implementations, the reading order model may include three models: a first reading order model, a second reading order model, and a third reading order model. Therefore, in the embodiments of the present disclosure, when referring specifically to a specific reading order model, "first", "second", or "third" will be used to distinguish the reading order models, and when referring generally to any one of the above-mentioned first reading order model, second reading order model, and third reading order model, the reading order model will be used to represent any one of the above-mentioned first reading order model, second reading order model, and third reading order model.

[0146] For example, Figure 13 The present invention provides some embodiments of the present invention to provide a method for performing feature processing on the target document image, the positions of the pre-sorted document elements and the pre-sorted text content to obtain multimodal features. Figure 13 As shown, the above method may include the following steps.

[0147] In step 1310, the target document image is segmented and encoded to obtain visual features.

[0148] In an embodiment of the present disclosure, the visual features may be features obtained by extracting features from a visual dimension of a target document image, representing features that can be seen from a user's perspective.

[0149] Specifically, the document restoration device can use the image processing layer in the reading order model to resize the target document image from its original size to a preset size, and then evenly segment the resized target document image to obtain a preset number of image blocks; each image block can be processed by a block encoder to obtain a corresponding block visual feature, and each image block can be processed by a block encoder to obtain a corresponding block position feature at its position in the target document image; then, the block visual feature and block position feature of each image block can be added together to obtain a block feature, and then the preset number of block features can be determined as the visual feature of the target document image. The preset size and preset number can be set according to actual conditions. For example, the preset size can be 224×224, and the preset number can be 49. The target document image can be segmented according to a 32×32 window to obtain 49 image blocks, and then 49 block features can be obtained, which are combined to obtain the visual feature of the target document image.

[0150] In step 1320 , the layout features and semantic features of the document elements are determined based on the positions of the preliminarily sorted document elements and the preliminarily sorted text contents, and the layout features and semantic features of the document elements are added together to obtain element features.

[0151] It should be noted that the arrangement order of the positions of the above-mentioned document elements corresponds to the arrangement order of the text content after preliminary sorting. In the embodiments of the present disclosure, the layout features of the document elements can be the arrangement of the document elements in the target document image, which can include indentation, line spacing, etc. The semantic features of the document elements can be the meaning, context, logic, etc. corresponding to the text content included in the document elements. The element features can be the comprehensive features of the combination of the layout features and semantic features of the document elements in the target document image.

[0152] In some embodiments, determining the layout features and semantic features of a document element based on the positions of the preliminarily sorted document elements and the preliminarily sorted textual content may include: performing linear feature extraction on the positions of the preliminarily sorted document elements to determine the layout features corresponding to the document elements; performing feature extraction on the preliminarily sorted textual content to obtain the semantic features of the document elements. Optionally, performing feature extraction on the preliminarily sorted textual content to obtain the semantic features of the document elements may include: performing word segmentation on the preliminarily sorted textual content to obtain multiple words; encoding the multiple words to obtain multiple word features; and determining a mean of the multiple word features to obtain the semantic features of the document element.

[0153] Specifically, the document restoration device can input the positions of the preliminarily sorted document elements into the linear feature layer of the reading order model, and output the layout features of the document elements; and use a tokenizer to segment the preliminarily sorted text content to obtain multiple words, where a word refers to the smallest unit (token) in the text processing process, which can be a word, letter, number, punctuation mark, etc., and then use an encoder to encode each word to obtain the corresponding word feature, and use a mean pooling layer to mean pool the multiple word features, and determine the mean of the output multiple word features as the semantic feature of the document element. Then, for each document element, its corresponding layout feature and semantic feature are added together to obtain the element feature.

[0154] In step 1330 , the element features and visual features of the document elements are concatenated to obtain multimodal features.

[0155] In an embodiment of the present disclosure, the document restoration device may splice the element features of each document element and multiple block features in the visual features according to a preset dimension. The preset dimension may be set according to actual circumstances, for example, the preset dimension may be 265. If the dimension is insufficient during the splicing process, padding information may be used to fill in the gaps until the preset dimension is reached, and the resulting features are determined as multimodal features.

[0156] With respect to the above step 270 , in an embodiment of the present disclosure, a decoder of a reading order model may be used to restore the target document content corresponding to the target document image based on multimodal features.

[0157] In an embodiment of the present disclosure, the reading order model can be a model that restores document content from an image using an autoregressive approach. As previously mentioned, the reading order model can include an encoder and a decoder. The encoder is used to encode input data, extract features, and input them into the decoder. The decoder is used to decode the input features using an autoregressive approach to obtain content sorted according to the reading order. In an embodiment of the present disclosure, the target document content is the full text obtained by restoring the target document image, including text, images, tables, and other content. Non-text items such as images and tables can be represented using special fields or strings, allowing for customized rendering. The target document content can subsequently be input into various models for further processing. The reading order can refer to the order in which a user reads a document. For example, a document can be read from top to bottom and from left to right. When encountering a document with columnar layout, the content of each column is typically read from left to right. Within each column, the content within the column can then be read from top to bottom.

[0158] In some embodiments, restoring the target document content corresponding to the target document image based on multimodal features may include: inputting the multimodal features into the decoder of the first reading order model, obtaining the text content corresponding to the document elements sorted in the reading order through an autoregressive method, and adding formatting characters to the text content corresponding to the document elements sorted in the reading order to obtain the target document content.

[0159] Among them, the first reading order model can be a model that directly restores the text content corresponding to the document elements of the target document image to obtain the text content sorted in the reading order. The format character can be a symbol that represents the arrangement of the document elements, and the format character can include a line break character, a first line indent, etc. Specifically, the document restoration device can input the multimodal features into the decoder of the first reading order model, and the decoder performs autoregressive decoding processing on the multimodal features according to the reading order to obtain the text content of each document element sorted in the reading order, and adds the format character thereto to obtain the target document content. For example, assuming that the target document image only includes text lines, the reading order of each text line can be represented by xy, where x represents the reading order of the text paragraph where the text line is located, and y represents the reading order of the text line in the corresponding text paragraph. This is only an example.

[0160] In other embodiments, restoring the target document content corresponding to the target document image based on multimodal features may include: inputting the multimodal features into the decoder of the second reading order model to obtain a document element index sequence corresponding to the document elements sorted in the reading order; replacing the document element indexes corresponding to the document elements sorted in the reading order with the text content corresponding to the document elements to obtain the target document content.

[0161] The second reading order model can be a model that can first use the document element index to represent each document element when restoring the text content corresponding to the document element of the target document image, and obtain a document element index sequence sorted in the reading order. Then, the text content can be replaced according to the document element index to obtain the target document content. As mentioned above, the document element index can be an index added by the embodiment of the present disclosure to represent a document element, and can be represented by several characters such as letters, numbers, symbols, etc., for example, by "<idx_n> " is used to identify the nth document element. In the above embodiment, the document element index sequence sorted in reading order output by the second reading order model may be in the following format (only as an example):<idx_0><idx_1><idx_3><idx_5> …. Since the second reading order model outputs document element indexes rather than the text content corresponding to the document elements themselves, compared with the first reading order model, the second reading order model compresses the output data, which can greatly improve the processing speed of the second reading order model and thus improve the efficiency of the second reading order model.

[0162] The document restoration device can input the multimodal features into the decoder of the second reading order model. The decoder can perform autoregressive decoding on the multimodal features according to the reading order to obtain the document element index of each document element sorted according to the reading order, and can further include a format identifier, each format identifier represents a unique format character, for example, <sep>As a separator between paragraphs, it represents a line break; then each document element index output by the decoder of the above-mentioned second reading order model can be replaced with the corresponding text content, and each format identifier can be replaced with the corresponding format character, so as to obtain the target document content.

[0163] In some further embodiments, restoring the target document content corresponding to the target document image based on the multimodal features may include: inputting the multimodal features into the decoder of the third reading order model to obtain a document element index sequence corresponding to the document elements that have been run-length encoded and sorted in the reading order; run-length decoding the document element index sequence corresponding to the document elements that have been run-length encoded and sorted in the reading order to obtain a document element index sequence corresponding to the document elements sorted in the reading order; and replacing the document element index in the document element index sequence corresponding to the document elements sorted in the reading order with the text content corresponding to the document elements to obtain the target document content.

[0164] The third reading order model can be a model that, when restoring the text content corresponding to the document elements of the target document image, first uses the document element index to represent each document element, obtaining a document element index sequence corresponding to the document elements that have been run-length encoded and sorted in reading order, then first performs run-length decoding and then replaces the text content based on the document element index. As mentioned above, the document element index can be an index added in the embodiment of the present disclosure to represent a document element, and can be represented by several characters such as letters, numbers, and symbols, for example, by "<idx_n> " is used to identify the nth document element. In addition, the third reading order model further performs run-length encoding on the document element indexes sorted according to the reading order. Therefore, compared with the second reading order model, the output data of the third reading order model is further compressed, thereby further improving the processing speed and efficiency of the third reading order model.

[0165] In an embodiment of the present disclosure, the run-length encoding may include: performing run-length encoding on the continuous document element indexes using the first document element index as the run value and the number of continuous document element indexes as the run length, thereby completing data compression on the continuous document element indexes. For example, assuming that the document element indexes sorted in the reading order obtained after sorting by the third reading order model contain 5 continuous document element indexes.<idx_2><idx_3><idx_4><idx_5><idx_6> ", then the above five consecutive document element indexes can be encoded as "<idx_2><count_5> ",in,<idx_2> Represents the first document element index (run value) in a continuous document element index.<count_5> Represents the number of document element indices in a continuous document element index (run length). Typically, "<idx_2><count_5> " represents that the current paragraph is a paragraph consisting of five consecutive document elements starting with the second document element, which means that the current paragraph includes the second to sixth document elements. It can be seen that through the above-mentioned run-length encoding, the output data of the third reading order model can be further compressed, thereby further improving the processing speed of the third reading order model and further improving the efficiency of the third reading order model.

[0166] Based on the above run-length encoding content, in some embodiments of the present disclosure, the document element index sequence corresponding to the above run-length encoded document elements sorted in reading order can generally include: alternately arranged separators and run-length encoding subsequences; wherein the run-length encoding subsequence includes: document element index and the number of document elements. The separator can also be replaced by a format identifier, wherein each format identifier represents a unique format character, for example, <sep>As a separator between paragraphs, it represents a line break. As an example, the output of the third reading order model, that is, the document element index sequence corresponding to the document elements sorted in reading order after run-length encoding, can be in the following format:<idx_0> <sep><idx_2><count_5> <sep><idx_7><count_8> <sep>…….in, <sep>is a separator;<idx_n><count_m> Represents a run-length coded subsequence, where n and m are both positive integers.

[0167] Based on this, the above-mentioned run-length decoding may include: reading the run-length encoding subsequences in sequence from the document element index sequence corresponding to the run-length encoded document elements sorted in reading order based on the delimiter; generating a document element index subsequence corresponding to the run-length encoding subsequence based on the document element index and the number of document elements in the run-length encoding subsequence for each run-length encoding subsequence; wherein the document element index subsequence includes one or more document element indexes that increase in sequence; the first document element index of the document element index subsequence is the document element index in the run-length encoding subsequence; the number of document element indexes included in the document element index subsequence is equal to the number of document elements in the run-length encoding subsequence; using the document element index subsequence to replace the run-length encoding subsequence corresponding to the document element index subsequence, to obtain a document element index sequence corresponding to the document elements sorted in reading order. For example, for the above-mentioned run-length encoding subsequence "<idx_2><count_5> "It can be determined that the corresponding document element index subsequence is the first document element index is<idx_2> 5 consecutive document element indexes, that is, "<idx_2><idx_3><idx_4><idx_5><idx_6> Then, use<idx_2><idx_3><idx_4><idx_5><idx_6> "replace"<idx_2><count_5> After performing the above operation on each run-length encoding subsequence, the document element index sequence corresponding to the document elements sorted in reading order can be obtained.

[0168] For example, Figure 14 Schematic diagram of target document content provided by some embodiments of the present disclosure. Figure 14 As shown in the figure, an exemplary target document content 1400 is shown. The target document content 1400 only includes texts sorted in reading order, which is only an example.

[0169] In related technologies, restoring document images requires identifying the characters in text lines and restoring them on a character-by-character basis, resulting in a large amount of data to be processed and low efficiency. In this solution, the input end of the reading order model is encoded on a document element basis, and document elements can be preliminarily sorted based on the order of document element blocks. Furthermore, document element indexes can be used at the output end to replace text lines for data compression, and run-length encoding can be used to further compress document element indexes. This significantly shortens the overall length of the output data, thereby significantly improving data processing efficiency.

[0170] The document restoration method provided by the embodiment of the present disclosure can extract the position and text content of the document elements in the target document image; determine the position and arrangement order of the document element blocks therein; then, preliminarily sort the text content based on the arrangement order of the document element blocks; then perform feature processing on the target document image, the position of the document elements and the preliminarily sorted text content to obtain multimodal features; finally, restore the multimodal features to obtain the document content. The above-mentioned document restoration method realizes the use of multimodal features to restore the document content corresponding to the image in reading order, especially when complex scenes are involved, which can greatly improve the accuracy of document restoration. In addition, by first preliminarily sorting the extracted text content based on the arrangement order of the document element blocks, and then generating multimodal features based on the preliminarily sorted text content, the complexity of restoring the document content corresponding to the image in reading order using multimodal features can be reduced, further effectively improving the accuracy of document restoration and processing efficiency.

[0171] This solution combines the semantics and layout information of images and text lines to extract features using a multimodal solution. It compresses data by replacing text content with text line indexes, or further compresses data through run-length encoding. It can accurately and efficiently extract the reading order of images, thereby restoring the text content of the document, maintaining a high reading order accuracy and low inference time.

[0172] Corresponding to the above-mentioned document restoration method, some embodiments of the present disclosure further disclose a document restoration device. Figure 15 The internal structure of the document restoration device according to the embodiment of the present disclosure is shown. Figure 15 As shown, the above-mentioned document restoration device may include the following multiple modules: an image acquisition module 1510, used to acquire a target document image; a document element extraction module 1520, used to extract the position of the document element in the target document image and the text content corresponding to the document element; a document element block extraction module 1530, used to determine the document element blocks in the target document image and the position of the document element blocks; a document element block sorting module 1540, used to determine the arrangement order of the document element blocks based on the position of the document element blocks; a document element sorting module 1550, used to preliminarily sort the position and text content of the document elements based on the arrangement order of the document element blocks to obtain the preliminarily sorted position of the document elements and the preliminarily sorted text content; a feature processing module 1560, used to perform feature processing on the target document image, the preliminarily sorted position of the document elements and the preliminarily sorted text content to obtain multimodal features; and a restoration module 1570, used to restore the target document content corresponding to the target document image based on the multimodal features; wherein the target document content includes the text content corresponding to the document elements sorted in reading order.

[0173] It should be noted that the implementation method of each module in the above device and the specific technical effects that can be achieved can refer to the implementation method of each step in the above embodiment, and will not be repeated here.

[0174] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the document restoration method described in any of the above-mentioned embodiments is implemented.

[0175] Figure 16 1 is a schematic diagram showing the hardware structure of a more specific electronic device provided in this embodiment. The device may include: a processor 2010, a memory 2020, an input / output interface 2030, a communication interface 2040, and a bus 2050. The processor 2010, the memory 2020, the input / output interface 2030, and the communication interface 2040 are communicatively connected to each other within the device via the bus 2050.

[0176] The processor 2010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0177] The memory 2020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 2020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 2010.

[0178] The input / output interface 2030 is used to connect input / output devices to enable information input and output. Input / output devices can be configured as components within the device or externally connected to the device to provide corresponding functions. Input devices may include microphones and various sensors, while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0179] The communication interface 2040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0180] The bus 2050 comprises a path for transmitting information between the various components of the device (eg, the processor 2010 , the memory 2020 , the input / output interface 2030 , and the communication interface 2040 ).

[0181] It should be noted that although the above device only shows the processor 2010, the memory 2020, the input / output interface 2030, the communication interface 2040, and the bus 2050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0182] The electronic device of the above embodiment is used to implement the corresponding document restoration method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0183] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the document restoration method described in any of the above embodiments.

[0184] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0185] The computer instructions stored in the storage medium of the above embodiment are used to enable a computer to execute the document restoration method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0186] Based on the same inventive concept, corresponding to any of the document restoration methods described in the aforementioned embodiments, the present disclosure further provides a computer program product comprising computer program instructions. In some embodiments, when the computer program instructions are executed on a computer, they cause the computer to execute each step in each embodiment of the document restoration method. For each step in each embodiment of the document restoration method, the processor executing the step may belong to the corresponding execution entity.

[0187] The computer program product of the above embodiment is used to enable the processor to execute the document restoration method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0188] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0189] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0190] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0191] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.< / sep> < / sep> < / sep> < / sep> < / sep> < / sep>

Claims

1. A document restoration method, comprising: Obtain target document image; Extracting the position of the document element in the target document image and the text content corresponding to the document element; Determining a document element block in the target document image and a position of the document element block; Determining an arrangement order of the document element blocks based on positions of the document element blocks; Preliminarily sorting the positions of the document elements and the text contents based on the arrangement order of the document element blocks to obtain the preliminarily sorted positions of the document elements and the preliminarily sorted text contents; Performing feature processing on the target document image, the positions of the preliminarily sorted document elements, and the preliminarily sorted text content to obtain multimodal features; Restoration is performed based on the multimodal features to obtain target document content corresponding to the target document image; wherein the target document content includes text content corresponding to the document elements sorted in reading order.

2. The method according to claim 1, wherein The document element includes text lines and non-text; and extracting the position of the document element in the target document image and the text content corresponding to the document element includes: Performing text line recognition on the target document image using a text line recognition model to obtain positions of a plurality of text lines and text contents corresponding to the plurality of text lines; The target document image is subjected to non-text detection using a layout analysis model to obtain the position of the non-text and the text content corresponding to the non-text.

3. The method according to claim 2, wherein: The document element blocks include text blocks and non-text blocks; and determining the document element blocks in the target document image and the positions of the document element blocks includes: sorting the plurality of text lines from top to bottom based on the positions of the text lines, and determining the vertical distance between any two upper and lower adjacent text lines after the sorting; Dividing the plurality of text lines into one or more candidate text blocks based on the vertical distance between any two of the text lines that are adjacent to each other; For each of the text lines in the candidate text block, determining a horizontal distance between any two left and right adjacent text lines based on the positions of the text lines, and dividing the candidate text block into one or more candidate text sub-blocks based on the horizontal distance between any two left and right adjacent text lines; respectively setting each of the candidate text sub-blocks as a text block, and determining the position of the text block based on the position of the candidate text sub-blocks; Each of the non-texts is set as a non-text block, and the position of the non-text block is determined based on the position of the non-text.

4. The method according to claim 1, wherein Determining the document element block in the target document image and the position of the document element block includes: Performing image binarization processing on the target document image to obtain a binarized image corresponding to the target document image; Segmenting the target document image in vertical and horizontal directions based on the binarized image to obtain one or more image blocks and positions of the image blocks; Each of the image blocks is respectively regarded as a document element block, and the position of the document element block is determined based on the position of the image block.

5. The method according to claim 1, wherein The target document image includes a plurality of document element blocks; and determining the arrangement order of the document element blocks based on positions of the document element blocks includes: Based on the positions of the plurality of document element blocks, read one of the document element blocks in order from top to bottom as a target document element block, and set an arrangement order of the target document element blocks according to the reading order; Based on the positions of the multiple document element blocks, determining whether there is another document element block on the right side of the target document element block; In response to determining that other document element blocks exist on the right side of the target document element block, reading the document element block to the right of the target document element block as the target document element block, setting the arrangement order of the target document element blocks according to the reading order, and returning to the step of determining whether other document element blocks exist on the right side of the target document element block based on the positions of the multiple document element blocks; In response to determining that no other document element blocks exist to the right of the target document element block, determining, based on positions of the plurality of document element blocks, whether other document element blocks exist below the target document element block; In response to determining that other document element blocks exist below the target document element block, returning to the step of reading one of the document element blocks as the target document element block in order from top to bottom based on the positions of the multiple document element blocks, and setting the arrangement order of the target document element blocks in accordance with the reading order; In response to determining that no other document element block exists below the target document element block, the process ends.

6. The method according to claim 1, wherein Preliminarily sorting the positions of the document elements and the text contents based on the arrangement order of the document element blocks to obtain the preliminarily sorted positions of the document elements and the preliminarily sorted text contents includes: Grouping the document elements according to the document elements included in the document element block to obtain one or more document element groups; Sorting the document element groups according to the arrangement order of the document element blocks to obtain the arrangement order of the document element groups; For each of the sorted document element groups, sorting the document elements included therein in order from top to bottom to obtain an arrangement order of the document elements included in each of the document element groups; determining an arrangement order of the document elements based on an arrangement order of the document element groups and an arrangement order of the document elements included in each of the document element groups; The positions of the document elements and the text contents corresponding to the document elements are sorted based on the arrangement order of the document elements to obtain the positions of the document elements and the text contents that have been preliminarily sorted.

7. The method according to claim 1, wherein The performing feature processing on the target document image, the positions of the preliminarily sorted document elements, and the preliminarily sorted text content to obtain multimodal features includes: Segmenting and encoding the target document image to obtain visual features; Determining the layout features and semantic features of the document elements based on the positions of the preliminarily sorted document elements and the preliminarily sorted text contents, and adding the layout features and semantic features of the document elements to obtain element features; The element features and the visual features are spliced together to obtain the multimodal features.

8. The method according to claim 1, wherein The target document content corresponding to the target document image is obtained by restoring the target document content based on the multimodal features, including: The multimodal features are input into the decoder of the first reading order model, and the text content corresponding to the document elements sorted in the reading order is obtained through an autoregressive method, and formatting characters are added to the text content corresponding to the document elements sorted in the reading order to obtain the target document content.

9. The method according to claim 1, wherein The target document content corresponding to the target document image is obtained by restoring the target document content based on the multimodal features, including: Inputting the multimodal features into a decoder of a second reading order model to obtain a document element index sequence corresponding to the document elements sorted according to the reading order; The document element index in the document element index sequence corresponding to the document element sorted in the reading order is replaced with the text content corresponding to the document element to obtain the target document content.

10. The method according to claim 1, wherein The target document content corresponding to the target document image is obtained by restoring the target document content based on the multimodal features, including: Inputting the multimodal features into a decoder of a third reading order model to obtain a document element index sequence corresponding to the document elements sorted in reading order after run-length encoding; Performing run-length decoding on the document element index sequence corresponding to the document elements sorted in reading order after run-length encoding to obtain a document element index sequence corresponding to the document elements sorted in reading order; and The document element index in the document element index sequence corresponding to the document element sorted in the reading order is replaced with the text content corresponding to the document element to obtain the target document content.

11. The method according to claim 10, wherein: The document element index sequence corresponding to the run-length-encoded document elements sorted in reading order comprises: alternately arranged delimiters and run-length-encoded subsequences; wherein the run-length-encoded subsequences comprise: document element indexes and the number of document elements; The step of performing run-length decoding on the document element index sequence corresponding to the document elements sorted in reading order after run-length encoding to obtain the document element index sequence corresponding to the document elements sorted in reading order comprises: Reading the run-length encoded subsequences in sequence from the document element index sequence corresponding to the run-length encoded document elements sorted in reading order based on the delimiter; For each of the run-length encoded subsequences, generating a document element index subsequence corresponding to the run-length encoded subsequence based on the document element index and the number of document elements in the run-length encoded subsequence; wherein the document element index subsequence includes one or more document element indexes that increase in sequence; the first document element index of the document element index subsequence is the document element index in the run-length encoded subsequence; and the number of the document element indexes included in the document element index subsequence is equal to the number of document elements in the run-length encoded subsequence; The run-length encoding subsequence corresponding to the document element index subsequence is replaced with the document element index subsequence to obtain the document element index sequence corresponding to the document elements sorted in reading order.

12. A document restoration device, comprising: An image acquisition module, used for acquiring a target document image; A document element extraction module, configured to extract the position of a document element in the target document image and text content corresponding to the document element; A document element block extraction module, configured to determine a document element block in the target document image and a position of the document element block; A document element block sorting module, configured to determine an arrangement order of the document element blocks based on positions of the document element blocks; A document element sorting module, configured to preliminarily sort the positions of the document elements and the text contents based on the arrangement order of the document element blocks, and obtain preliminarily sorted positions of the document elements and preliminarily sorted text contents; a feature processing module, configured to perform feature processing on the target document image, the positions of the preliminarily sorted document elements, and the preliminarily sorted text content to obtain multimodal features; A restoration module is used to restore the target document content corresponding to the target document image based on the multimodal features; wherein the target document content includes text content corresponding to the document elements sorted in reading order.

13. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the document restoration method according to any one of claims 1 to 11 when executing the computer program.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the document restoration method according to any one of claims 1 to 11.

15. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the document restoration method according to any one of claims 1 to 11.

Images