Document processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a document processing method, which can be applied to the technical field of big data, the technical field of artificial intelligence or the technical field of financial technology. The document processing method comprises: correcting an initial document image to obtain a target document image; obtaining a plurality of target contours based on foreground targets by separating pixels of the foreground targets and pixels of background targets in the target document image; performing pixel reconstruction based on the target contours to generate layout information of the target document image; extracting entity content and entity relationship in the target document image to obtain entity information of the target document image; and performing layout on the entity information of the target document image by using the layout information of the target document image to obtain a target file with a preset format. The present disclosure also provides a document processing apparatus, device, storage medium and program product.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of big data, the technical field of artificial intelligence, or the technical field of financial technology, and more particularly to a document processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the development of industrial informatization, data has become one of the indispensable elements of people's work and life. In the face of massive data, more and more enterprise users need to store a large amount of structured and unstructured data into documents for subsequent extraction and analysis. In the process of implementing the concept of the present disclosure, the inventors found that the related art generally has the following problems: a document generally contains multiple types of data such as tables, pictures, and text. The existing document processing method generally cannot accurately identify these data when processing them, resulting in inaccurate extraction and analysis of the content of the document, and reducing the accuracy of document processing. SUMMARY

[0003] In view of the above problems, the present disclosure provides a document processing method and device, an electronic device, a storage medium, and a program product for improving the accuracy of document processing.

[0004] One aspect of the present disclosure provides a document processing method, comprising: correcting an initial document image to obtain a target document image; separating pixels of a foreground target and pixels of a background target in the target document image to obtain a plurality of target contours based on the foreground target; performing pixel reconstruction based on the target contours to generate layout information of the target document image; extracting entity content and entity relationships in the target document image to obtain entity information of the target document image; and performing layout on the entity information of the target document image using the layout information of the target document image to obtain a target file with a preset format.

[0005] According to an embodiment of the present disclosure, the separating of the pixels of the foreground target and the pixels of the background target in the target document image to obtain the plurality of target contours based on the foreground target comprises: separating the pixels of the foreground target and the pixels of the background target to obtain a plurality of boundary pixels; and generating the plurality of target contours based on the foreground target based on the plurality of boundary pixels.

[0006] According to an embodiment of the present disclosure, the performing of pixel reconstruction based on the target contours to generate the layout information of the target document image comprises: reconstructing pixel relationships between the pixels of the foreground target, the pixels of the background target, and the boundary pixels in the target document image to obtain a construction result, wherein the boundary pixels are associated with the target contours; and compensating the boundary pixels using the construction result to obtain the layout information of the target document image.

[0007] According to an embodiment of the present disclosure, the entity content of the target document image comprises a table file; and the method further comprises: inputting the table file into a feature decoding model to output one-to-one corresponding cell tokens and position coordinates of the cells.

[0008] According to an embodiment of the present disclosure, the feature decoding model comprises a hidden layer, a structure decoding sub-model and a cell position decoding sub-model; and the inputting the table file into the feature decoding model to output one-to-one corresponding cell tokens and position coordinates of the cells comprises: inputting the table file into the feature decoding model, obtaining a hidden layer output result after the hidden layer; inputting the hidden layer output result into the structure decoding sub-model and the cell position decoding sub-model respectively to output the cell tokens and the position coordinates of the cells; and performing a connection operation on the cell tokens and the position coordinates of the cells to obtain the one-to-one corresponding cell tokens and position coordinates of the cells.

[0009] According to an embodiment of the present disclosure, the extracting the entity content and the entity relationship in the target document image to obtain the entity information of the target document image comprises: inputting the target document image into a multi-modal pre-training model to output the entity information of the target document image, wherein the multi-modal pre-training model comprises a semantic entity recognition sub-model and an entity relationship extraction sub-model, the semantic entity recognition sub-model is used to output the entity content of the target document image, and the entity relationship extraction sub-model is used to output the entity relationship of the target document image.

[0010] According to an embodiment of the present disclosure, the multi-modal pre-training model is obtained by the following manner: based on a unified mutual learning knowledge distillation algorithm, the semantic entity recognition sub-model and the entity relationship extraction sub-model are caused to perform mutual learning; in the process of the mutual learning, a mutual distillation loss function and a distance loss function between the semantic entity recognition sub-model and the entity relationship extraction sub-model are calculated; according to the mutual distillation loss function and the distance loss function, model parameters of the semantic entity recognition sub-model and model parameters of the entity relationship extraction sub-model are adjusted; and in the case that the mutual distillation loss function and the distance loss function are converged, the multi-modal pre-training model is constructed according to the semantic entity recognition sub-model and the entity relationship extraction sub-model obtained under the convergence condition.

[0011] According to an embodiment of the present disclosure, the semantic entity recognition sub-model is constructed based on a ranking algorithm containing a position offset threshold; and the semantic entity recognition sub-model is configured to output the entity content of the target document image by performing entity content reading on the target document image using the ranking algorithm containing the position offset threshold, wherein, in a case where a distance in a first reading direction is less than the position offset threshold, the entity content of the target document image is read in a reading order in a second direction, and the first reading direction and the second reading direction are perpendicular to each other.

[0012] Another aspect of the present disclosure further provides a document processing apparatus, comprising: a rectification module configured to rectify an initial document image to obtain a target document image; a separation module configured to separate pixels of a foreground object and pixels of a background object in the target document image to obtain a plurality of target contours based on the foreground object; a generation module configured to generate layout information of the target document image based on the target contours; an extraction module configured to extract entity content and entity relationships in the target document image to obtain entity information of the target document image; and a layout module configured to layout the entity information of the target document image using the layout information of the target document image to obtain a target file having a preset format.

[0013] Another aspect of the present disclosure further provides an electronic device, comprising: one or more processors; and a storage apparatus configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the document processing method.

[0014] Another aspect of the present disclosure further provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the document processing method.

[0015] Another aspect of the present disclosure further provides a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the document processing method.

[0016] According to the document processing method and device, the electronic device, the storage medium and the program product provided by the embodiments of the present disclosure, the initial document image is corrected to obtain a target document image; foreground target pixels and background target pixels in the target document image are separated to obtain a plurality of target contours; pixel reconstruction is performed based on the target contours to generate layout information of the target document image; and entity content and entity relationship extracted from the target document image are laid out based on the layout information to generate a target file with a preset format. Because the foreground target pixels and the background target pixels are separated in the process of document processing, the local information in the document image is considered; and the pixel reconstruction is performed based on the target contours to complete the information lost in the process of separating the foreground pixels and the background pixels, so that the global information of the document image is considered; the local information and the global information are fully combined in the process of obtaining the layout information, so that the accuracy of the layout information of the document image is improved; and the entity information recognized is laid out according to the layout information, so that the problem that the extraction and analysis of the document content are not in place due to the inaccurate recognition of the content in the document image in the related art can be at least partially overcome, and the technical effect of improving the accuracy of document processing is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1A An application scenario diagram of a document processing method and device according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 1B An application scenario diagram of a document processing method and device according to another embodiment of the present disclosure is schematically shown;

[0020] Figure 2 A flowchart of a document processing method according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 3 A structural schematic diagram of a structure and position information alignment feature decoding module according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 4 A structural schematic diagram of an SDM sub-module according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 5 A structural schematic diagram of a CLDM sub-module according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 6 A structural schematic diagram of a CSP unit according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 7 A structural diagram of a gated recurrent unit is schematically shown according to an embodiment of the present disclosure;

[0026] Figure 8A A read result of the prior art is schematically shown;

[0027] Figure 8B A read result based on a sorting algorithm containing a position offset threshold according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 9 A flowchart of obtaining entity information by using a multi-modal pre-training model according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 10 A system architecture diagram suitable for a document processing method according to an embodiment of the present disclosure is schematically shown;

[0030] Figure 11 A structural block diagram of a document processing apparatus according to an embodiment of the present disclosure is schematically shown; and

[0031] Figure 12 A block diagram of an electronic device suitable for implementing a document processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it is to be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it will be apparent to one skilled in the art that one or more embodiments can be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present disclosure.

[0033] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprise" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0034] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present specification, and should not be interpreted in an idealized or overly formal manner.

[0035] In the case of using expressions similar to "at least one of A, B, and C, etc.", it is generally intended to include any of A, B, and C alone, or in any combination, as well as others equivalent thereto.

[0036] The financial industry has a large amount of documents, and it takes a lot of manpower to find information from relevant documents and use it. The development and landing of emerging technologies such as big data, artificial intelligence, and cloud computing have realized the deep mining of data and improved the efficiency of the financial industry. However, for computers, the processing of financial documents is still difficult, and its complexity mainly lies in the following three aspects.

[0037] Complex document layout: Financial documents contain tables, pictures, text, and other types of data, and their layout in the document is also different. The existing document analysis model has not fully considered multi-modal information, and the model robustness is not strong, which leads to incomplete extraction and analysis of document knowledge.

[0038] Various table types: Financial product documents contain a large number of tables, which contain a large amount of data information, and the extraction of the internal relationship between the data is not in place. Moreover, the table types in financial documents are diverse, such as different row and column spans, different text types, etc., making table recognition a difficult task in document processing.

[0039] Difficult to extract key information: The document of the financial product is relatively long, and it will be difficult to rely on manpower to process the key information required by the user, and it will take a lot of time and effort.

[0040] Therefore, the embodiments of the present disclosure provide a document processing method, device, electronic equipment, storage medium and program product, which can improve the accuracy of document processing, and do not require manual processing, thereby reducing the cost of human resources and improving the intelligent degree of document processing. Specifically, the method comprises correcting an initial document image to obtain a target document image; separating the pixels of the foreground target and the pixels of the background target in the target document image to obtain a plurality of target contours based on the foreground target; performing pixel reconstruction based on the target contours to generate layout information of the target document image; extracting entity content and entity relationship in the target document image to obtain entity information of the target document image; and using the layout information of the target document image to layout the entity information of the target document image to obtain a target file with a preset format.

[0041] It should be noted that the document processing method and device determined by the embodiments of the present disclosure can be used in the field of big data technology, the field of artificial intelligence technology, or the field of financial technology, and can also be used in any field other than the field of big data technology, the field of artificial intelligence technology, or the field of financial technology. The embodiments of the present disclosure do not limit the application field of the determined document processing method and device.

[0042] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of the data (such as including but not limited to user personal information, recorded videos, extracted metadata, and extracted personalized data) involved in the technical solutions comply with the relevant legal regulations, necessary security measures are taken, and do not violate public order and good customs.

[0043] Figure 1A An application scenario diagram of the document processing method and device according to the embodiments of the present disclosure is schematically shown.

[0044] As shown in Figure 1A According to the application scenario of this embodiment, the application scenario can include a terminal device 101 and a document 102.

[0045] The terminal device 101 can be various electronic devices with processing functions, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0046] The document 102 can be an electronic document, which can include images in.tiff or.jpg format for text collection, or files in.pdf,.doc,.docx,.wps,.txt,.ppt,.xml format, or an.xls format table, etc. The document 103 can be an electronic document obtained by scanning or photographing a paper document, or an electronic document generated in response to a user operation.

[0047] Various communication client applications can be installed on the terminal device 101, such as financial applications, document processing applications, document editing applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as an example).

[0048] The terminal device 101 can process the document 102, for example, correct the initial document image to obtain a target document image; separate the pixels of the foreground target and the pixels of the background target in the target document image to obtain a plurality of target contours based on the foreground target; perform pixel reconstruction based on the target contours to generate layout information of the target document image; extract entity content and entity relationship in the target document image to obtain entity information of the target document image; and perform layout on the entity information of the target document image by using the layout information of the target document image to obtain a target file with a preset format.

[0049] Figure 1B An application scenario diagram of the document processing method and device according to another embodiment of the present disclosure is schematically shown.

[0050] As shown in Figure 1B , in another embodiment, the application scenario can further include a terminal device 101, a document 102, a network 103, and a server 104, wherein the terminal device 101 and the document 102 have the same attribute functions as the terminal device 101 and the document 102 shown in Figure 1A , and details are not repeated here.

[0051] The terminal device 101 can interact with the server 104 through the network 103, and the network 103 can include various connection types, such as wired and / or wireless communication links, etc.

[0052] The server 104 can be a background management server that provides support for a client application installed in the terminal device 101.

[0053] The terminal device 101 can send the document 102 obtained to the server 104, and the server 104 can correct the initial document image to obtain a target document image; separate the pixels of the foreground target and the pixels of the background target in the target document image to obtain a plurality of target contours based on the foreground target; perform pixel reconstruction based on the target contours to generate layout information of the target document image; extract entity content and entity relationship in the target document image to obtain entity information of the target document image; and perform layout on the entity information of the target document image by using the layout information of the target document image to obtain a target file with a preset format.

[0054] It should be noted that the document processing method provided by the embodiments of the present disclosure can generally be executed by the terminal device 101. Correspondingly, the document processing apparatus provided by the embodiments of the present disclosure can generally be arranged in the terminal device 101. The document processing method provided by the embodiments of the present disclosure can also be executed by a terminal device different from the terminal device 101 and capable of communicating with the terminal device 101. Correspondingly, the document processing apparatus provided by the embodiments of the present disclosure can also be arranged in a terminal device different from the terminal device 101 and capable of communicating with the terminal device 101.

[0055] Alternatively, the document processing method provided by the embodiments of the present disclosure can generally be executed by the server 104. Correspondingly, the document processing apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 104. The document processing method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 104 and capable of communicating with the terminal device 101 and / or the server 104. Correspondingly, the document processing apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 104 and capable of communicating with the terminal device 101 and / or the server 104.

[0056] It should be understood that Figure 1A or Figure 1B the number of documents, terminal devices, networks and servers in

[0057] The document processing method of the embodiments of the present disclosure will be described in detail below based on the scenario described in Figures 1A-1B . Figures 2-9

[0058] Figure 2 A flowchart of the document processing method according to the embodiments of the present disclosure is schematically shown.

[0059] As shown in Figure 2 , the document processing method of this embodiment includes operations S201-S205.

[0060] In operation S201, the initial document image is corrected to obtain a target document image.

[0061] In operation S202, a plurality of target contours based on foreground targets are obtained by separating the pixels of the foreground targets and the pixels of the background targets in the target document image.

[0062] In operation S203, the layout information of the target document image is generated by performing pixel reconstruction based on the target contours.

[0063] ​In operation S204, the entity content and the entity relationship in the target document image are extracted to obtain entity information of the target document image.

[0064] In operation S205, the entity information of the target document image is laid out by using the layout information of the target document image to obtain a target file with a preset format.

[0065] According to an embodiment of the present disclosure, the initial document image can include a document obtained after scanning or photographing a paper file. The paper file can include a financial form or a file of a financial product. When the paper file is photographed or scanned, there is an angle deviation problem, so the image obtained by scanning or photographing has an angle tilt, which is not conducive to the extraction of subsequent layout information and entity information. To improve the extraction effect of information, the initial document image can be angle corrected, for example, the orientation is corrected according to the angle of the image, and the corrected document image obtained can be used as the target document image. Preferably, the target document image can be an image with an angle deviation of 0.

[0066] According to an embodiment of the present disclosure, the target document image can include pixels of foreground targets and pixels of background targets. The foreground targets can include tables, pictures, text, and the like in the document. The background targets can be content other than the foreground targets, such as the page background of the document, noise in the document, and the like.

[0067] According to an embodiment of the present disclosure, when separating the pixels, due to the imbalance between the foreground and the background, a local knowledge distillation algorithm can be used to focus on key pixels and channels to separate the foreground target pixels and the background target pixels. The key pixels can refer to the boundary pixels between the foreground targets and the background targets, and the channels can refer to the color channels of the image, i.e., channels composed of R (Red), G (Green), and B (Blue). When separating the foreground targets and the background targets, each point can be separated according to the pixel position of the point, for example, if the position of the pixel point belongs to a ground truth box, the target can be determined as a foreground target, and if the position of the pixel point does not belong to the ground truth box, the target is a background target.

[0068] According to an embodiment of the present disclosure, the target contour can refer to a contour composed of boundary pixels, and the object surrounded by the target contour can be a foreground target.

[0069] According to an embodiment of the present disclosure, in the process of separating foreground target pixels and background target pixels by the local knowledge distillation algorithm, the original pixel relationship in the target document image can be disturbed, and thus the problem of information loss can be caused. To solve the problem, a global knowledge distillation algorithm can be used to reconstruct the pixel relationship between all pixels on the target document image, so as to complete the missing information in the process of processing by the local knowledge distillation algorithm.

[0070] According to an embodiment of the present disclosure, according to the target contour and the reconstructed pixel relationship, the distribution information of the foreground target on the target document image can be obtained. The layout information can refer to the distribution information of the foreground target, such as the positions of tables, pictures, and texts.

[0071] According to an embodiment of the present disclosure, the entity information can be composed of entity content and entity relationship. The entity content can be data on the target document image, such as data recorded in a table, data recorded on a text, data appearing on a picture, and the like. The entity relationship can refer to the corresponding relationship between each column / row data and each column / row data in a table, or the corresponding relationship between tables, or the corresponding relationship between tables and texts, pictures, and the like.

[0072] According to an embodiment of the present disclosure, the target file in the preset format can be understood as a document in an editable format. The editable format can be adaptively adjusted according to the file format that can be supported in the terminal device. Specifically, according to the layout information, the positions of tables, texts, and pictures can be obtained. According to the entity information, the specific recorded data and the corresponding relationship between the data can be obtained. By using the layout information to layout the text information, the original initial document image or target temperature image that cannot be edited can be converted into a document in an editable format, so as to be analyzed and operated by a user.

[0073] According to the document processing method, the device, the electronic device, the storage medium and the program product provided by the embodiments of the present disclosure, the initial document image is corrected to obtain a target document image; foreground target pixels and background target pixels in the target document image are separated to obtain a plurality of target contours; pixel reconstruction is performed based on the target contours to generate layout information of the target document image; and entity content and entity relationships extracted from the target document image are laid out based on the layout information to generate a target file with a preset format. Because the foreground target pixels and the background target pixels are separated in the process of document processing, the local information in the document image is considered; and the pixel reconstruction is performed based on the target contours to complete the information lost in the process of separating the foreground pixels and the background pixels, and the global information of the document image is considered; the local information and the global information are fully combined in the process of obtaining the layout information, so that the accuracy of the layout information of the document image is improved; and the layout of the recognized entity information is performed according to the layout information, so that the problem that the extraction and analysis of the document content are not in place due to the inaccurate recognition of the content in the document image in the related art can be at least partially overcome, and the technical effect of improving the accuracy of document processing is achieved.

[0074] According to the embodiments of the present disclosure, operation S202 can further include the following operation: obtaining a plurality of boundary pixels by separating the pixels of the foreground target and the pixels of the background target; and generating a plurality of target contours based on the foreground target based on the plurality of boundary pixels.

[0075] According to the embodiments of the present disclosure, operation S203 can further include the following operation: reconstructing the pixel relationship between the pixels of the foreground target, the pixels of the background target and the boundary pixels in the target document image to obtain a construction result, wherein the boundary pixels are associated with the target contours; and compensating the boundary pixels using the construction result to obtain the layout information of the target document image.

[0076] According to the embodiments of the present disclosure, because the document image contains rich information such as text, title, table and image, in order to efficiently understand the document content, layout analysis needs to be performed. Layout analysis is to divide the document into predefined regions to facilitate subsequent analysis and understanding.

[0077] According to the embodiments of the present disclosure, a lightweight detector can be used to detect the document image. For example, a PP-PicoDet (mobile terminal ultra-light target detection algorithm), MobileDets (search mobile terminal target detection network) and the like are used to detect the document image. Specifically, the detector can be internally provided with a label allocation strategy of text, table, image, title and the like, and the detector can search according to the different structural information of text, table, image, title and the like in the allocation strategy, so as to divide the recognized document into different regions.

[0078] According to an embodiment of the present disclosure, the detector can retrieve from the results obtained according to the local knowledge distillation algorithm and the global knowledge distillation algorithm. Specifically, the local distillation algorithm is used to separate the foreground and background of the document image, focusing on the boundary pixels and color channels of the document image; the global distillation algorithm reconstructs the pixel relationship between the pixels of the foreground target, the pixels of the background target, and the boundary pixels, obtains a construction result including new pixel relationships, and transmits the construction result to the result obtained by the local distillation to compensate for global information. By fusing local feature information and global feature information, the layout information of the target document image can be obtained. The detector can retrieve from the layout information according to the allocation strategy, and divide the retrieved content into corresponding regions.

[0079] According to an embodiment of the present disclosure, in the process of reconstructing the pixel relationship, the global distillation can obtain the global pixel relationship of the image through convolution operation according to the entire image. By reconstructing the relationship between pixels, the relationship between all pixels on the image can be focused on, unlike the local distillation which only focuses on the boundary pixels between the foreground and the background.

[0080] According to an embodiment of the present disclosure, by using a lightweight document detection model, effective analysis of the document layout can be realized, which provides convenience for subsequent document processing of financial products.

[0081] According to an embodiment of the present disclosure, the local distillation can separate the foreground and background of the image, and pay more attention to the critical pixels and channels of the image; the global distillation reconstructs the relationship between different pixels and transmissions, and makes up for the missing global information in the local distillation. By fusing the results obtained by the local knowledge distillation algorithm and the results obtained by the global knowledge distillation algorithm, the details and overall information of the document are fully considered, and the redundant information such as background and noise in the document is effectively removed.

[0082] According to an embodiment of the present disclosure, in the process of layout analysis of the target document image, table recognition and OCR (Optical Character Recognition) technology can be used to preliminarily recognize the document content, so as to combine the preliminarily recognized content with the layout information, and improve the accuracy of the obtained layout information.

[0083] According to an embodiment of the present disclosure, a table recognition algorithm in the related art generally connects the output of each hidden layer in series, and sends it into a structure decode module (SDM) and a cell location decode module (CLDM) to generate all cell tokens and cell location coordinates, ignoring the one-to-one correspondence between the cell tokens and the cell location coordinates. In view of this, the present disclosure proposes a structure and location alignment module (SLAM) to realize the alignment between the output cell tokens and the cell location coordinates.

[0084] According to an embodiment of the present disclosure, when recognizing a table, the following operation can be included: inputting a table file into a feature decoding model to output one-to-one corresponding cell tokens and cell location coordinates.

[0085] According to an embodiment of the present disclosure, the feature decoding model includes a hidden layer, a structure decode sub-model, and a cell location decode sub-model; and inputting a table file into the feature decoding model to output one-to-one corresponding cell tokens and cell location coordinates can include the following operation: inputting the table file into the feature decoding model, obtaining a hidden layer output result after passing through the hidden layer; inputting the hidden layer output result into the structure decode sub-model and the cell location decode sub-model respectively to output cell tokens and cell location coordinates; and performing a connection operation on the cell tokens and the cell location coordinates to obtain one-to-one corresponding cell tokens and cell location coordinates.

[0086] According to an embodiment of the present disclosure, the cell token can refer to the content of each cell in the table, and the cell location coordinate can refer to the position coordinate of each cell content in the table.

[0087] According to an embodiment of the present disclosure, the hidden layer in the feature decoding module can be used in the process of converting the input information of the module into content that can be used to obtain the output, such as cell structure information, cell tokens, cell location coordinates, etc.

[0088] According to an embodiment of the present disclosure, in the SLAM, the hidden layer output of each recognition step is respectively sent to the SDM and the CLDM to obtain the cell token and the cell position coordinate of the current step. The cell token and the cell position coordinate output by each recognition step are respectively concatenated (concat operation) to obtain the html (HyperText Mark-up Language, HyperText Mark-up Language) expression of the table and the coordinates of all cells. Preferably, the hidden layer output of each recognition step is simultaneously sent to the SDM and the CLDM to ensure that the output cell token and the cell position coordinate are one-to-one aligned.

[0089] Figure 3 An illustrative structural diagram of a structure and position information aligned feature decoding module according to an embodiment of the present disclosure is shown.

[0090] As shown in Figure 3 , the structure and position information aligned feature decoding module 300 includes a backbone network 301, a concatenation operator module 302, a dimension reduction submodule 303, an SDM submodule 304, a CLDM submodule 305, and a connection (Concat) submodule 306. The backbone network 301 can be composed of multiple convolution layers 3011 with different parameters. The dimension reduction submodule 303 can include an UP (up) sampling unit 3031, a DOWN (down) sampling unit 3032, a CSP (Cross Stage Partial, Cross Stage Partial) unit 3033, and a GRU (Gated Recurrent Unit, Gated Recurrent Unit) 3034.

[0091] Referring back to Figure 3 , first, the table is input into the backbone network 301, and the feature data in the table is extracted by the multiple convolution layers 3011 in the backbone network 301, and then multiple feature maps can be obtained through the multiple convolution layers. The multiple feature maps pass through the concatenation operator module 302. The dimension reduction submodule 303 can perform dimension reduction processing on the multiple feature maps after passing through the concatenation operator module 302 to reduce the dimension of the feature maps, and finally input the dimension-reduced feature maps into the SDM submodule 304 and the CLDM submodule 305 to obtain the cell token and the cell position coordinate of the current step. The connection submodule 306 is used to concatenate (concat operation) the cell token and the coordinate output by each step to obtain the html expression of the table and the coordinates of all cells.

[0092] Figure 4 An illustrative structural diagram of an SDM submodule according to an embodiment of the present disclosure is shown.

[0093] As shown in Figure 4As shown, the SDM submodule can include two linear transformation layers 3041. Specifically, the feature maps obtained by the backbone network or the dimensionality-reduced feature maps can be transformed twice to obtain the table's structural information, cell tokens, etc.

[0094] Figure 5 A schematic diagram of the structure of a CLDM submodule according to an embodiment of the present disclosure is shown.

[0095] like Figure 5 As shown, the CLDM submodule can include two linear transformation layers 3051 and one activation function layer 3052. Specifically, the feature maps or dimensionality-reduced feature maps obtained by the backbone network can be transformed into table position coordinate information after two linear transformations and one activation function calculation.

[0096] Figure 6 A schematic diagram of the structure of a CSP unit according to an embodiment of the present disclosure is shown.

[0097] like Figure 6 As shown, the CSP unit mainly divides the features output by the convolutional layer into two parts: table structure information and table position coordinate information, or into cell tokens and table position coordinate information. Taking the table structure information and table position information as an example, the first part 30331 represents the table structure information, and the second part 30332 represents the table position coordinate information. Exemplarily, the second part 30332 can continue to perform convolution operations, and the feature map obtained after the convolution operation of the first part 30331 and the second part 30332 can be fused. The combination of the upper and lower convolution operations of the first part 30331 and the second part 30332 can quickly reduce the dimensionality of the output feature map. In another embodiment, the first part 30331 can also represent the table position coordinate information, and 30332 can also represent the table structure information. In addition, CSP can increase the distinguishability between the table structure information or cell tokens and the cell position coordinates.

[0098] Figure 7 A schematic diagram of the structure of a gated loop unit according to an embodiment of the present disclosure is shown.

[0099] like Figure 7 As shown, the gated recurrent unit (GRU) is a gating mechanism in recurrent neural networks (RNNs). Similar to other gating mechanisms in RNNs, GRUs aim to solve the gradient vanishing or exploding problem in standard RNNs while preserving long-term information of the sequence.

[0100] According to an embodiment of the present disclosure, the SLAM is to send the hidden layer output into the SDM and the CLDM respectively, and then connect the obtained cell token and the position coordinates of the cell respectively, unlike the traditional method which causes the problem that the cell token and the position coordinates of the cell do not correspond due to the series connection before inputting into the SDM and the CLDM, thereby achieving the technical effect of improving the accuracy of table recognition.

[0101] According to an embodiment of the present disclosure, the operation S204 can further include the operation of inputting the target document image into a multi-modal pre-training model to output entity information of the target document image, wherein the multi-modal pre-training model includes a semantic entity recognition sub-model and an entity relationship extraction sub-model, the semantic entity recognition sub-model is configured to output entity content of the target document image, and the entity relationship extraction sub-model is configured to output entity relationship of the target document image.

[0102] According to an embodiment of the present disclosure, the multi-modal pre-training model can be irrelevant to visual features. Specifically, the multi-modal pre-training model can include a semantic entity recognition sub-model and an entity relationship extraction sub-model, both of which can extract entity information and entity relationship information in the target document image. This model does not involve extracting visual features such as pixels, so the multi-modal pre-training model is irrelevant to visual features. Because the multi-modal pre-training model does not involve processing of visual information, the processing time can be reduced in terms of processing time, and the efficiency of extracting text information can be improved.

[0103] According to an embodiment of the present disclosure, to improve the accuracy of the multi-modal pre-training model, a threshold-based YX sorting algorithm and a UDML (Unified-Deep Mutual Learning) knowledge distillation algorithm are used to train the semantic entity recognition sub-model and the entity relationship extraction sub-model.

[0104] According to an embodiment of the present disclosure, the semantic entity recognition sub-model is constructed based on a sorting algorithm containing a position offset threshold (threshold-based YX sorting algorithm); and the semantic entity recognition sub-model configured to output entity content of the target document image includes: using the sorting algorithm containing the position offset threshold to read the entity content of the target document image, wherein in the case that the distance in the first reading direction is less than the position offset threshold, the entity content of the target document image is read in the reading order in the second direction, and the first reading direction and the second reading direction are perpendicular to each other.

[0105] According to an embodiment of the present disclosure, in the process of reading the entity content of the target document image, the reading order of the text is important. In the existing OCR reading technology, the OCR engine will sort and read the text information from top to bottom and then from left to right according to the absolute coordinates (YX) of the detected text box. However, the error reading order generated by different OCR engines will directly affect the position embedding and the final reading result. Therefore, the embodiment of the present disclosure introduces a position offset threshold to constrain the YX sorting algorithm. Specifically, the text boxes are still sorted from top to bottom, but when the distance between two text boxes in the Y direction is less than the threshold, the reading order is determined by the order in the X direction. The position offset threshold can be adaptively adjusted according to actual needs.

[0106] Figure 8A The reading result of the prior art is schematically shown; Figure 8B The reading result based on the sorting algorithm containing the position offset threshold according to the embodiment of the present disclosure is schematically shown.

[0107] As Figures 8A-8B shown, the prior art only mechanically reads according to the order from top to bottom. However, the method provided by the embodiment of the present disclosure can read "data 1" and "data 2" to the same line from the X direction when the distance between "data 1" and "data 2" in the Y direction is less than the threshold. Compared with the prior art, the reading method provided by the embodiment of the present disclosure can prevent the loss and misplacement of information.

[0108] According to an embodiment of the present disclosure, the multi-modal pre-training model is obtained by: based on a unified mutual learning knowledge distillation algorithm, making the semantic entity recognition sub-model and the entity relationship extraction sub-model perform mutual learning; in the process of mutual learning, calculating a mutual distillation loss function and a distance loss function between the semantic entity recognition sub-model and the entity relationship extraction sub-model; according to the mutual distillation loss function and the distance loss function, adjusting the model parameters of the semantic entity recognition sub-model and the model parameters of the entity relationship extraction sub-model; and in the case that the mutual distillation loss function and the distance loss function are converged, constructing the multi-modal pre-training model according to the semantic entity recognition sub-model and the entity relationship extraction sub-model obtained under the convergence condition.

[0109] According to an embodiment of the present disclosure, UDML joint mutual learning is introduced, for example, to enable semantic entity recognition sub-models and entity relationship extraction sub-models to learn from each other, which can effectively improve the accuracy of the model. Specifically, the joint mutual learning can refer to the semantic entity recognition sub-model and the entity relationship extraction sub-model in the learning process, which will have two loss functions, one is the loss function of the semantic entity recognition sub-model, and the other is the loss function of the entity relationship extraction sub-model. By using these two loss functions, not only can the respective sub-models learn how to distinguish different categories, but also one of the sub-models can refer to the probability estimation of the other sub-model to improve its generalization ability, thereby improving the accuracy of the model.

[0110] According to an embodiment of the present disclosure, for the process of obtaining entity information, the embodiments of the present disclosure perform mutual learning on the semantic entity recognition sub-model and the relationship entity sub-model, which are two completely identical models, calculate the mutual distillation loss function (DML (deep metric learning) loss) between the two models, and at the same time calculate the distance loss function (L2 loss (also known as Mean Squred Error, MSE)) for the output result of the conversion layer. Among them, the DML loss mainly reduces the gap between the semantic entity sub-model and the entity relationship extraction sub-model; the L2 loss is used to measure the sum of squares of the distance between the predicted value and the true value, and in the embodiments of the present disclosure, it is mainly to minimize the error, so that the model can fit the training data.

[0111] Figure 9 A flowchart for obtaining entity information using a multi-modal pre-training model according to an embodiment of the present disclosure is schematically shown.

[0112] As shown in Figure 9 , the flowchart can include a multi-modal pre-training model (VI-LayoutXLM, Visual-feature Independent Layout XLM) 901, a UDML algorithm module 902, and a decoding layer module 903.

[0113] As shown in Figure 9As shown, first, information in the table is extracted, such as spatial, position, text, and the like, and then a feature map is generated using the network of the UDML algorithm module 902, and finally the structure information of the table is output by decoding the feature map through the decoding layer module 903. S5, S8, and S12 in the figure are used to represent that they are obtained by different convolutional networks, and mainly represent that the feature maps above and below are consistent, while the feature maps on the left and right are inconsistent. GT Loss (ground truth loss, the purpose is to ensure that the training is based on the true label), is a loss function used to evaluate the gap between the true value and the predicted value. The operation network in the decoding layer module 903 can integrate multiple sub-modules, and the operation network is used to operate the feature map and obtain the data required for calculating GTloss.

[0114] According to an embodiment of the present disclosure, by using a joint mutual learning algorithm to mutually learn two completely same models of semantic entity recognition and relationship entity, the effect of the model is effectively improved.

[0115] According to an embodiment of the present disclosure, operation S205 can further include the following operation: using a layout recovery module, according to the analysis result, and using layout information to layout the entity information, restoring the target document image into an editable text file, and the layout of the restored file can be consistent with the layout of the initial temperature image or the target document image. By converting into an editable document, it is convenient for the user to analyze and operate.

[0116] Figure 10 The system architecture diagram suitable for the document processing method according to an embodiment of the present disclosure is schematically shown.

[0117] As Figure 10 shown, the system architecture 1000 includes a direction correction module 1010, a layout information extraction module 1020, and a key information extraction module 1030. The layout information extraction module 1020 can include a layout analysis unit 1021, a table recognition unit 1022, a first OCR recognition unit 1023, a layout recovery unit 1024, and a recovery information unit 1025. The key information extraction module 1030 can include a second OCR recognition unit 1031, a semantic entity recognition unit 1032, a relationship entity recognition unit 1033, and a structure information unit 1034. The main difference between the first OCR recognition unit 1023 and the second OCR recognition unit 1031 is that the first OCR recognition unit 1023 is used to roughly recognize the content of the table, such as the title, the edge data, and the like, so as to more accurately obtain the result of layout analysis. The second OCR recognition unit 1031 is mainly used to accurately obtain all the content in the table, such as the data and information of each cell in the table.

[0118] According to the document processing system provided by the embodiment of the present disclosure, mainly composed of a layout information extraction module 1020 and a key information extraction module 1030. For the layout information extraction module 1020, first, the document image is divided into different regions such as text, table and image by the layout analysis unit 1021, and then the document information is analyzed using the table recognition unit 1022 and the first OCR recognition unit 1023. Finally, the image is restored to an editable text file consistent with the original image layout using the layout recovery unit 1024. For the key information extraction module 1030, first, the text content is extracted using the second OCR recognition unit 1031, and then the entities in the image and their relationships are obtained using the semantic entity recognition unit 1032 and the relationship entity recognition unit 1033, respectively, to extract the required key information.

[0119] It should be noted that the document processing system part of the embodiment of the present disclosure corresponds to the document processing method part of the embodiment of the present disclosure, and the description of the document processing system part is specifically referred to the document processing method part, which will not be repeated here.

[0120] According to the embodiment of the present disclosure, the document processing system enhances the layout analysis and entity recognition capabilities of the document by fusing layout analysis and key information extraction, and improves the processing efficiency of the business.

[0121] According to the embodiment of the present disclosure, by adopting a lightweight document detection model, the document layout analysis is effectively realized, providing a basis for subsequent product document content understanding. Through the local and global knowledge distillation algorithm, the detailed information and overall information of the document are fully considered, and the redundant information such as background and noise in the document is effectively removed. For different forms of tables in the document, a feature decoding module that aligns structure and position information is designed to realize the alignment between cell tokens and coordinates, improving the accuracy of table recognition. The joint mutual learning algorithm is used to mutually learn two completely same models of semantic entity recognition and relationship entity, effectively improving the effect of the model.

[0122] It should be noted that, unless it is explicitly stated that there is an execution sequence between different operations, or there is an execution sequence between different operations in the technical implementation, the execution sequence between multiple operations can not be distinguished, and multiple operations can be executed simultaneously.

[0123] Based on the above document processing method, the present disclosure further provides a document processing device. The following will be combined Figure 11 The device will be described in detail.

[0124] Figure 11 The structure block diagram of the document processing device according to the embodiment of the present disclosure is schematically shown.

[0125] AsFigure 11 As shown in the figure, the document processing apparatus 1100 of this embodiment includes a correction module 1110, a separation module 1120, a generation module 1130, an extraction module 1140, and a layout module 1150.

[0126] The correction module 1110 is configured to correct the initial document image to obtain a target document image.

[0127] The separation module 1120 is configured to separate the pixels of the foreground target and the pixels of the background target in the target document image to obtain a plurality of target contours based on the foreground target.

[0128] The generation module 1130 is configured to perform pixel reconstruction based on the target contours to generate layout information of the target document image.

[0129] The extraction module 1140 is configured to extract entity content and entity relationships in the target document image to obtain entity information of the target document image.

[0130] The layout module 1150 is configured to perform layout on the entity information of the target document image by using the layout information of the target document image to obtain a target file with a preset format.

[0131] According to the document processing method, apparatus, electronic device, storage medium, and program product provided by the embodiments of the present disclosure, the initial document image is corrected to obtain a target document image; the foreground target pixels and the background target pixels in the target document image are separated to obtain a plurality of target contours; pixel reconstruction is performed based on the target contours to generate layout information of the target document image; and the entity content and the entity relationships extracted from the target document image are laid out by using the layout information to generate a target file with a preset format. Because the foreground target pixels and the background target pixels are separated in the process of document processing, the local information in the document image is considered; and the pixel reconstruction is performed based on the target contours to complete the information lost in the process of separating the foreground pixels and the background pixels, and the global information of the document image is considered; the local information and the global information are fully combined in the process of obtaining the layout information, the accuracy of the layout information of the document image is improved; and the entity information recognized is laid out according to the layout information, which at least partially overcomes the problem that the extraction and analysis of the document content are not in place due to the inaccurate recognition of the content in the document image in the related art, and thus the technical effect of improving the accuracy of document processing is achieved.

[0132] According to the embodiments of the present disclosure, the separation module can further include a separation unit and a generation unit.

[0133] The separation unit is configured to separate the pixels of the foreground target and the pixels of the background target to obtain a plurality of boundary pixels.

[0134] The generating unit is configured to generate a plurality of target contours based on the foreground target based on the plurality of boundary pixels.

[0135] According to an embodiment of the present disclosure, the generating module further includes a reconstructing unit and a compensating unit.

[0136] The reconstructing unit is configured to reconstruct a pixel relationship between the pixels of the foreground target, the pixels of the background target and the boundary pixels in the target document image to obtain a reconstruction result, wherein the boundary pixels are associated with the target contours.

[0137] The compensating unit is configured to compensate the boundary pixels using the reconstruction result to obtain the layout information of the target document image.

[0138] According to an embodiment of the present disclosure, the document processing apparatus further includes an input module.

[0139] The input module is configured to input the table file into the feature decoding model to output the one-to-one corresponding cell token and the position coordinates of the cell.

[0140] According to an embodiment of the present disclosure, the input module further includes a first input unit, a second input unit and a connecting unit.

[0141] The first input unit is configured to input the table file into the feature decoding model to obtain a hidden layer output result after passing through a hidden layer.

[0142] The second input unit is configured to input the hidden layer output result into a structure decoding sub-model and a cell position decoding sub-model respectively to output the cell token and the position coordinates of the cell.

[0143] The connecting unit is configured to perform a connection operation on the cell token and the position coordinates of the cell to obtain the one-to-one corresponding cell token and the position coordinates of the cell.

[0144] According to an embodiment of the present disclosure, the extracting module further includes a third input unit.

[0145] The third input unit is configured to input the target document image into a multi-modal pre-training model to output entity information of the target document image, wherein the multi-modal pre-training model includes a semantic entity recognition sub-model and an entity relationship extraction sub-model, the semantic entity recognition sub-model is configured to output entity content of the target document image, and the entity relationship extraction sub-model is configured to output an entity relationship of the target document image.

[0146] According to an embodiment of the present disclosure, the document processing apparatus further includes a mutual learning module, a calculating module, an adjusting module and a constructing module.

[0147] The learning module is configured to enable the semantic entity recognition sub-model and the entity relationship extraction sub-model to perform mutual learning based on a unified mutual learning knowledge distillation algorithm.

[0148] The computing module is configured to calculate mutual distillation loss functions and distance loss functions between the semantic entity recognition sub-model and the entity relationship extraction sub-model in the process of mutual learning.

[0149] The adjusting module is configured to adjust model parameters of the semantic entity recognition sub-model and model parameters of the entity relationship extraction sub-model according to the mutual distillation loss functions and the distance loss functions.

[0150] The constructing module is configured to construct a multi-modal pre-training model according to the semantic entity recognition sub-model and the entity relationship extraction sub-model obtained under the convergence condition when the mutual distillation loss functions and the distance loss functions are converged.

[0151] According to an embodiment of the present disclosure, the third input unit can further include a reading sub-unit.

[0152] The reading sub-unit is configured to perform entity content reading on the target document image by using a sorting algorithm containing a position offset threshold, and in a case where a distance in a first reading direction is less than the position offset threshold, reading the entity content of the target document image in a reading order in a second direction, the first reading direction being perpendicular to the second reading direction.

[0153] According to an embodiment of the present disclosure, any one or more of the rectifying module 1110, the separating module 1120, the generating module 1130, the extracting module 1140, and the layout module 1150 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the rectifying module 1110, the separating module 1120, the generating module 1130, the extracting module 1140, and the layout module 1150 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or in any one of software, hardware, and firmware or in any appropriate combination of any of them. Alternatively, at least one of the rectifying module 1110, the separating module 1120, the generating module 1130, the extracting module 1140, and the layout module 1150 can be at least partially implemented as a computer program module which can perform corresponding functions when it is run.

[0154] It should be noted that the document processing apparatus part of the embodiments of the present disclosure corresponds to the document processing method part of the embodiments of the present disclosure, and the description of the document processing apparatus part is specifically referred to the document processing method part, which will not be repeated here.

[0155] Figure 12 A block diagram of an electronic device suitable for implementing the document processing method according to an embodiment of the present disclosure is schematically shown.

[0156] As shown in Figure 12 The electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1202 or loaded into a random access memory (RAM) 1203 from a storage portion 1208. The processor 1201 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), and / or the like. The processor 1201 can also include an on-board memory for cache use. The processor 1201 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present disclosure.

[0157] In the RAM 1203, various programs and data required for the operation of the electronic device 1200 are stored. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. The processor 1201 performs various operations of the method processes according to embodiments of the present disclosure by executing programs in the ROM 1202 and / or the RAM 1203. It should be noted that the programs can also be stored in one or more memories other than the ROM 1202 and the RAM 1203. The processor 1201 can also perform various operations of the method processes according to embodiments of the present disclosure by executing programs stored in the one or more memories.

[0158] According to an embodiment of the present disclosure, the electronic device 1200 can further include an input / output (I / O) interface 1205 that is also connected to the bus 1204. The electronic device 1200 can further include one or more of the following components connected to the I / O interface 1205: an input part 1206 including, for example, a keyboard and a mouse; an output part 1207 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage part 1208 including, for example, a hard disk; and a communication part 1209 including, for example, a LAN card, a modem, and the like. The communication part 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as necessary. A removable medium 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 1210 as necessary, so that a computer program read therefrom is installed in the storage part 1208 as necessary.

[0159] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to the embodiments of the present disclosure.

[0160] According to an embodiment of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium, for example, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more memories such as the ROM 1202 and / or the RAM 1203 described above and / or one or more memory devices other than the ROM 1202 and the RAM 1203.

[0161] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the document processing method provided by the embodiments of the present disclosure.

[0162] The above-described functions of the system / device defined in the embodiments of the present disclosure are performed when the computer program is executed by the processor 1201. According to the embodiments of the present disclosure, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0163] In one embodiment, the computer program can be stored in a tangible storage medium, such as an optical, magnetic, or other storage device. In another embodiment, the computer program can be transmitted over a network via a signal, and downloaded and installed by communication section 1209, and / or installed from removable medium 1211. The computer program embodied in the computer program code can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, or any suitable combination of the foregoing.

[0164] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by the processor 1201, the above-described functions of the system defined in the embodiments of the present disclosure are performed. According to the embodiments of the present disclosure, the system, apparatus, device, module, unit, etc. described above can be implemented by computer program modules.

[0165] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language, or similar programming language. The program code can be executed entirely on a user computing device, partially on a user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0166] The computer program product of the first aspect can include one or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform the operations of the first aspect. The one or more non-transitory computer-readable media can include, for example, magnetic media such as one or more magnetic disks, magnetic tapes or cassettes; optical media such as one or more compact discs (CD), optical discs or discs (for example, DVD, Blu-ray Disc®, digital video disc, ultra density disc, ultra-compact disc, any optical media, etc.); semiconductor media such as solid state hard drives (for example, flash memory, solid state USB drives, etc.); any other suitable medium; or any suitable combination of media.

[0167] Those skilled in the art will understand that features of the various embodiments and / or claims of the present disclosure can be combined or / and integrated with one another, even though such a combination or integration is not expressly disclosed in the present disclosure. In particular, the features of the various embodiments and / or claims of the present disclosure can be combined and / or integrated with one another in any number of ways, without departing from the spirit and scope of the present disclosure. All such combinations and / or integrations are within the scope of the present disclosure.

[0168] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and all such substitutions and modifications shall fall within the scope of the present disclosure.

Claims

1. A document processing method, comprising: The initial document image is corrected to obtain the target document image; By separating the pixels of the foreground target and the pixels of the background target in the target document image, multiple target contours based on the foreground target are obtained; Pixel reconstruction is performed based on the target contour to generate the layout information of the target document image; Extract entity content and entity relationships from the target document image to obtain entity information of the target document image; The entity information of the target document image is laid out using the layout information of the target document image to obtain a target file with a preset format; The entity content of the target document image includes a table file; the method further includes: inputting the table file into a feature decoding model, obtaining the hidden layer output after passing through the hidden layer included in the feature decoding model, wherein the feature decoding model further includes a structure decoding sub-model and a cell position decoding sub-model; inputting the hidden layer output into the structure decoding sub-model and the cell position decoding sub-model respectively, and outputting cell tokens and cell position coordinates; performing a concatenation operation on the cell tokens and the cell position coordinates to obtain a one-to-one correspondence between cell tokens and cell position coordinates.

2. The method according to claim 1, wherein, The step of obtaining multiple target contours based on the foreground target by separating the pixels of the foreground target and the pixels of the background target in the target document image includes: By separating the pixels of the foreground target and the pixels of the background target, multiple boundary pixels are obtained; Based on the multiple boundary pixels, multiple target contours based on the foreground target are generated.

3. The method according to claim 2, wherein, The step of reconstructing the target document image based on the target contour to generate the layout information includes: The pixel relationships between the pixels of the foreground target, the pixels of the background target, and the boundary pixels in the target document image are reconstructed to obtain the construction result, wherein the boundary pixels are associated with the target contour; The boundary pixels are compensated using the construction result to obtain the layout information of the target document image.

4. The method according to claim 1, wherein, The step of extracting entity content and entity relationships from the target document image to obtain entity information of the target document image includes: The target document image is input into a multimodal pre-trained model, which outputs the entity information of the target document image. The multimodal pre-trained model includes a semantic entity recognition sub-model and an entity relationship extraction sub-model. The semantic entity recognition sub-model is used to output the entity content of the target document image, and the entity relationship extraction sub-model is used to output the entity relationships of the target document image.

5. The method according to claim 4, wherein, The multimodal pre-trained model was obtained in the following way: Based on the unified mutual learning knowledge distillation algorithm, the semantic entity recognition sub-model and the entity relationship extraction sub-model learn from each other. During the mutual learning process, the mutual distillation loss function and the distance loss function between the semantic entity recognition sub-model and the entity relationship extraction sub-model are calculated; Based on the mutual distillation loss function and the distance loss function, adjust the model parameters of the semantic entity recognition sub-model and the model parameters of the entity relationship extraction sub-model; and When both the cross-distillation loss function and the distance loss function converge, the multimodal pre-trained model is constructed based on the semantic entity recognition sub-model and entity relationship extraction sub-model obtained under the convergence condition.

6. The method according to claim 4, wherein, The semantic entity recognition sub-model is constructed based on a ranking algorithm that includes a position offset threshold; The semantic entity recognition sub-model for outputting the entity content of the target document image includes: reading the entity content of the target document image using the sorting algorithm containing the position offset threshold, wherein, when the distance in the first reading direction is less than the position offset threshold, the entity content of the target document image is read in the reading order in the second reading direction, and the first reading direction and the second reading direction are perpendicular to each other.

7. A document processing apparatus, comprising: The correction module is used to correct the initial document image to obtain the target document image; A separation module is used to obtain multiple target contours based on the foreground target by separating the pixels of the foreground target and the pixels of the background target in the target document image; A generation module is used to reconstruct pixels based on the target contour to generate layout information of the target document image; An extraction module is used to extract entity content and entity relationships from the target document image to obtain entity information of the target document image; The layout module is used to lay out the entity information of the target document image using the layout information of the target document image, so as to obtain a target file with a preset format; The entity content of the target document image includes a table file. The device further includes: a first input unit for inputting the table file into a feature decoding model, and obtaining the hidden layer output result after passing through the hidden layer included in the feature decoding model; the feature decoding model further includes a structure decoding sub-model and a cell position decoding sub-model; a second input unit for inputting the hidden layer output result into the structure decoding sub-model and the cell position decoding sub-model respectively, and outputting cell tokens and cell position coordinates; and a connection unit for performing a connection operation on the cell tokens and the cell position coordinates to obtain a one-to-one corresponding cell token and cell position coordinates.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images