Document digital processing method and device and nonvolatile storage medium

Abstract

The invention discloses a document digital processing method and device and a nonvolatile storage medium. The method comprises the steps that a to-be-processed area is determined in a target document image, and the area type of the to-be-processed area comprises a text area and a table area; an encoder in the document digital processing model is adopted to extract area visual features of the to-be-processed area; processing the regional visual features by adopting a first decoder in a document digital processing model to obtain spatial layout information and regional content information of the to-be-processed region; processing the regional visual features by adopting a second decoder in a document digital processing model to obtain structural information of the to-be-processed region; and obtaining a digital document corresponding to the target document image according to the spatial layout information, the region content information and the structure information of each to-be-processed region. According to the method and the device, the technical problem that the document digitization effect is poor due to the fact that the table part of the document cannot be accurately recognized in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of electronic digital data processing, and in particular, to a document digital processing method, device and non-volatile storage medium. Background Art

[0002] In the related art, when digitalizing documents, for relatively complex documents containing table parts, it is impossible to efficiently and accurately extract the content of the table part and the original table structure, resulting in poor document digitization effect.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a document digitization processing method, device and non-volatile storage medium to at least solve the technical problem of poor document digitization effect caused by the inability to accurately identify the table part of the document in the related art.

[0005] According to one aspect of an embodiment of the present application, a document digitization processing method is provided, comprising: determining a region to be processed in a target document image, wherein the region types of the region to be processed include text regions and table regions; using an encoder in a document digitization processing model to extract regional visual features of the region to be processed; using a first decoder in the document digitization processing model to process the regional visual features to obtain spatial layout information and regional content information of the region to be processed; using a second decoder in the document digitization processing model to process the regional visual features to obtain structural information of the region to be processed, wherein, in a case where the region type of the region to be processed is a text region, the structural information includes text hierarchical structural information, and in a case where the region type of the region to be processed is a table region, the structural information includes table hierarchical information, cell span information, and cell relationship information in the table; based on the spatial layout information, regional content information, and structural information of each region to be processed, a digitized document corresponding to the target document image is obtained.

[0006] Optionally, using an encoder in a document digitization processing model to extract regional visual features of the area to be processed includes: determining a type token of the area to be processed, wherein the type token is used to indicate the regional type of the area to be processed; inputting the area to be processed and the type token corresponding to the area to be processed into the encoder, and obtaining regional visual features extracted by the encoder from the area to be processed based on the type token, wherein the feature type of the regional visual features includes at least one of the following: table line features, text line spacing features, and character shape features.

[0007] Optionally, the spatial layout information includes region bounding box position information of the region to be processed and element bounding box position information of each element in the region to be processed, wherein the element type of the element includes a text block and a cell.

[0008] Optionally, the first decoder in the document digitization processing model is used to process the regional visual features to obtain the regional content information of the area to be processed, including: obtaining the language information token output by the encoder, wherein the language information token is used to indicate the text language of the area to be processed; extracting and identifying text features from the regional visual features based on the language information token and the spatial layout information to obtain the regional content information, wherein the regional content information includes the text content recognition results in the area to be processed.

[0009] Optionally, based on the spatial layout information, regional content information and structural information of each area to be processed, a digitized document corresponding to the target document image is obtained, including: for the area to be processed whose area type is a text area, the row recognition result and the paragraph recognition result in the area to be processed are merged according to the spatial layout information, regional content information and structural information of the area to be processed to obtain a text recognition result; for the area to be processed whose area type is a table area, a target table structure is constructed according to the spatial layout information and structural information, and corresponding table content is filled in the target table structure according to the regional content information to obtain a table recognition result; and the text recognition result and the table recognition result are merged to obtain a digitized document.

[0010] Optionally, merging text recognition results and table recognition results to obtain a digitized document includes: determining position information of the text recognition results based on spatial layout information corresponding to the text recognition results; determining position information of the table recognition results based on spatial layout information corresponding to the table recognition results; and merging the text recognition results and the table recognition results based on the position information of the text recognition results and the position information of the table recognition results to obtain a digitized document.

[0011] Optionally, the document digitization processing model is trained in the following manner: determining the intersection-over-union loss function and the sequence cross entropy loss function of the document digitization processing model, the function value of the intersection-over-union loss function is used to determine the accuracy of the spatial layout information, and the function value of the sequence cross entropy loss function is used to determine the accuracy of the structural information; determining the priority of the table area and the text area, and determining the weights of the intersection-over-union loss function and the sequence cross entropy loss function based on the priority; determining the total loss function of the document digitization processing model based on the weights, the intersection-over-union loss function and the sequence cross entropy loss function, and using the total loss function to train the document digitization processing model.

[0012] According to another aspect of an embodiment of the present application, a document digitization processing device is also provided, including: a first processing module, used to determine a to-be-processed area in a target document image, wherein the area types of the to-be-processed area include text areas and table areas; a second processing module, used to extract regional visual features of the to-be-processed area using an encoder in a document digitization processing model; a third processing module, used to process the regional visual features using a first decoder in the document digitization processing model to obtain spatial layout information and regional content information of the to-be-processed area; a fourth processing module, used to process the regional visual features using a second decoder in the document digitization processing model to obtain structural information of the to-be-processed area, wherein, in the case where the area type of the to-be-processed area is a text area, the structural information includes text hierarchical structural information, and in the case where the area type of the to-be-processed area is a table area, the structural information includes table hierarchical information, cell span information, and cell relationship information in the table; a fifth processing module, used to obtain a digitized document corresponding to the target document image based on the spatial layout information, regional content information, and structural information of each to-be-processed area.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a program is stored, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute a document digitization processing method.

[0014] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the document digitization processing method is executed when the program is run.

[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, and the computer program implements a document digitization processing method when executed by a processor.

[0016] In an embodiment of the present application, a region to be processed is determined in a target document image, wherein the region types of the region to be processed include text regions and table regions; an encoder in a document digitization processing model is used to extract regional visual features of the region to be processed; a first decoder in the document digitization processing model is used to process the regional visual features to obtain spatial layout information and regional content information of the region to be processed; a second decoder in the document digitization processing model is used to process the regional visual features to obtain structural information of the region to be processed, wherein, in the case where the region type of the region to be processed is a text region, the structural information includes text hierarchical structure information, and in the case where the region type of the region to be processed is a table region, the structural information includes table hierarchical information, cell span information, and cell relationship information in the table; a method of obtaining a digitized document corresponding to the target document image is adopted based on the spatial layout information, regional content information, and structural information of each region to be processed, and the deep structure of the text and table is parsed by type by a dual decoder, so as to achieve the purpose of finely restoring the semantics and logical relationship of the document, thereby achieving the technical effect of high-fidelity digital reconstruction of the text and table content, thereby solving the technical problem of poor document digitization effect caused by the inability to accurately identify the table part of the document in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 It is a structural schematic diagram of a computer terminal (mobile terminal) provided according to an embodiment of the present application;

[0019] Figure 2 It is a flowchart of a document digitization processing method provided according to an embodiment of the present application;

[0020] Figure 3 It is a flowchart of a document digitization process provided according to an embodiment of the present application;

[0021] Figure 4 It is a structural schematic diagram of a document digitization processing device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0025] OCR (Optical Character Recognition): refers to the technology that reads text in paper documents through electronic devices (such as scanners or cameras) and converts them into editable digital text.

[0026] Document Layout Analysis (DLA): is a key step in converting unstructured documents into a structured format suitable for further processing. This process is crucial for tasks such as OCR, document retrieval, table extraction, and information extraction. The goal of DLA is to detect and classify different homogeneous regions in a document, such as text blocks, tables, and images, based on visual features and text patterns. This task can be viewed as a type of object detection or semantic segmentation problem in document images, by identifying regions of interest and annotating them in preparation for subsequent processing.

[0027] Document digitization plays a vital role in modern information management. As more and more organizations and institutions transition to digital solutions, the demand for digitizing physical documents is increasing. This transition is particularly important for historical documents, legal records, and multilingual materials that need to be preserved and easily retrieved. Digitization brings the following benefits: 1) Accessibility: Digitized documents can be made available to a wider audience, breaking down geographic and temporal barriers. 2) Searchability: Digital formats allow keyword searches, enabling users to easily locate specific information in large document collections. 3) Preservability: Digital copies can effectively prevent physical degradation and loss due to environmental factors.

[0028] Although OCR technology in related technologies has made significant progress, it still faces many challenges in accurately processing complex documents containing text, tables, and images. The main problems include: 1) Complex layout: Traditional OCR systems often make recognition and formatting errors when processing complex documents; 2) Multi-language processing: Many OCR solutions focus on the recognition of a single language and cannot effectively process documents containing multiple languages; 3) Table and image recognition: Extracting structured data from tables and interpreting image content remain difficult problems for standard OCR systems. Although current solutions, such as Google Tesseract and ABBYY FineReader, perform well in recognizing plain text, they often fall short in preserving the structural integrity of tables and images. In addition, advanced models such as LayoutLMv3 and Donut show potential in processing document structure and information extraction, but they still fail to fully solve the problems of multi-language processing or structured output generation.

[0029] In addition, despite many advances, open source and commercial document analysis algorithms in related technologies still have certain limitations. For example, due to the computational complexity and maximum sequence length limit of Transformer, capturing long-distance dependencies between cells may be challenging, resulting in information loss in long tables, thus affecting the model's accurate understanding of the context. In addition, when dealing with tables with a large number of empty cells or sparse content, the model may have difficulty distinguishing between meaningful empty cells and cells with missing data.

[0030] In order to solve this problem, a relevant solution is provided in the embodiments of the present application, which is described in detail below.

[0031] According to an embodiment of the present application, a method embodiment of a document digitization processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a document digitization processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0033] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the document digitization processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned document digitization processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0037] In the above operating environment, the embodiment of the present application provides a document digitization processing method, such as Figure 2 As shown, the method comprises the following steps:

[0038] Step S202, determining a region to be processed in the target document image, wherein the region types of the region to be processed include a text region and a table region;

[0039] In some embodiments of the present application, Figure 3 As shown, the area to be processed can be determined in the target document image by means of Document Layout Analysis (DLA), wherein the target document image can be a document image obtained by photographing, scanning, or other methods of a physical document such as a paper document.

[0040] As an optional implementation, Figure 3The document layout analysis module in can use Mask R-CNN to implement document layout analysis. Mask R-CNN is an extended version of Faster R-CNN, which adds a parallel branch for predicting segmentation masks, making it a powerful tool for document layout analysis. Unlike traditional methods that rely on multiple independent steps to detect regions and convert them into bounding boxes, Mask R-CNN is able to perform both tasks simultaneously, that is, to generate pixel-level bounding boxes and semantic masks at the same time. On the one hand, it reduces the need for additional post-processing (such as converting segmentation masks into bounding boxes), reduces the overall computational burden and simplifies the workflow; on the other hand, it can flexibly and accurately detect and depict subtle details in documents, such as curved or tilted text lines, complex table structures, and images embedded in other content. Masks can also be output in the form of polygons to better adapt to non-standard shapes that cannot be fully represented by bounding boxes, so they are particularly effective when processing documents containing irregular shapes, overlapping elements, or complex layouts. In this way, by integrating Ask R-CNN into the OCR preprocessing component, the document layout can be effectively segmented into independent regions and text, tables, and images can be accurately recognized. This helps ensure that each element can be processed according to its type in the subsequent processing flow. The types of elements include text area elements, table area elements and picture area elements. Text area elements include each character and paragraph in the text area, table area elements include cells and text content in the cells and table lines, and picture area elements may include each picture in the picture area.

[0041] Step S204, using the encoder in the document digitization processing model to extract regional visual features of the area to be processed;

[0042] In the technical solution provided in step S204, the step of using the encoder in the document digitization processing model to extract the regional visual features of the area to be processed includes: determining the type token of the area to be processed, wherein the type token is used to indicate the regional type of the area to be processed; inputting the area to be processed and the type token corresponding to the area to be processed into the encoder, and obtaining the regional visual features extracted by the encoder from the area to be processed based on the type token, wherein the feature type of the regional visual features includes at least one of the following: table line features, text line spacing features, and character shape features.

[0043] As an optional embodiment, the above-mentioned encoder may be a shared visual encoder. The shared visual encoder can extract visual features from the input data, regardless of whether the input data is table image data or plain text image data. The shared visual encoder uses a deep neural network architecture (Vision Transformer) to capture relevant visual features in the image, which not only include pixel-level information, but also capture high-level visual patterns (such as table lines, text line spacing, character shapes, etc.). In addition, in order to enhance the encoder's ability to distinguish different input types (such as tables and plain text), an input type token is also introduced in this application to indicate whether the input area to be processed is a table area or a plain text area, so that the encoder can adaptively focus on features related to a specific content type.

[0044] In some embodiments of the present application, in order to process multilingual text, the encoder also incorporates a language information token to indicate the language used in the input text, thereby optimizing the feature extraction process to improve the recognition accuracy of multiple languages.

[0045] Step S206, using the first decoder in the document digitization processing model to process the regional visual features to obtain spatial layout information and regional content information of the area to be processed;

[0046] In the technical solution provided in step S206, the spatial layout information includes the region bounding box position information of the area to be processed and the element bounding box position information of each element in the area to be processed, wherein the element type of the element includes a text block and a cell.

[0047] As an optional implementation, the first decoder in the document digitization processing model is used to process the regional visual features, and the step of obtaining the regional content information of the area to be processed includes: obtaining the language information token output by the encoder, wherein the language information token is used to indicate the text language of the area to be processed; extracting and identifying text features from the regional visual features based on the language information token and the spatial layout information to obtain the regional content information, wherein the regional content information includes the text content recognition results in the area to be processed.

[0048] In some embodiments of the present application, the first decoder is a bounding box and content decoder, and the output results include bounding box coordinates and content tokens containing regional content information. Specifically, the first decoder can predict the position of each text block or cell, and express the bounding box coordinates in the form of (x1, y1, x2, y2). These bounding boxes represent the physical positions of each element and are used to reconstruct the spatial layout of the document; the content tokens represent the text content in each bounding box. When predicting content tokens, the decoder uses language information tokens to adjust the text processing strategy to ensure that texts in different languages ​​maintain semantic coherence during the parsing process, and generate interlaced bounding box and content token sequences, thereby associating the spatial layout with the actual content.

[0049] It should be noted that the above coordinates are the coordinates of the bounding box in a preset plane rectangular coordinate system, which can be a plane rectangular coordinate system established with the vertices of the text image as the origin. In order to facilitate subsequent splicing, the bounding box coordinates can also include the page number of the area to be processed corresponding to each bounding box in the document. The page number can be used to quickly locate the approximate location information of the area to be processed corresponding to the bounding box or the content recognition result, and then the precise positioning can be performed according to the coordinates of the bounding box.

[0050] Specifically, considering that an actual document usually has multiple pages, in order to efficiently process multiple pages of a document at one time, the scanned images of each page of the document may be arranged and spliced ​​according to the page numbers, and the spliced ​​result may be used as the target document image.

[0051] Step S208, using the second decoder in the document digitization processing model to process the regional visual features to obtain structural information of the area to be processed, wherein, when the area type of the area to be processed is a text area, the structural information includes text hierarchical structural information, and when the area type of the area to be processed is a table area, the structural information includes table hierarchical information, cell span information, and cell relationship information in the table;

[0052] In the technical solution provided in step S208, the second decoder is also called a table structure decoder. The decoder uses the optimized table structure language (OTSL, Organized Table Structure Language) to generate tokens representing the input logical structure. When the input area to be processed is a table area, an OTSL token can be output, where the token includes the following structural information: 1) table level: including rows, columns and merged cells; 2) cell span: information about cells spanning multiple rows or columns; 3) cell relationship: parent-child relationship between table elements.

[0053] When the input is a plain text area, the second decoder can treat paragraphs and titles as different cells. The decoder generates tokens that describe the document's hierarchical structure (such as paragraphs and titles), and ensures the correct identification of the structural characteristics of different languages ​​when processing multilingual content. The decoding strategy is dynamically adjusted through the newly added language structure identifier, and texts in different languages ​​can be flexibly processed, thereby ensuring that the generated structured output can accurately reflect the hierarchy and logical relationship of the original document.

[0054] Step S210, obtaining a digitized document corresponding to the target document image according to the spatial layout information, regional content information and structural information of each to-be-processed region.

[0055] In the technical solution provided in step S210, the step of obtaining a digitized document corresponding to the target document image based on the spatial layout information, regional content information and structural information of each area to be processed includes: for the area to be processed whose area type is a text area, merging the row recognition result and the paragraph recognition result in the area to be processed based on the spatial layout information, regional content information and structural information of the area to be processed to obtain a text recognition result; for the area to be processed whose area type is a table area, constructing a target table structure based on the spatial layout information and structural information, and filling in the corresponding table content in the target table structure based on the regional content information to obtain a table recognition result; merging the text recognition result and the table recognition result to obtain a digitized document.

[0056] In some embodiments of the present application, the step of merging text recognition results and table recognition results to obtain a digitized document includes: determining position information of the text recognition results based on spatial layout information corresponding to the text recognition results; determining position information of the table recognition results based on spatial layout information corresponding to the table recognition results; and merging the text recognition results and the table recognition results based on the position information of the text recognition results and the position information of the table recognition results to obtain a digitized document.

[0057] As an optional implementation, the document digitization processing model is trained in the following manner: determining the intersection-over-union loss function and the sequence cross entropy loss function of the document digitization processing model, the function value of the intersection-over-union loss function is used to determine the accuracy of the spatial layout information, and the function value of the sequence cross entropy loss function is used to determine the accuracy of the structural information; determining the priority of the table area and the text area, and determining the weights of the intersection-over-union loss function and the sequence cross entropy loss function based on the priority; determining the total loss function of the document digitization processing model based on the weights, the intersection-over-union loss function and the sequence cross entropy loss function, and using the total loss function to train the document digitization processing model.

[0058] Specifically, the total loss function can be defined using the weighted sum of the loss functions of the two decoders. The bounding box loss calculates the position accuracy (IoU loss), while the OTSL loss measures the accuracy of structured annotations (sequence cross entropy loss). The weights of the two loss functions are adjusted according to the task requirements (priority of table or text). By dynamically adjusting the weights of the loss functions, the model training is optimized for the characteristics of different languages ​​to improve the accuracy and robustness of multilingual recognition. It also enables the model to learn and adapt more effectively when facing multilingual documents, improving the overall performance of the model.

[0059] In some embodiments of the present application, Figure 3 As shown, the outputs of the two decoders can be combined to post-process the structure and content of the segmented regions to generate the final output representation. The main task of this step is to integrate the information of different regions into a complete document representation and output it in a structured format to facilitate downstream document parsing and analysis.

[0060] First, for the plain text area, rows and paragraphs need to be merged to ensure the coherence of the text. The reconstruction of the table area is based on the predicted cell positions and structural relationships to reconstruct the complete table layout and restore the order and content of the cells. Specifically, the OTSL tokens generated by the table structure decoder are used to guide the reconstruction process of the table. By decoding the row, column and merged cell information in OTSL, a complete table structure is constructed. On the basis of the table structure, the information output by the bounding box and content decoder is added to format the table into a standard spreadsheet or database format for subsequent data processing and export. For table elements with nested relationships, they can be further refined to various levels of substructures.

[0061] The processed tables and text areas are merged into a unified document representation. According to the original document layout information, the text blocks, tables and image areas are reassembled and arranged into the structured order of the document. For embedded image areas, corresponding placeholders can be inserted or annotations can be added according to the bounding box position of the image to ensure the overall consistency of the document.

[0062] In the format conversion and export stage, the final document representation will be converted into a structured data format, such as JSON, XML or HTML, to meet specific application requirements. In addition, the results can be exported to different types of file formats such as PDF, Word or spreadsheet to meet the diverse needs of users. This can output highly structured document representations that conform to the original layout, which is suitable for a variety of application scenarios such as electronic archive management, data mining and information retrieval.

[0063] For example, in the case of archive management, historical archives in the office area can be digitized, and paper archives can be scanned into image files using a scanner. According to the steps of the algorithm, the scanned images are analyzed for document layout and content recognition, and content information such as text, tables, and images are extracted. When the system detects archives containing complex tables, it will reconstruct the table structure based on the analysis results, and typeset and format the text content to ensure the readability and accuracy of the digitized documents.

[0064] In addition, during the processing process, if the image quality is poor or the data is incomplete, a warning can be issued to the archive management personnel, and the corresponding image files and timestamps can be saved to facilitate subsequent review and correction.

[0065] The method adopts the method of determining a region to be processed in a target document image, wherein the region type of the region to be processed includes a text region and a table region; extracting the regional visual features of the region to be processed by using an encoder in a document digitization processing model; processing the regional visual features by using a first decoder in the document digitization processing model to obtain spatial layout information and regional content information of the region to be processed; processing the regional visual features by using a second decoder in the document digitization processing model to obtain structural information of the region to be processed, wherein, when the region type of the region to be processed is a text region, the structural information includes text hierarchical structural information, and when the region type of the region to be processed is a table region, the structural information includes table hierarchical information, cell span information and cell relationship information in the table; obtaining a digitized document corresponding to the target document image according to the spatial layout information, regional content information and structural information of each region to be processed, parsing the deep structure of text and table by type through a dual decoder, achieving the purpose of finely restoring the semantics and logical relationship of the document, thereby achieving the technical effect of high-fidelity digital reconstruction of text and table content, and further solving the technical problem of poor document digitization effect caused by the inability to accurately identify the table part of the document in the related technology.

[0066] Compared with related technologies, most of them use traditional deep learning models for document processing, which usually rely on fixed feature extraction and classification strategies and are difficult to adapt to complex and changing document layouts and content structures. At the same time, traditional models have high requirements for the amount and diversity of training data, resulting in poor adaptability when facing new types of documents, limiting the generalization ability of the model.

[0067] The method provided in the embodiment of the present application can effectively analyze and identify different types of documents. In the first step, Document Layout Analysis (DLA) introduces Mask R-CNN to achieve accurate detection and classification of different areas such as text blocks, tables, and images, and generate pixel-level segmentation masks. This method significantly improves the overall efficiency and accuracy of document processing, reduces post-processing requirements, reduces computational burden, and enables the model to flexibly capture subtle details in documents, such as curved text lines and complex table structures.

[0068] In addition, the method provided in this application is based on multi-decoder table and text recognition, and uses a shared visual encoder to extract visual features, thereby enhancing the model's ability to recognize multilingual texts. The introduction of input type tokens and language information tokens enables the encoder to adaptively focus on the features of specific content types, ensuring the semantic coherence of texts in different languages. The design of the two decoders effectively associates the outputs of bounding boxes and table structures, further improving the accuracy and flexibility of multimodal recognition.

[0069] Finally, the method provided by this application combines the information of different regions into a complete document representation through post-processing and integration of the generated results, outputs a highly structured data format, and supports multiple output formats, such as JSON, XML or HTML. This process not only retains the layout structure of the original document, but also uses natural language processing technology to perform semantic correction and context information fine-tuning, further improving the accuracy and consistency of the output results.

[0070] It should also be noted that the document digitization processing method provided in the embodiment of the present application has a wide range of application scenarios and can be applied in a variety of scenarios. For example, in the multilingual document digitization scenario, it can be used to process old documents containing multiple languages ​​and digitize them into a structured text format for easy archiving and retrieval. In the application scenario of automated processing of legal documents and contracts, legal documents, contracts and other documents can be automatically OCR and parsed to help law firms or enterprises to manage documents and conduct compliance inspections. In the medical record digitization scenario, it can support the digitization of documents containing handwritten text, tables and images such as medical records and examination reports, and improve the accuracy and availability of electronic medical record systems. In financial statement and invoice processing, it is suitable for automatic OCR and data extraction of documents such as financial statements and invoices containing text and tables, simplifying accounting and auditing work. In the scenario of digitalization projects in archives and libraries, archived historical documents, books and manuscripts can be effectively digitized, maintaining the original structure while improving retrieval efficiency. In the digitalization scenario of educational and scientific research documents, it can be used to convert documents in complex formats such as textbooks and scientific research papers into editable electronic versions for knowledge sharing and further analysis.

[0071] The present application provides a document digitization processing device. Figure 4 is a schematic diagram of the structure of the device. Figure 4 It can be seen that the device includes: a first processing module 40, which is used to determine the area to be processed in the target document image, wherein the area type of the area to be processed includes a text area and a table area; a second processing module 42, which is used to extract the regional visual features of the area to be processed by using the encoder in the document digitization processing model; a third processing module 44, which is used to process the regional visual features by using the first decoder in the document digitization processing model to obtain the spatial layout information and regional content information of the area to be processed; a fourth processing module 46, which is used to process the regional visual features by using the second decoder in the document digitization processing model to obtain the structural information of the area to be processed, wherein, in the case where the area type of the area to be processed is a text area, the structural information includes text hierarchical structure information, and in the case where the area type of the area to be processed is a table area, the structural information includes table hierarchical information, cell span information and cell relationship information in the table; a fifth processing module 48, which is used to obtain a digitized document corresponding to the target document image based on the spatial layout information, regional content information and structural information of each area to be processed.

[0072] In some embodiments of the present application, the second processing module 42 uses an encoder in a document digitization processing model to extract regional visual features of the area to be processed, including: determining a type token of the area to be processed, wherein the type token is used to indicate the area type of the area to be processed; inputting the area to be processed and the type token corresponding to the area to be processed into the encoder, and obtaining the regional visual features extracted by the encoder from the area to be processed based on the type token, wherein the feature type of the regional visual features includes at least one of the following: table line features, text line spacing features, and character shape features.

[0073] In some embodiments of the present application, the spatial layout information includes region bounding box position information of the area to be processed and element bounding box position information of each element in the area to be processed, wherein the element type of the element includes text blocks and cells.

[0074] In some embodiments of the present application, the third processing module 44 uses the first decoder in the document digitization processing model to process the regional visual features, and the step of obtaining the regional content information of the area to be processed includes: obtaining the language information token output by the encoder, wherein the language information token is used to indicate the text language of the area to be processed; extracting and identifying text features from the regional visual features based on the language information token and the spatial layout information to obtain the regional content information, wherein the regional content information includes the text content recognition results in the area to be processed.

[0075] In some embodiments of the present application, the fifth processing module 48 obtains a digitized document corresponding to the target document image based on the spatial layout information, regional content information and structural information of each area to be processed, and the steps include: for the area to be processed whose area type is a text area, merging the row recognition results and the paragraph recognition results in the area to be processed according to the spatial layout information, regional content information and structural information of the area to be processed to obtain a text recognition result; for the area to be processed whose area type is a table area, constructing a target table structure based on the spatial layout information and structural information, and filling in the corresponding table content in the target table structure based on the regional content information to obtain a table recognition result; merging the text recognition result and the table recognition result to obtain a digitized document.

[0076] In some embodiments of the present application, the fifth processing module 48 merges the text recognition results and the table recognition results, and the steps of obtaining a digitized document include: determining the position information of the text recognition results based on the spatial layout information corresponding to the text recognition results; determining the position information of the table recognition results based on the spatial layout information corresponding to the table recognition results; merging the text recognition results and the table recognition results based on the position information of the text recognition results and the position information of the table recognition results to obtain a digitized document.

[0077] In some embodiments of the present application, the document digitization processing model is trained in the following manner: determining the intersection-over-union loss function and the sequence cross entropy loss function of the document digitization processing model, the function value of the intersection-over-union loss function is used to determine the accuracy of the spatial layout information, and the function value of the sequence cross entropy loss function is used to determine the accuracy of the structural information; determining the priority of the table area and the text area, and determining the weights of the intersection-over-union loss function and the sequence cross entropy loss function based on the priority; determining the total loss function of the document digitization processing model based on the weight, the intersection-over-union loss function and the sequence cross entropy loss function, and using the total loss function to train the document digitization processing model.

[0078] It should be noted that the various modules in the above-mentioned document digitization processing device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0079] According to an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following document digitization processing method: determining a region to be processed in a target document image, wherein the region types of the region to be processed include text regions and table regions; using an encoder in a document digitization processing model to extract regional visual features of the region to be processed; using a first decoder in the document digitization processing model to process the regional visual features to obtain spatial layout information and regional content information of the region to be processed; using a second decoder in the document digitization processing model to process the regional visual features to obtain structural information of the region to be processed, wherein, when the region type of the region to be processed is a text region, the structural information includes text hierarchical structure information, and when the region type of the region to be processed is a table region, the structural information includes table hierarchical information, cell span information, and cell relationship information in the table; based on the spatial layout information, regional content information, and structural information of each region to be processed, a digitized document corresponding to the target document image is obtained.

[0080] According to an embodiment of the present application, an electronic device is also provided, including a memory and a processor, the processor being used to run a program stored in the memory, wherein the following document digitization processing method is executed when the program is running: determining an area to be processed in a target document image, wherein the area types of the area to be processed include text areas and table areas; using an encoder in a document digitization processing model to extract regional visual features of the area to be processed; using a first decoder in the document digitization processing model to process the regional visual features to obtain spatial layout information and regional content information of the area to be processed; using a second decoder in the document digitization processing model to process the regional visual features to obtain structural information of the area to be processed, wherein, when the area type of the area to be processed is a text area, the structural information includes text hierarchical structure information, and when the area type of the area to be processed is a table area, the structural information includes table hierarchical information, cell span information, and cell relationship information in the table; based on the spatial layout information, regional content information, and structural information of each area to be processed, a digitized document corresponding to the target document image is obtained.

[0081] According to an embodiment of the present application, a computer program product is also provided, including a computer program, which implements the following document digitization processing method when executed by a processor: determining a region to be processed in a target document image, wherein the region types of the region to be processed include text regions and table regions; using an encoder in a document digitization processing model to extract regional visual features of the region to be processed; using a first decoder in the document digitization processing model to process the regional visual features to obtain spatial layout information and regional content information of the region to be processed; using a second decoder in the document digitization processing model to process the regional visual features to obtain structural information of the region to be processed, wherein, when the region type of the region to be processed is a text region, the structural information includes text hierarchical structure information, and when the region type of the region to be processed is a table region, the structural information includes table hierarchical information, cell span information, and cell relationship information in the table; based on the spatial layout information, regional content information, and structural information of each region to be processed, a digitized document corresponding to the target document image is obtained.

[0082] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0085] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0087] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A document digital processing method, characterized in that: include: Determining a region to be processed in the target document image, wherein the region types of the region to be processed include a text region and a table region; Using an encoder in a document digitization processing model to extract regional visual features of the area to be processed; Using the first decoder in the document digitization processing model to process the regional visual features, and obtain the spatial layout information and regional content information of the to-be-processed region; The second decoder in the document digitization processing model is used to process the regional visual features to obtain structural information of the area to be processed, wherein, when the area type of the area to be processed is a text area, the structural information includes text hierarchical structural information, and when the area type of the area to be processed is a table area, the structural information includes table hierarchical information, cell span information, and cell relationship information in the table; According to the spatial layout information, regional content information and structural information of each of the to-be-processed regions, a digitized document corresponding to the target document image is obtained.

2. The document digitization processing method according to claim 1, characterized in that: The method of extracting the regional visual features of the area to be processed by using the encoder in the document digitization processing model includes: Determine a type token of the area to be processed, wherein the type token is used to indicate the area type of the area to be processed; The area to be processed and the type token corresponding to the area to be processed are input into the encoder, and the area visual features extracted by the encoder from the area to be processed based on the type token are obtained, and the feature types of the area visual features include at least one of the following: table line features, text line spacing features, and character shape features.

3. The document digitization processing method according to claim 1, characterized in that: The spatial layout information includes the region bounding box position information of the region to be processed and the element bounding box position information of each element in the region to be processed, wherein the element type of the element includes a text block and a cell.

4. The document digitization processing method according to claim 1, characterized in that: The first decoder in the document digitization processing model is used to process the regional visual features to obtain the regional content information of the area to be processed, including: Acquire a language information token output by the encoder, wherein the language information token is used to indicate the text language of the area to be processed; The text features are extracted and recognized from the regional visual features according to the language information token and the spatial layout information to obtain the regional content information, wherein the regional content information includes the text content recognition result in the area to be processed.

5. The document digitization processing method according to claim 1, characterized in that: According to the spatial layout information, regional content information and structural information of each of the to-be-processed regions, a digitized document corresponding to the target document image is obtained, including: For the area to be processed whose area type is a text area, combining the line recognition result and the paragraph recognition result in the area to be processed according to the spatial layout information, area content information and structure information of the area to be processed to obtain a text recognition result; For the area to be processed whose area type is a table area, a target table structure is constructed according to the spatial layout information and the structural information, and corresponding table content is filled in the target table structure according to the area content information to obtain a table recognition result; The text recognition result and the table recognition result are combined to obtain the digitized document.

6. The document digitization processing method according to claim 5, characterized in that: Combining the text recognition result and the table recognition result to obtain the digitized document includes: Determining position information of the text recognition result according to the spatial layout information corresponding to the text recognition result; Determining the position information of the table recognition result according to the spatial layout information corresponding to the table recognition result; The text recognition result and the table recognition result are combined according to the position information of the text recognition result and the position information of the table recognition result to obtain the digitized document.

7. The document digitization processing method according to claim 1, characterized in that: The document digitization processing model is trained in the following way: Determine an intersection-over-union loss function and a sequence cross entropy loss function of the document digitization processing model, wherein the function value of the intersection-over-union loss function is used to determine the precision of the spatial layout information, and the function value of the sequence cross entropy loss function is used to determine the accuracy of the structural information; Determine the priorities of the table area and the text area, and determine the weights of the intersection-over-union loss function and the sequence cross entropy loss function according to the priorities; The total loss function of the document digitization processing model is determined according to the weight, the intersection-over-union loss function and the sequence cross entropy loss function, and the document digitization processing model is trained using the total loss function.

8. A document digital processing device, characterized in that: include: A first processing module is used to determine a region to be processed in the target document image, wherein the region types of the region to be processed include a text region and a table region; A second processing module is used to extract regional visual features of the area to be processed by using an encoder in the document digitization processing model; A third processing module is used to process the regional visual features using the first decoder in the document digitization processing model to obtain the spatial layout information and regional content information of the to-be-processed region; A fourth processing module is used to process the regional visual features using the second decoder in the document digitization processing model to obtain structural information of the area to be processed, wherein, when the area type of the area to be processed is a text area, the structural information includes text hierarchical structural information, and when the area type of the area to be processed is a table area, the structural information includes table hierarchical information, cell span information, and cell relationship information in the table; The fifth processing module is used to obtain a digitized document corresponding to the target document image according to the spatial layout information, regional content information and structural information of each of the to-be-processed regions.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the document digitization processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the document digitization processing method described in any one of claims 1 to 7 when running.

11. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the document digitization processing method according to any one of claims 1 to 7.

Images