Document analysis method and device, equipment, storage medium and product

Abstract

The invention provides a document analysis method and device, equipment, a storage medium and a product, and the method comprises the steps: carrying out character prediction based on a document page image and text prompt information indicating analysis, and generating a target character; performing character prediction based on the target character and a set prediction guide unit, and generating at least two characters of which the character positions are ranked behind the target character; if the ending character is not obtained, taking at least two characters as target characters, and performing character prediction again based on the target characters and the prediction guide unit until the ending character is obtained; and based on the characters obtained by each prediction, an analysis result is generated, and the analysis result comprises text information used for representing the content elements in the document page image and the structural relationship between the content elements. According to the technical scheme, the prediction guiding unit is introduced, so that multiple characters can be predicted in parallel in single character prediction, and the document analysis speed is increased while the document analysis accuracy is kept.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and specifically relates to a document parsing method, apparatus, device, storage medium, and product. Background Technology

[0002] Document parsing is a technology that converts unstructured or semi-structured documents into structured data that electronic devices can directly process. Its core lies in accurately identifying and reconstructing content elements such as text, images, formulas, and tables in a document, and restoring their logical order and hierarchical relationship in the original document.

[0003] In related technologies, documents are usually parsed using large-scale visual language models (VLMs). While this method can improve the accuracy of document parsing, the model is based on an autoregressive sequence generation paradigm, which means that each character in the sequence is generated sequentially. The generation of each character depends on the characters that have been generated before, which means that the generation is done character by character in a single step. This method is slow and affects the efficiency of document parsing.

[0004] Therefore, how to improve document parsing speed while maintaining accuracy is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] This application provides a document parsing method, apparatus, device, storage medium, and product. By introducing a prediction guidance unit, the character prediction process can generate not only the corresponding predicted character based on the target character, but also the corresponding predicted character based on the prediction guidance unit. This allows multiple characters to be predicted in parallel in a single prediction process, thereby improving the speed of document parsing.

[0006] In a first aspect, embodiments of this application provide a document parsing method, the method comprising: Based on the document page image to be parsed and the text prompt information indicating that the document page image should be parsed, character prediction processing is performed to generate target characters; Based on the target character and the set prediction guidance unit, character prediction processing is performed to generate at least two characters whose positions are after the target character; wherein, the prediction guidance unit is used to predict characters whose positions are after the target character and are not adjacent to the target character; If the obtained at least two characters do not contain the end character, then the at least two characters are taken as the target characters, and character prediction processing is performed again based on the target characters and the prediction guidance unit until the obtained at least two characters contain the end character; Based on the characters obtained from each character prediction process, a parsing result corresponding to the document page image is generated. The parsing result contains text information that characterizes the content elements in the document page image and the structural relationships between the content elements.

[0007] Secondly, embodiments of this application provide a document parsing apparatus, the apparatus comprising: The first character prediction processing unit is used to perform character prediction processing based on the document page image to be parsed and the text prompt information indicating that the document page image should be parsed, so as to generate target characters; The second character prediction processing unit is used to perform character prediction processing based on the target character and the set prediction guidance unit to generate at least two characters whose character positions are after the target character; wherein, the prediction guidance unit is used to predict characters whose character positions are after the target character and are not adjacent to the target character; The third character prediction processing unit is used to, if the obtained at least two characters do not contain an end character, take the at least two characters as target characters and perform character prediction processing again based on the target characters and the prediction guidance unit until the obtained at least two characters contain the end character; The generation unit is used to generate a parsing result corresponding to the document page image based on the characters obtained from each character prediction process. The parsing result includes text information that characterizes the content elements in the document page image and the structural relationships between the content elements.

[0008] Thirdly, embodiments of this application provide an electronic device, which includes: one or more processors; and a memory for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the document parsing method of the first aspect described above.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the document parsing method of the first aspect described above.

[0010] Fifthly, embodiments of this application provide a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the document parsing method as described in the first aspect above.

[0011] In some embodiments of this application, character prediction processing is performed based on the document page image to be parsed and text prompts indicating that parsing is to be performed, to obtain a target character. Next, character prediction processing is performed based on the target character and a predefined prediction guidance unit to generate at least two characters whose positions follow the target character. The prediction guidance unit predicts characters whose positions follow the target character and are not adjacent to it. If the at least two characters obtained do not contain an end character, these at least two characters are used as the target character, and character prediction processing is performed again based on the target character and the prediction guidance unit until the at least two target characters contain an end character. Finally, based on the characters obtained from each character prediction process, a parsing result corresponding to the document page image is generated. This parsing result includes text information characterizing the content elements in the document page image and the structural relationships between the content elements.

[0012] Therefore, by introducing a pre-defined prediction guidance unit, in each character prediction process, not only can the corresponding predicted character be generated based on the target character, but also the corresponding predicted character can be generated based on the prediction guidance unit, allowing multiple characters to be generated in parallel during a single prediction process. Compared to prediction methods that can only generate a single character at a time, this method of generating multiple characters at a time can improve the speed of document parsing and reduce the latency of document parsing. Furthermore, by combining the multiple characters obtained from each character prediction process to obtain the parsing result, unstructured or semi-structured documents can be quickly converted into structured data that electronic devices can directly process, thereby improving the overall document parsing efficiency. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a document parsing method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a process for obtaining a document page image to be parsed, provided in an embodiment of this application. Figure 4 This is a schematic diagram of the attention feature cache replacement process provided in an embodiment of this application; Figure 5 This is a schematic diagram of the optimized attention mask provided in the embodiments of this application; Figure 6 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 7 This is a flowchart illustrating a document parsing method provided in an embodiment of this application; Figure 8 This is a schematic diagram of the process for obtaining training data provided in an embodiment of this application; Figure 9 This is a schematic diagram of the document parsing model provided in an embodiment of this application; Figure 10 This is a schematic diagram of the performance test results of the document parsing model provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of a document parsing device provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a computer system for an electronic device provided in an embodiment of this application. Detailed Implementation

[0015] It should be noted in advance that, in order to enable those skilled in the art to better understand the technical solutions proposed in the embodiments of this application, the embodiments of this application will be described clearly and completely in conjunction with one or more accompanying drawings. Furthermore, the various drawings shown in the embodiments of this application are merely illustrative examples; for example, the execution order of each step in the drawings can be adaptively adjusted according to the actual application scenario. In addition, in the embodiments of this application, the block diagrams shown in the various drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0016] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0017] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0018] Document parsing refers to the technology of converting unstructured or semi-structured documents (such as PPT, PDF, and DOC formats) into machine-readable structured data output (such as Markdown and JSON formats). Its core lies in accurately identifying and reconstructing various content elements in the original document, such as text, images, formulas, and tables, while restoring their logical order and hierarchical relationships within the original document. Document parsing is an important method for multimodal parsing, document- or knowledge-based question answering, retrieval-augmented generation (RAG), and building large model training corpora, and it is also a crucial pathway for machines to understand the digital world.

[0019] With the development of artificial intelligence technology, end-to-end document parsing based on Visual Learning Models (VLM) is gradually becoming mainstream. Due to VLM's powerful visual and semantic understanding capabilities, this method can improve the accuracy of document parsing. However, on the one hand, VLM uses an autoregressive sequence generation paradigm (Next-Token Prediction, NTP), which refers to generating each character (token) in a sequence sequentially. The generation of each character depends on previously generated characters, meaning only one character can be generated at a time, resulting in slow inference speed and latency bottlenecks in the inference process. On the other hand, end-to-end document parsing uses the entire document as input to obtain the parsed result from VLM. However, this approach may be limited by the context window length of VLM, making it unable to handle longer sequences, leading to higher computational complexity and problems such as task failure, decreased accuracy, and slow speed, further limiting document parsing efficiency.

[0020] Based on this, this application provides a document parsing method. By introducing a predefined prediction guidance unit, the method can not only predict characters based on the target character during character prediction processing, but also synchronously predict additional characters according to the prediction guidance unit, thereby achieving parallel generation of multiple characters in a single prediction. Compared to the character-by-character prediction (NTP) method, this parallel-to-token prediction (PTP) method can improve the speed of document parsing and reduce the latency of document parsing. Furthermore, the document page image to be parsed consists of multiple local images, each corresponding to different content elements. By performing character prediction processing on multiple local images in parallel, and then combining the characters obtained from each character prediction processing, the parsing result is obtained. The parsing result includes structured data representing various content elements and the structural relationships between content elements. Compared to directly parsing a high-resolution complete document page image, this method of parsing multiple local images in parallel can not only alleviate resolution limitations and reduce computational complexity, but also quickly convert unstructured or semi-structured documents into structured data that electronic devices can directly process, thereby improving the overall parsing efficiency and accuracy.

[0021] Based on the above description, please refer to Figure 1 , Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. For example... Figure 1 As shown, the implementation environment includes a document parsing device 101 and a display device 102. The document parsing device 101 and the display device 102 can be directly or indirectly connected via wired or wireless means. It should be noted that... Figure 1 The number and form of the devices shown are for illustrative purposes only and do not constitute a limitation on the embodiments of this application.

[0022] In some embodiments, the number of display devices 102 can be multiple, each corresponding to an electronic device used by a different user. The display device 102 can receive interactive content input by the user, such as selecting or uploading a document page to be parsed, and a text prompt indicating that parsing should be performed. It then sends this interactive content to the document parsing device 101 to trigger the document parsing process. Simultaneously, the display device 102 can also receive and output the parsing results generated by the document parsing device 101 for the user to view.

[0023] In some embodiments, the document parsing device 101 and the display device 102 may be the same electronic device, that is, the electronic device can be used not only for document parsing but also for displaying the parsing results.

[0024] The document parsing device 101 can be an electronic device for implementing document parsing. It can be equipped with a machine learning model (document parsing model) that performs character prediction processing on document page images. It can be used to parse the input document page images to generate document parsing results.

[0025] The display device 102 can be an electronic device used by a user, and may include output devices such as a display screen, speakers, etc., for displaying in real time the parsing results generated by the document parsing device 101 (including text information about content elements in the document page image and the structural relationships between content elements, i.e., structured data). Optionally, the display device 102 may also include input devices such as a touch screen, keyboard, mouse, microphone, etc. In real-time interactive application scenarios, users can input the document page to be parsed and text prompts indicating that parsing is to be performed through the input devices, and receive the corresponding parsing results.

[0026] In some embodiments, the implementation environment may also include a database, which can be used to cache data in document parsing. For example, after obtaining a target character during the document parsing process, the document parsing device 101 can calculate and cache the attention feature corresponding to the target character so that when predicting subsequent characters, the attention feature can be directly called from the database to further improve the parsing speed.

[0027] The document parsing method provided in this application can be executed by the document parsing device 101. The document parsing device 101 and the display device 102 can be terminal devices or servers. Terminal devices include, but are not limited to, smartphones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile internet devices (MIDs), smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc., and this application embodiment does not limit this. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and this application embodiment does not limit this.

[0028] The general process of the document parsing method provided in this application is as follows: The document parsing device 101 can perform character prediction processing based on the image of the document page to be parsed and textual indication information for instructing the parsing of the document page image to generate a target character. Furthermore, the document parsing device 101 can perform character prediction processing based on the target character and a set prediction guidance unit to generate at least two characters whose positions follow the target character. The prediction guidance unit is used to predict characters whose positions follow the target character and are not adjacent to it. If the resulting at least two characters do not contain an end character, the document parsing device 101 can use these at least two characters as the target character and perform character prediction processing again based on the target character and the prediction guidance unit until the resulting at least two characters contain an end character.

[0029] Subsequently, the document parsing device 101 can generate a parsing result corresponding to the document page image based on the characters obtained from each character prediction process. The parsing result includes text information that characterizes the content elements in the document page image and the structural relationships between the content elements. After obtaining the parsing result, the document parsing device 101 can send the parsing result to the display device 102 so that the display device 102 can display the parsing result.

[0030] In some embodiments, the document parsing device 101 may also send the obtained characters to the display device 102 in real time after each character prediction process, so that the display device 102 gradually displays the parsed content until the complete parsing result is presented.

[0031] In some embodiments, the display device 102 can also receive interactive content input by the user, and then send the interactive content to the document parsing device 101. The document parsing device 101 can perform document parsing based on the interactive content input by the user, and send the parsing result back to the display device 102.

[0032] Therefore, based on Figure 1 The implementation environment shown introduces a prediction guidance unit for character prediction processing, enabling multiple characters to be generated in parallel for each prediction, which significantly improves the document parsing speed while ensuring the accuracy of document parsing.

[0033] It is understood that the embodiments described in this application are as follows: Figure 1 The implementation environment shown is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0034] Based on the above implementation environment, this application provides a document parsing method. The video generation method described in this application can be executed by an electronic device, which can be... Figure 1 The document parsing device 101 in the implementation environment shown. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a document parsing method provided in an embodiment of this application. The document parsing method includes the following steps S201-S204: S201. Based on the document page image to be parsed and the text indication information for indicating that the document page image should be parsed, perform character prediction processing to generate target characters.

[0035] In this embodiment, the document parsing device can acquire the document page to be parsed and text prompts indicating that parsing is to be performed. The document page to be parsed can come from different types of original documents (e.g., PDF, PPT, DOC, etc.), and each original document can include one or more document pages. For example, the document parsing device can receive any original document uploaded or selected by the user on the display device, or any document page from that original document. The document parsing device can split the received original document into one or more document pages to be parsed, or use the received document pages as the document pages to be parsed.

[0036] Furthermore, document parsing devices can convert the document pages to be parsed into high-resolution images, for example, by scanning the document pages to obtain images of the pages to be parsed. This method transforms the parsing of document files into the parsing of images, reducing the overall complexity of document parsing and, to some extent, improving the speed of document parsing.

[0037] Specifically, the document parsing device can acquire an initial document image containing at least one content element. Different content elements are presented in different forms within the initial document image. The initial document image is input into a layout analysis model to segment local images corresponding to various content elements from the initial document image. These segmented local images are then identified as the document page images to be parsed.

[0038] Please refer to the following: Figure 3 , Figure 3This is a flowchart illustrating a process for obtaining a document page image to be parsed, provided in an embodiment of this application. The initial document image is the document parsing device converting the document page to be parsed into a high-resolution image. The initial document image may contain one or more content elements, such as text (paragraphs), tables, formulas, titles, headers and footers, annotations, etc., and different content elements are presented in different forms in the initial document image. The document parsing device can input the initial document image into a pre-trained layout analysis model. The layout analysis model can perform an overall visual understanding of the initial document image, identify one or more content elements contained in the initial document image and the positions of various content elements in the initial document image, and then segment the local images corresponding to various content elements from the initial document image. Furthermore, the document parsing device can use the local image corresponding to each content element as the document page image to be parsed.

[0039] Understandably, a document parsing device can parse multiple local images (i.e., images of the document page to be parsed) in parallel, obtain the parsing result corresponding to each local image, and finally arrange them according to the reading order to form a complete parsing result (such as a complete Markdown document) corresponding to the document page to be parsed.

[0040] It should be noted that the document parsing device can divide the initial document image into multiple independent local images and send them in parallel to the document parsing model for document parsing. This method of dividing a large image into multiple small images and then inputting them into the model can alleviate the resolution limitations of VLM to a certain extent compared to directly inputting a large image, thereby improving the ability to parse large or information-dense document pages.

[0041] For ease of description, this application uses a Document Parsing Model (VLM) as an example. The text prompt can be a prompt word for the VLM. This text prompt can be any type of information used to guide the VLM to perform language understanding on the image of the document page to be parsed, such as "Output the text from the image." In some embodiments, the text prompt can also use other modal information, such as image-type information or speech-type information, that is, guiding the VLM to parse the image of the document page through image or speech prompts. It can also be multimodal information (such as simultaneously including text prompts, image prompts, and speech prompts), which can be determined according to the actual application scenario. For example, the text prompt can be any text input by the user on the display device 102, the image prompt can be any image uploaded or imported by the user on the display device 102, and the speech prompt can be any speech collected by the user through the display device 102.

[0042] Character prediction processing refers to the process by which VLM predicts and generates characters in a serialized manner based on the input image and text prompts. The target character is the first character predicted by VLM based on the document page image and text prompts.

[0043] In one possible implementation, the document parsing device can extract image features corresponding to the document page image and text features corresponding to the text indication information. The image and text features are then input into a pre-trained document parsing model. The feature processing module in the document parsing model performs feature operations on the concatenated features obtained from the image and text features to obtain the first character prediction feature. Finally, the prediction module in the document parsing model performs character mapping processing on the first character prediction feature to obtain the target character.

[0044] Specifically, document parsing devices can employ preset rules, such as image feature extraction based on color, texture, and shape, and text feature extraction based on keywords and word frequency. Alternatively, they can use machine learning methods, such as VLM or other models, to automatically learn the semantic features of images or text from shallow to deep layers, in order to extract the image features of the document page image and the text features corresponding to the text indication information. Furthermore, the document parsing device can perform character prediction processing based on the image and text features to obtain the target characters.

[0045] In some embodiments, the VLM may include a feature processing module that maps the input image and text to corresponding image features and text features. For example, after inputting the document page image and text prompts into the document parsing model, the model can divide the document page image into multiple image units (patches). A linear transformation is performed on each image unit to obtain initial image features (patch embeddings), which are feature vector representations, also known as embedding vector representations. Then, a position embedding is assigned to each image unit based on its position information within the document page image. This position embedding is a feature vector representation indicating the image unit's position within the document page image. This yields the image features corresponding to each image unit, such as the sum / concatenation of the initial image features and the position embedding (patch embedding + position embedding). Thus, by combining the image features corresponding to multiple image units sequentially, the image features corresponding to the document page image, i.e., the initial vector representation for serialization, can be obtained.

[0046] Similarly, this feature processing module can segment the text prompt information to obtain multiple text units (prompt tokens), and determine the unique index (token ID) corresponding to each text unit based on a predefined vocabulary. Based on the index, it queries the preset embedding table for the initial prompt feature (token embedding) corresponding to each text unit, and combines this with the position information of each text unit within the text prompt information to assign a position encoding (position embedding) to each text unit. This yields the prompt feature corresponding to each text unit, such as (token embedding + position embedding) obtained by adding / concatenating the feature vectors of the initial prompt feature and the position encoding. Therefore, by combining the prompt features corresponding to multiple text units in sequence, the text features corresponding to the text prompt information can be obtained, which is the initial vector representation for serialization.

[0047] In some embodiments, the document parsing model may further include a feature processing module and a prediction module. The feature processing module can refer to the backbone of the VLM, which can be a neural network model / submodule based on a Transformer architecture. The Transformer relies on a self-attention mechanism with independent model parameters for sequence processing. This self-attention mechanism can model contextual relationships and semantic features at different levels to obtain a deep semantic representation that can be used to predict subsequent characters. The prediction module may contain a prediction head, which can be understood as a structure containing network layers, such as linear layers and fully connected layers. This head can be used to map the deep semantic representation output by the feature processing module to a predefined character space (token space) to obtain a probability distribution. Then, the currently predicted character, i.e., the target character, is generated through greedy selection (selecting the one with the highest probability) or sampling selection (selecting from the top k characters with the highest probability).

[0048] Specifically, please refer to Figure 1 In Step 0, the document parsing model performs the first character prediction process as follows: First, it concatenates the image features corresponding to the document page image and the text features corresponding to the text prompt information to obtain the concatenated features (also called the input sequence). For example: {Image Feature 1, Image Feature 2...Image Feature n, Text Feature 1, Text Feature 2,...Text Feature n}, where Image Feature 1 corresponds to image unit 1 of the document page image, and Text Feature 1 corresponds to text unit 1 obtained from the word segmentation of the text prompt information, and so on. Then, through the feature operation module, the input sequence is processed layer by layer (feature operation processing), which gradually transforms the initial vector representation in the input sequence from a shallow semantic representation to a deep semantic representation, obtaining the first character prediction feature.

[0049] For example, the concatenated input sequence is input into the feature processing module. The first network layer in this feature processing module performs a linear transformation on each feature in the input sequence to obtain the attention features corresponding to each unit (including image units and text units). These attention features include query features (Q), key features (K), and value features (V). The feature processing module can then perform self-attention processing based on these attention features. For instance, based on the Q, K, and V of the last unit (e.g., the text unit n corresponding to the last text feature n in the input sequence), and the key and value features of all units preceding it, a hidden vector (a shallower semantic representation) is calculated. This hidden vector is then used as the input to the second network layer in the feature processing module. The hidden vector corresponding to the second network layer is calculated based on the self-attention mechanism. This hidden vector represents a deeper semantic representation than the input hidden vector. This process continues until the hidden vector obtained after m network layers is the deepest semantic representation, i.e., the first character prediction feature.

[0050] Furthermore, the document parsing device can perform character mapping processing on the predicted features of the first character through the prediction module to obtain the probability of each character in the predefined character space, and then select a character as the target character based on preset rules, such as... Figure 1 Box 1 in Step 0 represents the target character (token1).

[0051] In some embodiments, an additional start character (such as...) can also be added to the input sequence. <bos> 、 <s>、 <start>), used to instruct the document parsing model to initiate character prediction processing.

[0052] In some embodiments, after obtaining QKV, the document parsing device can store the K and V of each image unit and each text unit in a designated storage space since the KV will be reused subsequently. This allows the data to be directly read during subsequent character prediction processing without recalculation, thereby effectively speeding up inference.

[0053] Step S202: Perform character prediction processing based on the target character and the set prediction guidance unit to generate at least two characters whose positions are after the target character; wherein, the prediction guidance unit is used to predict characters whose positions are after the target character and are not adjacent to the target character.

[0054] In this embodiment, the document parsing device can perform character prediction processing based on a target character and a set prediction guidance unit. The prediction guidance unit can be a feature vector used to guide the prediction of the character corresponding to a specified character position. During the character prediction processing, it can collect and summarize contextual information from the characters preceding the specified character position to predict the character at that position. Therefore, this prediction guidance unit can also be called a register. The number of prediction guidance units is a set number of characters (which can be one or more). The prediction guidance unit can be combined with the initial character features corresponding to the target character as input for the next character prediction processing.

[0055] Specifically, in the next character prediction process, the document parsing device can input the target character into a pre-trained document parsing model. The feature processing module within the model maps the target character to initial character features and generates a prediction guidance unit based on the character position corresponding to the target character. The feature operation module in the document parsing model then performs feature operations on the initial character features and the prediction guidance unit to obtain at least two second character prediction features. Finally, the prediction module in the document parsing model performs character mapping processing on these at least two second character prediction features to obtain at least two characters following the target character.

[0056] Taking the document parsing model VLM as an example, its feature processing module can not only map target characters to initial character features, but its mapping process is also the same as the feature processing module's process of mapping text prompts to text features, which will not be elaborated here. In the process of generating prediction guidance units based on the character positions corresponding to the target characters, pre-trained initial prediction guidance units can be obtained first. Based on the character positions corresponding to the target characters and a set number of characters, the position information corresponding to each initial prediction guidance unit is determined. Then, the initial prediction guidance units with a set number of characters and the position information corresponding to each initial prediction guidance unit are fused to generate prediction guidance units with a set number of characters.

[0057] Specifically, the feature processing module can determine the index of the character position (position ID) corresponding to the target character based on a predefined vocabulary, and then find the corresponding embedding feature vector (token embedding) from the pre-trained embedding table based on this index. The obtained embedding feature vector is then used as the initial prediction guidance unit. Setting the number of characters can refer to the number of characters to be predicted in parallel during a single character prediction process minus the number of predicted characters corresponding to the target character. For example... Figure 1 In Step 1, we want to predict characters 2, 3, and 4 in parallel, with a total of 3 characters. The target character (character 1) corresponds to a predicted character (character 2), with a total of 1 character. Therefore, the number of characters set in Step 1 can be obtained by subtracting one from the total number of characters to be predicted in parallel. For example... Figure 1 In Step 2, we want to predict characters 3, 4, 5, 6, and 7 in parallel, with a total of 5 characters. The target characters (characters 2, 3, and 4) have a total of 3 characters. Therefore, the number of characters to be predicted in Step 2 can be obtained by subtracting the number of predicted characters corresponding to the target characters from the number of characters to be predicted in parallel.

[0058] Furthermore, the feature processing module can determine the position information corresponding to each initial prediction guidance unit based on the character generation order (i.e., character position). For example, if the target character is the first character, its corresponding character position is 1, i.e., positionid is 1. Based on the preset positional relationship between the initial prediction guidance unit and the initial character features, a corresponding character position can be assigned to each initial prediction guidance unit. The character position corresponding to the first initial prediction guidance unit after the initial character features is 2, which can be obtained by adding one to the character position corresponding to the target character. The character position corresponding to the second initial prediction guidance unit after the initial character features is 3, and so on. Thus, the position information corresponding to each initial prediction guidance unit is determined based on the character position of the target character and the pre-trained position embedding table; this can also be called position encoding and can be represented by an embedded feature vector (position embedding). Finally, the feature processing module fuses each initial prediction guidance unit and its corresponding position information, such as by addition or concatenation, to obtain a feature vector (token embedding + position embedding) with a set number of characters, which is the obtained prediction guidance unit.

[0059] It is understandable that each prediction guidance unit can share the same initial prediction guidance unit, but the position information corresponding to each prediction guidance unit is different, that is, it has its own independent position code.

[0060] Specifically, in the process of performing feature operations on the initial character features and the prediction guidance unit to obtain at least two second character prediction features, the feature operation module can perform attention operations based on the initial character features and the prediction guidance unit to generate a first attention feature corresponding to the initial character features and a second attention feature corresponding to the prediction guidance unit. The feature operation module then performs feature operations on the first and second attention features to obtain at least two second character prediction features.

[0061] Please refer to the following: Figure 1 In Step 1 of the document parsing model, the feature processing module can map the target character (e.g., character 1) to initial character features, and determine each prediction guidance unit based on the character position corresponding to the target character and the set number of characters. Figure 1 The following diagram is drawn using two predictive guidance units as an example. Figure 1 (As shown in the two boxes R in the document parsing model). The feature processing module in the document parsing model can perform attention operations on the initial character features and the prediction guidance unit. For example, it first performs a linear transformation on the initial character features to obtain the first attention features (Q, K, V) corresponding to the initial character features, and performs a linear transformation on the prediction guidance unit to obtain the second attention features (Q, K, V) corresponding to the prediction guidance unit. Then, based on the self-attention mechanism and the optimized attention mask, at least two second character prediction features are calculated.

[0062] Subsequently, the prediction module in the document parsing model can perform character mapping processing on at least two second character prediction features to obtain the probability distribution of at least two character spaces. Based on preset rules, a character is selected from the probability distribution of different character spaces, and finally at least two characters whose positions are after the target character (such as character 2, character 3 and character 4 as shown in Step 1) are obtained. That is, the number of predicted characters is the sum of the number of target characters and the number of prediction guidance units (i.e., the set number of characters).

[0063] It's worth noting that by introducing prediction guidance units with a set number of characters, VLM can predict not only the character at the next position based on the target character during inference, but also the characters at the corresponding positions after the next character position based on the prediction guidance units. For example, setting one prediction guidance unit after the initial character feature corresponding to character 1 (the target character) can simultaneously predict characters 2 and 3, which are positioned after character 1. Setting two prediction guidance units after the initial character feature corresponding to character 1 can simultaneously predict characters 2, 3, and 4, which are positioned after character 1, and so on. Thus, a single character prediction process can predict multiple characters in parallel. While maintaining the accuracy of document parsing, VLM can overcome the inference latency caused by the character-by-character generation paradigm, thereby improving document parsing speed.

[0064] In some embodiments, after obtaining the first attention feature corresponding to the initial character features and the second attention feature corresponding to the prediction guidance unit, the key-value pairs in these attention features can be stored in a designated storage space. See also... Figure 1 and Figure 4 In Step 1, after the document parsing model obtains the first attention feature corresponding to the initial character feature of character 1 and the second attention feature corresponding to the two prediction guidance units, it can cache the K and V in these attention features. This specified storage space is the storage space used by the document parsing device to cache K and V, such as the video memory in a graphics processor.

[0065] This is because the second attention features corresponding to these two prediction guidance units can be directly used as the key-value pairs (KV) for characters 2 and 3 in some embodiments. In related technologies, in the next character prediction process, only the last character generated in this round of character prediction (e.g., character 4) can be used as the target character, combined with the set prediction guidance units (e.g., two R's) to calculate the first attention feature corresponding to the initial character feature of character 4 and the second attention feature of the prediction guidance unit, thereby predicting multiple characters (e.g., characters 5, 6, and 7). Although this method can reduce repeated predictions, the use of the less accurate second attention feature for subsequent inference will affect the accuracy of the generated characters, resulting in poor inference performance.

[0066] In one possible implementation, the document parsing device may store the first attention feature and the second attention feature in a designated storage space; wherein, after the first attention feature and the second attention feature are generated again, the stored second attention feature is deleted from the designated storage space.

[0067] Please see Figure 4 , Figure 4 This is a schematic diagram of the attention feature cache replacement process provided in an embodiment of this application. To balance document parsing accuracy and speed, key-value (KV) cache replacement can be performed. Specifically, after calculating the first attention feature corresponding to the initial character feature and the second attention feature corresponding to the prediction guidance unit in this round of character prediction processing, the KV values ​​in these attention features can be stored in a designated storage space. If at least two characters are obtained in this round of character prediction processing without including the end character, these at least two characters can be used as target characters (such as character 2, character 3, and character 4) for the next character prediction processing.

[0068] After calculating the first attention feature of the target character and the second attention feature of the prediction guidance unit, the second attention feature stored in the memory space is removed and replaced with the first attention feature of the target character. For example... Figure 4 Step 2: Replace the existing key-value pairs (KV) in the storage space with the newly generated key-value pairs for characters 2 and 3. Figure 4 In the diagram, box R represents the key-value pair corresponding to the prediction guidance unit stored in Step 1, and the key-value pair for character 4 is newly written. This allows subsequent inference to read a more accurate key-value cache, ensuring the accuracy of characters generated by the model during inference.

[0069] It's important to note that while this process introduces a slight computational overhead—namely, the frequent reading of stored weight parameters and key-value pairs during the document parsing model's inference phase—it may be limited by memory access speed, making it a memory-intensive task rather than a computationally intensive task limited by the CPU / GPU's processing power. Therefore, this additional computation can be completed during the time spent waiting for memory access, having almost no impact on the overall inference speed.

[0070] Specifically, during the feature operation module's processing of the first and second attention features, the module can perform feature operation based on the first attention feature to obtain the second predicted character feature corresponding to the target character. Furthermore, the feature operation module performs feature operation based on the first attention feature, the second attention features corresponding to each prediction guidance unit, and the second attention features corresponding to other prediction guidance units whose character positions precede those of the prediction guidance units, to obtain the second predicted character features corresponding to each prediction guidance unit with a set number of characters.

[0071] In some embodiments, the feature operation module of VLM uses an attention mask to filter the attention features when performing feature operations. The attention mask can be a causal mask, that is, the t-th character can only focus on the first t characters in the character sequence and cannot focus on future characters (such as the (t+1)-th character).

[0072] Please refer to the following: Figure 5 , Figure 5 This is a schematic diagram of the optimized attention mask provided in an embodiment of this application. Specifically, the first attention feature corresponding to the target character can only focus on itself and the first attention features corresponding to the target characters whose positions precede it, such as... Figure 5 As shown by the dark squares in rows and columns 1 and 2, no second attention features corresponding to any prediction guidance units will be paid attention to, such as... Figure 5 The white squares in rows and columns 1 and 2 are shown. The second attention feature corresponding to the prediction guidance unit only pays attention to itself, the first attention feature corresponding to the target character whose character position is before it, and the second attention feature corresponding to the prediction guidance unit that follows the same target character but whose character position is before it, as shown in the example. Figure 5 As shown by the dark squares in the rows and columns where r2, r3, and r4 are located, the second attention feature corresponding to the prediction guide unit following other target characters will not be paid attention to, such as... Figure 5 The white squares in the rows and columns containing r2, r3, and r4 are shown.

[0073] Please refer to the following: Figure 1 Taking Step 1 as an example, the target character is character 1, and there are no other characters before it. The network layers in the feature processing module can calculate the hidden vector based on the Q, K, V of the target character and the previously generated K and V (calculated based on image units and text units) until the hidden vector of the last network layer is obtained, in order to predict the character whose position is the first one after the target character, such as character 2. In the first prediction guide unit after the target character (represented by a box R), it can focus on its own Q, K, V, the K and V of the target character, and the K and V calculated based on image units and text units, in order to predict the character whose position is the second one after the target character, i.e., character 3. In the second prediction guide unit after the target character (also represented by a box R), it can focus on its own Q, K, V, the K and V of the first prediction guide unit, the K and V of character 1 (the target character), and the K and V calculated based on image units and text units, in order to predict the character whose position is the third one after the target character, i.e., character 4.

[0074] Taking Step 2 as an example, the target characters are characters 2, 3, and 4. For character 2, there is character 1 before it. Therefore, the network layer in the feature operation module can calculate the hidden vector layer by layer based on the Q, K, and V of character 2, the K and V of character 1, and the K and V calculated based on the image unit and the text unit until the hidden vector of the last layer is obtained, which is the second predicted character feature corresponding to character 2. It can predict the character whose position is after character 2, such as character 3.

[0075] It should be noted that the document parsing model in this application uses the same attention mask for inference operations during the inference phase as during the training phase. This approach can maintain the consistency of inference operations between the training and inference phases, ensuring that the training effect of the model can be reflected in the inference phase. This helps to reduce the workload of applying the pre-trained model to the inference phase, enabling the model to be put into use quickly.

[0076] Step S203: If the obtained at least two characters do not contain the end character, then the at least two characters are taken as the target characters, and character prediction processing is performed again based on the target characters and the prediction guidance unit until the obtained at least two characters contain the end character.

[0077] In this embodiment of the application, after the document parsing device obtains at least two characters through character prediction processing based on the target character and the set prediction guidance unit, it can determine whether the at least two characters contain an ending character (e.g., ...). <eos> 、< / eos> < / start> < / s> 、 <end>The end character is a specific character used to indicate that the model has determined that the document parsing task for the document page image has been fully executed. If at least two characters do not contain the end character, it means that the document parsing task is not yet complete. At least two characters can be used as target characters to perform character prediction processing again based on the target characters and the prediction guidance unit until at least two generated characters contain the end character.

[0078] It should be noted that the number of prediction guide units used in each character prediction process can be the same or different, depending on the actual needs, and the number of characters generated will also vary accordingly. For example, in one character prediction process, setting one prediction guide unit after the target character can predict the two characters following the target character. In the next character prediction process, setting one or more prediction guide units after the target character can predict two or more characters following the target character.

[0079] Step S204: Based on the characters obtained from each character prediction process, generate the parsing result corresponding to the document page image. The parsing result includes text information used to characterize the content elements in the document page image and the structural relationships between the content elements.

[0080] In this embodiment, the document parsing device determines that the document parsing task has been completed when at least two characters contain a closing character. Then, the document parsing device can combine the characters obtained from each character prediction process to generate a parsing result corresponding to the document page image, that is, a parsing result corresponding to the parsed document page. The parsing result includes text information characterizing the content elements in the document page image and the structural relationships between them.

[0081] For example, a document page image may include a 3x3 table. The first row could be {Name, Age, City}, the second row could be {Zhang San, 28, Beijing}, and the third row could be {Li Si, 32, Shanghai}. The parsed result could be four lines of text information: the first line could be |Name|Age|City|, the second line could be |:---||:---||:---|. This symbol can be used to separate or align lines, and the electronic device can determine from this symbol that the first line is the table header, and the subsequent content is the table's cell data. The third line could be |Zhang San|28|Beijing|, and the fourth line could be |Li Si|32|Shanghai|.

[0082] In some embodiments of this application, the technical solutions provide that by dividing the document page image to be parsed into blocks, parallel parsing of higher-resolution local images can be performed. Compared to directly inputting the entire document / document page for parsing, this alleviates resolution limitations and reduces computational complexity, thereby improving the overall parsing efficiency of the document. Furthermore, by introducing a prediction guidance unit, multiple characters can be predicted in parallel at a time, thereby increasing the speed and reducing the latency of document parsing. Moreover, through a KV cache replacement mechanism, the attention features calculated during each prediction are stored so that they can be directly read in the next prediction, avoiding repeated calculations in subsequent inference and improving inference efficiency. Finally, by combining the characters obtained from each character prediction process, the content elements in unstructured or semi-structured documents can be reconstructed, and their logical order and hierarchical relationships in the original document can be accurately restored, so as to quickly obtain structured parsing results that can be directly processed by electronic devices.

[0083] This application also provides a document parsing method. The document parsing method described in this application can be executed by an electronic device, which can be... Figure 1 or Figure 6 The document parsing device 101 in the implementation environment shown. See also... Figure 6 and Figure 7 , Figure 6 This is a schematic diagram of an implementation environment provided in an embodiment of this application. Figure 7 This is a flowchart illustrating a document parsing method provided in an embodiment of this application. The document parsing method includes the following steps S701-S704: Step S701: Obtain a training document page image, a tag character sequence corresponding to the training document page image, and training text prompts for indicating that the training document page image is parsed; wherein, at least two tag characters in the tag character sequence correspond to a character position.

[0084] In this embodiment of the application, the training document page image can refer to the document page image used to train the initial document parsing model, which can be one or more, and can be used... The training text prompt can refer to the text information (prompt) used to instruct the initial document parsing model to perform the document parsing task. It can be represented as... The tag character sequence refers to the text information used to train the initial document parsing model. It can also be understood as the parsing result paired with the training document page image, and can be represented as... It is used to represent the content elements in the training document page image and the structural relationships between the content elements. It is a kind of reference information and a training target for the document parsing model.

[0085] The tag character sequence can include multiple tag characters, each corresponding to a character position. For example, the tag character sequence can be represented as follows: ,in, Indicates label characters, This indicates the character position corresponding to the tag character.

[0086] In some embodiments, the document parsing device may acquire at least two initial document page images, input these images into a pre-trained discriminative model, and detect at least one of the image quality and document type corresponding to each of the initial document page images. Filtering is performed based on defined filtering rules and the image quality and document type corresponding to each initial document page image, and the acquired training document page image is determined based on the at least two candidate document page images obtained through filtering.

[0087] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating the process of acquiring training data provided in an embodiment of this application. The training data mainly includes training document page images and the tag character sequences corresponding to the training document page images. The document parsing device can collect usable basic data (at least two initial document page images) from diverse data sources, specifically including open source data sources, in-house data sources, and synthetic data sources.

[0088] The document parsing device can input at least two initial document page images into a pre-trained discriminative model to detect at least one of the following for each initial document page image: image quality and document type. Image quality can be determined by the distortion and blurriness of the initial document page image. Document type can include newspapers, papers, exam questions, handwritten text (such as handwritten formulas), textbooks, formulas, financial reports, and other types (such as...). Figure 8 (The musical score shown, etc.).

[0089] Understandably, if the initial document page image is severely distorted or blurry, document parsing becomes more difficult, and the image quality can be judged as low. Conversely, if the initial document page image is well-formed and clear, document parsing becomes easier, and the image quality can be judged as high. Therefore, the document parsing device can filter the initial document page image according to the set filtering rules to obtain at least two candidate document page images, and then determine the training document page image based on these at least two candidate document page images.

[0090] The filtering rules can be associated with the image quality and / or document type of the initial document page images. For example, initial document page images with image quality below a preset image quality threshold can be removed. Another example is deleting some initial document page images if there are too many for a particular document type. Yet another example is deleting initial document page images of one or more document types if their proportion of total initial document page images is too low, indicating insufficient representativeness of that document type, thus ensuring a more balanced number of initial document page images across all document types. This removal method can alleviate type imbalance, reduce model bias, and lower the risk of overfitting.

[0091] In some embodiments, the discriminative model can also detect the content density of each initial document page image to obtain a content density detection result. The document parsing device can also filter the initial document page images based on the content density detection result obtained by the discriminative model, for example, deleting initial document page images with less content.

[0092] In one possible implementation, the document parsing device can further filter the at least two candidate document page images obtained from the initial filtering process to obtain training document page images that possess both data quality and diversity. Specifically, the document parsing device can extract the image semantic features corresponding to the at least two candidate document page images respectively, and based on the image semantic features corresponding to the at least two candidate document page images respectively, remove candidate document page images whose image semantic feature similarity is greater than a first similarity threshold from the at least two candidate document page images. The removed candidate document page images are then determined as the obtained training document page images.

[0093] The image semantic feature can be the feature vector representation corresponding to at least two candidate document page images. It is a high-dimensional abstract semantic expression. The image semantic features corresponding to similar images are closer in the vector space, while the image semantic features corresponding to images that are significantly different are farther apart in the vector space. Therefore, the similarity of image semantic features can be used to remove overly similar document page images and obtain training document page images.

[0094] For example, a document parsing device can extract the image semantic features corresponding to at least two candidate document page images through machine learning models such as Contrastive Language–Image Pretraining (CLIP) models and Distillation with No Labels (DINO).

[0095] Document page images can be used to determine the similarity of their semantic features by calculating the Euclidean distance or cosine similarity between each pair of candidate document page images. If the semantic feature similarity between two candidate document page images is greater than a preset first similarity threshold, one of the candidate document page images can be deleted. This process continues until the semantic feature similarity between any two remaining candidate document page images is less than the first similarity threshold, thus completing the semantic deduplication. The document parsing device can then use the remaining candidate document page images as training document page images.

[0096] In another possible implementation, the document parsing device can also perform image representation deduplication on at least two candidate document page images using perceptual hashing (pHash). Specifically, the document parsing device can calculate the pixel hash value corresponding to each of the at least two candidate document page images; based on the pixel hash values ​​corresponding to the at least two candidate document page images, remove candidate document page images whose pixel hash value similarity is greater than a second similarity threshold from the at least two candidate document page images, and determine the removed candidate document page images as the obtained training document page images.

[0097] In some embodiments, the document parsing device can calculate the pixel hash values ​​corresponding to at least two document page images using a pHash calculation algorithm. These pixel hash values ​​can also be called perceptual hash values. The algorithm used to calculate the pHash is a hash value generation algorithm based on image content. Its core characteristic is that it is sensitive to the perceptual content of the image but insensitive to changes in detail. Unlike traditional cryptographic hashing, perceptual hashing generates hash values ​​that reflect the visual similarity of images; even if the images undergo resizing, brightness adjustments, or format compression, their hash values ​​remain similar.

[0098] For example, a document parsing device can calculate the perceptual hash value of different candidate document page images using a discrete cosine transform algorithm. This involves first converting the candidate document page image into a grayscale image, then performing a discrete cosine transform on the grayscale image to obtain a frequency domain representation. After extracting the low-frequency components, the grayscale image is binarized and quantized. Finally, the resulting binarized sequence is determined as the perceptual hash value. Other algorithms can also be used to calculate the perceptual hash value; this application does not limit the specific algorithms used.

[0099] Furthermore, the document parsing device can calculate the Hamming distance between each pair of candidate document page images based on their respective pixel hash values. This Hamming distance is calculated by counting the number of different bits (values) at corresponding positions in two equal-length binary hash values, using this number as the Hamming distance value. The similarity between the pixel hash values ​​of the two images can then be measured. If the pixel hash similarity between two candidate document page images is greater than a preset second similarity threshold, one of the candidate document page images can be deleted. This process continues until the pixel hash similarity between any two remaining candidate document page images is less than the second similarity threshold, thus completing the image representation deduplication. The document parsing device can then determine the remaining candidate document page images as training document page images.

[0100] It is understood that the embodiments of this application can perform deduplication simultaneously from both image semantics and image representation (pixels), i.e., dual deduplication. Alternatively, only one aspect can be selected for deduplication, depending on the application scenario. The aforementioned first similarity threshold and second similarity threshold can also be set based on the business scenario, and this application does not limit them.

[0101] In some embodiments, after obtaining the training document page image, the training document page image can be augmented by image processing methods such as blurring, distortion, adding noise, perspective, jittering, and reversal to increase the amount of data in the training document page image, thereby improving the robustness and adaptability of the trained document parsing model through rich training data.

[0102] In some embodiments, the document parsing device can also segment each remaining candidate document page image into multiple local images using a pre-trained layout analysis model. Each local image corresponds to a content element (e.g., text, formula, table, image, etc.). To further improve the quality of the training data, local images that are too large or too small, or whose content is incomplete, can be removed, and the remaining local images can be used as training document page images.

[0103] In some embodiments, the document parsing device can also perform character prediction processing on the training document page image using at least two pre-trained reference document parsing models to obtain reference label character sequences generated by at least two reference document parsing models respectively. If, in at least two reference label character sequences, the character similarity between the target reference label character sequence and other reference label character sequences is less than a set character similarity threshold, then the target reference label character sequence is determined as the label character sequence.

[0104] Specifically, document parsing devices can employ a multi-model strategy, generating character sequences through multiple reference document parsing models. For example, they can combine strong models (gemini-pro), large models (such as Qwen2.5-VL), and expert models (MonkeyOCR-pro) to perform character prediction processing on training document page images, resulting in three reference label character sequences.

[0105] Furthermore, the document parsing device can calculate the edit distance between each reference tag character sequence and other reference tag character sequences, such as the Levenshtein distance, which defines the minimum number of single-character editing operations required to transform one string into another. Therefore, this edit distance can be used to measure the character similarity between a given reference tag character sequence and other reference tag character sequences. In other words, the edit distance represents the minimum number of character insertion, deletion, and replacement operations required to transform one reference tag character sequence into another; the smaller the edit distance, the more similar the two reference tag character sequences are.

[0106] In some embodiments, if the character similarity between a certain reference label character sequence and other reference label character sequences is less than a set character similarity threshold, then the reference character label sequence can be determined as the target reference label sequence, thereby obtaining the label character sequence corresponding to the training document page image.

[0107] In some embodiments, the document parsing device may also calculate the sum of edit distances between each reference tag character sequence and other reference tag character sequences, and determine the reference tag character sequence with the smallest sum of edit distances as the target reference tag character sequence, thus obtaining the tag character sequence corresponding to the training document page image. It is understood that the smallest sum of edit distances indicates that the candidate result is more similar to most candidate results and is the result closest to the model consensus.

[0108] In some embodiments, the target reference label character sequence can also be optimized and verified by a small amount of manual intervention to ensure the quality of the final obtained label character sequence even when the confidence level is low or the discrepancies between models are large.

[0109] Therefore, this application provides an automated document parsing training data generation scheme to address the problem of insufficient training data. This scheme generates high-quality training data at low cost by constructing various data filtering and data augmentation strategies and integrating multiple different multimodal models for joint character prediction processing.

[0110] Step S702: Insert training prediction guidance units at specified character positions after the character positions corresponding to each tag character in the tag character sequence to obtain training input data.

[0111] In this embodiment of the application, the label character sequence can be... Each tag character in Then, training prediction guide units are inserted to obtain training input data. The training prediction guide unit can be one or multiple consecutive units, and the number of training prediction guide units inserted after each label character can be the same or different. For example, the training input data can be as shown in Formula 1: Formula 1 In Formula 1, each label character n training prediction guidance units were inserted after each one. The units are indicated by square brackets as individual training prediction guidance units. The indices are different, that is, the position IDs are different, which means the corresponding position codes are also different, but they share the same learnable vector (token embedding).

[0112] In other words, during training, the model needs to learn to predict the next label character (already predicted) based on the previous label character. Prediction is required. ), and based on the training prediction guidance unit located after the label character, predict subsequent label characters at specific offsets (such as predicting...). , , (etc.). Therefore, it can be designed in After The prediction target is , The prediction target is And so on. Thus, the training objective corresponding to the training input data can be represented as shown in Formula 2: Formula 2 In Formula 2, Indicates the training objective. It refers to the label character, where the label character at each character position is the prediction target corresponding to each character position in Formula 1.

[0113] Step S703: Input the training input data, the training document page image, and the training text prompt information into the initial document parsing model to perform character prediction processing through the initial document parsing model to obtain the training character sequence.

[0114] In this embodiment, the document parsing device can input training input data, training document page images, and training text prompts into an initial document parsing model, and perform character prediction through the initial document parsing model to obtain a training character sequence. The initial document parsing model can be, for example, a pre-trained large visual language model (VLM), which may include a feature extraction module, a feature operation module, and a prediction module, and may also include other modules; this application does not limit the scope of the application.

[0115] After the document parsing device inputs the training document page image and training text prompt information into the initial document parsing model, the initial document parsing model can divide the training document page image into multiple image units and map each image unit to an initial image feature. Then, it combines the position of the image unit in the training document page image with the position code assigned to each image unit. By concatenating or adding the feature vectors, the image features corresponding to each image unit are finally obtained, and the image features of the training document page image are obtained by combining them.

[0116] Furthermore, the initial document parsing model performs word segmentation on the training text prompts, obtaining multiple text units, and retrieves the index corresponding to each text unit based on a predefined vocabulary. Then, the initial document parsing model can look up the initial prompt features corresponding to each text unit from a predefined embedding table based on the index, and combine this with the positional encoding assigned to each text unit based on its position information in the training text prompts to obtain the text features corresponding to each text unit. Thus, the initial document parsing model can combine these two methods to obtain the text features of the training text prompts.

[0117] Similarly, the initial document parsing model can also retrieve tag characters from a predefined vocabulary. The index is used to determine the vector representation based on the index and a predefined embedding table. This can be called the embedding feature. Combined with the character position of the label character (as shown in the index below), the label character can be determined. The positional encoding is used to ultimately combine the embedded features and the positional encoding to obtain the character features corresponding to the tag characters.

[0118] Furthermore, the initial document parsing model can determine each training prediction guide unit following the tag character based on the tag character and a set number of characters. These training prediction guide units can be composed of a shared learnable vector representation and different positional codes, which can be preset or learnable. For example, the initial document parsing model can determine the character position corresponding to each training prediction guide unit based on the character position of the tag character. The character position corresponding to the subsequent training prediction guide unit is This allows us to obtain the corresponding positional encoding to determine the training prediction guidance unit. The character positions corresponding to the training input data can be represented as shown in Formula 3: Formula 3 In Formula 3, This represents a character position sequence with a length of . Each number represents the position of a character in the training data, and the number in square brackets represents the position of the character in the prediction guide unit.

[0119] Therefore, the initial document parsing model can perform character prediction processing on the image features corresponding to each image unit, the text features corresponding to each text unit, and the features corresponding to the training data (including character features and training prediction guidance units) to obtain the predicted character sequence, i.e., the training character sequence.

[0120] The process of character prediction processing in the initial document parsing model is the same as that in the document parsing model, and can be found in the specific description of the inference stage, which will not be repeated here.

[0121] It should be noted that the feature computation module (such as the backbone network of VLM) in the initial document parsing model performs parallel computation for each position during training. In this process, the feature computation module uses attention mask to ensure autoregressive properties, that is, to ensure that the character at position t does not focus on characters after its character position (such as the (t+1)th character).

[0122] like Figure 5 As shown, this attention mask ensures that the attention features corresponding to a label character only focus on the attention features corresponding to the label characters preceding it, ignoring the attention features corresponding to any training prediction guidance units. Similarly, the attention features corresponding to training prediction guidance units only focus on the attention features corresponding to the label characters preceding them, as well as the attention features corresponding to training prediction guidance units at adjacent character positions but preceding them—that is, the attention features corresponding to training prediction guidance units that are ranked after the same label character but preceding it. Furthermore, training prediction guidance units following different label characters do not focus on each other.

[0123] As can be seen, the initial document parsing model, through self-attention mechanism and attention mask, can precisely control the attention target corresponding to each character position. A single forward propagation can yield the predicted character for each character position, thus forming the predicted character sequence, i.e., the training character sequence, enabling efficient batch parallel training. This attention mask also follows causal masking (it cannot focus on future characters) and ensures that the attention features of each character position do not interfere with each other during training.

[0124] Step S704: Based on the difference between the tag character sequence and the training character sequence, train the initial document parsing model to obtain a document parsing model; wherein, the document parsing model is used to perform character prediction processing based on the document page image and the text indication information to generate the target character and at least two characters following the target character.

[0125] In this embodiment, the model can perform character prediction processing on each position in the training input data during a single forward propagation based on a self-attention mechanism and an optimized attention mask, obtaining the training character corresponding to each position, thus forming a training character sequence. The difference between the tag character sequence and the training character sequence can then be quantified to train the initial parsing model, resulting in a document parsing model.

[0126] Specifically, the document parsing device can generate target loss data based on the differences between each training character in the training character sequence and the corresponding label character (prediction target) in the label character sequence. Then, based on the target loss data, the model is trained to obtain the document parsing model. The document parsing model is a model that can be used in the inference stage to perform document parsing tasks (character prediction processing) based on document page images and text indication information. It can simultaneously generate the target character and at least two characters following the target character, thus achieving parallel character prediction without changing the model structure.

[0127] In one possible implementation, please refer to Figure 9 , Figure 9 This is a schematic diagram of the document parsing model provided in this application embodiment. Due to the use of attention masks, the label characters and the training prediction guidance unit do not interfere with each other, and different training loss functions can be used to solve the corresponding loss data. Specifically, the label characters can use the NTP loss function to quantify the difference between the true value and the predicted value. This loss data can be, for example, as shown in Equation 4: Formula 4 in, This represents the prediction loss for the label characters. Representation model, This indicates the model parameters to be adjusted. Represents a dataset, This represents the mean. Represents the training document page image. This indicates the training text prompt information. Represents the sequence of label characters. The character position indicating the tag character. Indicates the length of the character sequence.

[0128] For example, the loss function for training the prediction guide unit can be as shown in Equation 5: Formula 5 In Formula 5, This represents the prediction loss used to train the prediction guide unit. Representation model, This indicates the model parameters to be adjusted. Represents a dataset, This represents the mean. Represents the sequence of label characters. Indicates the position of a character in the sequence. tag characters, Indicates the position of a character in the sequence. Training prediction guidance unit, Indicates the length of the character sequence.

[0129] In some embodiments, weights can be set for the two loss functions respectively, and the target loss data can be expressed as Equation 6: Formula 6 in, The relative weights between the two loss functions control the relative contribution of the two loss terms. Loss data, as shown in Formulas 5 and 6, are calculated based on different loss functions, and then the two loss data are weighted and summed according to preset weights to obtain the target loss data.

[0130] Subsequently, the document parsing device can update the model parameters of the initial document parsing model based on the target loss function using methods such as gradient descent, for example, updating the attention parameters and updating the learnable embedded features shared by the training prediction guidance units, thereby obtaining the document parsing model.

[0131] Understandably, the trained document parsing model can fully utilize the prediction guidance units' ability to predict future (specific offset) characters learned during training during the inference phase. Adding a predetermined number of prediction guidance units after the target character ensures that the model generates at least the predetermined number of characters plus a new character for each character prediction, thus achieving the speedup shown below: Formula 7 in, The delay for the model's prediction processing step for each character. The model is given a forward delay for multiple characters, where n is the set number of characters. It is assumed that the number of prediction guidance units added in each character prediction process is the same, n. With sufficient computational resources, the difference between the two methods is negligible, meaning the document parsing model can achieve an n+1x speedup. Furthermore, this method fully conforms to the causal language model setup and requires no modification to the model structure.

[0132] It should be noted that the document parsing model trained in this application has good extrapolation capabilities. For example, during the training process, a training prediction guidance unit is inserted after each tag character, predicting two characters each time. During the inference phase, any set number of prediction guidance units (one or more) can be inserted after the target character, predicting two or more characters each time.

[0133] In some embodiments of this application, the technical solutions provide that, by introducing a training prediction guidance unit during training, the model is trained to not only predict the next character but also multiple characters following it simultaneously, thus improving the model's ability to model complex contexts. This method of introducing a prediction guidance unit for training allows the model to focus on future contexts at longer distances (such as more distant character positions), enhancing its ability to capture long-distance dependencies. Furthermore, compared to the paradigm of generating only one character at a time in autoregressive sequences, this approach achieves parallel character prediction without changing the model structure by adjusting the training data and training method, thereby improving document parsing efficiency.

[0134] The following is a quantitative evaluation of the effectiveness of the document parsing model provided in the embodiments of this application in performing document parsing tasks.

[0135] Experiments were conducted on publicly available document parsing benchmarks, such as the OmniDocBench dataset. The OmniDocBench dataset comprises 16,886 images, covering nine document types (slides, academic papers, books, textbooks, exam papers, magazines, news articles, notes, and financial reports), four layout types, and three language types, thus comprehensively covering various challenges faced in real-world document parsing and allowing for the evaluation of the model's ability to understand different document types in real-world scenarios.

[0136] Please refer to Table 1, which uses edit distance to evaluate the parsing capabilities of models PTP0, PTP1, and PTP2 for different document types. PTP0 indicates the model does not introduce any predictive guide units (RBUs), PTP1 indicates the model introduces one RBU, and PTP2 indicates the model introduces two RBUs. The values ​​in Table 1 refer to the edit distance, which measures how many insertion, deletion, and replacement operations are required between the character sequence generated by the model and the tag character sequence to achieve a perfect match. A smaller value indicates better model performance, with 0 representing a perfect match.

[0137]

[0138] Table 1 As shown in Table 1, the PTP1 model, which employs a prediction guidance unit for parallel inference, not only maintains its performance in document parsing tasks but also achieves further improvements in document parsing accuracy, surpassing models without a prediction guidance unit. This is because the PTP (Parallel Character Prediction) mechanism allows the model to better utilize the contextual information surrounding characters, reducing illusions and producing more accurate parsing results. Through the prediction guidance unit's ability to capture features of different regions of the document page image and predict future characters, the model can more comprehensively understand the semantic relevance of the document content. Therefore, the parallel character prediction technology provided in this application generates multiple characters simultaneously in multiple character prediction steps, fundamentally solving the inference delay caused by the serial prediction of autoregressive models.

[0139] Please see Figure 10 , Figure 10 This diagram illustrates the performance test results of the document parsing model provided in this application embodiment. Based on 16,886 images from the OmniDocBench dataset, the average character output time (TPOT), average latency, and inter-character latency (ITL) of NTP, PTP1, and PTP0 were tested on an H2O graphics processing unit (GPU) under different queries per second (QPS) metrics, as well as the decoding speed and speedup ratio in synchronous mode (same throughput). The performance test results show that PTP1 achieved a 1.6x speedup (relative to NTP), and PTP2 achieved a 2.2x speedup (relative to NTP). This speedup is achieved by introducing a prediction guidance unit. Based on the parallel prediction capability of the prediction guidance unit during the inference stage, the model can generate multiple characters in a single character prediction process, thereby effectively reducing the average character output time and average latency, while improving the decoding throughput.

[0140] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0141] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a document parsing device provided in an embodiment of this application. Figure 11 The document parsing device shown can be mounted in an electronic device, specifically a terminal device or a server. Figure 11 The document parsing device shown can be used to perform the above. Figure 2 and Figure 7 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 11 The document parsing device 1100 includes: The first character prediction processing unit 1110 is used to perform character prediction processing based on the document page image to be parsed and the text prompt information indicating that the document page image should be parsed, so as to generate target characters. The second character prediction processing unit 1120 is used to perform character prediction processing based on the target character and a set prediction guidance unit to generate at least two characters whose positions are after the target character; wherein, the prediction guidance unit is used to predict characters whose positions are after the target character and are not adjacent to the target character; The third character prediction processing unit 1130 is used to, if the obtained at least two characters do not contain an end character, take the at least two characters as target characters and perform character prediction processing again based on the target characters and the prediction guidance unit until the obtained at least two characters contain the end character; The generation unit 1140 is used to generate a parsing result corresponding to the document page image based on the characters obtained from each character prediction process. The parsing result includes text information that characterizes the content elements in the document page image and the structural relationships between the content elements.

[0142] In one possible implementation, the document parsing device further includes a layout analysis unit, which is used to: acquire an initial document image containing at least one content element; wherein different content elements are presented in different forms in the initial document image; The initial document image is input into the layout analysis model, so that the layout analysis model can segment local images corresponding to various content elements from the initial document image, and the local images segmented by the layout analysis model are determined as the document page images to be parsed.

[0143] In one possible implementation, the first character prediction processing unit 1110 is configured to perform character prediction processing based on the document page image to be parsed and text indication information indicating that the document page image should be parsed, in order to generate target characters, specifically for: Extract the image features corresponding to the document page image, and extract the text features corresponding to the text indication information; The image features and text features are input into a pre-trained document parsing model, and the concatenated features obtained based on the image features and text features are processed by the feature operation module in the document parsing model to obtain the first character prediction feature. The first character prediction feature is then processed by the prediction module in the document parsing model to obtain the target character.

[0144] In one possible implementation, the second character prediction processing unit 1120 is configured to perform character prediction processing based on the target character and a set prediction guidance unit to generate at least two characters whose positions follow the target character, specifically for: The target character is input into a pre-trained document parsing model, so that the feature processing module in the document parsing model maps the target character into initial character features, and the prediction guidance unit is generated based on the character position corresponding to the target character; The feature operation module in the document parsing model performs feature operation processing on the initial character features and the prediction guidance unit to obtain at least two second character prediction features. Then, the prediction module in the document parsing model performs character mapping processing on the at least two second character prediction features to obtain at least two characters that follow the target character.

[0145] In one possible implementation, the second character prediction processing unit 1120 is configured to generate the prediction guidance unit based on the character position corresponding to the target character, specifically for: Obtain the pre-trained initial prediction guidance unit, and based on the character position corresponding to the target character and the set number of characters, determine the position information corresponding to the set number of initial prediction guidance units respectively; The initial prediction guidance unit with the set number of characters and the position information corresponding to each initial prediction guidance unit are fused together to generate the prediction guidance unit with the set number of characters. The number of at least two characters following the target character is the sum of the number of the target character and the number of the set characters.

[0146] In one possible implementation, the second character prediction processing unit 1120 is used to perform feature operation processing on the initial character features and the prediction guidance unit through the feature operation module in the document parsing model to obtain at least two second character prediction features, specifically for: The feature processing module performs attention processing based on the initial character features and the prediction guidance unit to generate a first attention feature corresponding to the initial character features and a second attention feature corresponding to the prediction guidance unit. The feature operation module performs feature operation processing on the first attention feature and the second attention feature to obtain the at least two second character prediction features.

[0147] In one possible implementation, the number of prediction guidance units is a set number of characters, wherein each prediction guidance unit corresponds to a character position; The second character prediction processing unit 1120 is used to perform feature operation processing on the first attention feature and the second attention feature through the feature operation module to obtain the at least two second character prediction features, specifically for: The feature processing module performs feature processing based on the first attention feature to obtain the second predicted character feature corresponding to the target character; The feature processing module performs feature processing based on the first attention feature, the second attention feature corresponding to each prediction guidance unit, and the second attention feature corresponding to other prediction guidance units whose character positions are before the prediction guidance unit, to obtain the second predicted character feature corresponding to the prediction guidance unit with the set number of characters.

[0148] In one possible implementation, the second character prediction processing unit 1120 is further configured to: Store the first attention feature and the second attention feature in the specified storage space; Specifically, after the first attention feature and the second attention feature are generated again, the stored second attention feature is deleted from the designated storage space.

[0149] In one possible implementation, the document parsing device 1100 further includes a training unit 1150, configured to acquire a training document page image, a tag character sequence corresponding to the training document page image, and training text prompts indicating that the training document page image should be parsed; wherein at least two tag characters in the tag character sequence correspond to a character position; a training prediction guidance unit is inserted at a specified character position after each character position corresponding to a tag character in the tag character sequence to obtain training input data; the training input data, the training document page image, and the training text prompts are input into an initial document parsing model to perform character prediction processing through the initial document parsing model to obtain a training character sequence; the initial document parsing model is trained based on the difference between the tag character sequence and the training character sequence to obtain a document parsing model; wherein the document parsing model is used to perform character prediction processing based on the document page image and the text prompts to generate the target character and at least two characters following the target character.

[0150] In one possible implementation, at least two training characters in the training character sequence each correspond to a character position; The training unit 1150 is used to train the initial document parsing model based on the difference between the tag character sequence and the training character sequence to obtain a document parsing model, specifically for: Based on the differences between each training character in the training character sequence and the corresponding label character in the label character sequence, target loss data is generated; Based on the target loss data, the initial document parsing model is trained to obtain the document parsing model.

[0151] In one possible implementation, the training unit 1150 is used to acquire training document page images, specifically for: Obtain at least two initial document page images; The at least two initial document page images are input into a pre-trained discriminant model to detect at least one of the image quality and document type corresponding to each of the at least two initial document page images. The filtering process is performed based on the set filtering rules and the image quality and document type corresponding to each initial document page image. Based on at least two candidate document page images obtained from the filtering, the obtained training document page image is determined.

[0152] In one possible implementation, the training unit 1150 is used to determine the acquired training page image based on at least two candidate document page images obtained through filtering, specifically for: Extract the image semantic features corresponding to the at least two candidate document page images respectively; Based on the image semantic features corresponding to the at least two candidate document page images, candidate document page images with image semantic feature similarity greater than a first similarity threshold are removed from the at least two candidate document page images, and the removed candidate document page images are determined as the obtained training document page images.

[0153] In one possible implementation, the training unit 1150 is configured to determine the acquired training document page image based on at least two candidate document page images obtained through filtering, specifically for: Calculate the pixel hash values ​​corresponding to the at least two candidate document page images respectively; Based on the pixel hash values ​​corresponding to the at least two candidate document page images, candidate document page images with pixel hash value similarity greater than a second similarity threshold are removed from the at least two candidate document page images, and the removed candidate document page images are determined as the obtained training document page images.

[0154] In one possible implementation, the training unit 1150 is used to obtain the tag character sequence corresponding to the training document page image, specifically for: The training document page image is processed by character prediction using at least two pre-trained reference document parsing models to obtain reference tag character sequences generated by the at least two reference document parsing models respectively; If, among the at least two reference tag character sequences, the character similarity between the target reference tag character sequence and all other reference tag character sequences other than the target reference tag character sequence is less than a set character similarity threshold, then the target reference tag character sequence is determined as the tag character sequence.

[0155] According to one embodiment of this application, Figure 2 and Figure 7 The document parsing method shown can be partially implemented by... Figure 11 The document parsing apparatus shown is executed by individual units within it. For example, Figure 2 Step S201 shown can be performed by Figure 11 The first character prediction processing unit 1110 shown is executed, and step S202 can be performed by... Figure 11 The second character prediction processing unit 1120 shown executes step S203, which can be performed by... Figure 11 The third character prediction processing unit 1130 shown is executed, and step S204 can be performed by... Figure 7 The generation unit 1140 shown is executed. For example, Figure 7 Step S701 shown can be executed by the acquisition unit, and step S702 can be executed by... Figure 11 The insertion unit shown is executed, and step S703 can be performed by... Figure 11 The input unit shown is executed, and step S704 can be performed by... Figure 11 The model training unit shown is executed. Figure 11 The document parsing device shown can be composed of individual or combined units into one or more other units, or some of the units can be further divided into two or more functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can also be implemented by two or more units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the video generation device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by two or more units.

[0156] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 2 and Figure 7 The computer program (including program code) involved in each step of the corresponding method shown, to construct such... Figure 11 The document parsing apparatus shown herein, and the video generation method for implementing embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0157] Based on the same inventive concept, the principle and beneficial effects of the document parsing device provided in the embodiments of this application are similar to the principle and beneficial effects of the video generation method in the embodiments of this application. For details, please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, these will not be repeated here.

[0158] Embodiments of this application also provide an electronic device, including: at least one processor; and a storage device for storing at least one computer program, which, when executed by the at least one processor, causes the electronic device to implement the document parsing methods provided in the above embodiments.

[0159] Figure 12 A schematic diagram of a computer system suitable for implementing an electronic device according to embodiments of this application is shown. The electronic device may be... Figure 1 The device shown.

[0160] It should be noted that, Figure 12 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0161] like Figure 12 As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes based on a computer program stored in Read-Only Memory (ROM) 1202 or a computer program loaded from storage portion 1208 into Random Access Memory (RAM) 1203, such as executing the document parsing method in the above embodiments. Various computer programs and data required for system operation are also stored in RAM 1203. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. An input / output (I / O) interface 1205 is also connected to bus 1204.

[0162] In some embodiments, the following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.

[0163] In particular, according to embodiments of this application, a computer program implementing the document parsing method can be carried on a computer-readable medium, which can be downloaded and installed from a network via the communication section 1209, and / or installed from a removable medium 1211.

[0164] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or at least two wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer program contained in the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or at least two executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0166] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0167] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor of an electronic device, causes the electronic device to implement the aforementioned document parsing method. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0168] Another aspect of this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the document parsing methods provided in the various embodiments described above. The computer program can be stored in a computer-readable storage medium. The computer program product can be a computer program as a product, such as an APP (Application), webpage, mini-program, etc.; or, the computer program product can also be a storage medium, device, terminal, virtual machine, etc., containing the computer program.

[0169] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

[0170] It should also be noted that when the above embodiments of this application are applied to specific products or technologies, if it is necessary to obtain user data, the user's permission or consent must be obtained, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.< / end> < / bos>

Claims

1. A document parsing method, characterized in that, The method includes: Based on the document page image to be parsed and the text prompt information indicating that the document page image should be parsed, character prediction processing is performed to generate target characters; Based on the target character and the set prediction guidance unit, character prediction processing is performed to generate at least two characters whose positions are after the target character; wherein, the prediction guidance unit is used to predict characters whose positions are after the target character and are not adjacent to the target character; If the obtained at least two characters do not contain the end character, then the at least two characters are taken as the target characters, and character prediction processing is performed again based on the target characters and the prediction guidance unit until the obtained at least two characters contain the end character; Based on the characters obtained from each character prediction process, a parsing result corresponding to the document page image is generated. The parsing result contains text information that characterizes the content elements in the document page image and the structural relationships between the content elements.

2. The method according to claim 1, characterized in that, The method further includes: Obtain an initial document image containing at least one content element; wherein different content elements are presented in different forms in the initial document image; The initial document image is input into the layout analysis model, so that the layout analysis model can segment local images corresponding to various content elements from the initial document image, and the local images segmented by the layout analysis model are determined as the document page images to be parsed.

3. The method according to claim 1, characterized in that, The character prediction processing based on the document page image to be parsed and the text indication information for indicating that the document page image should be parsed, to generate target characters, includes: Extract the image features corresponding to the document page image, and extract the text features corresponding to the text indication information; The image features and text features are input into a pre-trained document parsing model, and the concatenated features obtained based on the image features and text features are processed by the feature operation module in the document parsing model to obtain the first character prediction feature. The first character prediction feature is then processed by the prediction module in the document parsing model to obtain the target character.

4. The method according to claim 1, characterized in that, The character prediction processing based on the target character and the set prediction guidance unit to generate at least two characters whose positions follow the target character includes: The target character is input into a pre-trained document parsing model, so that the feature processing module in the document parsing model maps the target character into initial character features, and the prediction guidance unit is generated based on the character position corresponding to the target character; The feature operation module in the document parsing model performs feature operation processing on the initial character features and the prediction guidance unit to obtain at least two second character prediction features. Then, the prediction module in the document parsing model performs character mapping processing on the at least two second character prediction features to obtain at least two characters that follow the target character.

5. The method according to claim 4, characterized in that, The step of generating the prediction guidance unit based on the character position corresponding to the target character includes: Obtain the pre-trained initial prediction guidance unit, and based on the character position corresponding to the target character and the set number of characters, determine the position information corresponding to the set number of initial prediction guidance units respectively; The initial prediction guidance unit with the set number of characters and the position information corresponding to each initial prediction guidance unit are fused together to generate the prediction guidance unit with the set number of characters. The number of at least two characters following the target character is the sum of the number of the target character and the number of the set characters.

6. The method according to claim 4, characterized in that, The feature processing module in the document parsing model performs feature operations on the initial character features and the prediction guidance unit to obtain at least two second character prediction features, including: The feature processing module performs attention processing based on the initial character features and the prediction guidance unit to generate a first attention feature corresponding to the initial character features and a second attention feature corresponding to the prediction guidance unit. The feature operation module performs feature operation processing on the first attention feature and the second attention feature to obtain the at least two second character prediction features.

7. The method according to claim 6, characterized in that, The number of prediction guidance units is a set number of characters, wherein each prediction guidance unit corresponds to one character position; The step of performing feature operations on the first attention feature and the second attention feature through the feature operation module to obtain the at least two second character prediction features includes: The feature processing module performs feature processing based on the first attention feature to obtain the second predicted character feature corresponding to the target character; The feature processing module performs feature processing based on the first attention feature, the second attention feature corresponding to each prediction guidance unit, and the second attention feature corresponding to other prediction guidance units whose character positions are before the prediction guidance unit, to obtain the second predicted character feature corresponding to the prediction guidance unit with the set number of characters.

8. The method according to claim 6, characterized in that, After the feature processing module performs attention processing based on the initial character features and the prediction guidance unit to generate a first attention feature corresponding to the initial character features and a second attention feature corresponding to the prediction guidance unit, the method further includes: Store the first attention feature and the second attention feature in the specified storage space; Specifically, after the first attention feature and the second attention feature are generated again, the stored second attention feature is deleted from the designated storage space.

9. The method according to claim 1, characterized in that, The method further includes: Acquire a training document page image, a tag character sequence corresponding to the training document page image, and training text prompts for indicating that the training document page image is to be parsed; wherein, at least two tag characters in the tag character sequence correspond to a character position respectively; Training prediction guidance units are inserted at specified character positions after the character positions corresponding to each label character in the label character sequence to obtain training input data; The training input data, the training document page image, and the training text prompt information are input into the initial document parsing model to perform character prediction processing through the initial document parsing model to obtain the training character sequence. Based on the difference between the tag character sequence and the training character sequence, the initial document parsing model is trained to obtain the document parsing model; The document parsing model is used to perform character prediction processing based on the document page image and the text indication information to generate the target character and at least two characters following the target character.

10. The method according to claim 9, characterized in that, At least two training characters in the training character sequence each correspond to a character position; The step of training the initial document parsing model based on the difference between the tag character sequence and the training character sequence to obtain the document parsing model includes: Based on the differences between each training character in the training character sequence and the corresponding label character in the label character sequence, target loss data is generated; Based on the target loss data, the initial document parsing model is trained to obtain the document parsing model.

11. The method according to claim 9, characterized in that, The step of obtaining the training document page image includes: Obtain at least two initial document page images; The at least two initial document page images are input into a pre-trained discriminant model to detect at least one of the image quality and document type corresponding to each of the at least two initial document page images. The filtering process is performed based on the set filtering rules and the image quality and document type corresponding to each initial document page image. Based on at least two candidate document page images obtained from the filtering, the obtained training document page image is determined.

12. The method according to claim 11, characterized in that, The determination of the acquired training document page image based on at least two candidate document page images obtained through filtering includes: Extract the image semantic features corresponding to the at least two candidate document page images respectively; Based on the image semantic features corresponding to the at least two candidate document page images, candidate document page images with image semantic feature similarity greater than a first similarity threshold are removed from the at least two candidate document page images, and the removed candidate document page images are determined as the obtained training document page images.

13. The method according to claim 11, characterized in that, The determination of the acquired training document page image based on at least two candidate document page images obtained through filtering includes: Calculate the pixel hash values ​​corresponding to the at least two candidate document page images respectively; Based on the pixel hash values ​​corresponding to the at least two candidate document page images, candidate document page images with pixel hash value similarity greater than a second similarity threshold are removed from the at least two candidate document page images, and the removed candidate document page images are determined as the obtained training document page images.

14. The method according to claim 9, characterized in that, The step of obtaining the tag character sequence corresponding to the training document page image includes: The training document page image is processed by character prediction using at least two pre-trained reference document parsing models to obtain reference tag character sequences generated by the at least two reference document parsing models respectively; If, among the at least two reference tag character sequences, the character similarity between the target reference tag character sequence and all other reference tag character sequences other than the target reference tag character sequence is less than a set character similarity threshold, then the target reference tag character sequence is determined as the tag character sequence.

15. A document parsing device, characterized in that, The device includes: The first character prediction processing unit is used to perform character prediction processing based on the document page image to be parsed and the text prompt information indicating that the document page image should be parsed, so as to generate target characters; The second character prediction processing unit is used to perform character prediction processing based on the target character and the set prediction guidance unit to generate at least two characters whose character positions are after the target character; wherein, the prediction guidance unit is used to predict characters whose character positions are after the target character and are not adjacent to the target character; The third character prediction processing unit is used to, if the obtained at least two characters do not contain an end character, take the at least two characters as target characters and perform character prediction processing again based on the target characters and the prediction guidance unit until the obtained at least two characters contain the end character; The generation unit is used to generate a parsing result corresponding to the document page image based on the characters obtained from each character prediction process. The parsing result includes text information that characterizes the content elements in the document page image and the structural relationships between the content elements.

16. An electronic device, characterized in that, include: One or more processors; as well as A memory for storing one or more computer programs, which, when executed by one or more processors, cause the electronic device to implement the document parsing method according to any one of claims 1-14.

17. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the document parsing method according to any one of claims 1-14.

18. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and the processor of the electronic device reads from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the document parsing method according to any one of claims 1-14.

Images