Document understanding model pre-training method, document understanding method, electronic equipment and storage medium

Abstract

The invention provides a document understanding model pre-training method, a document understanding method, electronic equipment and a storage medium, and relates to the technical field of document understanding, in the method, semantic chunk information and text information are combined, the input sequence of the text information is determined, the reading process of human beings can be simulated, and the user experience is improved. Therefore, the initial document understanding model can more easily capture the logic structure and context dependence of the sample document, the semantic coherence of the sample document pair is enhanced, and the initial document understanding model can determine the priority or importance of each element without designing a complex rule or algorithm. Moreover, the input sequence is kept, so that context information loss caused by rearrangement of the text information of each element can be avoided, information loss is reduced, and the model understanding accuracy is improved. An initial document understanding model adopted by the method comprises an encoder constructed based on a spatial state model, the calculation complexity is reduced from secondary reduction to linear reduction, the document processing efficiency is remarkably improved, the calculation time required by the model is shortened, and the memory overhead of the model is reduced.

Description

Technical Field

[0001] The present invention relates to the field of document understanding technology, and in particular to a document understanding model pre-training method, a document understanding method, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of technology, documents have become an essential tool for information transmission, driving a growing demand for document processing technologies. To reduce the tedious and complex human labor involved in traditional document processing, visually rich document understanding (VrDU) technology has emerged and has garnered widespread attention from both academia and industry. This technology aims to automate document information extraction, thereby improving the efficiency and accuracy of various applications.

[0003] Existing visually rich document understanding technologies are usually based on deep learning models, especially the Transformer architecture, such as LayoutLM and its successors. However, models based on the Transformer architecture have disadvantages such as high computational complexity and high memory consumption. Specifically, given a length of Due to the inherent self-attention mechanism, the computational complexity of the Transformer-based model is When processing long, text-intensive documents, the model's computational time and memory usage increase dramatically, making it inefficient for the model to be applied to long documents. Furthermore, existing document understanding technologies are affected by document layout, such as column divisions, tables, and images, which can prevent the model from accurately understanding the document's content.

[0004] Therefore, how to shorten the calculation time required for the model and reduce the model memory overhead while ensuring the accuracy of model understanding has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The present invention provides a document understanding model pre-training method, a document understanding method, an electronic device and a storage medium, which are used to solve the defects existing in the related art.

[0006] The present invention provides a document understanding model pre-training method, comprising: Obtaining text information of each element in a sample document and semantic block information of the sample document; determining an input order of the text information based on the semantic block information and the text information; Based on the input sequence, applying the text information to pre-train an initial document understanding model; The initial document understanding model includes an encoder constructed based on a spatial state model.

[0007] According to a document understanding model pre-training method provided by the present invention, the initial document understanding model is a spatial state model encoder or a bidirectional spatial state model encoder.

[0008] According to a document understanding model pre-training method provided by the present invention, the text information includes position information of each element, and the semantic block information includes position information of each semantic block in the sample document; and determining the input order of the text information based on the semantic block information and the text information includes: Performing position matching between the elements and the semantic blocks to determine the semantic blocks to which the elements belong; Based on the semantic blocks to which the elements belong, a semantic block priority scanning strategy is applied to sort the elements to obtain the input order.

[0009] According to a document understanding model pre-training method provided by the present invention, the semantic block priority scanning strategy includes a semantic block scanning order and an element scanning order; The semantic block scanning order and the element scanning order both include an order from top to bottom and from left to right, or an order from bottom to top and from right to left.

[0010] According to a document understanding model pre-training method provided by the present invention, the text information includes the text content and position information of each element; and the pre-training of the initial document understanding model based on the input order and applying the text information includes: Determining a word-gram sequence of the sample document based on the input order and the text content of each element, and determining a semantic vector sequence corresponding to the word-gram sequence; Determining a spatial vector sequence corresponding to the word-gram sequence based on the position information of each element; splicing the vectors in the semantic vector sequence and the spatial vector sequence according to word units to obtain a splicing result; Based on the splicing result, the initial document understanding model is pre-trained.

[0011] According to a document understanding model pre-training method provided by the present invention, the initial document understanding model is pre-trained based on the input sequence and using the text information, including: Based on the input sequence and taking the text information as a basis, a masked language prediction task is applied to pre-train the initial document understanding model.

[0012] According to a document understanding model pre-training method provided by the present invention, obtaining text information of each element in a sample document and semantic block information of the sample document includes: Acquiring the text information based on a text recognition tool; The semantic block information is obtained based on a document layout analysis tool.

[0013] The present invention also provides a document understanding method, comprising: Acquire target text information of each element in the target document and target semantic block information of the target document; determining a target input order of the target text information based on the target semantic block information and the target text information; Based on the target input order, the target text information is input into a document understanding model to obtain a feature vector of at least one target word in the target document output by the document understanding model; The document understanding model is obtained based on the above-mentioned document understanding model pre-training method.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements any one of the document understanding model pre-training methods or document understanding methods described above.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the document understanding model pre-training methods or document understanding methods described above.

[0016] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the document understanding model pre-training methods or document understanding methods described above.

[0017] The document understanding model pre-training method, document understanding method, electronic device and storage medium provided by the present invention combine semantic block information and text information to determine the input order of text information, which can simulate the human reading process, making it easier for the initial document understanding model to capture the logical structure and contextual dependency of the sample document, and enhance the semantic coherence of the sample document pair. There is no need to design complex rules or algorithms to enable the initial document understanding model to determine the priority or importance of each element. Moreover, maintaining the input order can avoid the loss of contextual information due to rearrangement of the text information of each element, reduce information loss, and improve the accuracy of model understanding. The initial document understanding model adopted by the method includes an encoder constructed based on a spatial state model. The spatial state model can model sequence data by updating the hidden state space, reducing the computational complexity from quadratic to linear, significantly improving document processing efficiency, shortening the computational time required for the model, and reducing the model memory overhead, so that the pre-trained document understanding model can more efficiently process text-intensive documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is one of the flow charts of the document understanding model pre-training method provided by the present invention.

[0020] Figure 2 It is a structural diagram of the bidirectional SSM encoder provided by the present invention.

[0021] Figure 3 This is one of the input sequence diagrams in the document understanding model pre-training method provided by the present invention.

[0022] Figure 4 This is the second input sequence diagram in the document understanding model pre-training method provided by the present invention.

[0023] Figure 5 This is the second flow chart of the document understanding model pre-training method provided by the present invention.

[0024] Figure 6 It is a flowchart of the document understanding method provided by the present invention.

[0025] Figure 7 It is a structural diagram of the document understanding model pre-training device provided by the present invention.

[0026] Figure 8 It is a structural diagram of the document understanding device provided by the present invention.

[0027] Figure 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0029] Existing visually rich document understanding technologies are typically based on pre-trained Transformer models and can be divided into three categories based on the input modality. The first approach extracts the text from the document and treats it as plain text, such as BERT. The second approach additionally utilizes layout information. By jointly modeling text and layout information, it can better understand the multimodal characteristics of the document, such as LayoutLM. The third approach uniformly models text, layout, and image information, strengthening the alignment of cross-modal information and further improving the model's adaptability to complex documents.

[0030] However, for long text inputs, the Transformer model suffers from high computational complexity and memory consumption. Furthermore, the model cannot accurately understand the document content due to document layout factors, such as column divisions, tables, and images.

[0031] To address this issue, a sliding window strategy is often used for text-intensive input documents, where the model processes elements in batches, processing fixed-length sequences at a time. For example, the LayoutLM family processes a maximum length of 512 characters at a time, meaning the sliding window size is 512 characters. However, this approach can lead to loss of global information within the document, compromising the quality of document understanding.

[0032] Based on this, an embodiment of the present invention provides a document understanding model pre-training method, so that the pre-trained document understanding model can shorten the calculation time required for the model and reduce the model memory overhead while ensuring the accuracy of model understanding.

[0033] Figure 1 A flowchart of a document understanding model pre-training method provided in an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes: S11, obtaining text information of each element in a sample document and semantic block information of the sample document; S12, determining an input order of the text information based on the semantic block information and the text information; S13, based on the input sequence, applying the text information to pre-train an initial document understanding model; The initial document understanding model includes an encoder constructed based on a spatial state model.

[0034] Specifically, the document understanding model pre-training method provided in the embodiment of the present invention is executed by a document understanding model pre-training device, which can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0035] First, step S11 is executed to obtain the text information of each element in the sample document and the semantic block information of the sample document. The sample document can be in image format or PDF format. The elements in the sample document refer to individual characters, words, or text lines contained in the sample document.

[0036] The text information of each element may include the text content of the element and the position information of the element. The position information of each element refers to the position information of the element in the sample document, which can be represented by the position information of the element's bounding rectangle. The position information of the bounding rectangle can be determined by the coordinates of the vertices of the bounding rectangle or the coordinates of the center point of the bounding rectangle, which is not specifically limited here.

[0037] The semantic block information of a sample document includes the semantic blocks contained in the sample document and their location information. Each semantic block is a portion of the sample document with a specific semantic function, such as a title or paragraph. The location information of each semantic block refers to its position within the sample document and can also be represented by the location information of the bounding rectangle of each semantic block.

[0038] Here, both text information and semantic block information can be obtained by processing sample documents using text recognition tools. These tools can include neural network models, optical character recognition (OCR) tools, and PDF parsers, depending on the format of the sample documents. Neural network models and OCR tools can process sample documents in both document and image formats, while PDF parsers can process sample documents in PDF format.

[0039] In addition, semantic block information can also be obtained by parsing sample documents using document layout analysis tools.

[0040] Then, step S12 is executed. Since each element in the sample document may have a complex two-dimensional layout, such as columns, tables, and images, the order in which the text information is input can be determined using the semantic block information and text information. This input order can be the order in which the text information of each element is input. This input order can be achieved by first sorting each semantic block in the sample document based on the semantic block information to obtain a semantic block sorting, then traversing each semantic block in the semantic block sorting to sort each element contained in the traversed semantic block.

[0041] The resulting ordering of the elements within each semantic block represents the input order of the text. Because the input order incorporates semantic block information and takes into account the semantic information of the sample document, it simulates the human reading process. This ensures that the text input to the initial document understanding model in this order has stronger semantic relevance, improving the initial document understanding model's understanding of the sample document.

[0042] Moreover, maintaining the input order can avoid the loss of contextual information due to rearranging the text information of each element and reduce information loss.

[0043] In this process, the semantic block where each element is located can be determined based on the position information of the semantic block and the position information of each element. Furthermore, all elements included in each semantic block can be determined.

[0044] Finally, step S13 is executed to pre-train the initial document understanding model based on the input order and the text information of each element. Specifically, the text information of each element is input into the initial document understanding model in the input order to pre-train the initial document understanding model. This input order takes into account the semantic information of the sample documents, making it easier for the initial document understanding model to capture the logical structure and contextual dependencies of the sample documents, enhancing the semantic coherence of the sample document pairs. This eliminates the need to design complex rules or algorithms for the initial document understanding model to determine the priority or importance of each element.

[0045] In the embodiment of the present invention, the initial document understanding model used includes an encoder built based on the State Space Model (SSM). SSM is different from the implementation principle of the Transformer architecture. SSM does not rely on the self-attention mechanism, but models the sequence data by updating the hidden state space, so it has a linear computational complexity. .

[0046] SSM is a continuous system, which is defined by a hidden state function with the independent variable t. , the input variable is t Mapped to the output of the independent variable t : ; ; in, is the evolution matrix, is the input mapping matrix, is the output mapping matrix, for In order to apply SSM to deep learning systems, the SSM is discretized and the zero-order hold method is used to solve the above differential equation to obtain: ; ; in, , , is the hidden state at time t-1, is the hidden state at time t, is the input at time t, is the output at time t, is the time step. In order to make SSM have stronger representation ability, Set as Input function, that is, these three parameters are input Obtained through linear transformation and other calculations.

[0047] It can be understood that before inputting the text information corresponding to each element into the initial document understanding model in the input order, it is also necessary to convert the text information corresponding to each element into an embedding vector that can be processed by the model, and then input the embedding vector corresponding to each element into the initial document understanding model in the input order.

[0048] The output of the initial document understanding model may include at least one predicted word of the sample document and position information of each predicted word.

[0049] Pre-training the initial document understanding model can determine sample labels by combining various downstream tasks. The sample labels and model outputs are then combined to calculate a loss function. The initial document understanding model is then iteratively updated using the loss function. When the loss function converges or a preset number of iterations is reached, the pre-trained document understanding model is obtained. Alternatively, the document understanding model can be obtained by performing unsupervised training on the initial document understanding model, which is not specifically limited here.

[0050] After pre-training the initial document understanding model, a working document understanding model is generated. The input of this document understanding model includes the text information of each element in the document arranged in a predetermined input order. The output of this document understanding model includes the feature vector of each word in the document. This feature vector can be applied to various downstream tasks, such as determining understanding information for the document. This understanding information can include paragraph information, key-value information, etc.

[0051] A document understanding model pretraining method provided in an embodiment of the present invention first obtains text information for each element in a sample document and semantic block information for the sample document; then uses the semantic block information and text information to determine the input order of the text information; and finally, uses this input order and the text information to pretrain an initial document understanding model. This method, which combines semantic block information and text information to determine the input order of text information, can simulate the human reading process, making it easier for the initial document understanding model to capture the logical structure and contextual dependencies of the sample document, enhancing the semantic coherence of the sample document pairs. This eliminates the need to design complex rules or algorithms for the initial document understanding model to determine the priority or importance of each element. Furthermore, maintaining the input order avoids the loss of contextual information due to rearranging the text information of each element, reducing information loss and improving model understanding accuracy. The initial document understanding model employed in this method includes an encoder based on a spatial state model. The spatial state model can model sequential data by updating the hidden state space, reducing computational complexity from quadratic to linear, significantly improving document processing efficiency, shortening the model's computational time, and reducing the model's memory overhead. This enables the pretrained document understanding model to more efficiently process text-intensive documents.

[0052] Based on the above embodiment, the initial document understanding model is a spatial state model encoder or a bidirectional spatial state model encoder.

[0053] Specifically, the initial document understanding model used in the embodiment of the present invention may be an SSM encoder or a bidirectional SSM encoder.

[0054] The SSM encoder may be a Mamba encoder having a forward path including SSM.

[0055] The bidirectional SSM encoder may be a Vision Mamba encoder having a forward path containing SSM and a reverse path containing SSM.

[0056] Taking the bidirectional SSM encoder as an example, the bidirectional SSM encoder consists of layer The input of the first layer is the embedding vector of the text information of each element, which includes n vector components.

[0057] Afterwards In the layer, Output of the layer It will be the first The input of the layer is connected through residual connection to obtain the Output of the layer : ; exist In, such as Figure 2 As shown in Figure 2, a bidirectional scanning strategy is adopted to better model context information. The input of the layer is first normalized and linearly transformed through two linear layers to generate the first feature and the second characteristic Z. Among them, Represents the eigenvalue corresponding to the nth vector component in the first feature X.

[0058] Then, the first feature X is processed in both the forward and reverse directions. In the forward processing, the first feature X passes through a one-dimensional convolution layer and applies an activation function to generate the third feature Next, the third feature The input mapping matrix variables are obtained by linear transformation through three linear transformation layers. , output mapping matrix variables and the time step variable , these variables and the third characteristic are input into SSM together to calculate the discretized and , thus obtaining the output of SSM . Reverse output It is obtained by flipping X, that is, X from Reverse to . Parameters for forward and backward processing are not shared.

[0059] After applying the activation function to the second feature Z, the positive output Perform gate operation, apply activation function to the second feature Z and then flip it, and output the reverse Perform gate operation and reverse output The corresponding gate operation result is flipped and compared with the positive output The corresponding gate operation results are added element by element and then passed through a linear layer to obtain The final output.

[0060] In the embodiment of the present invention, the bidirectional SSM encoder can simultaneously capture the context information of the current word due to the existence of a forward path and a reverse path, so as to more comprehensively capture the global information of the sample document with the initial document understanding model, thereby improving the understanding ability.

[0061] Based on the above embodiment, the text information includes position information of each element, and the semantic block information includes position information of each semantic block in the sample document; and determining the input order of the text information based on the semantic block information and the text information includes: Performing position matching between the elements and the semantic blocks to determine the semantic blocks to which the elements belong; Based on the semantic blocks to which the elements belong, a semantic block priority scanning strategy is applied to sort the elements to obtain the input order.

[0062] Specifically, the text information includes the position information of each element in the sample document, and the semantic block information includes the position information of each semantic block in the sample document. Therefore, when determining the input order of the text information, each element can be first positionally matched with each semantic block, that is, the position information of each element can be matched with the position information of each semantic block to determine the semantic block to which each element belongs. If the position information of an element is within the position information of a semantic block, it can be determined that the element is located in the semantic block and belongs to the semantic block.

[0063] Using the semantic blocks to which each element belongs, a semantic block-first scanning strategy is applied to sort the elements to obtain the input order. Here, the semantic block-first scanning strategy means first determining the semantic block scanning order. Then, when scanning each semantic block according to the semantic block scanning order, each element within the current semantic block is scanned according to the element scanning order.

[0064] Finally, the scanned elements are sorted to get the input order.

[0065] It is understandable that the semantic block scanning order and the element scanning order can be set as needed, for example, both can be set in order from top to bottom and from left to right, or both can be set in order from bottom to top and from right to left. No specific limitation is given here.

[0066] The input order is obtained from top to bottom and from left to right. Figure 3 direction of the arrow.

[0067] The input order is from bottom to top and from right to left. Figure 4 direction of the arrow.

[0068] by Figure 3 For example, when scanning each semantic block from top to bottom and from left to right, each element in the current semantic block is scanned from top to bottom and from left to right. Only after the entire current semantic block is scanned will the next semantic block be entered.

[0069] In an embodiment of the present invention, by position matching, the semantic block to which each element belongs is determined, and then a semantic block priority scanning strategy is applied to determine the input order, which can improve the consistency of layout and semantics and reduce ambiguity. Moreover, the method can also dynamically adapt to different document types such as legal documents and medical reports, enhancing generalization. The order from top to bottom and from left to right, or from bottom to top and from right to left, can make the model input meet the semantic relevance order of the user's reading, reducing the difficulty of model processing.

[0070] Based on the above embodiment, the text information includes the text content and position information of each element; and pre-training the initial document understanding model based on the input order and applying the text information includes: Determining a word-gram sequence of the sample document based on the input order and the text content of each element, and determining a semantic vector sequence corresponding to the word-gram sequence; Determining a spatial vector sequence corresponding to the word-gram sequence based on the position information of each element; splicing the vectors in the semantic vector sequence and the spatial vector sequence according to word units to obtain a splicing result; Based on the splicing result, the initial document understanding model is pre-trained.

[0071] Specifically, in an embodiment of the present invention, since the text information of each element may include text content and position information, when pre-training the initial document understanding model, the input order and the text content of each element can be used to first determine the word sequence of the sample document. The word sequence includes multiple tokens, and each word is a symbolic representation of the text content of the corresponding element.

[0072] After that, text embedding is performed on each word in the word sequence, that is, each word is mapped to a semantic vector, and the semantic vectors of each word are sorted according to the order of arrangement of each word in the word sequence to obtain a semantic vector sequence.

[0073] The position information of each element is mapped into a spatial vector. This spatial vector is the spatial vector of the word element corresponding to the element. The spatial vectors of each word element are sorted according to the arrangement order of each word element in the word element sequence to obtain a spatial vector sequence.

[0074] After that, the vectors in the semantic vector sequence and the spatial vector sequence are concatenated by word unit to obtain a concatenation result. This concatenation result is the embedding vector of the text information, which includes the concatenation vector of the semantic vector of each word unit and the spatial vector.

[0075] Finally, the concatenation result is input into the initial document understanding model to pre-train the initial document understanding model.

[0076] In an embodiment of the present invention, by combining text content and position information, the initial document understanding model can receive multimodal information, so that the initial document understanding model can pay attention to "text content" and "layout position" at the same time, and thus can understand the document logic more comprehensively, significantly improving the practicality of the initial document understanding model in real scenarios, and is especially suitable for complex tasks that require the combination of content and layout, such as contract parsing, table recognition, etc.

[0077] Based on the above embodiment, the method of pre-training the initial document understanding model based on the input order and applying the text information includes: Based on the input sequence and taking the text information as a basis, a masked language prediction task is applied to pre-train the initial document understanding model.

[0078] Specifically, in an embodiment of the present invention, when pre-training the initial document understanding model, a Masked Language Modeling (MLM) task may be used to pre-train the initial document understanding model.

[0079] The MLM task is to randomly select some tokens from the token sequence of a sample document and replace them with the special "[MASK]" symbol. Then, the initial document understanding model predicts the original masked words before the replacement based on the concatenation results containing the "[MASK]" symbol.

[0080] Obtain at least one predicted word and position information of each predicted word output by the initial document understanding model, calculate the loss function using at least one real word and the position information of each real word, at least one predicted word and the position information of each predicted word, and iteratively update the parameters of the initial document understanding model using the loss function until a preset number of iterations is reached or the loss function converges, thereby obtaining a document understanding model.

[0081] In the embodiment of the present invention, through the masked language prediction task, part of the input text is masked and the model is allowed to predict the masked content, which provides the model with powerful semantic understanding and generalization capabilities.

[0082] Based on the above embodiment, the step of obtaining text information of each element in the sample document and semantic block information of the sample document includes: Acquiring the text information based on a text recognition tool; The semantic block information is obtained based on a document layout analysis tool.

[0083] Specifically, since the granularity of text information and semantic block information is different, in the embodiment of the present invention, different tools are used to obtain text information and semantic block information respectively, so that the text information and semantic block information can be accurately obtained.

[0084] like Figure 5 As shown, for a sample document, a text recognition tool is used to obtain text information of each element in the sample document; and a document layout analysis tool is used to obtain semantic block information of the sample document.

[0085] Then, the semantic block-first scanning strategy is applied to sort the elements in the sample document to obtain the input order.

[0086] Thereafter, the word sequence and position sequence of the sample document are determined using the input sequence. Figure 5 In the example, the word sequence includes 6 words, which are represented as T1-T6, and T3 is replaced by [MASK]. The position sequence includes 6 position information, which are represented as Box1-Box6.

[0087] Determine the semantic vector sequence corresponding to the word sequence and space vector sequence , and concatenate the vectors in the semantic vector sequence and the spatial vector sequence according to word units to obtain the concatenation result.

[0088] The concatenation result is input into the initial document understanding model, the initial document understanding model predicts the word at [MASK], and the prediction result is used to calculate the loss function to pre-train the initial document understanding model.

[0089] Based on the above embodiments, Figure 6 As shown, an embodiment of the present invention further provides a document understanding method, which includes: S21, obtaining target text information of each element in the target document and target semantic block information of the target document; S22, determining a target input order of the target text information based on the target semantic block information and the target text information; S23, inputting the target text information into a document understanding model based on the target input order, and obtaining a feature vector of at least one target word in the target document output by the document understanding model; The document understanding model is obtained based on the document understanding model pre-training method provided in the above embodiments.

[0090] Specifically, the document understanding method provided in the embodiment of the present invention is executed by a document understanding device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0091] First, step S21 is performed to obtain target text information of each element in the target document and target semantic block information of the target document. The target text information and target semantic block information can be obtained by processing the target document using a text recognition tool, and the target semantic block information can also be obtained by parsing the target document using a document layout analysis tool.

[0092] Then, step S22 is executed to determine the target input order of the target text information using the target semantic block information and the target text information. The process of determining the target input order is consistent with the process of determining the input order in the above embodiments, which is detailed in the above embodiments and will not be repeated here.

[0093] Finally, step S23 is executed to input the target text information into the document understanding model using the target input order, and the document understanding model outputs a feature vector for at least one target word in the target document. This process is consistent with the process of inputting text information into the initial document understanding model using the input order and obtaining the output of the initial document understanding model in the aforementioned embodiments. For details, please refer to the aforementioned embodiments and will not be repeated here.

[0094] A document understanding method provided in an embodiment of the present invention first obtains target text information and target semantic block information for each element in a target document; then, using the target semantic block information and target text information, determines a target input order for the target text information; and finally, using the target input order, inputs the target text information into a document understanding model, outputting a feature vector for at least one target word in the target document. This method, which combines semantic block information with text information to determine the target input order, simulates the human reading process, enabling the document understanding model to more easily capture the logical structure and contextual dependencies of the target document, enhancing the semantic coherence of the target document pair, and eliminating the need to design complex rules or algorithms for the document understanding model to determine the priority or importance of each element. Furthermore, maintaining the target input order avoids the loss of contextual information caused by rearranging the target text information of each element, reducing information loss and improving model understanding accuracy. The document understanding model employed in this method includes an encoder based on a spatial state model. The spatial state model can model sequential data by updating the hidden state space, reducing computational complexity from quadratic to linear, significantly improving document processing efficiency, shortening the model's computational time, and reducing the model's memory overhead, enabling more efficient processing of text-intensive documents.

[0095] Based on the above embodiment, the target text information includes position information of the at least one target word; the step of inputting the target text information into a document understanding model based on the target input order, obtaining at least one target word in the target document and a feature vector of the at least one target word output by the document understanding model, and then comprising: Understanding information for the target document is determined based on the feature vector of the at least one target word and the position information of the at least one target word.

[0096] like Figure 7 As shown, based on the above embodiment, an embodiment of the present invention provides a document understanding model pre-training device, including: A first acquisition module 71 is used to acquire text information of each element in a sample document and semantic block information of the sample document; A first order determination module 72, configured to determine an input order of the text information based on the semantic block information and the text information; A pre-training module 73 is configured to pre-train an initial document understanding model based on the input sequence and applying the text information; The initial document understanding model includes an encoder constructed based on a spatial state model.

[0097] On the basis of the above embodiments, in the document understanding model pre-training device provided in the embodiments of the present invention, the initial document understanding model is a spatial state model encoder or a bidirectional spatial state model encoder.

[0098] On the basis of the above embodiment, in the document understanding model pre-training device provided in the embodiment of the present invention, the text information includes the position information of each element, and the semantic block information includes the position information of each semantic block in the sample document; The first sequence determination module is specifically configured to: Performing position matching between the elements and the semantic blocks to determine the semantic blocks to which the elements belong; Based on the semantic blocks to which the elements belong, a semantic block priority scanning strategy is applied to sort the elements to obtain the input order.

[0099] On the basis of the above embodiment, in the document understanding model pre-training device provided in the embodiment of the present invention, the semantic block priority scanning strategy includes a semantic block scanning order and an element scanning order; The semantic block scanning order and the element scanning order both include an order from top to bottom and from left to right, or an order from bottom to top and from right to left.

[0100] On the basis of the above embodiment, in the document understanding model pre-training device provided in the embodiment of the present invention, the text information includes the text content and position information of each element; The pre-training module is specifically used for: Determining a word-gram sequence of the sample document based on the input order and the text content of each element, and determining a semantic vector sequence corresponding to the word-gram sequence; Determining a spatial vector sequence corresponding to the word-gram sequence based on the position information of each element; splicing the vectors in the semantic vector sequence and the spatial vector sequence according to word units to obtain a splicing result; Based on the splicing result, the initial document understanding model is pre-trained.

[0101] Based on the above embodiments, the document understanding model pre-training device provided in the embodiments of the present invention is specifically used for: Based on the input sequence and taking the text information as a basis, a masked language prediction task is applied to pre-train the initial document understanding model.

[0102] Based on the above embodiment, in the document understanding model pre-training device provided in the embodiment of the present invention, the first acquisition module is used to: Acquiring the text information based on a text recognition tool; The semantic block information is obtained based on a document layout analysis tool.

[0103] Specifically, the functions of each module in the document understanding model pre-training device provided in the embodiment of the present invention correspond one-to-one to the operating procedures of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.

[0104] like Figure 8 As shown, based on the above embodiment, an embodiment of the present invention provides a document understanding device, including: The second acquisition module 81 is used to acquire target text information of each element in the target document and target semantic block information of the target document; A second order determination module 82 is configured to determine a target input order of the target text information based on the target semantic block information and the target text information; A document understanding module 83 is configured to input the target text information into a document understanding model based on the target input order, and obtain a feature vector of at least one target word in the target document output by the document understanding model; The document understanding model is obtained based on the document understanding model pre-training method provided in the above embodiments.

[0105] Figure 9 An example of a physical structure diagram of an electronic device is shown below. Figure 9 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logic instructions in the memory 830 to execute the document understanding model pre-training method or document understanding method provided in the above embodiments.

[0106] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the relevant art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0107] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the document understanding model pre-training method or document understanding method provided in the above embodiments.

[0108] In yet another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the document understanding model pre-training method or document understanding method provided in each of the above embodiments. The computer-readable storage medium may be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and is not specifically limited herein.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0110] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A document understanding model pre-training method, characterized in that: include: Obtaining text information of each element in a sample document and semantic block information of the sample document; determining an input order of the text information based on the semantic block information and the text information; Based on the input sequence, applying the text information to pre-train an initial document understanding model; The initial document understanding model includes an encoder constructed based on a spatial state model.

2. The document understanding model pre-training method according to claim 1, characterized in that: The initial document understanding model is a spatial state model encoder or a bidirectional spatial state model encoder.

3. The document understanding model pre-training method according to claim 1, characterized in that: The text information includes the position information of each element, and the semantic block information includes the position information of each semantic block in the sample document; The step of determining the input order of the text information based on the semantic block information and the text information includes: Performing position matching between the elements and the semantic blocks to determine the semantic blocks to which the elements belong; Based on the semantic blocks to which the elements belong, a semantic block priority scanning strategy is applied to sort the elements to obtain the input order.

4. The document understanding model pre-training method according to claim 3, characterized in that: The semantic block priority scanning strategy includes a semantic block scanning order and an element scanning order; The semantic block scanning order and the element scanning order both include an order from top to bottom and from left to right, or an order from bottom to top and from right to left.

5. The document understanding model pre-training method according to claim 1, characterized in that: The text information includes the text content and position information of each element; and the pre-training of the initial document understanding model based on the input order by applying the text information includes: Determining a word-gram sequence of the sample document based on the input order and the text content of each element, and determining a semantic vector sequence corresponding to the word-gram sequence; Determining a spatial vector sequence corresponding to the word-gram sequence based on the position information of each element; splicing the vectors in the semantic vector sequence and the spatial vector sequence according to word units to obtain a splicing result; Based on the splicing result, the initial document understanding model is pre-trained.

6. The document comprehension model pre-training method according to any one of claims 1 to 5, characterized in that: The pre-training of the initial document understanding model based on the input sequence and applying the text information includes: Based on the input sequence and taking the text information as a basis, a masked language prediction task is applied to pre-train the initial document understanding model.

7. The document comprehension model pre-training method according to any one of claims 1 to 5, characterized in that: The obtaining of text information of each element in the sample document and semantic block information of the sample document includes: Acquiring the text information based on a text recognition tool; The semantic block information is obtained based on a document layout analysis tool.

8. A document understanding method, characterized in that: include: Acquire target text information of each element in the target document and target semantic block information of the target document; determining a target input order of the target text information based on the target semantic block information and the target text information; Based on the target input order, the target text information is input into a document understanding model to obtain a feature vector of at least one target word in the target document output by the document understanding model; The document understanding model is obtained based on the document understanding model pre-training method according to any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the document understanding model pre-training method according to any one of claims 1 to 7, or the document understanding method according to claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the document understanding model pre-training method according to any one of claims 1 to 7, or the document understanding method according to claim 8.

Images