Document processing method and apparatus

Abstract

The application provides a document processing method and device, and belongs to the technical field of artificial intelligence. The document processing method comprises the following steps: obtaining a to-be-processed document; identifying document information of the to-be-processed document, wherein the document information comprises at least one of the following types: text information, image information and layout information; performing feature extraction on different types of document information respectively to obtain a feature vector corresponding to each type of document information; and fusing the feature vectors of different types of document information to obtain a document feature vector of the to-be-processed document. The technical scheme of the application can process different types of documents.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a document processing method and apparatus. Background Technology

[0002] With the development of internet technology and the digital economy, structured analysis and content extraction of document data have become crucial for industrial upgrading and digital transformation. Automatic, accurate, and rapid document information processing is essential for improving the efficiency and productivity of society as a whole. Document intelligence models and algorithms aim to automatically classify, extract, and structure information in documents, thereby accelerating automated document processing. Document intelligence tasks are more challenging than traditional Natural Language Processing (NLP) tasks: 1) In addition to textual information, documents also contain structured or layout information, such as document layout and the font, color, and position of text; 2) Documents may exist in image form, such as scanned documents and web pages. Existing NLP models cannot directly process image documents, while directly using visual models cannot model the semantic information in documents.

[0003] In related technologies, methods based on pre-trained models typically first use content extraction tools to extract textual, layout, and image information from documents of different layouts. Then, large-scale pre-trained models are used to model and understand this textual, layout, and image information, ultimately completing various downstream tasks. Representative tasks include document layout analysis, entity recognition, relation extraction, document classification, and document question answering. However, these methods usually only handle text data with specific layouts and modalities. Furthermore, different document understanding models typically need to be designed and trained for different document layouts, resulting in poor scalability and reusability. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a document processing method and apparatus that can process different types of documents.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solutions:

[0006] On one hand, embodiments of the present invention provide a document processing method, including:

[0007] Get the document to be processed;

[0008] The document information of the document to be processed is identified, and the document information includes at least one of the following types: text information, image information, and layout information;

[0009] Feature extraction is performed on different types of document information to obtain the feature vector corresponding to each type of document information;

[0010] The feature vectors of the document information of different types are fused to obtain the document feature vector of the document to be processed.

[0011] In some embodiments, feature extraction of the text information includes:

[0012] The text information is divided into multiple text words;

[0013] Encode the multiple text words respectively to obtain multiple first vectors;

[0014] The position information of the multiple text words is encoded to obtain multiple second vectors;

[0015] The plurality of first vectors and the plurality of second vectors are input into the first Transformer model to obtain text feature information, wherein the text feature information includes the feature vector of each text word and the feature vector of the entire text information;

[0016] The relation weights of the first Transformer model are determined by a first weight parameter, a second weight parameter, and a third weight parameter. The second weight parameter represents the self-attention weight of the i-th word relative to the j-th word in the text information. The third weight parameter represents the weight generated by the layout information of the i-th word relative to the j-th word. This represents the weight of the image features corresponding to the i-th word relative to the j-th word, where i and j are positive integers.

[0017] In some embodiments, the relation weights of the first Transformer model

[0018] Among them, g L and g V This is a gated function.

[0019] In some embodiments, feature extraction of the image information includes:

[0020] The ResNet model is used to extract multiple image blocks corresponding to the text from the document to be processed;

[0021] Encode the multiple image blocks separately to obtain multiple third vectors;

[0022] The position information of the multiple image blocks is encoded to obtain a fourth vector;

[0023] The plurality of third vectors and the plurality of fourth vectors are input into the second Transformer model to obtain image feature information, which includes the feature vector of each image block and the feature vector of the entire image information.

[0024] In some embodiments, the same location information encoder is used to encode the location information of the image blocks and the location information of the text words.

[0025] In some embodiments, feature extraction of the layout information includes:

[0026] Determine whether the document to be processed is an image document or a web page document;

[0027] When the document to be processed is an image document, the discrete position information of the image document is encoded to obtain the initial layout features; when the document to be processed is a web page document, the tag sequence and position order of each text word are encoded separately, and the encoding results are concatenated and input into the LSTM model to obtain the initial layout features.

[0028] The initial layout features are input into the third Transformer model to obtain layout feature information, which includes the feature vector of each text word and the feature vector of the entire layout information.

[0029] In some embodiments, fusing feature vectors of different types of document information to obtain the document feature vector of the document to be processed includes:

[0030] The document feature vector F of the document to be processed is obtained using the following formula:

[0031] F = F T +g ′ L (D)F L +g ′ V (D)F v

[0032] Among them, F T For text feature information, F L For image feature information, F v For layout feature information, g ′ L (D) and g ′ V (D) is a modality-aware gating function. When the document to be processed, D, is a plain text document, g... ′ L (D) and g ′ V (D) are all 0.

[0033] This invention also provides a document processing apparatus, comprising:

[0034] The acquisition module is used to acquire documents to be processed.

[0035] The recognition module is used to recognize the document information of the document to be processed, wherein the document information includes at least one of the following types: text information, image information, and layout information;

[0036] The feature extraction module is used to extract features from different types of document information to obtain feature vectors corresponding to each type of document information.

[0037] The fusion module is used to fuse the feature vectors of document information of different types to obtain the document feature vector of the document to be processed.

[0038] In some embodiments, the feature extraction module is specifically used to divide the text information into multiple text words; encode the multiple text words respectively to obtain multiple first vectors; encode the position information of the multiple text words respectively to obtain multiple second vectors; input the multiple first vectors and the multiple second vectors into a first Transformer model to obtain text feature information, wherein the text feature information includes the feature vector of each text word and the feature vector of the entire text information;

[0039] The relation weights of the first Transformer model are determined by a first weight parameter, a second weight parameter, and a third weight parameter. The second weight parameter represents the self-attention weight of the i-th word relative to the j-th word in the text information. The third weight parameter represents the weight generated by the layout information of the i-th word relative to the j-th word. This represents the weight of the image features corresponding to the i-th word relative to the j-th word, where i and j are positive integers.

[0040] In some embodiments, the relation weights of the first Transformer model

[0041] Among them, g L and g V This is a gated function.

[0042] In some embodiments, the feature extraction module is specifically used to extract multiple image blocks corresponding to text from the document to be processed using a ResNet model; encode the multiple image blocks respectively to obtain multiple third vectors; encode the position information of the multiple image blocks respectively to obtain fourth vectors; input the multiple third vectors and the multiple fourth vectors into a second Transformer model to obtain image feature information, wherein the image feature information includes the feature vector of each image block and the feature vector of the entire image information.

[0043] In some embodiments, the feature extraction module is specifically used to encode the positional information of the image block and the positional information of the text word using the same positional information encoder.

[0044] In some embodiments, the feature extraction module is specifically used to determine whether the document to be processed is an image document or a web page document; if the document to be processed is an image document, the discrete position information of the image document is encoded to obtain initial layout features; if the document to be processed is a web page document, the tag sequence and position order of each text word are encoded respectively, and the encoding results are concatenated and input into an LSTM model to obtain initial layout features; the initial layout features are input into a third Transformer model to obtain layout feature information, which includes the feature vector of each text word and the feature vector of the entire layout information.

[0045] In some embodiments, the fusion module is specifically used to obtain the document feature vector F of the document to be processed using the following formula:

[0046] F = F T +g ′ L (D)F L +g ′ V (D)F v

[0047] Among them, F T For text feature information, F L For image feature information, F v For layout feature information, g ′ L (D) and g ′ V (D) is a modality-aware gating function. When the document to be processed, D, is a plain text document, g... ′ L (D) and g ′ V (D) are all 0.

[0048] This invention also provides a document processing apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the program, it implements the document processing method described above.

[0049] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the document processing method described above.

[0050] The embodiments of the present invention have the following beneficial effects:

[0051] In the above scheme, feature extraction is performed on different types of document information to obtain the feature vector corresponding to each type of document information. The feature vectors of different types of document information are fused to obtain the document feature vector of the document to be processed. The document processing method of this embodiment can process documents containing text information, image information, and layout information. It is not limited by a specific document format and structure, and can uniformly process different types of document data, effectively utilizing the massive data resources of different types of documents. Attached Figure Description

[0052] Figure 1 A schematic diagram illustrating the breakdown of atomic capabilities and a large closed-loop AI system;

[0053] Figure 2 This is a flowchart illustrating the document processing method according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the document processing framework according to an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram illustrating the processing of text information according to an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram illustrating the image information processing according to an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram illustrating the layout information of image documents and web page documents;

[0058] Figure 7 This is a schematic diagram illustrating the encoding of layout information according to an embodiment of the present invention;

[0059] Figure 8 This is a schematic diagram of the document processing device according to an embodiment of the present invention;

[0060] Figure 9 This is a schematic diagram of the composition of the document processing device according to an embodiment of the present invention. Detailed Implementation

[0061] To make the technical problems, technical solutions and advantages of the embodiments of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0062] Methods based on pre-trained models typically first use content extraction tools to extract textual, layout, and image information from documents of different layouts. Then, large-scale pre-trained models are used to model and understand this textual, layout, and image information, ultimately completing various downstream tasks. Representative tasks include document layout analysis, entity recognition, relation extraction, document classification, and document question answering. When modeling multimodal features, these methods often directly add layout and image information onto textual features. While this integrates data features from multiple modalities, it suffers from high coupling. It not only struggles to avoid severe interference from positional and image information on the semantic content of the text but also fails to capture the spatial and visual relationships between different semantic elements within the document. Furthermore, these methods usually only handle textual data with specific layouts and containing specific modalities. Different document understanding models typically need to be designed and trained for different document layouts, resulting in poor scalability and reusability.

[0063] Related technologies propose a development direction for Holistic AI, researching new theories, mechanisms, paradigms, and frameworks needed for the systematic reconstruction of AI technologies. The goal is to achieve efficient training, learning inference, and generalized deployment in open environments, leveraging communication networks and diverse AI computing power. Atomistic reconstruction of AI technologies, as the foundation for Holistic AI, primarily involves the atomistic decomposition and reconstruction of current AI technologies based on principles such as reusability, low complexity, easy scheduling, and self-consistency / closed-loop operation. For example... Figure 1 The diagram illustrates the breakdown of atomic capabilities and a large-scale closed-loop AI example. Atomic capabilities include the interface layer, adaptation layer, and general intelligence layer. In the actual operation of a systematic AI system, the optimal sequence of AI atomic capabilities is determined based on business needs. Once the end-to-end AI application is running, business stakeholders can choose whether to promptly feed the received data back to the system, depending on whether there is a need for continuous optimization. If there is a need for continuous optimization, the system performs large-scale closed-loop optimization after receiving the business feedback data. This large-scale closed-loop optimization primarily involves continuously optimizing the adaptation layer of the AI ​​atomic capabilities, thereby continuously improving business performance metrics.

[0064] Document intelligence technology, as a current hot research direction, has broad application prospects. However, document data is typically characterized by modal diversity and structural complexity. Therefore, document intelligence technology usually needs to meet characteristics such as reusability, low complexity, and ease of scheduling to adapt to complex and ever-changing real-world application scenarios. Thus, atomically decomposing and reconstructing document intelligence technology is one of the important demonstration application scenarios for realizing systematic artificial intelligence.

[0065] Based on the idea of ​​atomized reconstruction in systematic AI and addressing the problems existing in current document intelligence technology, this embodiment provides a document processing method and apparatus that structurally decouples the understanding processes of three modalities: text content, layout information, and image information, enabling the processing of different types of documents.

[0066] Embodiments of the present invention provide a document processing method, such as... Figure 2 As shown, it includes:

[0067] Step 101: Obtain the document to be processed;

[0068] Step 102: Identify the document information of the document to be processed, wherein the document information includes at least one of the following types: text information, image information, and layout information;

[0069] Step 103: Extract features from different types of document information to obtain the feature vector corresponding to each type of document information;

[0070] Step 104: Fuse the feature vectors of the document information of different types to obtain the document feature vector of the document to be processed.

[0071] In this embodiment, feature extraction is performed on different types of document information to obtain feature vectors corresponding to each type of document information. The feature vectors of different types of document information are fused to obtain the document feature vector of the document to be processed. The document processing method of this embodiment can process documents containing text information, image information, and layout information. It is not limited by specific document format and structure, and can uniformly process different types of document data, effectively utilizing the massive data resources of different types of documents. Figure 3 This is a schematic diagram of the document processing framework according to an embodiment of the present invention, such as... Figure 3 As shown, given a document to be processed, the first step is to use a layout analysis tool to identify the text, image, and layout information contained within the document. Then, based on the modality-aware gate, the appropriate modality adaptation layer is selected according to the modality information for single-modality feature processing. Finally, multiple modality features are fused to execute downstream tasks. Specifically, the layout information adaptation layer is used for feature extraction of layout information, the text information adaptation layer for feature extraction of text information, and the image information adaptation layer for feature extraction of image information. The modality-aware gate enables dynamic activation and scheduling of different modality adaptation layers, thereby meeting the model's need for a unified understanding of document data containing different modalities.

[0072] In this embodiment, different models can be used to process different types of document information in a targeted manner. For example, the main difference between web page documents and image documents lies in their layout. The layout information of image documents can usually be represented by the coordinate positions of text, while the layout information of web page documents can be represented not only by coordinate positions but also by dependency paths generated by XML markup language. Therefore, the layout information can be decoupled from the document and a layout information processing model can be used to extract features from the layout information. The text information processing model can process the text information of documents with different layouts. If the layout remains unchanged, only the text information processing model needs to be modified to process documents in other languages, without modifying or retraining the layout information processing model. Therefore, this embodiment can effectively improve the flexibility and adaptability of the model.

[0073] In some embodiments, feature extraction of the text information includes:

[0074] The text information is divided into multiple text words;

[0075] Encode the multiple text words respectively to obtain multiple first vectors;

[0076] The position information of the multiple text words is encoded to obtain multiple second vectors;

[0077] The plurality of first vectors and the plurality of second vectors are input into the first Transformer model to obtain text feature information, wherein the text feature information includes the feature vector of each text word and the feature vector of the entire text information;

[0078] The relation weights of the first Transformer model are determined by a first weight parameter, a second weight parameter, and a third weight parameter. The second weight parameter represents the self-attention weight of the i-th word relative to the j-th word in the text information. The third weight parameter represents the weight generated by the layout information of the i-th word relative to the j-th word. This represents the weight of the image features corresponding to the i-th word relative to the j-th word, where i and j are positive integers.

[0079] In some embodiments, the relation weights of the first Transformer model

[0080] Among them, g L and g V This is a gated function.

[0081] In this embodiment, for a given text sequence in a document, a multi-layer Transformer model from the field of natural language processing can be used to extract the text feature representation of the document, denoted as text feature information FT = [f cls ,f t1 ,f t2 ,……,f tn ], f tn The feature representation of the nth word tn, f cls The feature representation of the entire text sequence, the specific process is as follows: Figure 4 As shown. To better perceive the dependencies between different text words during text modeling, the influence of positional relationships covered by layout information and attribute features contained in image information on text features is considered simultaneously during text representation. Specifically, this embodiment designs a self-attention control mechanism based on vision and layout awareness to achieve flexible adaptation to multiple modal information, that is, during the calculation of self-attention, a gating function g is used. L and g V Determine whether to consider the influence of other modal data.

[0082]

[0083] in This represents the self-attention weight of the i-th word relative to the j-th word, obtained from the features of the original text. This represents the weight generated by the layout information of the i-th word relative to the j-th word. This represents the weight of the i-th word relative to the image features corresponding to the j-th word. and The values ​​are calculated by the layout information adaptation layer and the image information adaptation layer, respectively, where D represents the entire document. The gating function controls whether the model considers information from other modalities when calculating the final attention weights. If the document is a plain text document, then g... L and g V The value of is 0, meaning that the contribution of other modal information to the current text features is not considered.

[0084] In some embodiments, feature extraction of the image information includes:

[0085] The ResNet model is used to extract multiple image blocks corresponding to the text from the document to be processed;

[0086] Encode the multiple image blocks separately to obtain multiple third vectors;

[0087] The position information of the multiple image blocks is encoded to obtain a fourth vector;

[0088] The plurality of third vectors and the plurality of fourth vectors are input into the second Transformer model to obtain image feature information, which includes the feature vector of each image block and the feature vector of the entire image information.

[0089] In some embodiments, the same location information encoder is used to encode the location information of the image blocks and the location information of the text words.

[0090] Compared to textual information, image information typically contains more intuitive and finer-grained visual information, such as font, color, and size. Therefore, utilizing the semantic features of image information can compensate for the missing information in plain text, helping the model to comprehensively and deeply understand the document content. The image blocks corresponding to the document text are determined by the bounding boxes extracted by the document parsing tool. For a given sequence of images corresponding to the text in a document, the image information processing process is as follows: Figure 5 As shown, the ResNet model is first used to extract the image feature information corresponding to the text. To capture the visual relationship between different image blocks, a multi-layer Transformer can be used to update the image features of each block and obtain the attention score of image block i relative to image block j. The final image features are denoted as FV = [f img ,f v1 ,f v2 ,……,f vn ], where f img This represents the image features of the entire document. To ensure alignment between image regions and text information, the same positional information encoder as the text information encoding layer can be used to encode the sequence information of the image.

[0091] In some embodiments, feature extraction of the layout information includes:

[0092] Determine whether the document to be processed is an image document or a web page document;

[0093] When the document to be processed is an image document, the discrete position information of the image document is encoded to obtain the initial layout features; when the document to be processed is a web page document, the tag sequence and position order of each text word are encoded separately, and the encoding results are concatenated and input into the LSTM model to obtain the initial layout features.

[0094] The initial layout features are input into the third Transformer model to obtain layout feature information, which includes the feature vector of each text word and the feature vector of the entire layout information.

[0095] As a unique attribute of document data, layout information can not only represent the absolute position information of the text in the document page, but also be used to represent the relative spatial position relationship between different texts in the document. Therefore, modeling the layout information helps the model to deeply understand the document content. Different types of documents have different layout forms and layout information expression methods, so it is difficult to model the layout information through a unified model. For example, the layout information of image documents is usually represented by the discrete coordinates of the area where the text is located, while the layout information of web documents can be represented by the DOM Tree or XPath generated by its markup language, specifically as Figure 6 shown.

[0096] In the related art, different model structures usually need to be trained for documents of different formats, and the generality and reusability of the model are weak. To address the above problems, in this embodiment, a general layout information encoding layer is designed to obtain layout features with a unified format, improving the adaptability of the model to documents of different formats. The general layout information encoding layer includes an image document layout encoder for encoding the layout information of image documents, a web document layout encoder for encoding the layout information of web documents, and a layout gating unit that can select the corresponding layout encoder according to the document type. The specific framework is as Figure 7 shown.

[0097] For the image document layout encoder, a neural network with 6 lightweight linear layers + activation layers is designed as the position information encoder to encode the discrete position (upper left corner, lower left corner, upper right corner, lower right corner, height, and width) information, and then 6 features with different dimensions are combined to obtain the initial layout features.

[0098] Considering that the layout information of web documents presents serialized characteristics, the web document layout encoder uses an LSTM network to encode the XPath corresponding to the web text. For example, given Figure 6 the XPath corresponding to the Chinese text word "full of New Year flavor": <html,body,div[1],div[2],h1>, first it is parsed into a tag sequence <html,body,div,div,h1> and a tag occurrence position sequence <0,0,1,2,0>, and then the WordEmbedding layer and the position information encoding layer are used to encode the tag sequence and the position order respectively, and then the two are concatenated and input into the LSTM to obtain the initial global feature representation of the layout information XPath of this node.

[0099] After obtaining the layout features, a multi-layer Transformer network architecture can be used to further understand the layout information, so as to capture the positional semantic association relationship between different words or different nodes, that is, layout attention Finally, the layout feature FL = [f lay,f l1 ,f l2 ,……,f ln ], where f ln f represents the layout feature corresponding to the nth word in the document sequence. lay This represents the feature representation of the entire document layout. Optionally, a positional feature representation layer from the text sequence can be added to the input to ensure the correspondence between layout information and text and image information.

[0100] In some embodiments, fusing feature vectors of different types of document information to obtain the document feature vector of the document to be processed includes:

[0101] The document feature vector F of the document to be processed is obtained using the following formula:

[0102] F = F T +g ′ L (D)F L +g ′ V (D)F v

[0103] Among them, F T For text feature information, F L For image feature information, F v For layout feature information, g ′ L (D) and g ′ V (D) is a modality-aware gating function. When the document to be processed, D, is a plain text document, g... ′ L (D) and g ′ V (D) are all 0. F will be applied to downstream tasks of document understanding, such as semantic analysis, automatic question and answer completion, and whole-page classification.

[0104] This invention also provides a document processing device, such as... Figure 8 As shown, it includes:

[0105] Module 21 is used to acquire the document to be processed;

[0106] The recognition module 22 is used to recognize the document information of the document to be processed, wherein the document information includes at least one of the following types: text information, image information, and layout information;

[0107] Feature extraction module 23 is used to extract features from different types of document information to obtain feature vectors corresponding to each type of document information;

[0108] The fusion module 24 is used to fuse the feature vectors of document information of different types to obtain the document feature vector of the document to be processed.

[0109] In this embodiment, feature extraction is performed on different types of document information to obtain feature vectors corresponding to each type of document information. The feature vectors of different types of document information are fused to obtain the document feature vector of the document to be processed. The document processing method of this embodiment can process documents containing text information, image information, and layout information. It is not limited by specific document format and structure, and can uniformly process different types of document data, effectively utilizing the massive data resources of different types of documents.

[0110] In some embodiments, the feature extraction module 23 is specifically used to divide the text information into multiple text words; encode the multiple text words respectively to obtain multiple first vectors; encode the position information of the multiple text words respectively to obtain multiple second vectors; input the multiple first vectors and the multiple second vectors into a first Transformer model to obtain text feature information, wherein the text feature information includes the feature vector of each text word and the feature vector of the entire text information;

[0111] The relation weights of the first Transformer model are determined by a first weight parameter, a second weight parameter, and a third weight parameter. The second weight parameter represents the self-attention weight of the i-th word relative to the j-th word in the text information. The third weight parameter represents the weight generated by the layout information of the i-th word relative to the j-th word. This represents the weight of the image features corresponding to the i-th word relative to the j-th word, where i and j are positive integers.

[0112] In some embodiments, the relation weights of the first Transformer model

[0113] Among them, g L and g V This is a gated function.

[0114] In some embodiments, the feature extraction module 23 is specifically used to extract multiple image blocks corresponding to text from the document to be processed using a ResNet model; encode the multiple image blocks respectively to obtain multiple third vectors; encode the position information of the multiple image blocks respectively to obtain fourth vectors; input the multiple third vectors and the multiple fourth vectors into a second Transformer model to obtain image feature information, wherein the image feature information includes the feature vector of each image block and the feature vector of the entire image information.

[0115] In some embodiments, the feature extraction module 23 is specifically used to encode the position information of the image block and the position information of the text word using the same position information encoder.

[0116] In some embodiments, the feature extraction module 23 is specifically used to determine whether the document to be processed is an image document or a web page document; if the document to be processed is an image document, the discrete position information of the image document is encoded to obtain initial layout features; if the document to be processed is a web page document, the tag sequence and position order of each text word are encoded respectively, and the encoding results are concatenated and input into the LSTM model to obtain initial layout features; the initial layout features are input into the third Transformer model to obtain layout feature information, the layout feature information including the feature vector of each text word and the feature vector of the entire layout information.

[0117] In some embodiments, the fusion module 24 is specifically used to obtain the document feature vector F of the document to be processed using the following formula:

[0118] F = F T +g ′ L (D)F L +g ′ V (D)F v

[0119] Among them, F T For text feature information, F L For image feature information, F v For layout feature information, g ′ L (D) and g ′ V (D) is a modality-aware gating function. When the document to be processed, D, is a plain text document, g... ′ L (D) and g ′ V (D) are all 0.

[0120] This invention also provides a document processing device, such as... Figure 9 As shown, it includes a memory 31, a processor 32, and a computer program stored on the memory 31 and executable on the processor 32; when the processor 32 executes the program, it implements the document processing method described above.

[0121] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the document processing method described above.

[0122] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage, or any other non-transferable medium that can be used to store information accessible to the computer-readable terminal device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0123] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A document processing method, characterized in that, include: Get the document to be processed; The document information of the document to be processed is identified, and the document information includes at least one of the following types: text information, image information, and layout information; Feature extraction is performed on different types of document information to obtain the feature vector corresponding to each type of document information; The feature vectors of the document information of different types are fused to obtain the document feature vector of the document to be processed; Feature extraction of the text information includes: The text information is divided into multiple text words; Encode the multiple text words respectively to obtain multiple first vectors; The position information of the multiple text words is encoded to obtain multiple second vectors; The plurality of first vectors and the plurality of second vectors are input into the first Transformer model to obtain text feature information, wherein the text feature information includes the feature vector of each text word and the feature vector of the entire text information; The relation weights of the first Transformer model are determined by a first weight parameter, a second weight parameter, and a third weight parameter. The second weight parameter represents the self-attention weight of the i-th word relative to the j-th word in the text information. The third weight parameter represents the weight generated by the layout information of the i-th word relative to the j-th word. This represents the weight of the i-th word relative to the image features corresponding to the j-th word, where i and j are positive integers; Relationship weights of the first Transformer model ;in, and is the gate function, and D is the document to be processed.

2. The document processing method according to claim 1, characterized in that, Feature extraction of the image information includes: The ResNet model is used to extract multiple image blocks corresponding to the text from the document to be processed; Encode the multiple image blocks separately to obtain multiple third vectors; The position information of the multiple image blocks is encoded to obtain multiple fourth vectors; The plurality of third vectors and the plurality of fourth vectors are input into the second Transformer model to obtain image feature information, which includes the feature vector of each image block and the feature vector of the entire image information.

3. The document processing method according to claim 2, characterized in that, The same location information encoder is used to encode the location information of the image block and the location information of the text word.

4. The document processing method according to claim 1, characterized in that, Feature extraction of the layout information includes: Determine whether the document to be processed is an image document or a web page document; When the document to be processed is an image document, the discrete position information of the image document is encoded to obtain the initial layout features; when the document to be processed is a web page document, the tag sequence and position order of each text word are encoded separately, and the encoding results are concatenated and input into the LSTM model to obtain the initial layout features. The initial layout features are input into the third Transformer model to obtain layout feature information, which includes the feature vector of each text word and the feature vector of the entire layout information.

5. The document processing method according to claim 1, characterized in that, The step of fusing feature vectors of different types of document information to obtain the document feature vector of the document to be processed includes: The document feature vector F of the document to be processed is obtained using the following formula: in, For text feature information, Image feature information, To lay out feature information, and For modality-aware gating functions, when the document D to be processed is a plain text document, and All are 0.

6. A document processing apparatus, characterized in that, include: The acquisition module is used to acquire documents to be processed. The recognition module is used to recognize the document information of the document to be processed, wherein the document information includes at least one of the following types: text information, image information, and layout information; The feature extraction module is used to extract features from different types of document information to obtain feature vectors corresponding to each type of document information. The fusion module is used to fuse the feature vectors of document information of different types to obtain the document feature vector of the document to be processed; The feature extraction module is specifically used to divide the text information into multiple text words; encode the multiple text words respectively to obtain multiple first vectors; encode the position information of the multiple text words respectively to obtain multiple second vectors; input the multiple first vectors and the multiple second vectors into a first Transformer model to obtain text feature information, wherein the text feature information includes the feature vector of each text word and the feature vector of the entire text information; The relation weights of the first Transformer model are determined by a first weight parameter, a second weight parameter, and a third weight parameter. The second weight parameter represents the self-attention weight of the i-th word relative to the j-th word in the text information. The third weight parameter represents the weight generated by the layout information of the i-th word relative to the j-th word. This represents the weight of the i-th word relative to the image features corresponding to the j-th word, where i and j are positive integers; Relationship weights of the first Transformer model ;in, and is the gate function, and D is the document to be processed.

7. The document processing apparatus according to claim 6, characterized in that, The feature extraction module is specifically used to extract multiple image blocks corresponding to text from the document to be processed using a ResNet model; encode the multiple image blocks to obtain multiple third vectors; encode the position information of the multiple image blocks to obtain multiple fourth vectors; input the multiple third vectors and the multiple fourth vectors into a second Transformer model to obtain image feature information, wherein the image feature information includes the feature vector of each image block and the feature vector of the entire image information.

8. The document processing apparatus according to claim 7, characterized in that, The feature extraction module is specifically used to encode the positional information of the image block and the positional information of the text word using the same positional information encoder.

9. The document processing apparatus according to claim 6, characterized in that, The feature extraction module is specifically used to determine whether the document to be processed is an image document or a web page document; if the document to be processed is an image document, the discrete position information of the image document is encoded to obtain initial layout features; When the document to be processed is a web page document, the tag sequence and position order of each text word are encoded separately, and the encoding results are concatenated and input into the LSTM model to obtain the initial layout features; the initial layout features are input into the third Transformer model to obtain layout feature information, which includes the feature vector of each text word and the feature vector of the entire layout information.

10. The document processing apparatus according to claim 6, characterized in that, The fusion module is specifically used to obtain the document feature vector F of the document to be processed using the following formula: in, For text feature information, Image feature information, To lay out feature information, and For modality-aware gating functions, when the document D to be processed is a plain text document, and All are 0.

11. A document processing apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that, When the processor executes the program, it implements the document processing method as described in any one of claims 1-5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the document processing method as described in any one of claims 1-5.

Images