Image document interpretation method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides an image document interpretation method and device, equipment, a storage medium and a product. The method comprises the following steps: receiving and preprocessing an image document to obtain preprocessed texts, images and formats; inputting the preprocessed text, image and format into a trained multi-modal encoder, and outputting multi-modal features in an image document; the multi-modal features are input to a modal alignment module; a marking sub-module in the modal alignment module divides the multi-modal features to obtain keys and values in the image document; a connection sub-module in the modal alignment module uses a local attention layer and a global interaction layer guided by a space prefix to obtain a matching relationship between the key and the value; and inputting the matching relationship into a trunk of the large language model, and interpreting the image document. According to the embodiment of the invention, the accuracy of image document interpretation can be improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of natural language processing tasks, and in particular, to a method, device, equipment, storage medium and product for interpreting image documents. Background Art

[0002] The purpose of interpreting image documents is to extract and understand information from documents containing images. Since image documents contain multimodal features such as images, texts, and tables, most existing technologies interpret image documents through large multimodal models. However, when processing image-dense documents, the computational complexity of the existing large multimodal models increases quadratically with the increase in image resolution in dense documents. In order to prevent high computational costs caused by complexity, the existing large multimodal models usually reduce the resolution of the input image, which ultimately results in the inability of the existing large multimodal models to accurately interpret image-dense documents.

[0003] Therefore, there is an urgent need for an image document interpretation method that can improve the accuracy of image document interpretation. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a method, device, equipment, storage medium and product for interpreting an image document to improve the accuracy of interpreting an image document.

[0005] To achieve the above objectives, on the one hand, an embodiment of this specification provides a method for interpreting an image document, including:

[0006] Receiving and preprocessing image documents to obtain preprocessed text, images and layouts;

[0007] Input the preprocessed text, image and layout into the trained multimodal encoder and output the multimodal features in the image document;

[0008] Inputting the multimodal features into a modality alignment module;

[0009] The marking submodule in the modality alignment module divides the multimodal features to obtain keys and values ​​in the image document;

[0010] The connection submodule in the modality alignment module uses a local attention layer and a global interaction layer guided by a spatial prefix to obtain a matching relationship between the key and the value;

[0011] The matching relationship is input into the large language model backbone to interpret the image document.

[0012] Preferably, the receiving and preprocessing of the image document to obtain the preprocessed text, image and format further comprises:

[0013] receiving image documents;

[0014] Use the OCR toolkit to extract the content and two-dimensional position of text in image documents;

[0015] Based on the pre-trained model, word embedding is initialized according to the content and two-dimensional position of the text to obtain an embedding vector of the text word;

[0016] Adjust the image document to a set size to obtain an adjusted image;

[0017] After dividing the adjusted image into P×P image blocks, flatten it into the embedding vector of the image block;

[0018] According to the one-dimensional index position and two-dimensional border coordinates of the text in the image document, as well as the one-dimensional index position and two-dimensional border coordinates of the image, they are converted into a layout embedding vector.

[0019] Preferably, the trained multimodal encoder is trained by the following method:

[0020] Receiving and preprocessing a known image document to obtain preprocessed known text, known image and known format;

[0021] Masking a first set proportion of text words in the preprocessed known text to obtain a masked text;

[0022] Masking an image block of a second set ratio in the preprocessed known image to obtain a masked image;

[0023] Aligning the masked text with the masked image to obtain alignment information between the masked text and the masked image;

[0024] The masked text, the masked image and the alignment information are input into a multimodal encoder, and the multimodal encoder is trained using a target loss function.

[0025] Preferably, the objective loss function is:

[0026] L=L MIM +L MIM +L WPA ;

[0027] Among them, L is the target loss function, L MLM is the first loss function corresponding to the masked text, L MIM is the second loss function corresponding to the masked image, L WPA is the third loss function corresponding to the alignment information.

[0028] Preferably, the first loss function is:

[0029]

[0030] Among them, M′ is the total position of the masked image blocks in the masked image, X M′ is the masked image, L′ is the total position of the masked text words in the masked text, and Y L′ is the masked text, yl is the masked text word at the lth position, θ is the model parameter of the multimodal encoder, and p θ is the predicted probability of the encoder module.

[0031] Preferably, the second loss function is:

[0032]

[0033] Among them, M′ is the total position of the masked image blocks in the masked image, X M′ is the masked image, L′ is the total position of the masked text words in the masked text, and Y L′ is the masked text, x m is the masked image block at the mth position, θ is the model parameter of the multimodal encoder, and p θ is the predicted probability of the encoder module.

[0034] Preferably, the third loss function is:

[0035]

[0036] Among them, LL′ is the total position of the unmasked text words in the masked text, X M′ is the masked image, Y L′ is the masked text, z l is the binary label corresponding to the unmasked text word at the lth position, θ is the model parameter of the multimodal encoder, and p θ is the predicted probability of the encoder module.

[0037] Preferably, the local attention layer guided by the spatial prefix is:

[0038]

[0039] in, is the output of the local attention layer guided by the spatial prefix, λ(R (t) ) is the weight coefficient, Attn(·) is the attention function, is the query vector of the tth iteration in the standard local attention layer, is the key vector for the tth iteration in the standard local attention layer, is the value vector of the tth iteration in the standard local attention layer, S kis the key vector of the t-th iteration in the spatial prefix guided local attention layer, S v is the value vector for the tth iteration in the spatial prefix guided local attention layer.

[0040] Preferably, the global interaction layer is:

[0041]

[0042] in, is the output of the global interaction layer, Attn(·) is the attention function, is the query vector of the tth iteration in the standard local attention layer, is the key vector of the tth iteration in the global interaction layer, is the value vector of the tth iteration in the global interaction layer.

[0043] On the other hand, an embodiment of the present specification provides a device for interpreting an image document, the device comprising:

[0044] A receiving and preprocessing module receives and preprocesses image documents to obtain preprocessed text, images and layouts;

[0045] Multimodal encoder, which inputs the preprocessed text, image and layout into the trained multimodal encoder and outputs the multimodal features in the image document;

[0046] An input module, inputting the multimodal features into a modality alignment module;

[0047] A modality alignment module, wherein the marking submodule in the modality alignment module divides the multimodal features to obtain keys and values ​​in the image document;

[0048] Matching module, the connection submodule in the modality alignment module uses the local attention layer and the global interaction layer guided by the spatial prefix to obtain the matching relationship between the key and the value;

[0049] The large language model backbone inputs the matching relationship into the large language model backbone to interpret the image document.

[0050] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of any one of the above methods are executed.

[0051] On the other hand, an embodiment of the present specification further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of a computer device, the computer program executes instructions of any one of the above-described methods.

[0052] On the other hand, the embodiments of this specification further provide a computer program product, which, when executed by a processor of a computer device, executes instructions according to any one of the methods described above.

[0053] It can be seen from the technical solutions provided by the above embodiments of this specification that, through the method of the embodiments of this specification, in the process of processing image documents, first, the document is received and preprocessed to extract text, image and layout information. Then, these preprocessed data are input into the trained multimodal encoder to generate multimodal features of the image document. These features are then passed to the modal alignment module, in which the marking submodule classifies the features and identifies the keys and values ​​in the document, that is, the structured information and actual data in the document.

[0054] In the modality alignment module, the connection submodule analyzes and determines the matching relationship between keys and values ​​through the local attention layer and the global interaction layer guided by the spatial prefix. This method uses the spatial and contextual information of text and images to accurately pair related content. Finally, these matching relationships are input into the large language model backbone for in-depth interpretation, thereby generating a comprehensive understanding and analysis of image documents.

[0055] This processing flow ensures the accuracy of information extraction and interpretation from image documents, and can effectively identify the structure and content in the document. Through the joint application of multimodal encoders and large language models, complex documents can be processed, and efficient information extraction and analysis can be achieved, which is suitable for various practical application scenarios.

[0056] In order to make the above and other purposes, features and advantages of the present specification more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A schematic diagram showing a flow chart of a method for interpreting an image document provided in an embodiment of this specification;

[0059] Figure 2 A schematic diagram of a process of receiving and preprocessing an image document to obtain preprocessed text, image and format provided in an embodiment of this specification is shown;

[0060] Figure 3A schematic diagram of a flow chart of a training method for a multimodal encoder provided in an embodiment of this specification is shown;

[0061] Figure 4 A schematic diagram of the module structure of an image document interpretation device provided in an embodiment of this specification is shown;

[0062] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of this specification is shown.

[0063] Description of the accompanying symbols:

[0064] 100. Receiving and preprocessing module;

[0065] 200, multimodal encoder;

[0066] 300, input module;

[0067] 400. Modal alignment module;

[0068] 500, matching module;

[0069] 600, Large Language Model Backbone;

[0070] 502. Computer equipment;

[0071] 504, processor;

[0072] 506. Memory;

[0073] 508, driving mechanism;

[0074] 510, input / output module;

[0075] 512. Input devices;

[0076] 514. Output device;

[0077] 516. Presentation equipment;

[0078] 518. Graphical user interface;

[0079] 520, network interface;

[0080] 522, communication link;

[0081] 524. Communication bus. DETAILED DESCRIPTION

[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0083] This application provides users with corresponding big data analysis (such as personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) operation entrances for users to choose to agree or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0084] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of this specification.

[0085] The purpose of interpreting image documents is to extract and understand information from documents containing images. Since image documents contain multimodal features such as images, texts, and tables, most existing technologies interpret image documents through large multimodal models. However, when processing image-dense documents, the computational complexity of the existing large multimodal models increases quadratically with the increase in image resolution in dense documents. In order to prevent high computational costs caused by complexity, the existing large multimodal models usually reduce the resolution of the input image, which ultimately results in the inability of the existing large multimodal models to accurately interpret image-dense documents.

[0086] In order to solve the above problems, an embodiment of the present specification provides a method for interpreting an image document. Figure 1 This is a flowchart of a method for interpreting an image document provided in an embodiment of this specification. This specification provides method operation steps as described in the embodiment or flowchart, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system or device product is executed, it can be executed in the order of the method shown in the embodiment or the drawings or in parallel.

[0087] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of this specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0088] Reference Figure 1 , the embodiment of this specification provides a method for interpreting an image document, including:

[0089] S101: receiving and preprocessing an image document to obtain preprocessed text, image and layout;

[0090] S102: input the preprocessed text, image and layout into the trained multimodal encoder, and output the multimodal features in the image document;

[0091] S103: Inputting the multimodal features into a modality alignment module;

[0092] S104: The marking submodule in the modality alignment module divides the multimodal features to obtain keys and values ​​in the image document;

[0093] S105: The connection submodule in the modality alignment module obtains the matching relationship between the key and the value using the local attention layer and the global interaction layer guided by the spatial prefix;

[0094] S106: Input the matching relationship into the large language model backbone to interpret the image document.

[0095] Image documents refer to documents that exist in the form of images, such as scanned PDF, Word or picture documents. The image documents need to be preprocessed to obtain preprocessed text, images and layouts, and input into the trained multimodal encoder. The multimodal encoder can convert information of different modalities (text, image and layout) into a feature vector in a common embedding space to capture the correlation and interaction between the modalities.

[0096] The multimodal features output by the multimodal encoder refer to information representations extracted and fused from multiple different modalities (text, image, and layout). Features of different modalities can provide complementary information. In order to effectively fuse these different modal information, they are usually mapped to a common embedding space, that is, text, image, and layout are fused together to form a unified feature representation for a more comprehensive interpretation.

[0097] The multimodal features are further input into the modal alignment module, which is responsible for processing the relationship between different modal features and aligning the features of different modalities (text, image, and layout) to ensure that they can be associated and understood in a unified document. In general, a key usually represents a query or identifier for a specific modal feature. For example, in a text feature, a key can be a word or phrase; in an image feature, a key can be a feature vector of an image region. The value is the feature or information associated with the key. For example, in a text feature, a value can be the specific content or embedded vector corresponding to the key; in an image feature, a value can be the detailed features of the region indicated by the key. Through the modal alignment module, the matching of key-value pairs enables the information from different modalities to be effectively aligned and fused, thereby improving the ability to understand and analyze multimodal information.

[0098] Finally, the obtained matching relationship is input into the backbone of the large language model. Based on this, the backbone of the large language model can interpret the image document. For example, some information that needs attention can be obtained by interpreting the image document. Assuming that the image document is a PDF of a shopping invoice, the final interpretation is: invoice date, name of the purchaser, name of the seller, tax number and other information that needs attention.

[0099] In the embodiments of this specification, refer to Figure 2 The receiving and preprocessing of the image document to obtain the preprocessed text, image and layout further includes:

[0100] S201: receiving an image document;

[0101] S202: extracting the content and two-dimensional position of the text in the image document using an OCR toolkit;

[0102] S203: Based on the pre-trained model, initialize word embedding according to the content and two-dimensional position of the text to obtain an embedding vector of the text word;

[0103] S204: adjusting the image document to a set size to obtain an adjusted image;

[0104] S205: After dividing the adjusted image into P×P image blocks, flattening them into embedding vectors of the image blocks;

[0105] S206: Convert the one-dimensional index position and two-dimensional border coordinates of the text in the image document and the one-dimensional index position and two-dimensional border coordinates of the image into a layout embedding vector.

[0106] During the extraction process, the OCR toolkit generally extracts the content of each word in the text and the two-dimensional position coordinates of the word in the image document, and then initializes the word embedding based on a pre-trained model (such as Word2Vec or BERT) to obtain the embedding vector of the text word. The embedding vector of the text word contains the embedding vector of the text word content and the embedding vector of the text word two-dimensional position, which is used to represent the content and structure of the text in the image document.

[0107] After the image document is adjusted to a set size, the size is set to C×H×W, where C, H and W are the channel, width and height of the image respectively. A unified image segmentation can be performed, where the value of P in the segmentation into P×P image blocks can be set according to actual conditions. Finally, the image blocks are converted into embedded vectors of the image blocks to characterize the content and structure of the image document.

[0108] The one-dimensional index position refers to the position number or index of the text or image, and the two-dimensional border coordinates refer to the border position coordinates of the text or image. The one-dimensional index position and the two-dimensional border coordinates are mapped to a vector space and converted into a layout embedding vector to represent the spatial layout of the image document.

[0109] The preprocessed text, image and layout obtained are the embedding vectors of text words, the embedding vectors of image blocks and the embedding vectors of layouts.

[0110] Among them, refer to Figure 3 , the trained multimodal encoder is trained by the following method:

[0111] S301: receiving and preprocessing a known image document to obtain preprocessed known text, known image and known format;

[0112] S302: Masking a first set ratio of text words in the preprocessed known text to obtain a masked text;

[0113] S303: Masking the image blocks of the second set ratio in the preprocessed known image to obtain a masked image;

[0114] S304: performing alignment processing on the masked text and the masked image to obtain alignment information between the masked text and the masked image;

[0115] S305: Input the masked text, the masked image and the alignment information into a multimodal encoder, and train the multimodal encoder using a target loss function.

[0116] The preprocessing process of the known image document is similar to the above steps S201-S206, and this manual will not go into details about it. The preprocessed known text, known image and known layout can be obtained, that is, the embedding vector of the known text word, the embedding vector of the known image block and the known layout embedding vector.

[0117] Furthermore, randomly mask the text words of the first set ratio in the preprocessed known text to obtain partially masked text words and remaining unmasked text words, and mask the image blocks of the second set ratio in the preprocessed known image to obtain partially masked image blocks and remaining unmasked image blocks, wherein the first set ratio and the second set ratio can be determined according to actual conditions, and generally take any value between 30% and 40%. Then align the masked text with the masked image, and the alignment information includes the alignment relationship between the text words and the image blocks, for example, the alignment relationship between text word A and image block 1, and the alignment relationship between text word B and image block 3.

[0118] For a known image document, the masked text words are known, the masked image blocks are known, and the alignment relationship between the text words and the image blocks is known. The multimodal encoder is trained using the known image document.

[0119] The objective loss function during training is:

[0120] L=K MLM +L MIM +L WPA ;

[0121] Among them, L is the target loss function, L MLM is the first loss function corresponding to the masked text, L MIM is the second loss function corresponding to the masked image, L WPA is the third loss function corresponding to the alignment information.

[0122] The masked text words and the masked image blocks are used to restore the masked text words, wherein the first loss function is:

[0123]

[0124] Among them, M′ is the total position of the masked image blocks in the masked image, X M′ is the masked image, L′ is the total position of the masked text words in the masked text, and Y L′ is the masked text, y l is the masked text word at the lth position, θ is the model parameter of the multimodal encoder, and p θ is the predicted probability of the encoder module.

[0125] The masked image block is restored by the masked text word and the masked image block, wherein the second loss function is:

[0126]

[0127] Among them, M′ is the total position of the masked image blocks in the masked image, X M′ is the masked image, L′ is the total position of the masked text words in the masked text, and Y L′ is the masked text, x m is the masked image block at the mth position, θ is the model parameter of the multimodal encoder, and p θ is the predicted probability of the encoder module.

[0128] The third loss function is:

[0129]

[0130] Among them, LL′ is the total position of the unmasked text words in the masked text, X M′ is the masked image, Y L′ is the masked text, z l is the binary label corresponding to the unmasked text word at the lth position, θ is the model parameter of the multimodal encoder, and p θ is the predicted probability of the encoder module. The binary label is used to represent whether the unmasked text word at the lth position is aligned with an unmasked image block or not. For example, if they are aligned, the binary label is 1, and if they are not aligned, the binary label is 0.

[0131] The masked text contains all masked text words and unmasked text words, and the masked image contains all masked image blocks and unmasked image blocks. Since the first loss function, the second loss function and the third loss function need to be added to obtain the target loss function, the three share model parameters, which generally include weights and biases.

[0132] As the model training is continuously updated, the model parameters are adjusted to minimize the loss function, thereby improving the model's predictive ability. During each training, the prediction probability is determined after the model parameters are determined.

[0133] In the embodiment of this specification, the marking submodule can determine the category of the image document entity, and divide the document entity into four categories: header, key, value, and other. The document entity helps define and classify different parts in the document, making the parsing and processing of the document more systematic and efficient. By identifying and marking these entities, the document content can be better organized. Among them, the title is the title part of the document as the overall structure, which is usually used to indicate the subject or paragraph title of the document or part of the content. The key refers to the document used to describe or identify a specific information, usually in pairs with the value. The value is the specific information corresponding to the key in the document, providing the information content described by the key. The division of keys and values ​​helps the model understand which texts are used to describe information (keys) and which texts or images provide actual information (values), such as keys: "name", "age", values: "Zhang X", "25 years old". Others are the remaining parts except the above three.

[0134] Furthermore, the connection submodule in the modality alignment module utilizes the local attention layer and the global interaction layer guided by the spatial prefix to obtain the matching relationship between keys and values.

[0135] The spatial prefix guided local attention layer is:

[0136]

[0137] in, is the output of the local attention layer guided by the spatial prefix, λ(R (t) ) is the weight coefficient, Attn(·) is the attention function, is the query vector of the tth iteration in the standard local attention layer, is the key vector for the tth iteration in the standard local attention layer, is the value vector of the tth iteration in the standard local attention layer, S k is the key vector of the t-th iteration in the spatial prefix guided local attention layer, S v is the value vector for the tth iteration in the spatial prefix guided local attention layer.

[0138] In the embodiments of this specification,

[0139] S k =W S_key g i,j +b S_key ;

[0140]

[0141] Among them, W S_key ,W S_value are transformation matrices, bS_key ,b S_value are bias matrices, g i,j =[w θ θ i,j ;w d d i,j ],W θ and W d are transformation matrices, (x i ,y i ) and (x j ,y j ) are the two-dimensional position coordinates of the key and value respectively.

[0142] Specifically, W S_key Used to transform the input of the spatial prefix guided local attention layer into S k , b S_key To adjust S k The output of W is used to adapt to more feature changes. S_value Used to transform the input of the local attention layer guided by the spatial prefix into S v , b S_value To adjust S v The output of W is used to adapt to more feature changes. θ To convert θ i,j The calculation results are transformed into a new space, W d Used to i,j The calculation results are transformed into another new space.

[0143] The weight coefficient is a coefficient used to adjust the ratio of two different attention calculation methods, which can be set according to actual conditions. λ is used to balance the standard local attention calculation with the attention calculation guided by the spatial prefix. Its value is between 0 and 1. When λ = 0, it completely relies on the standard attention calculation. When λ = 1, it completely relies on the attention calculation guided by the spatial prefix. Between 0 and 1, it represents a weighted combination of the two. At the tth layer, R(t) can be regarded as the feature vector processed by the local attention layer. It is the intermediate result of the standard local attention and the spatial prefix guided attention, and will participate in the calculation of the weight coefficient.

[0144] The global interaction layer is:

[0145]

[0146] in, is the output of the global interaction layer, Attn(·) is the attention function, is the query vector of the tth iteration in the standard local attention layer, is the key vector of the tth iteration in the global interaction layer, is the value vector of the tth iteration in the global interaction layer.

[0147] In the embodiments of this specification,

[0148]

[0149] in, is the query vector of the tth iteration in the global interaction layer, W τ_key , W τ_value are transformation matrices, b τ_key , b τ_value are bias matrices, is the key vector for the tth iteration in the standard local attention layer, is the value vector of the t-th iteration in the standard local attention layer.

[0150] Specifically, W τ_key Used to Transformed to b τ_key Used to adjust the transformed results to adapt to more feature changes, W τ_value Used to Transformed to b τ_value Used to adjust the transformed results to adapt to more feature changes.

[0151] Finally passed Get the matching relationship of key-value pairs.

[0152] Generally speaking, the attention mechanism enables the model to dynamically focus on different parts of the input vector when processing the current input. The attention mechanism usually involves three key components, namely the query vector: the input vector currently being processed, or the request used by the model to find and match relevant information. Key vector: contains all possible features, which are the objects used by the model to compare with the query vector. Each key vector represents a potential key in the document (such as "name", "date"). Value vector: This is the feature vector corresponding to the key vector, which contains the specific data associated with each key. Each value vector represents the actual information associated with a key (such as "Zhang Mou", "September 5, 2024").

[0153] By the method of the embodiment of this specification, in the process of processing image documents, first, the document is received and preprocessed to extract text, image and layout information. Then, these preprocessed data are input into the trained multimodal encoder to generate multimodal features of the image document. These features are then passed to the modal alignment module, in which the marking submodule classifies the features and identifies the keys and values ​​in the document, that is, the structured information and actual data in the document.

[0154] In the modality alignment module, the connection submodule analyzes and determines the matching relationship between keys and values ​​through the local attention layer and the global interaction layer guided by the spatial prefix. This method uses the spatial and contextual information of text and images to accurately pair related content. Finally, these matching relationships are input into the large language model backbone for in-depth interpretation, thereby generating a comprehensive understanding and analysis of image documents.

[0155] This processing flow ensures the accuracy of information extraction and interpretation from image documents, and can effectively identify the structure and content in the document. Through the joint application of multimodal encoders and large language models, complex documents can be processed, and efficient information extraction and analysis can be achieved, which is suitable for various practical application scenarios.

[0156] Based on the above-mentioned method for interpreting an image document, an embodiment of this specification also provides a corresponding device for interpreting an image document. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiment of this specification and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0157] Specifically, Figure 4 This is a schematic diagram of the module structure of an embodiment of an image document interpretation device provided in the embodiment of this specification, referring to Figure 4 As shown, an image document interpretation device provided by an embodiment of the present specification includes: a receiving and preprocessing module 100, a multimodal encoder 200, an input module 300, a modality alignment module 400, a matching module 500, and a large language model backbone 600.

[0158] A receiving and preprocessing module 100 receives and preprocesses an image document to obtain preprocessed text, image and format;

[0159] The multimodal encoder 200 inputs the preprocessed text, image and layout into the trained multimodal encoder and outputs the multimodal features in the image document;

[0160] An input module 300 inputs the multimodal features into a modality alignment module;

[0161] A modality alignment module 400, a marking submodule in the modality alignment module divides the multimodal features to obtain keys and values ​​in the image document;

[0162] The matching module 500, the connection submodule in the modality alignment module uses the local attention layer and the global interaction layer guided by the spatial prefix to obtain the matching relationship between the key and the value;

[0163] The large language model backbone 600 inputs the matching relationship into the large language model backbone to interpret the image document.

[0164] Reference Figure 5 As shown, based on the above-mentioned method for interpreting an image document, a computer device 502 is also provided in an embodiment of the present specification, wherein the above-mentioned method is run on the computer device 502. The computer device 502 may include one or more processors 504, such as one or more central processing units (CPUs) or graphics processing units (GPUs), and each processing unit may implement one or more hardware threads. The computer device 502 may also include any memory 506, which is used to store any kind of information such as code, settings, data, etc. In a specific implementation, a computer program on the memory 506 and executable on the processor 504, when the computer program is executed by the processor 504, can execute instructions according to the above-mentioned method. Non-limitingly, for example, the memory 506 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 502. In one embodiment, when the processor 504 executes the associated instructions stored in any memory or combination of memories, the computer device 502 can perform any operation of the associated instructions. The computer device 502 also includes one or more drive mechanisms 508 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0165] The computer device 502 may also include an input / output module 510 (I / O) for receiving various inputs (via input devices 512) and for providing various outputs (via output devices 514). A specific output mechanism may include a presentation device 516 and an associated graphical user interface 518 (GUI). In other embodiments, the input / output module 510 (I / O), input device 512, and output device 514 may not be included, and the computer device 502 may be used as a computer device in a network. The computer device 502 may also include one or more network interfaces 520 for exchanging data with other devices via one or more communication links 522. One or more communication buses 524 couple the components described above together.

[0166] The communication link 522 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 522 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0167] Corresponds to Figure 1-Figure 3 The method in the embodiment of the present specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0168] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 3 The method shown.

[0169] The present specification also provides a computer program product. When the computer program product is executed by a processor of a computer device, the computer program product performs the following steps: Figures 1 to 3 The method shown.

[0170] The computer program product described in this specification is a software product that mainly implements the method described in this specification through a computer program.

[0171] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0172] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the embodiments of this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0173] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of this specification can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of this specification.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0175] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0177] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0179] Specific embodiments are used in this specification to illustrate the principles and implementation methods of the embodiments of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of this specification. At the same time, for those skilled in the art, according to the ideas of the embodiments of this specification, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of this specification.

Claims

1. A method for interpreting an image document, characterized in that: include: Receiving and preprocessing image documents to obtain preprocessed text, images and layouts; Input the preprocessed text, image and layout into the trained multimodal encoder and output the multimodal features in the image document; Inputting the multimodal features into a modality alignment module; The marking submodule in the modality alignment module divides the multimodal features to obtain keys and values ​​in the image document; The connection submodule in the modality alignment module uses a local attention layer and a global interaction layer guided by a spatial prefix to obtain a matching relationship between the key and the value; The matching relationship is input into the large language model backbone to interpret the image document.

2. The method according to claim 1, characterized in that The receiving and preprocessing of the image document to obtain the preprocessed text, image and format further comprises: receiving image documents; Use the OCR toolkit to extract the content and two-dimensional position of text in image documents; Based on the pre-trained model, word embedding is initialized according to the content and two-dimensional position of the text to obtain an embedding vector of the text word; Adjust the image document to a set size to obtain an adjusted image; After dividing the adjusted image into P×P image blocks, flatten it into the embedding vector of the image block; According to the one-dimensional index position and two-dimensional border coordinates of the text in the image document, as well as the one-dimensional index position and two-dimensional border coordinates of the image, they are converted into a layout embedding vector.

3. The method according to claim 1, characterized in that The trained multimodal encoder is trained by the following method: Receiving and preprocessing a known image document to obtain preprocessed known text, known image and known format; Masking a first set proportion of text words in the preprocessed known text to obtain a masked text; Masking an image block of a second set ratio in the preprocessed known image to obtain a masked image; Aligning the masked text with the masked image to obtain alignment information between the masked text and the masked image; The masked text, the masked image and the alignment information are input into a multimodal encoder, and the multimodal encoder is trained using a target loss function.

4. The method according to claim 3, characterized in that The objective loss function is: L=L MLM +L MIM +L WPA ; Among them, L is the target loss function, L MLM is the first loss function corresponding to the masked text, L MIM is the second loss function corresponding to the masked image, L WPA is the third loss function corresponding to the alignment information.

5. The method according to claim 4, characterized in that The first loss function is: Among them, M′ is the total position of the masked image blocks in the masked image, X M′ is the masked image, L′ is the total position of the masked text words in the masked text, and Y L′ is the masked text, y l is the masked text word at the lth position, θ is the model parameter of the multimodal encoder, and p θ is the predicted probability of the encoder module.

6. The method according to claim 4, characterized in that The second loss function is: Among them, M′ is the total position of the masked image blocks in the masked image, X M′ is the masked image, L′ is the total position of the masked text words in the masked text, and Y L′ is the masked text, x m is the masked image block at the mth position, θ is the model parameter of the multimodal encoder, and p θ is the predicted probability of the encoder module.

7. The method according to claim 4, characterized in that The third loss function is: Among them, LL′ is the total position of the unmasked text words in the masked text, X M′ is the masked image, Y L′ is the masked text, z l is the binary label corresponding to the unmasked text word at the lth position, θ is the model parameter of the multimodal encoder, and p θ is the predicted probability of the encoder module.

8. The method according to claim 1, characterized in that The spatial prefix guided local attention layer is: in, is the output of the local attention layer guided by the spatial prefix, λ(R (t) ) is the weight coefficient, Attn(·) is the attention function, is the query vector of the tth iteration in the standard local attention layer, is the key vector for the tth iteration in the standard local attention layer, is the value vector of the tth iteration in the standard local attention layer, S k is the key vector of the t-th iteration in the spatial prefix guided local attention layer, S v is the value vector for the tth iteration in the spatial prefix guided local attention layer.

9. The method according to claim 1, characterized in that: The global interaction layer is: in, is the output of the global interaction layer, Attn(·) is the attention function, is the query vector of the tth iteration in the standard local attention layer, is the key vector of the tth iteration in the global interaction layer, is the value vector of the tth iteration in the global interaction layer.

10. An image document interpretation device, characterized in that: The device comprises: A receiving and preprocessing module receives and preprocesses image documents to obtain preprocessed text, images and layouts; Multimodal encoder, which inputs the preprocessed text, image and layout into the trained multimodal encoder and outputs the multimodal features in the image document; An input module, inputting the multimodal features into a modality alignment module; A modality alignment module, wherein the marking submodule in the modality alignment module divides the multimodal features to obtain keys and values ​​in the image document; Matching module, the connection submodule in the modality alignment module uses the local attention layer and the global interaction layer guided by the spatial prefix to obtain the matching relationship between the key and the value; The large language model backbone inputs the matching relationship into the large language model backbone to interpret the image document.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the computer program is executed by the processor, the computer program executes the instructions of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the computer program executes the instructions of the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that When the computer program product is executed by a processor of a computer device, the computer program product executes the instructions of the method according to any one of claims 1 to 9.

Images