Audit information extraction method and device based on multi-modal fusion and electronic equipment

Abstract

The application provides an audit information extraction method and device based on multi-modal fusion and electronic equipment, which can be applied to the technical field of document processing. The method comprises the following steps: in response to receiving an extraction request for audit information, obtaining a field to be queried; according to the modal type of the data in the audit information, calling an encoder corresponding to the modal type to encode the audit information to generate feature vectors of at least two modes; taking the vector obtained by encoding the field to be queried as an initial query vector, calling a cross-attention module in layers, and respectively interacting and fusing with the feature vectors of the at least two modes to realize feature interaction and alignment between modes and within modes, and outputting target modal feature vectors matched with the field to be queried; and decoding the target modal feature vectors to extract target fields matched with the field to be queried from the audit information.

Description

Technical Field

[0001] This invention relates to the field of document processing technology, and more specifically, to a method, apparatus, and electronic device for extracting audit information based on multimodal fusion. Background Technology

[0002] In auditing, it is necessary to identify key modules and extract core field information from a large number of audit documents for audit verification. However, current technologies for extracting key field information from audit documents often struggle to accommodate multimodal information, resulting in low efficiency in audit information extraction. Summary of the Invention

[0003] In view of this, the present invention provides a method, apparatus and electronic device for extracting audit information based on multimodal fusion.

[0004] One aspect of the present invention provides an audit information extraction method based on multimodal fusion, comprising: in response to receiving an extraction request for audit information, obtaining a field to be queried; wherein the audit information includes data of at least two modalities: text, image, and table; according to the modal type of the data in the audit information, calling an encoder corresponding to the modal type to encode the audit information, generating feature vectors of at least two modalities; wherein the feature vectors include at least two items: text feature vector, image feature vector, and table feature vector; the table feature vector is used to characterize the absolute position feature of the table and the relative position feature of each cell in the table relative to the table; using the vector obtained by encoding the field to be queried as the initial query vector, calling a stacked cross-attention module to interact and fuse with the feature vectors of the at least two modalities respectively, so as to realize feature interaction and alignment between and within modalities, and outputting a target modal feature vector that matches the field to be queried; decoding the target modal feature vector to extract the target field that matches the field to be queried from the audit information.

[0005] According to an embodiment of the present invention, the aforementioned cross-attention module includes M layers, where M is an integer greater than 1. The above-mentioned method uses the vector obtained by encoding the query field as the initial query vector, calls the stacked cross-attention modules, and performs interactive fusion with the feature vectors of at least two modalities respectively to achieve feature interaction and alignment between and within modalities, outputting a target modal feature vector matching the query field. This includes: linearly mapping the feature vectors of the at least two modalities to obtain key-value pairs of the first and second modalities corresponding to the feature vectors of the at least two modalities respectively; using the fused feature vector output by the (m-1)th layer as the query vector, performing dot product attention calculations with the key-value pairs of the first and second modalities respectively to achieve interaction and alignment within modalities, generating aggregated features of the first and second modalities; wherein, the fused feature vector output by the (m-1)th layer... The feature vector is obtained by processing the initial query vector in the first m-1 layers; the fusion feature vector output by the (m-1)th layer aggregates the local features of each modality in the audit information; m is greater than 1 and less than or equal to M; the aggregated features of the first modality and the aggregated features of the second modality are weighted and fused to achieve interaction and alignment between modalities, generating the fusion feature vector output by the mth layer; in response to determining that m is less than M, dot product attention calculation and weighted fusion operation are performed on the fusion feature vector output by the mth layer, and m is incremented; in response to determining that m is equal to M, the query field is repeatedly interacted with the feature vectors of at least two modalities at different scales to generate the target fusion feature vector; wherein, the target fusion feature vector fuses the local features and global semantic features in the audit information.

[0006] According to an embodiment of the present invention, the above-mentioned method of encoding the audit information by calling an encoder corresponding to the modality type of the data in the audit information to generate feature vectors of at least two modalities includes: in response to determining that the modality type is a table, reconstructing the cell boundaries of the table modality data in the audit information to generate reconstructed table modality data; and calling a table encoder to generate table modality feature vectors by capturing the row and column relationships and cell merging information in the table modality data, using a predetermined cell adjacency matrix and geometric position encoding as bias terms.

[0007] According to an embodiment of the present invention, in response to determining that the modality type is a table, the above-mentioned cell boundary reconstruction of the table modality data in the audit information to generate reconstructed table modality data includes: in response to determining that the modality type is a table, performing text position detection on the table modality data to obtain the position coordinates of each text box; clustering the ordinates of the position coordinates of each text box and a predetermined ordinate threshold to obtain row boundary lines; clustering the abscissas of the position coordinates of each text box and a predetermined abscissa threshold to obtain column boundary lines; creating an initial network based on the row boundary lines and the column boundary lines; wherein the height of each row in the initial network is the same and the width of each column is the same; and filling the content of each text box into the initial network to obtain the reconstructed table modality data.

[0008] According to an embodiment of the present invention, the above-mentioned method of filling the content of each text box into the initial grid to obtain the reconstructed table modal data further includes: in response to determining that the width of any text box is greater than the width of a column in the initial grid and that the horizontal center of any text box overlaps with multiple columns, determining that any text box is a cross-column cell; in response to determining that the height of any text box is greater than the height of a row in the initial grid and that the vertical center of any text box overlaps with multiple rows, determining that any text box is a cross-row cell; and performing local correction on the initial grid based on the cross-column cell and the cross-row cell, and filling the content of each text box into the corrected grid to obtain the reconstructed table modal data.

[0009] According to embodiments of the present invention, the multimodal fusion model includes a text encoder, an image encoder, a table encoder, stacked cross-attention modules, and a decoder. The method further includes: acquiring sample text, a sample image, and a sample table; wherein the sample text is obtained by performing text recognition on the sample image; the sample image is obtained by image acquisition of the sample table; masking the sample text, the sample image, and the sample table respectively to generate masked sample text, a masked mask image, and a masked sample table; calling the text encoder to encode the masked sample text to generate a sample text feature vector; calling the image encoder to encode the masked mask image to generate a sample image feature vector; and calling the table encoder to encode the masked sample table to generate sample table features. Vectors; wherein, the sample table feature vector is used to characterize the absolute position features of the sample table and the relative position features of each cell in the sample table relative to the sample table; the stacked cross-attention module is invoked to interactively fuse the sample text feature vector, sample image feature vector and sample table feature vector to achieve feature interaction and alignment between and within modalities, and output the expected text, expected image and expected table; based on the loss between the sample text and the expected text, the loss between the sample image and the expected image and the loss between the sample table and the expected table, the model parameters of the stacked cross-attention module are adjusted to obtain the trained multimodal fusion model.

[0010] According to an embodiment of the present invention, the method further includes: using the vector obtained by encoding the sample query field as the initial sample query vector, calling the stacked cross-attention module to interact and fuse with the sample text feature vector, sample image feature vector and sample table feature vector respectively, so as to realize feature interaction and alignment between and within modalities, and outputting a target sample modal feature vector that matches the sample query field; calling the decoder to decode the target sample modal feature vector, and outputting a target sample field that matches the sample query field; extracting field labels from the sample text based on the sample query field; and adjusting the model parameters of the stacked cross-attention module based on the loss between the target sample field and the field labels to obtain the trained multimodal fusion model.

[0011] According to an embodiment of the present invention, the method further includes: performing the above method using at least two multimodal fusion models with the same structure, each outputting a first target sample field and a second target sample field that match the above sample query field; calculating the inter-distillation loss between the at least two multimodal fusion models based on the label loss between the first target sample field and the field label and the label loss between the second target sample field; and simultaneously optimizing the model parameters of the stacked cross-attention modules of the at least two multimodal fusion models using the inter-distillation loss and the label loss of each of the at least two multimodal fusion models, so as to determine the trained multimodal fusion model from the at least two multimodal fusion models.

[0012] Another aspect of the present invention provides an audit information extraction device based on multimodal fusion, comprising: an acquisition module, configured to acquire a field to be queried in response to receiving an extraction request for audit information; wherein the audit information includes data of at least two modalities: text, image, and table; an encoding module, configured to encode the audit information according to the modal type of the data in the audit information by calling an encoder corresponding to the modal type, generating feature vectors of at least two modalities; wherein the feature vectors include at least two items: text feature vector, image feature vector, and table feature vector; the table feature vector is used to characterize the absolute position feature of the table and the relative position feature of each cell in the table relative to the table; a fusion module, configured to use the vector obtained by encoding the field to be queried as the initial query vector, call a stacked cross-attention module to interact and fuse with the feature vectors of the at least two modalities respectively, so as to realize feature interaction and alignment between and within modalities, and output a target modal feature vector matching the field to be queried; and a decoding module, configured to decode the target modal feature vector and extract the target field matching the field to be queried from the audit information.

[0013] Another aspect of the present invention provides an electronic device comprising:

[0014] One or more processors;

[0015] Memory, used to store one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0017] According to an embodiment of the present invention, in response to receiving a request to extract audit information, a field to be queried is obtained to determine the need for extracting audit information. Based on the modality type of the data in the audit information, an encoder corresponding to the modality type is invoked to encode the audit information, generating feature vectors for at least two modalities, providing a data foundation for multimodal feature interaction. Using the vector obtained from the encoded field to be queried as the initial query vector, a layered cross-attention module is invoked to interact and fuse with the feature vectors of at least two modalities respectively, to achieve feature interaction and alignment between and within modalities, outputting a target modality feature vector that matches the field to be queried, thereby extracting relevant features from the multimodal features and aligning them into a unified feature vector. The target modality feature vector is decoded to extract the target field that matches the field to be queried from the audit information, realizing the extraction of audit information under multimodal information fusion, and improving the efficiency of audit information extraction. Attached Figure Description

[0018] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which the accompanying drawings are shown.

[0019] Figure 1 The present invention illustrates an application scenario where an audit information extraction method based on multimodal fusion can be applied according to an embodiment of the present invention.

[0020] Figure 2 A flowchart of an audit information extraction method based on multimodal fusion according to an embodiment of the present invention is shown.

[0021] Figure 3 A schematic diagram of a multimodal fusion model predicting a target field according to an embodiment of the present invention is shown.

[0022] Figure 4 An architecture diagram of a multimodal fusion model according to an embodiment of the present invention is shown.

[0023] Figure 5 A schematic diagram of the preset query fields in the audit information extraction method based on multimodal fusion according to an embodiment of the present invention is shown.

[0024] Figure 6 A block diagram of an audit information extraction device based on multimodal fusion according to an embodiment of the present invention is shown.

[0025] Figure 7 A block diagram of an electronic device suitable for implementing a multimodal fusion-based audit information extraction method according to an embodiment of the present invention is shown. Detailed Implementation

[0026] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] Audit documents can be applied in various fields, including compliance reviews and contract performance. Audit documents can consist of multiple parts such as reports, contracts, supporting documents, and attachments, and are characterized by being multi-page, multimodal, and semi-structured. When conducting audits, it is necessary to accurately identify target fields from a large number of audit documents based on requests for audit information. For example, target fields matching query fields such as contract numbers, dates, responsible parties, or approval opinions can be extracted from audit information.

[0031] In related technologies, extracting target fields from audit documents is limited to single-page documents, lacking the ability to model the overall structure of cross-page documents and struggling to handle structures with strong contextual dependencies between pages. Directly extracting target fields from audit documents lacks document structure awareness, resulting in a high error rate. Extracting relevant information from audit documents using visual or textual modal models struggles to consider the multimodal relationships between image layout, text semantics, and positional structure, leading to insufficient modal information fusion. Audit information extraction lacks a mechanism for understanding the unique modular structure of audit documents, making it difficult to adapt to audit documents with varied formats and discrete field distributions. Therefore, in related technologies, extracting target fields from audit documents is inefficient.

[0032] In view of this, embodiments of the present invention provide a method, apparatus and electronic device for extracting audit information based on multimodal fusion.

[0033] Figure 1 This illustrates an application scenario where the multimodal fusion-based audit information extraction method according to embodiments of the present invention can be applied. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to embodiments of the present invention, in order to help those skilled in the art understand the technical content of the present invention, but do not mean that embodiments of the present invention cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0035] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the audit information extraction method based on multimodal fusion provided in this disclosure can generally be executed by server 105. The audit information extraction method based on multimodal fusion provided in this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the audit information extraction method based on multimodal fusion provided in this disclosure can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0039] For example, audit information may originally be stored in any one of the first terminal device 101, the second terminal device 102, or the third terminal device 103 (e.g., the first terminal device 101, but not limited thereto), or it may be stored on an external storage device and imported into the first terminal device 101. Then, the first terminal device 101 may locally execute the audit information extraction method based on multimodal fusion provided in the embodiments of this disclosure, or send the audit information to other terminal devices, servers, or server clusters, and have the other terminal devices, servers, or server clusters that receive the audit information execute the audit information extraction method based on multimodal fusion provided in the embodiments of this invention.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0041] Figure 2 A flowchart of an audit information extraction method based on multimodal fusion according to an embodiment of the present invention is shown.

[0042] like Figure 2 As shown, the audit information extraction method based on multimodal fusion may include operations S210 to S240.

[0043] In operation S210, in response to receiving a request to extract audit information, the field to be queried is obtained.

[0044] Audit information may include data in at least two of the following modalities: text, images, and tables.

[0045] Audit information can be data related to audit objectives such as compliance review and contract performance in the audit business scenario. Audit information can be obtained by extracting key fields from audit documents. The fields to be queried can be fields preset by the user or the system in the audit information extraction task, and can be used for audit verification. Key fields to be queried can include contract number, date, responsible party, etc.

[0046] In operation S220, based on the modality type of the data in the audit information, the encoder corresponding to the modality type is called to encode the audit information and generate feature vectors of at least two modalities.

[0047] The feature vector includes at least two of the following: text feature vector, image feature vector, and table feature vector; the table feature vector is used to characterize the absolute position features of the table and the relative position features of each cell in the table relative to the table.

[0048] The encoder can convert audit information such as text and table cells into feature vectors. Text feature vectors can represent the semantic features of text, such as the responsible person or location. Image feature vectors can represent visual features, such as seal information. Table feature vectors can represent cell-level table structure information, such as the coordinates of a cell in the document.

[0049] By calling the encoder corresponding to the modality type to encode the audit information, the information of each modality is fully extracted, thereby providing accurate data for subsequent extraction of target fields.

[0050] In operation S230, the vector obtained by encoding the field to be queried is used as the initial query vector. The stacked cross-attention module is called to interact and fuse with the feature vectors of at least two modalities respectively, so as to realize feature interaction and alignment between and within modalities, and output the target modal feature vector that matches the field to be queried.

[0051] The target modality feature vector is a vector from the audit information of different modalities that is related to the field to be queried. By calling the stacked cross-attention module, the initial query vector can be matched with the feature vectors of at least two corresponding modalities, thereby obtaining the target modality feature vector that matches the query field.

[0052] In operation S240, the target modal feature vector is decoded, and the target field that matches the field to be queried is extracted from the audit information.

[0053] The information represented by the target modality feature vector exists in vector form and cannot be used directly. Decoding the target modality feature vector can convert the vector data into specific information that can be used directly.

[0054] According to an embodiment of the present invention, in response to receiving a request to extract audit information, a field to be queried is obtained to determine the need for extracting audit information. Based on the modality type of the data in the audit information, an encoder corresponding to the modality type is invoked to encode the audit information, generating feature vectors for at least two modalities, providing a data foundation for multimodal feature interaction. Using the vector obtained from the encoded field to be queried as the initial query vector, a layered cross-attention module is invoked to interact and fuse with the feature vectors of at least two modalities respectively, to achieve feature interaction and alignment between and within modalities, outputting a target modality feature vector that matches the field to be queried, thereby extracting relevant features from the multimodal features and aligning them into a unified feature vector. The target modality feature vector is decoded to extract the target field that matches the field to be queried from the audit information, realizing the extraction of audit information under multimodal information fusion, and improving the efficiency of audit information extraction.

[0055] The training process of the multimodal fusion model is explained below.

[0056] Multimodal fusion models can include text encoders, image encoders, table encoders, stacked cross-attention modules, and decoders.

[0057] Training a multimodal fusion model can include operations A1 to A7.

[0058] In operation A1, obtain sample text, sample image, and sample table.

[0059] The sample text is obtained by performing text recognition on the sample image. The sample image is obtained by capturing images of the sample table.

[0060] Optical Character Recognition (OCR) methods can be used to perform text recognition on sample images, thereby generating structured text with location information.

[0061] In operation A2, the sample text, sample image, and sample table are masked to generate masked sample text, masked masked image, and masked sample table, respectively.

[0062] Masking sample text, sample images, and sample tables separately can annotate more effective information in each sample text, sample image, and sample table. This reduces invalid information in the masked sample text, sample image, and sample table, allowing the multimodal fusion model to fully learn the features of each modality while using the effective information.

[0063] The masked sample text can be represented by formula (1).

[0064] (1) Among them, This is the masked sample text. It is a word embedding linear mapping matrix. This indicates that the sample text will be displayed. Mapped to a sequence of word vectors. Represents the text position embedding matrix. This represents the position encoding function.

[0065] The masked sample image generated by masking the sample image can be represented by formula (2).

[0066] (2)

[0067] in, It is the masked sample image. It is a clear image after preprocessing. This represents a linear mapping that segments the image and projects it into the embedding space. Represents the normalized cell boundary vector The generated coordinate encoding, Indicates the visual position projection.

[0068] The masked sample table can be represented by formula (3).

[0069] (3)

[0070] in, This is the masked sample table. It is a projection of cell content. This indicates that the content of each cell in the embedded table is aggregated. Indicates the table's position projection; This represents the normalized cell boundary vector.

[0071] In operation A3, the text encoder is invoked to encode the masked sample text and generate the sample text feature vector.

[0072] A text encoder can encode masked sample text, extract semantic features of the sample text, and generate a sample text feature vector that contains text semantics and contextual logical relationships.

[0073] The generated sample text feature vector can be represented by formula (4).

[0074] (4)

[0075] in, It is the feature vector of the sample text. It is a text encoder.

[0076] In operation A4, the image encoder is invoked to encode the masked image and generate the sample image feature vector.

[0077] Image encoders can capture local visual details such as pixels and global features such as the row and column arrangement of tables in an image.

[0078] The generated sample image feature vector can be represented by formula (5).

[0079] (5)

[0080] in, It is the feature vector of the sample image. It is an image encoder.

[0081] In operation A5, the table encoder is called to encode the masked sample table and generate the sample table feature vector.

[0082] Among them, the sample table feature vector is used to characterize the absolute position features of the sample table and the relative position features of each cell in the sample table relative to the sample table.

[0083] The table encoder can introduce a structure-aware subnetwork to enhance structure recognition capabilities and solve the problem of blurred row and column boundaries in tables.

[0084] The feature vector of the generated sample table can be represented by formula (6).

[0085] (6)

[0086] in, It is the feature vector of the sample table. It is a table encoder.

[0087] In operation A6, the stacked cross-attention module is invoked to interactively fuse the feature vectors of sample text, sample image, and sample table, realizing feature interaction and alignment between and within modalities, and outputting the expected text, expected image, and expected table.

[0088] The stacked cross-attention module can perform multiple rounds of intramodal and intermodal feature interactions on sample text feature vectors, sample image feature vectors, and sample table feature vectors, thereby establishing deep feature fusion across multiple modalities.

[0089] In operation A7, the model parameters of the stacked cross-attention modules are adjusted based on the loss between sample text and expected text, the loss between sample image and expected image, and the loss between sample table and expected table to obtain a trained multimodal fusion model.

[0090] The total loss of the multimodal fusion model is constructed based on the loss between sample text and expected text, the loss between sample image and expected image, and the loss between sample table and expected table.

[0091] The total loss of the multimodal fusion model can be expressed by formula (7).

[0092] (7)

[0093] in. It is the total loss of the multimodal fusion model. It is the loss between the sample image and the expected image in the image modality. It is the loss between the sample text and the expected text. It is the loss between the sample table and the expected table. These are the weight hyperparameters for each modal loss. The weights of each modal loss can be set as learnable parameters, thereby improving training efficiency and robustness.

[0094] As an implementation method, the goal can also be to minimize the total loss of the multimodal fusion model, using an uncertainty-weighted approach to dynamically adjust the weights of each modality's loss. The total loss of the multimodal fusion model determined using the uncertainty-weighted approach can be expressed by formula (8).

[0095] (8)

[0096] in, It is the total loss of the multimodal fusion model; Let be the uncertainty variable learned through backpropagation; i is the modality number, which is 3 when the multimodalities are text, image, and table, respectively. Let represent the loss of the i-th mode.

[0097] By using uncertain variables to determine the weights of different modal losses, the text modal loss, image modal loss, and table modal loss can be automatically balanced, thereby reducing manual parameter tuning time and accelerating the convergence of multimodal fusion models.

[0098] According to embodiments of the present invention, a joint self-supervised training task is constructed by using a text encoder, an image encoder, and a table encoder, thereby improving the adaptability of the multimodal fusion model to different types of audit information. The loss of different modalities is dynamically adjusted using weight hyperparameters or uncertain variables to avoid weight imbalance in multi-task training, thus improving the model's training efficiency and adaptability to multimodal information.

[0099] The following explains how to train a multimodal fusion model by adjusting the model parameters of the stacked cross-attention modules.

[0100] Adjusting the model parameters of the stacked cross-attention modules can include operations B1 through B4.

[0101] In operation B1, the vector obtained by encoding the sample query field is used as the initial sample query vector. The stacked cross-attention module is called to interact and fuse with the sample text feature vector, sample image feature vector and sample table feature vector respectively, so as to realize feature interaction and alignment between and within modalities, and output the target sample modal feature vector that matches the sample query field.

[0102] The sample query fields can be key audit information fields preset during the training phase of the multimodal fusion model, such as the name of the unit, the name of the project, the contract number, the location information, or the seal information, which are key fields in the audit business scenario.

[0103] The initial sample query vector allows the stacked cross-attention modules to focus on information more relevant to the sample query fields, thereby improving the accuracy of field extraction.

[0104] The process of generating the initial sample query vector can be represented by formulas (9)-(10).

[0105] (9)

[0106] (10)

[0107] in, This is the initial sample query vector. It is the embedding of key fields. It is a word embedding linear mapping matrix. It is a key field. This indicates that the key field information will be in text form. Mapped to a sequence of word vectors.

[0108] By calling the stacked cross-attention module, the initial sample query vector can be interacted and fused with the sample text feature vector, sample image feature vector and sample table feature vector through multiple rounds of iterative interaction, thereby realizing feature interaction and alignment between and within modalities.

[0109] In operation B2, the decoder is invoked to decode the modal feature vector of the target sample and output the target sample field that matches the sample query field.

[0110] The decoder can transform the modal feature vector of the target sample into specific field information that can be directly applied in auditing business scenarios. By calculating the start and end points of the target sample fields, the probability distribution of the start and end points of the target sample fields that match the sample query fields can be obtained, thus revealing the target sample fields.

[0111] The matching process can be represented by formula (11)-formula (12).

[0112] (11)

[0113] (12)

[0114] in, It is the probability of the starting point of the target sample field. It is the probability of the target sample field endpoint. It is the first in the sequence and the The context representation of each position; It is the query vector of the Lth layer.

[0115] In operation B3, field labels are extracted from sample text based on sample query fields.

[0116] In operation B4, the model parameters of the stacked cross-attention modules are adjusted based on the loss between the target sample field and the field label to obtain the trained multimodal fusion model.

[0117] Field labels are the true labels corresponding to the sample query fields during the training of the multimodal fusion model, and can be used to measure the accuracy of the target sample fields.

[0118] The loss between the target sample field and the field label reflects the difference between the target sample field matched by the stacked cross-attention module and the true field label. Based on the loss between the target sample field and the field label, the model parameters of the stacked cross-attention module are adjusted to avoid the randomness of unsupervised training, making the audit information extraction of the trained multimodal fusion model more accurate.

[0119] According to embodiments of the present invention, an initial sample query vector guides a stacked cross-attention module to match the target sample modal feature vector from the multimodal feature vector, thereby improving the accuracy of field extraction. The decoder outputs the target sample field, and the loss is calculated with the field labels extracted from the sample text. The parameters of the cross-attention module are then adjusted in reverse, thereby improving the accuracy of field recognition.

[0120] Training a multimodal fusion model can also include operations C1 to C3.

[0121] In operation C1, at least two multimodal fusion models with the same structure are used to perform the method of adjusting the model parameters of the stacked cross-attention modules, and each outputs a first target sample field and a second target sample field that match the sample query field.

[0122] In operation C2, the inter-distillation loss between at least two multimodal fusion models is calculated based on the label loss between the first target sample field and the field label, and the label loss between the second target sample field.

[0123] In operation C3, the model parameters of the stacked cross-attention modules of at least two multimodal fusion models are simultaneously optimized using the inter-distillation loss and the label loss of each of the at least two multimodal fusion models to determine the trained multimodal fusion model from the at least two multimodal fusion models.

[0124] The above process is illustrated using two multimodal fusion models with identical structures, referred to as the first model. Second Model The two models share the same initial parameters.

[0125] The vector obtained by encoding the sample query field is used as the initial sample query vector, and the initial sample query vector is simultaneously input into the first model. Second Model From the first model Output the starting probability distribution of the first target sample field. and endpoint probability distribution By the second model Output the starting probability distribution of the second target sample field and endpoint probability distribution Thus, the first target sample field and the second target sample field are obtained.

[0126] Construct a joint self-supervised training task and calculate the cross-distillation loss between the two models. The cross-distillation loss can be expressed by formula (13).

[0127] (13)

[0128] in, It is the loss from inter-distillation. It is the probability distribution of the first target sample field output by the first model. It is the probability distribution of the second target sample field output by the second model. express and The relative entropy (Kullback-Leibler, KL) divergence between them express and The KL divergence between them.

[0129] The model loss is constructed using the inter-distillation loss and the label losses of at least two multimodal fusion models, and the first model is simultaneously optimized using the model loss. Second Model The model parameters of the stacked cross-attention modules.

[0130] The model loss can be expressed by formula (14).

[0131] (14)

[0132] in, It is the model loss. It refers to the label loss of each individual model in the multimodal fusion model. This represents the weight of the inter-distillation loss.

[0133] When both multimodal fusion models have been optimized, the multimodal fusion model with better performance is selected as the trained multimodal fusion model.

[0134] According to an embodiment of the present invention, multiple multimodal fusion models with the same structure are trained in parallel. The cross-distillation loss between models and the label loss of each model are used to simultaneously optimize the multiple multimodal fusion models, thereby reducing the overfitting of a single model to the training data. This enables the model to stably extract target fields from audit information in audit documents of different formats and qualities, thereby improving the generalization ability of the model.

[0135] At the field-level granularity, stacked cross-attention modules are invoked to achieve feature interaction and alignment between and within modalities. The output target modal feature vector matching the query field may include operations D1 to D5. The cross-attention module can use a hierarchical attention structure, and the cross-attention module may include M layers, where M is an integer greater than 1.

[0136] In operation D1, the feature vectors of at least two modes are linearly mapped to obtain key-value pairs of the first mode and the second mode corresponding to the feature vectors of at least two modes, respectively.

[0137] A projection matrix can be used to linearly map the eigenvectors of at least two modes, thereby obtaining key-value pairs of the first mode and the second mode corresponding to the eigenvectors of the at least two modes, respectively.

[0138] The process of obtaining key-value pairs can be represented by formula (15).

[0139] (15)

[0140] in, It is an n-modal key. It is the value of the n-mode. It is the bond projection matrix of the n-mode. It is the value projection matrix of the n-mode. It is an n-modal feature vector, where n can be an image v, text t, or table b.

[0141] In operation D2, the fused feature vector output from layer m-1 is used as the query vector, and dot product attention is calculated with the key-value pairs of the first modality and the key-value pairs of the second modality respectively to achieve intramodal interaction and alignment, and generate aggregated features of the first modality and aggregated features of the second modality.

[0142] The fusion feature vector output by the (m-1)th layer is obtained by processing the initial query vector by the first (m-1)th layers; the fusion feature vector output by the (m-1)th layer aggregates the local features of each modality in the audit information; m is greater than 1 and less than or equal to M.

[0143] Aggregated features are a set of feature vectors that aggregate specific fields under a specific modality.

[0144] When performing dot product attention calculations on the query vector with the key-value pairs of the first and second modalities respectively, the raw attention scores of each modality are first calculated. Then, softmax calculation is performed on the raw attention scores of each modality to obtain the attention weight matrix of each modality. Finally, the attention weight matrix of each modality is multiplied by the values ​​of each modality to generate the aggregated features of each modality.

[0145] The aggregated feature for generating the nth mode can be represented by formula (16).

[0146] (16)

[0147] in, It is the aggregated feature of the (m-1)th layer of the n-mode. These are key-value pairs in the n-modal sense. It is the query vector of the (m-1)th layer. It is the hidden vector dimension used to scale the dot product. It is an activation function.

[0148] In operation D3, the aggregated features of the first mode and the aggregated features of the second mode are weighted and fused to achieve interaction and alignment between modes, generating the fused feature vector output by the m-th layer.

[0149] The fusion feature vector of the m-th layer output can be represented by formula (17).

[0150] (17)

[0151] in, It is the fused feature vector output by the m-th layer. This is a layer normalization operation. It normalizes the scale of the feature vectors to maintain numerical stability in subsequent layers. It is the weight coefficient assigned to the n-th mode at the (m-1)th layer, representing the degree of attention the vector receives in different modes. The specific value of the weight coefficient is adjusted according to the layer.

[0152] In operation D4, in response to determining that m is less than M, return to perform dot product attention calculation and weighted fusion operation on the fused feature vector output of the m-th layer, and increment m.

[0153] m is the current number of cross-attention module layers, and M is the preset total number of cross-attention mechanism module layers. If m is determined to be less than M, it means that the aggregated features have not been fully fused and feature fusion at the next level (i.e., layer m+1) is required to further perform feature interaction and alignment between and within modalities.

[0154] The stacked cross-attention mechanism module uses residual connections between each layer to ensure stable transmission of the queried field between layers, injecting modal features from the previous layer into feature fusion.

[0155] In operation D5, in response to determining that m equals M, the target fused feature vector is generated by repeatedly interacting with the feature vectors of at least two modalities at different scales through the field to be queried.

[0156] Among them, the target fusion feature vector integrates local features and global semantic features from the audit information.

[0157] When m equals M, the number of cross-attention module layers m has reached the preset total number of layers M. At this point, the iteration stops and the final target fusion feature vector is generated.

[0158] The multi-layered stacked cross-attention mechanism allows the initial query vector to interact repeatedly with modal features at different levels. From an interpretability perspective, lower layers can aggregate local evidence (such as numbers within cells or pixels in a seal), while higher layers can aggregate semantic evidence (such as the distribution of contract numbers across lines of text).

[0159] Local features in audit information can retain specific details related to the field being queried, such as contract numbers or invoice numbers. Global semantic features can retain contextual or structural features of multimodal information, such as cross-page contextual semantic relationships and field features of cross-row lists.

[0160] According to embodiments of the present invention, dot product attention is performed on key-value pairs of multiple modalities to generate aggregated features of multiple modalities. Different weight coefficients are used at different layers to perform weighted operations on the aggregated features of different modalities, thereby suppressing noisy modalities and amplifying evidential modalities at the feature level. By iteratively stacking M layers, features interact repeatedly at different scales, gradually focusing on relevant information of the target field, filtering redundant interference, and improving the matching accuracy between features and query fields.

[0161] Based on the modality type of the data in the audit information, the encoder corresponding to the modality type is invoked to encode the audit information, generating feature vectors of at least two modalities, which may include operations E1 to E2.

[0162] In operation E1, in response to determining that the modality type is table, cell boundaries are reconstructed for the table modality data in the audit information to generate the reconstructed table modality data.

[0163] When tabular modal data is identified from audit information, the tabular modal data may have problems such as blurred boundaries and cells spanning multiple rows or columns, making it difficult to directly identify cell boundaries. Therefore, it is necessary to reconstruct the cell boundaries.

[0164] When the modality type is determined to be table, the normalized cell boundaries can be obtained by using normalized coordinates to reconstruct the cell boundaries of the table modality data.

[0165] The process of normalizing coordinates can be represented by formula (18).

[0166] (18)

[0167] in, It is the normalized cell boundary vector. These are the coordinates of the cell boundaries in the table, W is the table width, and H is the table height.

[0168] In operation E2, the table encoder is invoked, using a predefined cell adjacency matrix and geometric position encoding as bias terms. By capturing the row and column relationships and cell merging information in the table modal data, a table modal feature vector is generated.

[0169] A predefined cell adjacency matrix can be a matrix used to describe the spatial relationships between multiple cells within a table, guiding the table encoder to capture the relationships between rows and columns. Geometric position encoding can represent the relative position of a cell within the entire table.

[0170] According to embodiments of the present invention, by reconstructing cell boundaries of table modal data in audit information, reconstructed table modal data is generated, giving the table modal data clear row and column boundaries and a unified structure. A table encoder is invoked, guided by a predetermined cell adjacency matrix and geometric position encoding, to generate table modal feature vectors, thereby preserving the structural information and semantic information of the table content, thus improving the efficiency of audit information extraction.

[0171] The following explains the reconstructed table modal data.

[0172] Generating the reconstructed tabular modal data can include operations F1 to F5.

[0173] In operation F1, in response to determining that the modal type is table, text position detection is performed on the table modal data to obtain the position coordinates of each text box.

[0174] One approach is to use OCR methods to detect text positions in tabular modal data, thereby obtaining the position information of a set of text boxes. For example, the position information of a text box may include the top x-coordinate (x_min), the top y-coordinate (y_min), the bottom x-coordinate (x_max), the bottom y-coordinate (y_max), and the text content (text).

[0175] In operation F2, clustering is performed using the ordinate of each text box's position coordinates and a predetermined ordinate threshold to obtain the row boundary lines.

[0176] With the position coordinates of each text box obtained, all text boxes can be initially sorted by y_min to preliminarily determine the row order.

[0177] As one implementation method, horizontal projection or Y-coordinate clustering can be used to determine a suitable row height threshold. All y_min and y_max values ​​are clustered, and the resulting cluster centers can be used as the target row boundary line. With the target row boundary line obtained, the sorted text box position information is traversed, and text boxes whose y-coordinates overlap with the current row's text box within a predetermined y-coordinate threshold range are considered as belonging to the same row, thus obtaining the row boundary line.

[0178] The predetermined vertical axis threshold can be determined based on the height of the text boxes. For example, text boxes with 50% overlap in height can be considered as being in the same row.

[0179] In operation F3, clustering is performed using the x-coordinate of each text box's position coordinates and a predetermined x-coordinate threshold to obtain column boundary lines.

[0180] With the position coordinates of each text box obtained, all text boxes can be initially sorted by x_min to preliminarily determine the column order.

[0181] One implementation method is to perform vertical projection or X-coordinate clustering on all text boxes or row by row to determine a suitable column width threshold. Clustering all x_min and x_max values ​​yields cluster center points that can be used as target row boundary lines. With the target column boundary lines obtained, the sorted text box position information is traversed, and text boxes whose x-coordinates overlap with the current column's text box within a predetermined x-coordinate threshold range are considered as belonging to the same row, thus obtaining the column boundary lines.

[0182] To align the text boxes in each column, it is necessary to either project all rows vertically or cluster all text boxes together by their x-coordinates.

[0183] In operation F4, an initial mesh is created based on the row and column boundary lines.

[0184] In the initial grid, all rows have the same height and all columns have the same width.

[0185] As one implementation, a two-dimensional list grid[i][j] can be constructed based on row and column boundary lines, representing the grid in row i and column j, which contains the grid's boundary coordinates and text content. The initial grid can be a grid with uniform row and column distribution.

[0186] Press F5 to fill the initial grid with the content of each text box, and you will get the reconstructed table modal data.

[0187] The content of each text box can be extracted from the audit information of the table modality using OCR methods. The extracted text content is not completely arranged according to the grid distribution and may span columns, rows, or be offset. The content of each text box is filled into the initial grid according to its original arrangement position to obtain structured text data with clear rows and columns.

[0188] According to an embodiment of the present invention, an initial grid with uniform row and column distribution is obtained by reconstructing the audit information of the table modality. The content of each text box is then filled into the initial grid according to its original position to obtain reconstructed table modality data with clear row and column boundaries. This allows the table encoder to more clearly capture the structural and content information of the table modality audit information, providing reliable table modality data for the multimodal fusion model, thereby improving the model's document structure awareness capability.

[0189] Fill the initial grid with the contents of each text box to obtain the reconstructed table modal data, which may include operations G1 to G3.

[0190] In operation G1, in response to determining that the width of any text box is greater than the width of a column in the initial grid, and that the horizontal center of any text box overlaps with multiple columns, any text box is determined to be a cross-column cell.

[0191] If the width of a text box is greater than the width of a column in the initial network, it means that the text box cannot be contained within a single column cell, i.e., the text box may span multiple columns. Further checks are made to determine if the horizontal center of the text box overlaps with multiple columns, i.e., whether the width of the text box covers the column start positions of multiple cells in the row below. If both the width and horizontal center of the text box meet the criteria for spanning multiple columns, then the text box is determined to be a spanning column cell.

[0192] In operation G2, in response to determining that the height of any text box is greater than the height of the rows in the initial network, and that the vertical center of any text box overlaps with multiple rows, any text box is determined to be a cross-row cell.

[0193] If the height of a text box is greater than the height of a row in the initial network, it means that the text box cannot be contained within a single-line cell, i.e., the text box may span multiple lines. Further analysis determines if the vertical center of the text box overlaps with multiple lines. If both the height and vertical center of the text box meet the criteria for a spanned cell, then the text box is confirmed as a spanned cell.

[0194] By determining the column width and horizontal center of the text box, as well as the row height and vertical center, and using two conditions to determine whether a cell spans multiple rows or columns, misjudgments caused by a single constraint can be avoided.

[0195] If the text box is determined to be a column-spanning cell, merge the corresponding column grids in grid[i][j] to generate a column-spanning cell. If the text box is determined to be a row-spanning cell, merge the corresponding row grids in grid[i][j] to generate a row-spanning cell.

[0196] In operation G3, the initial grid is locally modified based on the spanning of column cells and the spanning of row cells, and the content of each text box is filled into the modified grid to obtain the reconstructed table modal data.

[0197] Local corrections to the initial grid can be made by fine-tuning the column boundaries based on the median or mean of x_min and x_max of the cells spanning columns, and fine-tuning the row boundaries based on the median or mean of y_min and y_max of the cells spanning rows, thus completing the local correction of the initial grid.

[0198] When filling the initial grid, some grid cells may not have corresponding text boxes. In this case, these grid cells can be marked as empty cells.

[0199] This invention also includes a verification mechanism to ensure that the content of each text box is correctly filled into its corresponding position in the corrected grid. For example, a text box content index is constructed to record the filling status of each text box's content in the initial grid. When the content of a text box is filled into the corrected grid, the filling status of that text box in the index is marked as filled. After the text box content filling is complete, the text box content index is traversed, and text boxes with an unfilled status are filtered. If an unfilled text box exists, its content is filled into the corresponding grid; otherwise, the text box content filling is complete, resulting in the reconstructed table modal data.

[0200] According to embodiments of the present invention, by identifying and merging cells that span multiple rows or columns, the problem of standard grids being unable to adapt to irregular cells is solved, thereby fully restoring the true structural semantics of the audit table. The content of each text box is then filled into the corrected grid, ensuring the completeness and accurate attribution of content in merged cells, and avoiding errors in text box content filling due to structural misjudgment.

[0201] Figure 3 A schematic diagram is shown of a multimodal fusion model predicting a target field according to an embodiment of the present disclosure.

[0202] like Figure 3 As shown, the multimodal fusion model first uses an encoder to encode the audit information for the image, text, and table modalities respectively, obtaining feature vectors for each modality, thus constructing a sequence of modal vectors. Then, a stacked cross-attention module is invoked to linearly map the feature vectors of each modality using a projection matrix, generating key-value pairs for each feature vector. The stacked cross-attention module consists of M layers, with residual links between each layer. Multi-layer attention residual propagation completes the inter-layer accumulation of cross-modal features. Finally, the starting position of the target predicted field is obtained through the probability of the starting position of the predicted field.

[0203] Figure 4 An architecture diagram of a multimodal fusion model according to an embodiment of the present disclosure is shown.

[0204] like Figure 4 As shown, the architecture of the multimodal fusion model can include a multimodal preprocessing layer, a text recognition layer, a multimodal feature encoding layer, a multimodal feature fusion layer, a structured output and anomaly detection layer, and a field expansion module. The cross-attention module in the multimodal fusion model architecture is trained using a multi-task joint self-supervised training mechanism.

[0205] The multimodal preprocessing layer preprocesses the audit documents in image, text, and table modes respectively, including deblurring the scanned images, segmenting the text content into sentences, words, and cleaning symbols, and aligning the rows and columns and reconstructing the cell boundaries of the table data.

[0206] In the text recognition layer, the OCR method is used to recognize characters in the text image and generate structured text with location information.

[0207] In the multimodal feature encoding layer, an image encoder is used to encode the audit information of the image modality, obtaining image-level feature vectors. A text encoder is used to encode the audit information of the text modality, obtaining semantic-level feature vectors. A table encoder is used to encode the audit information of the table modality, obtaining cell-level structured feature vectors.

[0208] In the multimodal feature fusion layer, guided by the initial query vector at the field level, the layered cross-attention module is called. Using the hierarchical attention mechanism, the initial query vector and the multimodal feature vector interact and fuse. Feature interaction and alignment are achieved through intramodal attention and intermodal attention calculation, and the target modal feature vector that matches the field to be queried is output.

[0209] In the structured output and anomaly detection layer, structured field results are output and anomaly detection is performed. In the structured output, the target modality feature vector is decoded, and target fields matching the query fields are extracted from the audit information. The output shows structured field results related to the audit information extraction task, including but not limited to: company name, project name, contract number, responsible person, location information, seal information, invoice number, and anomaly alert fields.

[0210] In anomaly detection, the anomaly alert mechanism constructs a comprehensive anomaly score based on cross-modal attention distribution and content consistency metrics, and then calculates confidence and determines thresholds. When the same field exhibits missing, conflicting, or significantly different characteristics across different modalities, it can automatically trigger anomaly alert output to assist in subsequent manual review.

[0211] In the exception indication mechanism, the target field is calculated first. Comprehensive abnormality score Comprehensive anomaly score The calculation can be expressed by formula (19).

[0212] (19)

[0213] in, It is a measure of the consistency of attention distribution across modalities. It is a text modal value With table modal values The relative differences Represents a measure of string inconsistency. These are entity strings from the text modality and the table modality, respectively. These are the weighting coefficients.

[0214] Then calculate the target field. confidence level And perform anomaly detection. Confidence level. It can be expressed by formula (20).

[0215] (20)

[0216] in, It is the attention distribution after multimodal aggregation. It is an entropy function. Is the target field In position The predicted probability, This represents the weighting coefficient.

[0217] Finally, the calculated comprehensive anomaly score was analyzed. and confidence level Perform threshold determination; if the following conditions are met: or Then mark the target field The result is "abnormal," triggering manual review. This represents the threshold for abnormal alerts in the open settings.

[0218] The multimodal fusion model also features a dynamic field expansion mechanism, allowing users to add query fields. This dynamic field expansion mechanism shares features with the existing multimodal representation space, enabling rapid adaptation to new task requirements without retraining the entire model. When a user adds a query field, the newly added field query vector is first initialized in the shared representation space. The process of initializing the query vector can be represented by formula (21).

[0219] (twenty one)

[0220] in, This is a newly added field query vector. This is a new query field. This is the encoded result of the field name. It is the first Local context vectors for each example. This is the number of example samples for the newly added field. For pooling operations, This is a normalization operation.

[0221] The prototype vector and the parameter matrix are then adapted through a low-cost update. The prototype vector can be represented by formula (22).

[0222] (twenty two)

[0223] in, It is a prototype vector. It is the average embedding of the newly confirmed samples. It is the momentum coefficient.

[0224] While keeping the main model parameters frozen, only a small-scale low-rank adaptation layer is inserted or learned into the model for low-rank adaptation updates. The size of the low-rank adaptation layer is much smaller than the original parameter matrix. The prototypes of the low-rank parameters and the newly added query fields are fine-tuned step by step, and the update rule can be expressed by formula (23).

[0225] (twenty three)

[0226] This is the model parameter matrix to be updated. , It is a low-rank update matrix.

[0227] After a new query field is added, confidence is calculated using prototype similarity and aggregated attention distribution. A threshold or calibration module is then used to determine whether manual review should be triggered. If a decrease in accuracy is detected, it is rapidly improved through incremental online learning without retraining all parameters.

[0228] During the training phase of the multimodal fusion model, a multi-task joint self-supervised training mechanism is used to train the multimodal fusion model. In training, sample text, sample images, and sample tables are first acquired, and then the sample text, sample images, and sample tables are masked respectively. The encoder is then called to generate feature vectors for the sample text, sample images, and sample tables.

[0229] The stacked cross-attention module is invoked to interactively fuse the feature vectors of sample text, sample image, and sample table, and output the expected text, expected image, and expected table.

[0230] Based on the loss between sample text and expected text, the loss between sample image and expected image, and the loss between sample table and expected table, the model parameters of the stacked cross-attention modules are adjusted according to the dynamic task weights to obtain a trained multimodal fusion model.

[0231] Figure 5 A schematic diagram of the preset query fields in the audit information extraction method based on multimodal fusion according to an embodiment of the present invention is shown.

[0232] like Figure 5 As shown, given a preset query field, firstly, audit documents for which information extraction needs to be performed are collected, and the multimodal audit information in the audit documents is preprocessed. Then, a vector encoder is used to encode the features of the image modality, text modality, and table modality data respectively, obtaining multimodal feature vectors. Finally, the multimodal feature vectors are input into a multimodal fusion model based on an attention mechanism. The multimodal feature vectors are interactively fused based on a cross-modal cross-attention mechanism, achieving feature interaction and alignment between and within modalities, resulting in a field representation of the query field. The query field is then preset based on its representation. For example, the query field could be represented as company name, project name, contract number, responsible person, location information, seal information, invoice number, or anomaly alert field, etc.

[0233] Preprocessing can include adaptive deblurring and reconstruction of the original scanned image in the image modality to obtain a clear image; regular cleaning of the text content in the text modality, such as sentence segmentation, word segmentation and symbol cleaning, to obtain a cleaned effective text word sequence; and cell reconstruction of the table data in the table modality, such as row and column alignment and cell boundary reconstruction, to obtain normalized table boundaries.

[0234] In the image modality, adaptive deblurring reconstruction of the original scanned image can be performed. A joint deblurring algorithm based on convolution and transform domain can be used to reconstruct the low-quality original scanned image, thereby obtaining a clear image. This process can be represented by formula (24).

[0235] (twenty four)

[0236] in, It is a clear image. It is the original scanned image. Indicates retrieving the image The minimum error, This indicates the use of fuzzy convolution kernels during the deblurring and reconstruction process. The convolution operation performed Represents the image gradient. Denotes the Frobenius norm used for regularization. Indicates the balance parameters; Represents the L1 norm; This represents the L2 norm.

[0237] In text modality, regular cleaning of text content can include five steps: symbol normalization, full / half-width character unification, number format standardization, date standardization, and removal of invalid characters. The process of obtaining the cleaned valid text word sequence can be represented by formula (25).

[0238] (25)

[0239] in, It is the cleaned and valid text word sequence. This represents the original text sequence before cleaning. This indicates regularized cleaning. Represents a vocabulary list. This represents the set of invalid characters that need to be removed.

[0240] Figure 6 A block diagram of an audit information extraction apparatus based on multimodal fusion according to an embodiment of the present invention is shown.

[0241] like Figure 6 As shown, the audit information extraction device 600 based on multimodal fusion includes an acquisition module 610, an encoding module 620, a fusion module 630, and a decoding module 640.

[0242] The acquisition module 610 is used to acquire the fields to be queried in response to receiving an extraction request for audit information. The audit information includes data in at least two of the following modalities: text, images, and tables.

[0243] The encoding module 620 is used to encode the audit information according to the modality type of the data in the audit information by calling the encoder corresponding to the modality type, and generating feature vectors of at least two modalities. Among them, the feature vectors include at least two of the following: text feature vector, image feature vector, and table feature vector; the table feature vector is used to characterize the absolute position feature of the table and the relative position feature of each cell in the table relative to the table.

[0244] The fusion module 630 is used to call the stacked cross-attention module with the vector obtained by encoding the field to be queried as the initial query vector, and to interact and fuse with the feature vectors of at least two modalities respectively, so as to realize feature interaction and alignment between and within modalities, and output the target modal feature vector that matches the field to be queried.

[0245] The decoding module 640 is used to decode the target modality feature vector and extract the target field that matches the field to be queried from the audit information.

[0246] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0247] For example, any plurality of the acquisition module 610, encoding module 620, fusion module 630, and decoding module 640 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present disclosure, at least one of the acquisition module 610, encoding module 620, fusion module 630, and decoding module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 610, encoding module 620, fusion module 630 and decoding module 640 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0248] Figure 7 A block diagram of an electronic device suitable for implementing the multimodal fusion-based audit information extraction method described above, according to an embodiment of the present invention, is shown. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0249] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0250] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0251] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0252] According to embodiments of the present invention, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of the embodiments of the present invention. According to embodiments of the present invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0253] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0254] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0255] For example, according to embodiments of the present invention, a computer-readable storage medium may include the ROM 702 described above, or RAM 703, or ROM 702 and RAM 703, or one or more memories other than ROM 702 and RAM 703.

[0256] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the audit information extraction method based on multimodal fusion provided in the embodiments of the present invention.

[0257] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0258] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0259] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0260] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or pairings fall within the scope of this invention.

[0261] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for extracting audit information based on multimodal fusion, characterized in that, include: In response to receiving a request to extract audit information, the fields to be queried are obtained; wherein, the audit information includes data in at least two of the following modalities: text, image, and table; Based on the modality type of the data in the audit information, the encoder corresponding to the modality type is invoked to encode the audit information, generating feature vectors for at least two modalities; wherein, the feature vectors include at least two of the following: text feature vector, image feature vector, and table feature vector; the table feature vector is used to characterize the absolute position features of the table and the relative position features of each cell in the table relative to the table. Using the vector encoded from the field to be queried as the initial query vector, a stacked cross-attention module is invoked to interact and fuse with the feature vectors of at least two modalities, respectively, to achieve feature interaction and alignment between and within modalities, and output a target modal feature vector that matches the field to be queried; and Decode the target modality feature vector and extract the target field that matches the query field from the audit information; The cross-attention module comprises M layers, where M is an integer greater than 1; The process involves using the vector encoded from the field to be queried as the initial query vector, invoking a layered cross-attention module to interact and fuse with the feature vectors of at least two modalities, respectively, to achieve feature interaction and alignment between and within modalities, and outputting a target modal feature vector that matches the field to be queried, including: Linearly map the feature vectors of the at least two modes respectively to obtain key-value pairs of the first mode and key-value pairs of the second mode corresponding to the feature vectors of the at least two modes respectively; The fused feature vector output from the (m-1)th layer is used as the query vector. Dot product attention is then performed with the key-value pairs of the first modality and the second modality to achieve intra-modal interaction and alignment, generating aggregated features of the first and second modalities. The fused feature vector output from the (m-1)th layer is obtained by processing the initial query vector in the first (m-1)th layers. The fused feature vector output from the (m-1)th layer aggregates the local features of each modality in the audit information; m is greater than 1 and less than or equal to M. The aggregated features of the first modality and the aggregated features of the second modality are weighted and fused to achieve interaction and alignment between modalities, generating the fused feature vector output by the m-th layer; In response to determining that m is less than M, return to perform dot product attention calculation and weighted fusion operation on the fused feature vector output by the m-th layer, and increment m; and In response to determining that m equals M, the target modality feature vector is generated by repeatedly interacting with the feature vectors of the at least two modalities at different scales through the field to be queried; wherein the target modality feature vector integrates local features and global semantic features in the audit information.

2. The method according to claim 1, characterized in that, The step involves calling an encoder corresponding to the modality type of the data in the audit information to encode the audit information, generating feature vectors for at least two modalities, including: In response to determining that the modality type is a table, cell boundaries are reconstructed in the table modality data in the audit information to generate reconstructed table modality data; and The table encoder is invoked, using a predefined cell adjacency matrix and geometric position encoding as bias terms. By capturing the row and column relationships and cell merging information in the table modal data, a table modal feature vector is generated.

3. The method according to claim 2, characterized in that, In response to determining that the modality type is a table, the cell boundaries of the table modality data in the audit information are reconstructed to generate reconstructed table modality data, including: In response to determining that the modality type is a table, text position detection is performed on the table modality data to obtain the position coordinates of each text box; Clustering is performed using the ordinate of the position coordinates of each text box and a predetermined ordinate threshold to obtain the row boundary lines; Clustering is performed using the horizontal coordinates of the position coordinates of each text box and a predetermined horizontal coordinate threshold to obtain column boundary lines; An initial grid is created based on the row boundary lines and the column boundary lines; in the initial grid, all rows have the same height and all columns have the same width; and The content of each text box is filled into the initial grid to obtain the reconstructed table modal data.

4. The method according to claim 3, characterized in that, The step of filling the content of each text box into the initial grid to obtain the reconstructed table modal data further includes: In response to determining that the width of any text box is greater than the width of a column in the initial grid, and that the horizontal center of any text box overlaps with multiple columns, the any text box is determined to be a cross-column cell; In response to determining that the height of any text box is greater than the height of a row in the initial grid, and that the vertical center of any text box overlaps with multiple rows, the text box is determined to be a multi-row cell; and The initial grid is locally modified based on the cross-column cells and the cross-row cells, and the content of each text box is filled into the modified grid to obtain the reconstructed table modal data.

5. The method according to claim 1, wherein the multimodal fusion model comprises a text encoder, an image encoder, a table encoder, stacked cross-attention modules, and a decoder; the method further comprises: The sample text, sample image, and sample table are obtained; wherein the sample text is obtained by performing text recognition on the sample image; and the sample image is obtained by image acquisition from the sample table. The sample text, the sample image, and the sample table are respectively masked to generate masked sample text, masked masked image, and masked sample table; The text encoder is invoked to encode the masked sample text, generating a sample text feature vector; The image encoder is invoked to encode the masked image to generate a sample image feature vector; The table encoder is invoked to encode the masked sample table to generate a sample table feature vector; wherein, the sample table feature vector is used to characterize the absolute position features of the sample table and the relative position features of each cell in the sample table relative to the sample table. The stacked cross-attention module is invoked to interactively fuse the feature vectors of the sample text, sample image, and sample table, thereby achieving feature interaction and alignment between and within modalities, and outputting the expected text, expected image, and expected table. Based on the loss between the sample text and the expected text, the loss between the sample image and the expected image, and the loss between the sample table and the expected table, the model parameters of the stacked cross-attention modules are adjusted to obtain a trained multimodal fusion model.

6. The method according to claim 5, characterized in that, Also includes: Using the vector obtained by encoding the sample query field as the initial sample query vector, the stacked cross-attention module is called to interact and fuse with the sample text feature vector, the sample image feature vector and the sample table feature vector respectively, so as to realize feature interaction and alignment between and within modalities, and output the target sample modal feature vector that matches the sample query field; The decoder is invoked to decode the modal feature vector of the target sample, and the target sample field that matches the sample query field is output. Extract field labels from the sample text based on the sample query fields; as well as Based on the loss between the target sample field and the field label, the model parameters of the stacked cross-attention modules are adjusted to obtain the trained multimodal fusion model.

7. The method according to claim 6, characterized in that, The method further includes: At least two multimodal fusion models with the same structure are used to execute the method as described in claim 6, and each outputs a first target sample field and a second target sample field that match the sample query field; Calculate the inter-distillation loss between at least two multimodal fusion models based on the label loss between the first target sample field and the field label and the label loss between the second target sample fields; Using the inter-distillation loss and the label loss of each of the at least two multimodal fusion models, the model parameters of the stacked cross-attention modules of the at least two multimodal fusion models are simultaneously optimized to determine the trained multimodal fusion model from the at least two multimodal fusion models.

8. An audit information extraction device based on multimodal fusion, characterized in that, include: The acquisition module is used to acquire the fields to be queried in response to receiving an extraction request for audit information; wherein the audit information includes data in at least two of the following modalities: text, image, and table; The encoding module is used to encode the audit information according to the modality type of the data in the audit information by calling the encoder corresponding to the modality type, and generating feature vectors of at least two modalities; wherein, the feature vectors include at least two of the following: text feature vector, image feature vector and table feature vector; the table feature vector is used to characterize the absolute position feature of the table and the relative position feature of each cell in the table relative to the table; The fusion module is used to take the vector encoded by the field to be queried as the initial query vector, call the stacked cross-attention module, and interact and fuse it with the feature vectors of at least two modalities respectively, so as to achieve feature interaction and alignment between and within modalities, and output the target modal feature vector that matches the field to be queried; and The decoding module is used to decode the target modality feature vector and extract the target field that matches the field to be queried from the audit information; wherein, the cross-attention module includes M layers, where M is an integer greater than 1; The fusion module is used for: Linearly map the feature vectors of the at least two modes respectively to obtain key-value pairs of the first mode and key-value pairs of the second mode corresponding to the feature vectors of the at least two modes respectively; The fused feature vector output from the (m-1)th layer is used as the query vector. Dot product attention is then performed with the key-value pairs of the first modality and the second modality to achieve intra-modal interaction and alignment, generating aggregated features of the first and second modalities. The fused feature vector output from the (m-1)th layer is obtained by processing the initial query vector in the first (m-1)th layers. The fused feature vector output from the (m-1)th layer aggregates the local features of each modality in the audit information; m is greater than 1 and less than or equal to M. The aggregated features of the first modality and the aggregated features of the second modality are weighted and fused to achieve interaction and alignment between modalities, generating the fused feature vector output by the m-th layer; In response to determining that m is less than M, return to perform dot product attention calculation and weighted fusion operation on the fused feature vector output by the m-th layer, and increment m; and In response to determining that m equals M, the target modality feature vector is generated by repeatedly interacting with the feature vectors of the at least two modalities at different scales through the field to be queried; wherein the target modality feature vector integrates local features and global semantic features in the audit information.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 7.

Images