Intelligent material auditing method and system based on deep learning
By using deep fusion technology of multimodal feature mapping and Transformer encoder, the problem of incompatibility between template features and semantic features in the existing system is solved, thereby improving the accuracy and interpretability of intelligent material review.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI CREDIT INFORMATION CO LTD
- Filing Date
- 2026-02-06
- Publication Date
- 2026-05-19
AI Technical Summary
Existing intelligent material review systems suffer from incompatibility between the structured features output by template matching and the semantic features extracted by deep neural networks in deep learning applications. This leads to black-box decision-making that is difficult to interpret and makes it difficult to distinguish between template problems and semantic problems.
Multimodal features are mapped to the same dimension through a multimodal feature extraction model, deep fusion is performed using a Transformer encoder, and the initial review results are corrected through a chain of evidence to generate a global document vector to improve the accuracy and interpretability of the review.
It achieves deep fusion of multimodal features to generate review results that include document context-aware representations, thereby improving the accuracy and interpretability of the review results.
Smart Images
Figure CN122066441A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of materials review technology, and in particular to a deep learning-based intelligent materials review method and system. Background Technology
[0002] Intelligent review of application materials, which improves the efficiency and accuracy of application material processing through computational means, has been widely applied in many fields, such as education, government affairs, and finance.
[0003] The application of deep learning in intelligent review of application materials is evolving from traditional rule matching and shallow machine learning to a stage of intelligent understanding and decision-making capable of handling complex semantics, multimodal data, and hidden patterns. For example, deep learning methods combining template matching and semantic understanding have made significant progress in intelligent review of application materials.
[0004] However, many existing systems use parallel or serial methods for feature fusion rather than deep fusion. The structured features output by template matching and the semantic features (text embedding vectors) extracted by deep neural networks may be incompatible at the representation level. Moreover, when a deep neural network gives a low score, it is difficult to distinguish whether it is a template problem or a semantic problem, resulting in the uninterpretability of black-box decision-making. Summary of the Invention
[0005] This invention processes multimodal features through a multimodal feature extraction model, maps multimodal features to the same dimension, and establishes the relationship between text embedding vectors and image embedding vectors to achieve cross-modal alignment, thereby facilitating subsequent deep fusion.
[0006] The technical solution proposed in this invention is as follows: acquire the target document, extract multimodal features from the target document, generate fused embedding features, and perform multimodal encoding on the fused embedding features;
[0007] Multimodal coding is used to obtain the global document vector of the target document. Based on the global document vector, the initial review result of each document under each review condition is obtained.
[0008] Extract the chain of evidence from the multimodal encoding of the target document, and use the chain of evidence to correct the preliminary review results.
[0009] Preferably, the step of acquiring the target document, extracting multimodal features from the target document, generating fused embedding features, and performing multimodal encoding on the fused embedding features includes:
[0010] Multimodal features are extracted from the target document using a pre-trained multimodal feature extraction model, and the obtained multimodal features are used to construct fused embedding features.
[0011] The obtained fused embedding features are encoded using the Transformer encoder to obtain a document context-aware representation.
[0012] Preferably, the step of extracting multimodal features from the target document using a pre-trained multimodal feature extraction model and constructing fused embedding features using the obtained multimodal features includes:
[0013] Obtain a pre-trained multimodal feature extraction model;
[0014] By using the text feature extraction layer in a multimodal feature extraction model, the text in the target document is segmented into tokens, and each token is converted into a token sequence. ;in, Indicates the first Each word element, Indicates the number of lexical units;
[0015] Identify the bounding box coordinates of each lexical unit, and after normalization, construct the lexical layout feature sequence of the target text. ;in Indicates the first The coordinates of the word border. Indicates the first The width and height of the word border;
[0016] The images in the document are divided into multiple image blocks to form an image block sequence. ,in, Indicates the first Image blocks;
[0017] Get text embedding vector ,in, ; Indicates word embedding;
[0018] Get layout embedding vector ,in, Layout features ; Represents a linear transformation;
[0019] Obtain image embedding vector ,in, ; Represents a convolutional neural network;
[0020] By using the cross-modal attention mechanism layer in the multimodal feature extraction model, text embedding vectors, layout embedding vectors, and image embedding vectors are mapped to the same dimension to obtain fused embedding features, including:
[0021] Map the elements embedded in the text vector to dimension . Given a context space, obtain the text embedding vector elements within that context space. ;
[0022] Map the elements embedded in the layout vector to dimensions [1]. Within the context space, obtain the layout embedding vector elements within the context space. ;
[0023] Map the elements of the image embedding vector to a dimension of... Within the context space, obtain the image embedding vector elements within the context space. ;in, , , Represents the projection matrix; , , Indicates bias;
[0024] Alignment through attention mechanisms and Generate image-text aligned representation ;
[0025] Obtain the fused embedding features for each lexical unit. ,in, , , Represents the normalized result , , ;
[0026] Sort all fused embedded features according to the reading order to obtain a unified sequence. ;
[0027] The process of encoding the obtained fused embedding features using an encoder (Transformer) to obtain a document context-aware representation includes:
[0028] Obtaining a unified sequence ;
[0029] Using Transformer to process uniform sequences Encode to obtain a document context-aware representation ,in, , express The encoding.
[0030] Preferably, obtaining the global document vector of the target document using multimodal coding includes:
[0031] Add corresponding markers within each uniform sequence ;
[0032] Using Transformer to mark Encode to obtain hidden state features ;
[0033] A global document vector is constructed by utilizing hidden state features and document context-aware representation. ;
[0034] The process of obtaining the initial review result for each document under each review condition based on the global document vector includes:
[0035] It has The review criteria that require independent judgment, specifically for the first... Each review condition is used to perform a linear transformation on the global document vector to obtain the transformed global document vector. ;in, Indicates the first The first review condition applies to the first document vector within the global document vector. The weight of each element; Indicates the first The bias vector of each audit condition; ;
[0036] Regarding the first Based on the review criteria, the probability of obtaining initial approval for the target document is determined. ,in, This represents the activation function. .
[0037] Preferably, obtaining the initial review result for each document under each review condition based on the global document vector further includes:
[0038] Considering the dependency between the two review conditions, the global document vector is transformed as follows: ;in, Indicates condition With conditions The correlation, Indicates the first The first review condition applies to the first document vector within the global document vector. The weight of each element;
[0039] Regarding the first Each review criterion determines the probability that the target document will pass the initial review. ;
[0040] Calculate the uncertainty of obtaining probability estimates .
[0041] Preferably, extracting the evidence chain from the multimodal encoding of the target document includes:
[0042] Generate a corresponding static query vector for each review condition. , ; Indicates the first ID for each audit condition; This indicates a learnable embedding layer;
[0043] Based on the description text of the preliminary review results and review conditions, a multilayer perceptron is used. Generate dynamic query vectors ;in, Indicates the first Text description of each review condition Convert to a description vector; This indicates vector concatenation;
[0044] For each Calculate the relevance score based on the review criteria. ; Indicates the first The weight of each review condition;
[0045] To ensure the coherence of evidence capture, a local window is introduced. ;
[0046] In computational document context-aware representation, the first arrive The average of the elements ;
[0047] use The corrected correlation score yields the corrected result. ;
[0048] right Perform Top-K sparsity processing to obtain attention weights ;
[0049] Based on attention weights Select evidence unit ;in, This represents the attention weight threshold. This represents the highest attention weight in document context-aware representation. One element;
[0050] Merging evidence units that are spatially similar and semantically similar, including:
[0051] The evidence unit is an element within a document context-aware representation;
[0052] Calculate the similarity measure of evidence units. ;in, Similarity weight; Indicates the first The coordinates of the word border. Indicates the first The word element and the first The variance of word-to-word distance; express Cosine similarity;
[0053] if, ,but, and Combined into one evidence cluster; Indicates an approximate threshold;
[0054] Calculate the weights of the evidence clusters ,in, Indicates a cluster of evidence. Indicates the number of elements contained in the evidence cluster;
[0055] Regarding the first Based on the review criteria, the evidence cluster with the highest weight is obtained, and the evidence unit with the highest attention weight is selected from the evidence cluster as representative evidence;
[0056] Using representative evidence, evidence cluster weights, and the timing of word occurrences, a method is constructed targeting the first... The chain of evidence for each review condition: ;
[0057] in, Indicates the first Representative evidence for each cluster of evidence, Indicates the first The weight of each evidence cluster, Indicates and The occurrence time of the corresponding word or image block; ; Indicates the number of evidence units;
[0058] Mapping representative evidence from each evidence cluster back to the target document, including:
[0059] The Lookup algorithm is used to query the lexical units of the target document that represent the evidence.
[0060] Obtain the corresponding image-text aligned representation based on the lexical units of the target document;
[0061] Image patches associated with lexical units are obtained based on image-text alignment representation;
[0062] Obtain the coordinates of the tokens representing the evidence and identify the positions of the tokens in the target document;
[0063] The obtained lexical units, images, and the positions of lexical units in the target document are output to form the explanatory text of the evidence chain.
[0064] Preferably, the step of correcting the preliminary review result using the chain of evidence includes:
[0065] Get the The chain of evidence for each review condition;
[0066] Calculation obtained The value of the chain of evidence ;
[0067] Constructing the corrected input vector ;
[0068] Obtain the corrected global document vector using the corrected input vector. ;
[0069] based on Calculate to obtain the first The probability of the target text passing the review under certain review conditions. .
[0070] Preferred options also include:
[0071] Identifying and optimizing low-confidence chains of evidence, including:
[0072] Get the The chain of evidence for each review condition;
[0073] In the weight calculation, the random deactivation Dropout algorithm is applied multiple times to obtain... Weight distribution: ;
[0074] Calculate attention concentration Among them, attention distribution entropy ;
[0075] Calculate attention stability ; Indicates the first Each weight distribution value; This represents the average value of the weighted distribution. Indicates JS divergence;
[0076] Calculate the specificity of the evidence unit relative to the target document. ;in, Indicates representative evidence Average of target document features Cosine similarity;
[0077] Construct attention weight vector Attention weights are input into a pre-trained neural network to obtain the confidence level of the evidence unit. ;
[0078] if If the evidence unit is determined to be a low-confidence unit, then the evidence chain including the low-confidence evidence unit is a low-confidence evidence chain.
[0079] Calculate the gradient of the attention weights with respect to the final prediction. ;
[0080] Adjust the attention weights along the gradient direction to obtain the adjusted attention weights. , Indicates the activation function; Represents the learning coefficient;
[0081] Constructing evidence diagrams ;in, This represents a set of nodes, where each node is an evidence unit. Let represent the set of edges, where each edge is a similarity measure of the evidence units;
[0082] By utilizing the propagation of attention weights within the evidence graph, updated evidence units are obtained. , Indicates the activation function; Indicates the propagation coefficient;
[0083] Using the updated evidence units and adjusted attention weights, the confidence level of the updated evidence units is calculated to obtain... ;
[0084] if If the chain of evidence is correct, then the adjusted chain of evidence will be used; otherwise, repeat the above steps.
[0085] A deep learning-based intelligent material review system is provided, wherein the system is used to execute the deep learning-based intelligent material review method.
[0086] A computer-readable storage medium storing a computer program that is executed by a processor to implement the aforementioned deep learning-based intelligent material review method.
[0087] The beneficial effects of this invention are:
[0088] 1. This invention maps multimodal features into the same space and then encodes a unified sequence through an attention mechanism set within the Transformer to obtain a document context-aware representation containing all modal features, thereby achieving deep fusion of multimodal features.
[0089] 2. This invention utilizes document context-aware representation to generate an initial review result that does not rely on a specific chain of evidence. Elements relevant to the current review conditions are extracted from the target document vector to form a chain of evidence, and the initial review result is corrected based on this chain of evidence to improve the accuracy and interpretability of the review result. Attached Figure Description
[0090] Figure 1 This is a flowchart of a deep learning-based intelligent material review method according to the present invention. Detailed Implementation
[0091] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious modifications will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.
[0092] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number. Example 1:
[0093] refer to Figure 1 The technical solution provided by this invention is: a material intelligent review method based on deep learning, the method comprising:
[0094] Step 1: Obtain the target document, extract multimodal features from the target document, generate fused embedding features, and perform multimodal encoding on the fused embedding features. This specifically includes the following steps:
[0095] Multimodal features are extracted from the target document using a pre-trained multimodal feature extraction model, and the obtained multimodal features are used to construct fused embedding features.
[0096] The obtained fused embedding features are encoded using the Transformer encoder to obtain a document context-aware representation, specifically:
[0097] Obtaining a unified sequence ;
[0098] Using Transformer to process uniform sequences Encode to obtain a document context-aware representation ,in, , express The encoding.
[0099] In this embodiment, the encoder Transformer includes L layers, each of which includes a multi-head attention mechanism and a feedforward neural network.
[0100] Obtain a pre-trained multimodal feature extraction model, such as LayoutLMv3;
[0101] By using the text feature extraction layer in a multimodal feature extraction model, the text in the target document is segmented into tokens, and each token is converted into a token sequence. ;in, Indicates the first Each word element, Indicates the number of lexical units;
[0102] Identify the bounding box coordinates of each lexical unit, and after normalization, construct the lexical layout feature sequence of the target text. ;in Indicates the first The coordinates of the word border. Indicates the first The width and height of the word border;
[0103] The images in the document are divided into multiple image blocks to form an image block sequence. ,in, Indicates the first Image blocks;
[0104] Get text embedding vector ,in, ; Indicates word embedding;
[0105] Get layout embedding vector ,in, Layout features ; Represents a linear transformation, for example By embedding vectors in the layout, spatial location and structural information of lexical units can be provided, such as titles, tables, and signature areas.
[0106] Obtain image embedding vector ,in, ; It represents a convolutional neural network; it provides visual style information such as texture, font, and stamp through image embedding vectors.
[0107] By using the cross-modal attention mechanism layer in the multimodal feature extraction model, text embedding vectors, layout embedding vectors, and image embedding vectors are mapped to the same dimension (i.e., aligned), resulting in fused embedding features, including:
[0108] Map the elements embedded in the text vector to dimension . Given a context space, obtain the text embedding vector elements within that context space. ;
[0109] Map the elements embedded in the layout vector to dimensions [1]. Within the context space, obtain the layout embedding vector elements within the context space. ;
[0110] Map the elements of the image embedding vector to a dimension of... Within the context space, obtain the image embedding vector elements within the context space. ;in, , , Represents the projection matrix; , , Indicates bias;
[0111] Alignment through attention mechanisms and Generate image-text aligned representation ;
[0112] Obtain the fused embedding features for each lexical unit. ,in, , , Represents the normalized result , , ;
[0113] Sort all fused embedded features according to the reading order to obtain a unified sequence. .
[0114] Step 2: Obtain the global document vector of the target document using multimodal coding. Based on the global document vector, obtain the initial review result for each document under each review condition. This includes the following steps:
[0115] Add corresponding markers within each uniform sequence ; Using Transformer to mark Encode to obtain hidden state features ;
[0116] A global document vector is constructed by utilizing hidden state features and document context-aware representation. ;
[0117] The process of obtaining the initial review result for each document under each review condition based on the global document vector includes:
[0118] It has The review criteria that require independent judgment, specifically for the first... Each review condition is used to perform a linear transformation on the global document vector to obtain the transformed global document vector. ;in, Indicates the first The first review condition applies to the first document vector within the global document vector. The weight of the i-th element represents the weight of the i-th element within the global document vector. The element is the first The importance of each review condition; Indicates the first The bias vector of each audit condition; .
[0119] Regarding the first Based on the review criteria, the probability of obtaining initial approval for the target document is determined. ,in, This represents the activation function. .
[0120] In some preferred embodiments, it further includes:
[0121] Considering the dependency between the two review conditions, the global document vector is transformed as follows: ;in, Indicates condition With conditions The correlation, that is, the condition conditions The impact, Indicates the first The first review condition applies to the first document vector within the global document vector. The weight of each element.
[0122] Regarding the first Each review criterion determines the probability that the target document will pass the initial review. ; Calculate the uncertainty of obtaining the probability estimate The larger this value, the more uncertain the estimated probability is, and the more evidence is needed.
[0123] Step 3: Extract the chain of evidence from the multimodal encoding of the target document, and use the chain of evidence to revise the preliminary review results. This includes the following steps:
[0124] Step 3.1: Extract the chain of evidence from the multimodal encoding of the target document, including:
[0125] Generate a corresponding static query vector for each review condition. , This is used to query content in the target document that is related to the aforementioned review criteria; Indicates the first ID for each audit condition; This indicates a learnable embedding layer;
[0126] Based on the description text of the preliminary review results and review conditions, a multilayer perceptron is used. Generate dynamic query vectors ;in, Indicates the first Text description of each review condition Convert to a description vector; This indicates vector concatenation;
[0127] For each Calculate the relevance score based on the review criteria. ; Indicates the first The weight of each review condition;
[0128] To ensure the coherence of evidence capture, a local window is introduced. ; The first in the computation of document context-aware representation arrive The average of the elements .
[0129] use The corrected correlation score yields the corrected result. ;right Perform Top-K sparsity processing to obtain attention weights ;
[0130] Based on attention weights Select evidence unit ;in, This represents the attention weight threshold (e.g., 0.05). This represents the highest attention weight in document context-aware representation. There are M elements to ensure that at least M are selected.
[0131] Merging evidence units that are spatially similar and semantically similar, including:
[0132] The evidence unit is an element within a document context-aware representation;
[0133] Calculate the similarity measure of evidence units. ;in, Similarity weight; Indicates the first The coordinates of the word border. Indicates the first The word element and the first The variance of word-to-word distance; express Cosine similarity;
[0134] if, ,but, and Combined into one evidence cluster; Indicates an approximate threshold;
[0135] Calculate the weights of the evidence clusters ,in, Indicates a cluster of evidence. Indicates the number of elements contained in the evidence cluster;
[0136] Regarding the first Based on the review criteria, the evidence cluster with the highest weight is obtained, and the evidence unit with the highest attention weight is selected from the evidence cluster as representative evidence;
[0137] Using representative evidence, evidence cluster weights, and the time of word occurrence (reading time order), a method is constructed targeting the first... The chain of evidence for each review condition:
[0138] ;in, Indicates the first Representative evidence for each cluster of evidence, Indicates the first The weight of each evidence cluster, Indicates and The occurrence time of the corresponding word or image block; ; Indicates the number of evidence units;
[0139] Mapping representative evidence from each evidence cluster back to the target document, including:
[0140] The Lookup algorithm is used to query the lexical units of the target document that represent the evidence.
[0141] Obtain the corresponding image-text aligned representation based on the lexical units of the target document;
[0142] Image patches associated with lexical units are obtained based on image-text alignment representation;
[0143] Obtain the coordinates of the tokens representing the evidence and identify the positions of the tokens in the target document;
[0144] The obtained lexical units, images, and their positions in the target document are output to form the explanatory text of the evidence chain. For example, the evidence chain for the applicant's identity verification in the target document is the organization name, with a weight of 0.95; the organization name matches the official seal, appears on the first line, and appears 1 second later.
[0145] Step 3.1: Revise the preliminary review results using the chain of evidence, including:
[0146] Get the The chain of evidence for each review condition;
[0147] Calculation obtained The value of the chain of evidence ;
[0148] Constructing the corrected input vector ;
[0149] Obtain the corrected global document vector using the corrected input vector. ;
[0150] based on Calculate to obtain the first The probability of the target text passing the review under certain review conditions. . Example 2:
[0151] If the confidence level of the evidence chain is low, it indicates that the evidence chain's interpretation of the examination results is insufficiently accurate. Therefore, to ensure the accuracy of the evidence chain's interpretation of the examination results, it is necessary to identify and optimize the evidence chain with low confidence. To this end, based on Embodiment 1, we propose the following technical solution:
[0152] Identifying and optimizing low-confidence chains of evidence, including:
[0153] Get the The chain of evidence for each review condition;
[0154] In the weight calculation, the random deactivation Dropout algorithm is applied multiple times to obtain... Weight distribution: ;
[0155] Calculate attention concentration Among them, attention distribution entropy ;
[0156] Calculate attention stability ; Indicates the first Each weight distribution value; This represents the average value of the weighted distribution. Indicates JS divergence;
[0157] Calculate the specificity of the evidence unit relative to the target document. ;in, Indicates representative evidence Average of target document features Cosine similarity;
[0158] Construct attention weight vector Attention weights are input into a pre-trained neural network to obtain the confidence level of the evidence unit. ;
[0159] if If the evidence unit is determined to be a low-confidence unit, then the evidence chain including the low-confidence evidence unit is a low-confidence evidence chain.
[0160] Calculate the gradient of the attention weights with respect to the final prediction. ;
[0161] Adjust the attention weights along the gradient direction to obtain the adjusted attention weights. , This represents the activation function (i.e., sparse max). Represents the learning coefficient;
[0162] Constructing evidence diagrams ;in, This represents a set of nodes, where each node is an evidence unit. Let represent the set of edges, where each edge is a similarity measure of the evidence units;
[0163] By utilizing the propagation of attention weights within the evidence graph, updated evidence units are obtained. , Indicates the activation function; Indicates the propagation coefficient;
[0164] Using the updated evidence units and adjusted attention weights, the confidence level of the updated evidence units is calculated to obtain... ;
[0165] if If the chain of evidence is correct, then the adjusted chain of evidence will be used; otherwise, repeat the above steps.
[0166] The present invention also provides a deep learning-based intelligent material review system, the system being used to execute the aforementioned deep learning-based intelligent material review method.
[0167] The present invention also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the aforementioned deep learning-based intelligent material review method.
[0168] The processes described above with reference to the flowcharts in the embodiments disclosed in this invention can be implemented as computer software programs. The embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical fibers, RF, etc., or any suitable combination thereof.
[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0170] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the principles described, the implementation of the present invention may have any changes or modifications.
Claims
1. A deep learning-based intelligent material review method, characterized in that, The method includes: Obtain the target document, extract multimodal features from the target document, generate fused embedding features, and perform multimodal encoding on the fused embedding features; Multimodal coding is used to obtain the global document vector of the target document. Based on the global document vector, the initial review result of each document under each review condition is obtained. Extract the chain of evidence from the multimodal encoding of the target document, and use the chain of evidence to correct the preliminary review results.
2. The intelligent material review method based on deep learning according to claim 1, characterized in that, The steps of acquiring the target document, extracting multimodal features from the target document, generating fused embedding features, and performing multimodal encoding on the fused embedding features include: Multimodal features are extracted from the target document using a pre-trained multimodal feature extraction model, and the obtained multimodal features are used to construct fused embedding features. The obtained fused embedding features are encoded using the Transformer encoder to obtain a document context-aware representation.
3. The intelligent material review method based on deep learning according to claim 2, characterized in that, The method involves extracting multimodal features from the target document using a pre-trained multimodal feature extraction model, and then using the obtained multimodal features to construct fused embedding features, including: Obtain a pre-trained multimodal feature extraction model; By using the text feature extraction layer in a multimodal feature extraction model, the text in the target document is segmented into tokens, and each token is converted into a token sequence. ;in, Indicates the first Each word element, Indicates the number of lexical units; Identify the bounding box coordinates of each lexical unit, and after normalization, construct the lexical layout feature sequence of the target text. ； in Indicates the first The coordinates of the word border. Indicates the first The width and height of the word border; The images in the document are divided into multiple image blocks to form an image block sequence. ,in, Indicates the first Image blocks; Get text embedding vector ,in, ; Indicates word embedding; Get layout embedding vector ,in, Layout features ; Represents a linear transformation; Obtain image embedding vector ,in, ; Represents a convolutional neural network; By using the cross-modal attention mechanism layer in the multimodal feature extraction model, text embedding vectors, layout embedding vectors, and image embedding vectors are mapped to the same dimension to obtain fused embedding features, including: Map the elements embedded in the text vector to dimension . Given a context space, obtain the text embedding vector elements within that context space. ; Map the elements embedded in the layout vector to dimensions [1]. Within the context space, obtain the layout embedding vector elements within the context space. ; Map the elements of the image embedding vector to a dimension of... Within the context space, obtain the image embedding vector elements within the context space. ;in, , , Represents the projection matrix; , , Indicates bias; Alignment through attention mechanisms and Generate image-text aligned representation ; Obtain the fused embedding features for each lexical unit. ,in, , , Represents the normalized result , , ; Sort all fused embedded features according to the reading order to obtain a unified sequence. ; The process of encoding the obtained fused embedding features using an encoder (Transformer) to obtain a document context-aware representation includes: Obtaining a unified sequence ; Using Transformer to process uniform sequences Encode to obtain a document context-aware representation ,in, , express The encoding.
4. The intelligent material review method based on deep learning according to claim 3, characterized in that, The process of obtaining the global document vector of the target document using multimodal coding includes: Add corresponding markers within each uniform sequence ; Using Transformer to mark Encode to obtain hidden state features ; A global document vector is constructed by utilizing hidden state features and document context-aware representation. ； The process of obtaining the initial review result for each document under each review condition based on the global document vector includes: It has The review criteria that require independent judgment, specifically for the first... Each review condition is used to perform a linear transformation on the global document vector to obtain the transformed global document vector. ;in, Indicates the first The first review condition applies to the first document vector within the global document vector. The weight of each element; Indicates the first The bias vector of each audit condition; ; Regarding the first Based on the review criteria, the probability of obtaining an initial pass for the target document is determined. ,in, This represents the activation function. .
5. The intelligent material review method based on deep learning according to claim 4, characterized in that, The method of obtaining the initial review result of each document under each review condition based on the global document vector also includes: Considering the dependency between the two review conditions, the global document vector is transformed as follows: ;in, Indicates condition With conditions The correlation, Indicates the first The first review condition applies to the first document vector within the global document vector. The weight of each element; Regarding the first Each review criterion determines the probability that the target document will pass the initial review. ; Calculate the uncertainty of obtaining probability estimates .
6. The intelligent material review method based on deep learning according to claim 5, characterized in that, Extracting the evidence chain from the multimodal encoding of the target document includes: Generate a corresponding static query vector for each review condition. , ; Indicates the first ID for each audit condition; This indicates a learnable embedding layer; Based on the description text of the preliminary review results and review conditions, a multilayer perceptron is used. Generate dynamic query vectors ； in, Indicates the first Text description of each review condition Convert to a description vector; This indicates vector concatenation; For each Calculate the relevance score based on the review criteria. ; Indicates the first The weight of each review condition; To ensure the coherence of evidence capture, a local window is introduced. ; In computational document context-aware representation, the first arrive The average of the elements ; use The corrected correlation score yields the corrected result. ; right Perform Top-K sparsity processing to obtain attention weights ; Based on attention weights Select evidence unit ;in, This represents the attention weight threshold. This represents the highest attention weight in document context-aware representation. One element; Merging evidence units that are spatially similar and semantically similar, including: The evidence unit is an element within a document context-aware representation; Calculate the similarity measure of evidence units. ； in, Similarity weight; Indicates the first The coordinates of the word border. Indicates the first The word element and the first The variance of word-to-word distance; express Cosine similarity; if, ,but, and Combined into a single evidence cluster; Indicates an approximate threshold; Calculate the weights of the evidence clusters ,in, Indicates a cluster of evidence. Indicates the number of elements contained in the evidence cluster; Regarding the first Based on the review criteria, the evidence cluster with the highest weight is obtained, and the evidence unit with the highest attention weight is selected from the evidence cluster as representative evidence; Using representative evidence, evidence cluster weights, and the timing of word occurrences, a method is constructed targeting the first... The chain of evidence for each review condition: ； in, Indicates the first Representative evidence for each cluster of evidence, Indicates the first The weight of each evidence cluster, Indicates and The occurrence time of the corresponding word or image block; ; Indicates the number of evidence units; Mapping representative evidence for each evidence cluster back to the target document, including: The Lookup algorithm is used to query the lexical units of the target document that represent the evidence. Obtain the corresponding image-text aligned representation based on the lexical units of the target document; Image patches associated with lexical units are obtained based on image-text alignment representation; Obtain the coordinates of the tokens representing the evidence and identify the positions of the tokens in the target document; The obtained lexical units, images, and the positions of lexical units in the target document are output to form the explanatory text of the evidence chain.
7. The intelligent material review method based on deep learning according to claim 6, characterized in that, The method of using the chain of evidence to correct the preliminary review results includes: Get the The chain of evidence for each review condition; Calculation obtained The value of the chain of evidence ; Constructing the corrected input vector ; Obtain the corrected global document vector using the corrected input vector. ; based on Calculate to obtain the first The probability of the target text passing the review under certain review conditions. .
8. The intelligent material review method based on deep learning according to claim 7, characterized in that, Also includes: Identify and optimize low-confidence chains of evidence, including: Get the The chain of evidence for each review condition; In the weight calculation, the random deactivation Dropout algorithm is applied multiple times to obtain... Weight distribution: ; Calculate attention concentration Among them, attention distribution entropy ; Calculate attention stability ; Indicates the first Each weight distribution value; This represents the average value of the weighted distribution. Indicates JS divergence; Calculate the specificity of the evidence unit relative to the target document. ； in, Indicates representative evidence Average of target document features Cosine similarity; Construct attention weight vector Attention weights are input into a pre-trained neural network to obtain the confidence level of the evidence unit. ; if If the evidence unit is determined to be a low-confidence unit, then the evidence chain including the low-confidence evidence unit is a low-confidence evidence chain. Calculate the gradient of the attention weights with respect to the final prediction. ; Adjust the attention weights along the gradient direction to obtain the adjusted attention weights. , Indicates the activation function; Represents the learning coefficient; Constructing evidence diagrams ;in, This represents a set of nodes, where each node is an evidence unit. Let represent the set of edges, where each edge is a similarity measure of the evidence units; By utilizing the propagation of attention weights within the evidence graph, updated evidence units are obtained. , Indicates the activation function; Indicates the propagation coefficient; Using the updated evidence units and adjusted attention weights, the confidence level of the updated evidence units is calculated to obtain... ; if If the above conditions are met, the adjusted chain of evidence will be used; otherwise, repeat the above steps.
9. A deep learning-based intelligent material review system, characterized in that, The system is used to execute the deep learning-based intelligent material review method described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the intelligent material review method based on deep learning as described in any one of claims 1-8.