Intelligent document comparison method based on multi-modal feature fusion and dynamic weight distribution

Abstract

The invention discloses an intelligent document comparison method based on multi-modal feature fusion and dynamic weight distribution, which is used for carrying out feature extraction on various types of information in documents in order to meet the comparison requirements of different types of documents; all modal features are aligned, a trained feature fusion model is adopted to carry out feature fusion on the extracted features, feature vectors are output and used for further comparing the overall content of the document, a multi-modal technical means is adopted, and multi-type comparison objects are provided, so that the comparison of the document is not limited to text features, but also to text features; document comparison can be carried out in combination with semantics and context structures; in order to improve the stability and effectiveness of multi-modal feature fusion data, a dynamic weight distribution mechanism is added, weight distribution is carried out on a modal fusion process, and weighted fusion is carried out on each modal feature through a trained feature fusion model, so that the document comparison precision and practicability are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and document processing technology, and in particular to an intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation. Background Technology

[0002] In modern life, documents are involved in every aspect of people's lives. A large number of documents are generated and transmitted in electronic form, such as contracts, books, and papers. With the development of the times, the content of documents is no longer limited to text; it may also include images, tables, audio, and video. For computers to understand this information, they must first perform relevant transformations. We can represent this data modally, that is, convert data from different senses or interaction methods into a form that computers can understand and process, so as to perform subsequent calculations, analysis, and fusion.

[0003] Multimodal joint representation is a method that maps information from multiple modalities (such as text, images, and sound) into a unified multimodal space. It uses a neural network model to fuse data from different modalities, generating a unified representation that incorporates information from multiple modalities. This representation not only preserves the key information of each modality but also establishes connections between different modalities, thus supporting cross-modal tasks such as multimodal semantic analysis and language recognition of emojis. Multimodal fusion fully utilizes the complementarity between different modalities, integrating information extracted from different modalities into a stable multimodal representation.

[0004] In the real world, due to external environmental factors, the quality of features extracted from different modalities often varies. Some studies, both empirically and theoretically, have shown that multimodal fusion may have shortcomings, such as dealing with imbalanced, noisy, or corrupted multimodal data. Therefore, to fully unlock the value of each modality and mitigate the impact of low-quality multimodal data, introducing dynamic fusion mechanisms has become a promising approach for obtaining reliable predictions.

[0005] Chinese invention patent CN119670724A discloses a method, device, and electronic device for comparing and detecting contract texts. It identifies character pairs between two contract texts, determines whether the two characters in each pair are identical, and identifies the pair as an anomaly if they are not identical. Each character is represented by at least one 8-bit byte. The information entropy and similarity of any anomaly pair are then calculated to determine text differences and errors. Its shortcomings include: 1. Focusing on surface-level text differences and using fixed feature extraction, it ignores semantic, format, and structural information, leading to inaccurate or uninterpretable comparison results. 2. In real life, documents are frequently updated, and document content may include text, tables, and images; therefore, the comparison cannot provide a comprehensive document comparison technology. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, there is a need for a document comparison method that combines multimodal features such as text semantics, structural information, and visual style to adapt to the comparison needs of different documents and can dynamically adjust feature weights, thereby improving the accuracy and practicality of the comparison.

[0007] This invention provides an intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation. To meet the comparison needs of different document types, it extracts features from various types of information in the document, aligns the features of each modality, and uses a trained feature fusion model to fuse the extracted features and output feature vectors for further comparison of the overall document content. By employing multimodal techniques, it provides multiple types of comparison objects, enabling document comparison not only to text features but also to incorporate semantics and contextual structure. To improve the stability and effectiveness of multimodal feature fusion data, a dynamic weight allocation mechanism is added to assign weights to the modality fusion process, and the trained feature fusion model performs weighted fusion of each modality feature to improve the accuracy and practicality of document comparison.

[0008] The specific contents of this invention are as follows: An intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation includes the following steps: S1. Preprocess the document to be compared and extract its multimodal features; S2. Perform cross-modal alignment on the multimodal features to map them to a unified feature space; S3. Based on the aligned multimodal features, a corresponding dynamic weight is determined for each modality feature in the multimodal features through a trained feature fusion model. The multimodal features are then weighted and fused using the dynamic weights to generate a fused feature vector. S4. Based on the fused feature vector, perform comparative analysis on the documents to be compared to determine the similarity and differences between the documents.

[0009] Preferably, the multimodal features include semantic features, structural features, visual features, and metadata features.

[0010] Preferably, the semantic features are embedded in paragraph-level text using a pre-trained language model.

[0011] Preferably, the visual features are extracted using a visual model to extract document layout features.

[0012] Preferably, the trained feature fusion model includes a self-attention module; the method for determining the dynamic weights includes: calculating the contribution of each modality feature based on the output of the self-attention module, and determining its corresponding dynamic weight based on the contribution.

[0013] Preferably, the method for determining the dynamic weights further includes: using the Gumbel-Softmax method to achieve differentiable discrete weight allocation; and using weight entropy regularization terms to prevent modal degradation.

[0014] Preferably, the dynamic weighting method further includes at least one of the following self-calibration steps: Limit the magnitude of dynamic weight changes; Set the valid range of values ​​for the dynamic weights; When the input signal of a certain modality is abnormal, the dynamic weights corresponding to that modality are reset or frozen.

[0015] Preferably, the trained feature fusion model further includes: Local sequence dependencies in document content are captured using a bidirectional long short-term memory network. Graph attention networks are used to model the global structural relationships of documents.

[0016] Beneficial effects This invention employs a pre-trained network model to extract multimodal features from documents, aligns and fuses these features, and dynamically adjusts the weights of each feature to promote better fusion, providing a high-quality and more stable feature vector for document comparison. By extracting multimodal features and performing modal alignment and fusion, the system can be applied to more document types. Furthermore, the multimodal feature comparison method allows for document comparison beyond just semantics, incorporating contextual semantics, document structure, and other information. Feature alignment and fusion ensures that identical or related content has similar representations across different modalities, improving data reliability and stability. Multimodal collaborative processing also increases system processing speed. The pre-trained network model enhances the system's adaptability; during use, the network models continuously self-train, expanding its applicability, adapting to more document types, and improving accuracy. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation provided in a preferred embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings. In the description of this invention, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention.

[0019] First Embodiment like Figure 1 As shown, firstly, the documents to be compared are preprocessed, and the multimodal features of the documents to be compared are extracted.

[0020] Preprocessing the documents to be compared can be understood as extracting information from the entire document. If the input is in image format, OCR recognition can be performed, and text parsing and structure extraction can be performed on the document to extract structural information such as paragraphs, headings, lists, and tables. The extracted information is then transformed into modal features. Compared with traditional document comparison methods, it can be applied to a wider range of document types, and by integrating multimodal features, it can achieve a more comprehensive and accurate comparison.

[0021] Extracting multimodal features from the documents to be compared involves encoding and processing each feature, converting the relevant data into a form that computers can understand and process.

[0022] Then, cross-modal alignment of multimodal features is performed to map them to a unified feature space. Since the feature extraction methods are different, the semantic spaces of different modalities may be different. Cross-modal alignment of multimodal features aims to establish a common understanding of different modalities and align the semantic spaces of different modal data (such as images, text, audio, etc.). That is, after extracting features from different modal data with similar meanings, similar feature vectors are obtained, which can better understand and analyze the data and improve the accuracy and robustness of the model.

[0023] In some preferred embodiments, cross-modal alignment can map the extracted features to a unified feature space by using linear projection or nonlinear transformation. Alignment loss (such as contrast loss or mutual information maximization) can also be used to constrain the learning of multimodal features to ensure that the same or related content has similar representations in different modalities.

[0024] Cross-modal alignment is mainly divided into two categories: explicit alignment and implicit alignment. Explicit alignment directly establishes a correspondence between different modalities; while implicit alignment achieves cross-modal alignment implicitly through internal model mechanisms, including attention alignment and semantic alignment. Attention alignment dynamically generates weight vectors between different modalities through attention mechanisms, achieving weighted fusion of cross-modal information. For example, a network model based on the transformer architecture uses automatically learned attention distributions between images and text to achieve modal alignment; another example is a BERT-based network model, which, in text-image retrieval, combines BERT's pre-trained representations and attention mechanisms to implicitly align text queries and image content.

[0025] Secondly, the aligned multimodal features are input into the trained feature fusion model for multimodal feature fusion. The trained model determines the corresponding dynamic weights for each modality feature in the multimodal features, and the multimodal features are weighted and fused using the dynamic weights to generate a fused feature vector.

[0026] Multimodal fusion is the process of integrating input information from different modalities to achieve multimodal feature extraction. By fusing features from different modalities, it can fully utilize the complementarity between different modalities and integrate information extracted from different modalities into a stable multimodal representation.

[0027] Multimodal model fusion methods fall into two main categories: model-independent methods and model-based methods. The difference lies in that the former does not directly rely on specific deep learning methods, while the latter explicitly solves the multimodal fusion problem using deep learning models, such as neural network models based on the transformer architecture. This invention employs a pre-trained feature fusion model, aiming to determine corresponding dynamic weights for each modality feature in the multimodal features. It uses reinforcement learning or neural network regression models, trained based on task feedback or user click data, to achieve context-based adaptive weight optimization. In traditional feature fusion, static weight allocation strategies are often used, but the quality of features extracted from different modes usually varies. By dynamically allocating weights, the model can dynamically adjust the weights of each modality based on feedback during training, thereby achieving a better balance between model performance and training stability. This method overcomes the limitations of traditional static weight allocation and is particularly suitable for scenarios with complex tasks and dynamically changing data distribution, improving the accuracy and robustness of document comparison.

[0028] Finally, the documents to be compared are analyzed based on the fused feature vectors to determine the similarity and differences between the documents.

[0029] After feature fusion is completed, the model outputs the fused feature vector. The multimodal features after dynamic weight allocation are then input into the comparison network to determine the similarity between documents. If there are differences between documents, the key differences are checked.

[0030] To determine the similarity of documents, it is better to use similarity metrics for document matching, such as cosine similarity and Euclidean distance.

[0031] Second Embodiment In some preferred embodiments, the intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation includes semantic features, structural features, visual features, and metadata features.

[0032] Semantic features refer to the representation and understanding of semantic information obtained through different modalities (such as text, images, audio, etc.).

[0033] Structural features include information such as the document's hierarchical structure, heading hierarchy, and paragraph order.

[0034] Visual features involve extracting and fusing visual information from multiple modalities (such as images and text) to gain a deeper understanding of the meaning of the data.

[0035] Metadata features include auxiliary information such as document creation time, author, and version number.

[0036] By setting multimodal features, we can adapt to different document types, classify different categories of signals, facilitate the extraction of similar or nearly identical features, maintain information consistency, and the comparison content is not limited to text content. We can combine contextual semantics and article structure for comparative analysis.

[0037] The metadata features are optimally configured, allowing for comparison of document creation information and providing a more convenient management approach.

[0038] Third Embodiment In some preferred embodiments, the intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation also includes embedding paragraph-level text representations using a pre-trained language model to represent semantic features.

[0039] By inputting text into a pre-trained model, representation vectors for each word or sentence can be obtained. Using pre-trained semantic models as feature extractors is an important method to improve the performance of natural language processing tasks. Common pre-trained semantic models include ord2Vec, GloVe, ELMo, BERT, and GPT. When selecting a suitable pre-trained model, factors such as the training corpus, training task, and model structure need to be considered to match the requirements of the target task. In this invention, the following models are preferred for pre-training: BERT-base, RoBERTa, and DeBERTa.

[0040] Embedding representations are commonly used in Natural Language Processing (NLP). Word embeddings are typically used to convert words or phrases into vectors so that models can assist in understanding the semantics of language. This is particularly important in fields such as text classification, sentiment analysis, machine translation, question answering systems, and language models. Currently, the most popular representation method is word embedding. However, due to the complexity of semantics and context, especially in NLP, polysemy and contextual relevance are crucial for semantic understanding. To better understand contextual semantics, this invention preferably uses a pre-trained language model to embed semantic features into paragraph-level text. The model then performs relevant learning, facilitating a better understanding of the structure and meaning of the input text.

[0041] Fourth embodiment In some preferred embodiments, the intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation also includes extracting document layout features using a visual model; layout features include the document's page layout, font, style, font size, color, etc.

[0042] Visual representation involves two main issues: first, how to reasonably model visual input features; and second, how to fully learn representations through pre-training. These two points are fundamental to completing specific algorithmic tasks based on vision. Many convolutional neural network models are applied to visual representation, such as VGG and ResNet. The purpose of using visual models to extract features is to model vision and natural language into the same representation space and fuse them, achieving interoperability between natural language and visual semantics, thereby enabling downstream application tasks. This invention preferably uses visual models to extract document layout features, which can adapt to different document comparison needs. It can compare not only document semantics but also page layout, font, style, font size, color, etc.

[0043] Fifth Embodiment In some preferred embodiments, the intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation includes a self-attention module in the trained feature fusion model; the method for determining the dynamic weight includes: calculating the contribution of each modal feature based on the output of the self-attention module, and determining its corresponding dynamic weight based on the contribution.

[0044] The trained fusion feature module uses a self-attention mechanism based on the Transformer architecture to fuse multimodal features and establish matching relationships between paragraphs or elements in documents. It uses an attention mechanism to dynamically capture the correlation between different modalities and adopts a multi-head attention mechanism to learn the interaction relationship between modalities, providing more attention to better perceive contextual relationships.

[0045] The aligned feature sequences are used as input, and each layer is encoded using a multi-head attention mechanism to obtain a deep fusion multimodal feature representation. The degree of attention given to different information by the attention mechanism is reflected by weights. Each layer calculates attention weights using query vectors, key vectors, and value vectors, thereby generating a context-sensitive output representation.

[0046] The core calculation formula for the attention mechanism is: Where: Q is the query vector, K is the key vector, V is the value vector, and softmax is the normalization function used to convert the integral into a probability distribution to ensure that the sum of the weights is 1; Let be the dimension of the key vector. This is an attention score matrix, representing the similarity between each query and all keys; Scaling factor This helps prevent overscoring and maintains numerical stability. The final attention output is generated by multiplying the normalized weights by the value vector.

[0047] Employing a self-attention mechanism can better capture long-distance dependencies in the input sequence. It can dynamically generate a weight matrix based on the content of the input sequence and flexibly adjust the length of the output sequence. To adapt to different document comparison needs and improve the fusion quality of various modalities, it outputs feature vectors with higher fusion degree, which is beneficial for further document comparison and improves comparison accuracy.

[0048] Dynamic weight allocation is a technique built upon the attention mechanism. It allows the model to dynamically adjust attention weights based on different input data. Its basic principle is to dynamically adjust weights according to the importance of the input data to better represent the correlations between data points. The process includes the following steps: 1. The input data undergoes feature extraction to obtain a vector representing the data features.

[0049] 2. Calculate the importance of each feature. This step is achieved by calculating the degree of correlation between each feature and other features.

[0050] 3. Based on the calculated importance, the weights are dynamically updated, allowing for flexible adjustment of weights according to the current situation during model training.

[0051] Dynamic weight allocation based on neural networks makes full use of the model's nonlinear fitting ability and can better capture the characteristics of the data.

[0052] Dynamic weight allocation can make models more flexible and intelligent, enabling them to adjust attention allocation flexibly when processing different data, thereby improving generalization ability and adaptability.

[0053] Sixth Embodiment In some preferred embodiments, the intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation further includes the following methods for determining dynamic weights: using the Gumbel-Softmax method to achieve differentiable discrete weight allocation; and using weight entropy regularization terms to prevent modal degradation.

[0054] Gumbel-Softmax allows the process of sampling from a discrete distribution (such as a categorical distribution) in the model to become differentiable, thus allowing the model parameters to be updated using gradients during backpropagation. By introducing Gumbel distribution noise and temperature parameters, it transforms the non-differentiable discrete sampling process into a differentiable continuous approximation, thereby allowing end-to-end training using descent. It is used in neural networks for modeling and sampling discrete categorical variables.

[0055] During training, due to the different conditions of the training data, overfitting and underfitting may occur, which may cause the model output to tend to be homogenized or too diverse. By applying a weight entropy regularization term to the model, the robustness of the model is improved by adding additional constraints. Maintaining a certain degree of randomness during training can improve the exploration ability and prevent the model from converging to a local optimum too early.

[0056] In reinforcement learning, entropy regularization is a technique that increases policy randomness and improves exploration capabilities. It is commonly used in policy gradient algorithms. By adding an entropy term to the objective function, it encourages the policy to maintain a certain degree of randomness, thereby avoiding premature convergence to a local optimum.

[0057] This method is based on the probability distribution of the model output, and the specific steps are as follows: 1. Calculate the probability distribution: Based on the model's prediction results, calculate the predicted probability distribution for each sample.

[0058] 2. Calculate model entropy: Use the Shannon entropy formula to calculate model entropy.

[0059] 3. Add regularization term: Add model entropy as a regularization term to the loss function.

[0060] By combining attention mechanism weighted algorithm with multi-head attention, the weights of each modality feature are dynamically assigned, improving the effectiveness of weight allocation and adapting to the modality features obtained from different documents.

[0061] Seventh Embodiment In some preferred embodiments, the intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation further includes at least one self-correction step in determining the dynamic weights: Limit the magnitude of dynamic weight changes; Set the valid range of values ​​for the dynamic weights; When the input signal of a certain modality is abnormal, the dynamic weights corresponding to that modality are reset or frozen.

[0062] Limiting the magnitude of dynamic weight changes and employing self-correcting smoothing constraints in the dynamic region aims to smooth the weights of different loss terms as the training progresses, gradually focusing on the main objective of model performance. In effect, this is a way to control the complexity of the model, allowing it to maintain a certain degree of simplicity while fitting the data, and also preventing overfitting.

[0063] A better approach is to limit the magnitude of weight changes, such as... .

[0064] Where: new_weight is the updated weight, old_weight is the current weight value, and target_weight is the target weight value. It is a smoothing coefficient that controls the degree of mixing between the old and new weights.

[0065] Set the effective range of dynamic weight values. Ideally, the weight range should be set between [0.1, 0.9] to prevent extreme weight situations.

[0066] When the input signal of a certain modality is abnormal, the dynamic weight corresponding to that modality is reset or frozen. This can be understood as automatically reducing the modality weight value when the input of a certain modality is abnormal. For example, when a blurred image is detected, the visual weight attenuation coefficient is set to 0.2.

[0067] By setting a self-correction step for dynamic weights, the feature weights are dynamically adjusted to ensure the stability and robustness of the weight adjustment, so that key features can be given higher weights, thereby improving the sensitivity and accuracy of document comparison.

[0068] Eighth embodiment In some preferred embodiments, the trained feature fusion model of the intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation further includes: Local sequence dependencies in document content are captured using a bidirectional long short-term memory network. Graph attention networks are used to model the global structural relationships of documents.

[0069] Capturing local sequence dependencies in document content using a bidirectional long short-term memory (BiLSTM) network can be understood as employing BiLSTM to capture these dependencies. BiLSTM is a combination of forward and backward LSTMs, used in natural language processing tasks to model contextual information. LSTM models are better able to capture longer-distance dependencies. By using BiLSTM, bidirectional semantic dependencies can be captured more effectively.

[0070] Graph attention networks introduce an attention mechanism to achieve better neighbor aggregation. By learning the weights of neighbors, graph attention networks can achieve weighted aggregation of neighbors. Therefore, the attention mechanism also gives the model a certain degree of interpretability.

[0071] The trained feature fusion model is further enhanced by establishing text context attention and using a graph attention network to model the global structural relationships of the document. This enables multimodal features to consider not only the text content but also the actual context, article structure, and semantics when comparing them. Furthermore, multi-objective collaborative processing helps improve the precision and recall of document analysis.

[0072] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation, characterized in that, Including the following steps: S1. Preprocess the document to be compared and extract its multimodal features; S2. Perform cross-modal alignment on the multimodal features to map them to a unified feature space; S3. Based on the aligned multimodal features, a corresponding dynamic weight is determined for each modality feature in the multimodal features through a trained feature fusion model. The multimodal features are then weighted and fused using the dynamic weights to generate a fused feature vector. S4. Based on the fused feature vector, perform comparative analysis on the documents to be compared to determine the similarity and differences between the documents.

2. The intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation as described in claim 1, characterized in that: The multimodal features include semantic features, structural features, visual features, and metadata features.

3. The intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation as described in claim 2, characterized in that: The semantic features are represented by embedding paragraph-level text using a pre-trained language model.

4. The intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation as described in claim 2, characterized in that: The visual features are extracted using a visual model to capture document layout features.

5. The intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation as described in claim 1, characterized in that: The trained feature fusion model includes a self-attention module; The method for determining the dynamic weights includes: calculating the contribution of each modal feature based on the output of the self-attention module, and determining the corresponding dynamic weights based on the contribution.

6. The intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation as described in claim 5, characterized in that, The method for determining the dynamic weights also includes: using the Gumbel-Softmax method to achieve differentiable discrete weight allocation; and using weight entropy regularization terms to prevent modal degradation.

7. The intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation as described in claim 1, characterized in that, The method for determining the dynamic weights further includes at least one of the following self-calibration steps: Limit the magnitude of dynamic weight changes; Set the valid range of values ​​for the dynamic weights; When the input signal of a certain modality is abnormal, the dynamic weights corresponding to that modality are reset or frozen.

8. The intelligent document comparison method based on multimodal feature fusion and dynamic weight allocation as described in claim 1, characterized in that, The trained feature fusion model also includes: Local sequence dependencies in document content are captured using a bidirectional long short-term memory network. Graph attention networks are used to model the global structural relationships of documents.

Images