A multi-modal feature fusion method and device based on semantic reasoning

Abstract

The application discloses a kind of multi-modal feature fusion method and device based on semantic reasoning, belong to natural language processing and computer vision technical field, including steps: using DeBERTa language model and Swin Transformer visual model respectively text and image are extracted in depth feature;In cross-modal alignment, different modal characteristics are used modal center perception alignment mechanism, and the feature center of the same object is used as anchor point for global alignment;In the alignment process, combined with contrast learning and contrast loss, the cross-modal representation of the same target is aggregated in the public feature space while different targets remain distinguished;Introduce cross-modal attention calculation and gating screening mechanism to realize explicit semantic alignment and noise suppression, and use graph convolution network for implicit semantic reasoning;The multi-modal semantic representation obtained by fusion is input into the classifier for classification determination.The application can improve the accuracy and reliability of semantic analysis.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and computer vision, and more specifically, to a multimodal feature fusion method and apparatus based on semantic reasoning. Background Technology

[0002] In semantic-text applications such as natural disaster monitoring, road traffic control, and medical record analysis, data typically contains information in both text and image modalities. Textual descriptions provide rich semantic content, while related images (such as medical images, charts, and illustrations in analysis reports) provide intuitive background or supplementary information. Relying solely on a single modality for semantic analysis often fails to fully utilize the clues provided by the other modality, leading to incomplete or inaccurate semantic understanding.

[0003] Currently, artificial intelligence in multimodal semantic fusion mainly focuses on a few directions. For example, attention mechanisms or Transformer-based models are used to establish correspondences between images and text to align and fuse semantic features from images and text. These methods have made some breakthroughs in feature alignment and semantic analysis, but they still face the following key challenges when applied to multimodal feature fusion: (1) Lack of in-depth modeling of alignment feature relationships: Existing multimodal fusion methods usually directly use the alignment features for prediction after aligning the relevant features of the image and text, but do not fully model the interaction relationships between the alignment features. In other words, when some parts of the image and text correspond semantically, simple alignment processing alone cannot capture the more complex dependencies between them, which may lead to insufficient understanding of the overall semantics and affect the accuracy of the task.

[0004] (2) Cross-modal alignment is susceptible to interference from redundant information: Image and text data often contain noise or redundant information that is irrelevant to the main semantics. For example, cluttered background details in an image or irrelevant words and phrases in text can interfere with the image-text alignment process. Traditional attention alignment mechanisms may produce incorrect feature matching when such irrelevant information is present, incorrectly associating parts of the text and image that do not actually correspond. This feature mismatch reduces the credibility of the fusion result and affects the performance of downstream semantic analysis tasks.

[0005] For the reasons mentioned above, existing multimodal semantic fusion methods may exhibit problems such as incomplete semantic extraction, low accuracy, or insufficient robustness when handling complex semantic tasks. Therefore, there is an urgent need for an improved method that can more effectively align relevant semantic information and deeply mine the relationships within image-text multimodal scenarios, thereby improving the accuracy and reliability of semantic analysis. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal feature fusion method and apparatus based on semantic reasoning, which can improve the accuracy and reliability of semantic analysis.

[0007] The objective of this invention is achieved through the following solution: A multimodal feature fusion method based on semantic reasoning includes: For the multimodal feature fusion task, the DeBERTa language model and the Swin Transformer visual model are first used to extract deep features from text and images respectively. The DeBERTa language model models content and positional relationships through a decoupled attention mechanism, while the Swin Transformer visual model extracts multi-scale local and global visual features through windowing and sliding window mechanisms. After feature extraction from text and images, cross-modal alignment and fusion processing are performed to obtain multimodal semantic representations. In cross-modal alignment, a modality center-aware alignment mechanism is used for features of different modalities, with the feature center of the same object as the anchor point for global alignment. In the alignment process, contrastive learning and contrastive loss are combined to aggregate cross-modal representations of the same target in a common feature space while maintaining the distinction between different targets. In the fusion processing, cross-modal attention computation and gating filtering mechanisms are introduced to achieve explicit semantic alignment and noise suppression, and graph convolutional networks are used for implicit semantic reasoning. The fused multimodal semantic representation is input into the classifier for classification and judgment, and the semantic reasoning result is output.

[0008] Furthermore, before proceeding to cross-modal alignment and fusion processing, the process includes a step of ensuring that the extracted text features and image features are consistent in the output dimension.

[0009] Furthermore, the integration of contrastive learning during the alignment process specifically involves combining Siamese networks or contrastive networks during the alignment process.

[0010] Furthermore, the introduction of cross-modal attention calculation and gating screening mechanism to achieve explicit semantic alignment and noise suppression specifically includes the following sub-steps: calculating bidirectional matching scores using cross-modal attention and forming attention weights through Softmax normalization; and the gating unit adaptively adjusting the fusion weights of each modality information based on the attention weights and feature importance.

[0011] Furthermore, in the fusion process, the implicit semantic reasoning using a graph convolutional network specifically includes the following sub-steps: constructing a graph structure for the aligned features and using a graph convolutional network for implicit semantic reasoning to model high-order dependencies between nodes.

[0012] Furthermore, the classifier is specifically a bilinear classifier.

[0013] Furthermore, the step of inputting the fused multimodal semantic representation into the classifier for classification includes the following sub-steps: during the training phase, the model is optimized end-to-end using symmetric cross-entropy or a hybrid loss function combining contrastive loss, and the model is tuned through hyperparameter tuning.

[0014] Furthermore, the multimodal feature fusion task specifically includes semantic retrieval task, image-text matching task, and question answering and multimodal classification task.

[0015] A multimodal feature fusion device based on semantic reasoning includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes the method described in any of the preceding methods.

[0016] The beneficial effects of this invention include: (1) The present invention adopts a dual-path feature extraction structure combining DeBERTa and Swing Transformer, which can simultaneously capture the deep semantics of text and the multi-scale details of images, effectively improving the model's semantic understanding ability in complex scenarios.

[0017] (2) This invention introduces graph convolutional networks to model implicit semantic relationships, thereby realizing the mining and propagation of potential dependencies between cross-modal features, making the fusion results more comprehensive and robust.

[0018] (3) In the explicit semantic alignment process, the present invention adopts a cross-modal attention mechanism combined with gating correction to ensure the accuracy of alignment features and noise suppression capability, thereby enhancing the discrimination effect of semantic information. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a diagram of the multimodal feature fusion network structure for semantic reasoning in the method of this invention. Detailed Implementation

[0021] All features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0022] The specific implementation process of this invention is as follows: This invention specifically presents a multimodal semantic fusion method that combines image and text information for semantic analysis. Through cross-modal alignment and semantic reasoning, it achieves deep fusion of semantic information from both image and text modalities, making it suitable for tasks such as semantic tag extraction, semantic text generation, and classification.

[0023] This invention addresses the shortcomings of existing multimodal semantic fusion methods, such as low accuracy and uncontrollable semantic generation when handling complex semantic tasks. It proposes a multimodal feature fusion scheme based on semantic reasoning. This scheme utilizes an improved cross-modal attention mechanism to selectively focus on semantically relevant parts of images and text, and filters out incorrect matches through a gating mechanism to obtain accurate alignment features. By introducing a graph convolutional network to reason about the alignment features and the overall context, it models the dependency relationship between image and text features, compensating for the insufficient semantic analysis caused by only aligning without modeling interactions, and obtaining richer semantic representations. By fusing alignment features and semantic relationships and applying them to tasks such as semantic label extraction, text generation, or classification, the accuracy and completeness of semantic analysis can be significantly improved, enhancing the ability to understand semantic information in complex scenarios.

[0024] In a preferred embodiment, a multimodal feature fusion method based on semantic reasoning is provided. The DeBERTa language model and the Swin Transformer visual model are used to extract deep features from text and images respectively. DeBERTa improves the text context representation ability by modeling content and positional relationships separately through a decoupled attention mechanism. The Swin Transformer achieves efficient extraction of multi-scale local-global visual features through windowing and sliding window mechanisms. The text features and image features are kept consistent in output dimension to facilitate subsequent cross-modal alignment and fusion processing.

[0025] Furthermore, in cross-modal alignment, a modality center-aware alignment mechanism is adopted for different modal features, using the feature center of the same object as the anchor point for global alignment; during the alignment process, contrastive learning (e.g., Siamese / contrastive network) and contrastive loss are combined to aggregate cross-modal representations of the same target in the common feature space while different targets remain distinguishable. The alignment strategy improves the discriminativeness and transferability of cross-modal retrieval and matching.

[0026] Furthermore, in the fusion processing, a cross-modal attention calculation and gating screening mechanism is introduced to achieve explicit semantic alignment and noise suppression. The cross-modal attention calculation results in a bidirectional matching score, which is then normalized using Softmax to form attention weights. The gating unit adaptively adjusts the fusion weights of each modality based on the importance of attention and features. In addition, a graph structure is constructed for the aligned features, and a graph convolutional network is used for implicit semantic reasoning to model the high-order dependencies between nodes, thereby enhancing the coherence and reasoning ability of multimodal semantic representation.

[0027] Furthermore, the method includes the following steps: inputting the fused multimodal semantic representation into a bilinear (or other form of multilayer fully connected) classifier / discriminator to output semantic inference results; during the training phase, using symmetric cross-entropy or a hybrid loss function combining contrastive loss to perform end-to-end optimization of the model; and using hyperparameter tuning to balance accuracy and robustness. The method is applicable to various downstream tasks such as semantic retrieval, image-text matching, question answering, and multimodal classification and has good generalization performance.

[0028] In a preferred embodiment, based on the above embodiments, the present invention further provides a multimodal feature fusion method based on semantic reasoning, involving text feature extraction, image feature extraction, multimodal semantic fusion and prediction, specifically including the following steps: (1) The text is encoded using a DeBERTa pre-trained language model; (2) The image is encoded using the Swing Transformer (Shifted Window Transformer) model; (3) Based on the text features and image features obtained in steps (1) and (2), perform deep fusion and utilize implicit semantic extraction and explicit semantic extraction to further enhance the multimodal fusion capability of semantic reasoning; (4) After the explicit and implicit extractions obtained in step (3), a bilinear classifier is used to classify and determine the fused features.

[0029] More specifically, step (1) includes the following sub-steps: Step (1) Given the input text sequence Each word is encoded using DeBERTa. The context vectors of dimension form the text feature representation matrix. As shown in formula (1): (1); in, The input text sequence; The text feature matrix; It is a text feature matrix The row vector; It is an input text sequence The number of words contained; DeBERTa employs a decoupled attention mechanism, processing word content and position information separately in self-attention computation, thereby enhancing the model's ability to represent semantic and syntactic positional dependencies in text. Compared to the traditional BERT model, which directly superimposes position and content before calculating attention, DeBERTa makes fuller use of relative positional information, improving the accuracy of text feature representation.

[0030] More specifically, step (2) includes the following sub-steps: Step (2) First, input the image. The image is divided into fixed-size patches, and then each patch is mapped to... The 3D feature vectors are input into the Swing Transformer for hierarchical encoding, ultimately forming the overall feature representation of the image. As shown in formula (2): (2); in, For is the image feature matrix The row vector, The number of feature blocks obtained by dividing the image. is the dimension of the image feature vector.

[0031] More specifically, step (3) includes the following sub-steps: In step (3), implicit extraction and explicit extraction are used to perform deep fusion of the text features and image features obtained in step (2).

[0032] Implicit semantic extraction: Utilizing graph convolutional networks (GCNs) to mine latent relationships between image and text features. The first GCN... The update rule for layer nodes is shown in formula (3): (3); in, In the l-th layer of GCN, the first The feature vector of each node For GCN In the layer, the first The feature vector of each node Adjacency matrix The element represents the first element. The node and the first The strength of association between nodes For GCN The trainable weight matrix of the layer, For GCN The trainable bias vector of a layer; n is the total number of nodes in the GCN; Through multi-layered GCN iterative updates, each node can receive relevant information from its neighboring nodes, thereby learning more comprehensive implicit semantic features.

[0033] Explicit semantic extraction: First, cross-modal matching scores are calculated using bilinear mapping: (4); (5); in, This is a text-image matching score matrix. For text feature matrix, The projection matrix of the text features. For image feature matrix, The projection matrix of the text features. The image-text matching score matrix. is the projection matrix of the image features.

[0034] The matching score matrix above is normalized by... Obtain the attention weight matrix of text words in each image region. Attention weight matrix of image regions on each text word : (6); (7); Based on the obtained attention matrix, the image and text features are weighted and aggregated to obtain a preliminary aligned cross-modal feature representation: (8); (9); in, Image alignment features guided by text. Text alignment features guided by images; Subsequently, a gating mechanism was introduced to filter redundant information: (10); (11); in, This is a gating signal for the text direction. This is a gating signal for the image direction. For a trainable projection matrix, For a trainable projection matrix, Use the Sigmoid activation function; The alignment features are adaptively adjusted using a gating signal to obtain the corrected alignment features: (12); (13); in, For text-image fusion features that have been filtered out of noise, Image-text fusion features with noise filtered out; Through a gated fusion mechanism, the model can selectively retain relevant parts in explicitly aligned text and image features, suppress noise introduced during the alignment process, and thus adaptively correct the text and image alignment results.

[0035] Subsequently, the corrected alignment features are obtained through nonlinear mapping: (14); (15); in, For nonlinear transformation of text features, a trainable weight matrix is ​​used. For nonlinear transformation of image features, a trainable weight matrix is ​​used. This is a trainable bias vector for the nonlinear transformation of text features. It is a trainable bias vector for nonlinear transformation of image features; Furthermore, the present invention will also correct the aligned graphic features. and As the new initial node set of GCN, the aforementioned distance-based adjacency matrix construction method and graph convolution operation are repeatedly used to further enhance the multimodal fusion capability of semantic reasoning.

[0036] More specifically, step (4) includes the following sub-steps: In step (4), these two semantic features are fused, and the final analysis result is output through a classifier. First, the explicit and implicit features obtained above are concatenated as input features for classification. : (16); in, The number of words in the text. The number of image patches; A bilinear classifier is used to classify the fused features. The bilinear classifier consists of two cascaded fully connected layers: the first layer fuses implicit and explicit semantics. (17); in, The transformed multimodal features, The multimodal splicing feature matrix output by formula (16) For a trainable weight matrix, It is a trainable bias vector; The second layer is used to output the classification results: (18); Where Y represents the intermediate output feature of the classification task. For a trainable classification weight matrix, This is a trainable classification bias vector; In order to obtain the final discrimination result, the present invention... The output is processed using average pooling and softmax normalization: (19); in, For the final category probability distribution, For intermediate output features Average pooling aggregates the features from multiple nodes into a single global feature vector. for dimensional space; Finally, the model is trained and optimized using the cross-entropy loss function: (20); in, is the batch cross-entropy loss value, and s is the training batch size. For the first The true label of each sample For the first The predicted probability of each sample; By minimizing the aforementioned cross-entropy loss, the model's parameters are continuously updated and iterated to improve the accuracy of image and text classification.

[0037] In other embodiments, based on the above embodiments, such as Figure 1 As shown, this invention also provides a multimodal feature fusion method based on semantic reasoning, specifically including the following steps: (1) Dataset Construction: For the dataset used for semantic reasoning training, Conceptual Caption is adopted. This dataset contains a large number of image-text pairing datasets, covering a variety of scenarios and targets. It is divided into training set and test set. The training set is used for the model to learn the feature relationships between image and text modalities, and the test set serves as an image-text association retrieval library.

[0038] (2) Feature Extraction and Cross-Modal Alignment: For the input multimodal samples, semantic representations are obtained through text feature extraction and image feature extraction modules, respectively. For the text modality, a pre-trained language model based on DeBERTa is used to extract contextual semantic features; for the image modality, Swin Transformer is used to extract multi-scale visual features. After feature extraction, modality center perception technology is used to align features from different modalities. The multimodal feature centers of the same object are used as anchor points, and a contrastive learning loss function is introduced to optimize the consistency of the feature space, enabling the aligned features to calculate similarity in a unified semantic space.

[0039] The aligned cross-modal features are input into a cross-modal attention fusion model, which utilizes explicit semantic alignment and implicit semantic reasoning mechanisms to achieve multi-level fusion. The implicit reasoning part models the potential dependencies between graph and text features based on graph convolutional networks (GCNs), while the explicit part aligns and filters irrelevant information through attention weights and gating mechanisms to ensure the accuracy and robustness of the fusion results.

[0040] (3) Semantic Reasoning and Result Prediction: The fused features are input into a bilinear classifier for semantic discrimination and reasoning. The first layer of the fully connected network achieves deep fusion of multimodal features, and the second layer outputs the final reasoning result. To enhance discrimination accuracy, the system uses a symmetric cross-entropy loss function for training and optimization to ensure high consistency of the model in semantic recognition and cross-modal retrieval.

[0041] During the testing phase, cross-modal matching was performed by calculating the cosine similarity between text and images, enabling multimodal content retrieval based on semantic reasoning.

[0042] (4) System Deployment and Operation: This invention is implemented using the PyTorch framework and deployed on a high-performance server platform equipped with NVIDIA GPUs. The system integrates text encoding, image encoding, feature alignment, fusion inference, and result output modules, forming an end-to-end semantic analysis process. Extensive testing has verified that the system meets the expected performance indicators in terms of cross-modal feature alignment accuracy, semantic inference accuracy, and robustness.

[0043] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0044] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0045] In another aspect, embodiments of the present invention also provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

Claims

1. A multimodal feature fusion method based on semantic reasoning, characterized in that, include: For the multimodal feature fusion task, the DeBERTa language model and the Swin Transformer visual model are first used to extract deep features from text and images respectively. The DeBERTa language model models content and positional relationships through a decoupled attention mechanism, while the Swin Transformer visual model extracts multi-scale local and global visual features through windowing and sliding window mechanisms. After feature extraction from text and images, cross-modal alignment and fusion processing are performed to obtain multimodal semantic representations. In cross-modal alignment, a modality center-aware alignment mechanism is used for features of different modalities, with the feature center of the same object as the anchor point for global alignment. In the alignment process, contrastive learning and contrastive loss are combined to aggregate cross-modal representations of the same target in a common feature space while maintaining the distinction between different targets. In the fusion processing, cross-modal attention computation and gating filtering mechanisms are introduced to achieve explicit semantic alignment and noise suppression, and graph convolutional networks are used for implicit semantic reasoning. The fused multimodal semantic representation is input into the classifier for classification and judgment, and the semantic reasoning result is output.

2. The multimodal feature fusion method based on semantic reasoning according to claim 1, characterized in that, Before proceeding to cross-modal alignment and fusion processing, the process includes a step of ensuring that the extracted text features and image features are consistent in the output dimension.

3. The multimodal feature fusion method based on semantic reasoning according to claim 1, characterized in that, The integration of contrastive learning into the alignment process specifically involves combining twin networks or contrastive networks during the alignment process.

4. The multimodal feature fusion method based on semantic reasoning according to claim 1, characterized in that, The introduction of cross-modal attention calculation and gating screening mechanism to achieve explicit semantic alignment and noise suppression specifically includes the following sub-steps: calculating bidirectional matching scores using cross-modal attention and forming attention weights through Softmax normalization; and the gating unit adaptively adjusting the fusion weights of each modality information based on the attention weights and feature importance.

5. The multimodal feature fusion method based on semantic reasoning according to claim 1, characterized in that, In the fusion process, the implicit semantic reasoning using a graph convolutional network specifically includes the following sub-steps: constructing a graph structure for the aligned features and using a graph convolutional network for implicit semantic reasoning to model high-order dependencies between nodes.

6. The multimodal feature fusion method based on semantic reasoning according to claim 1, characterized in that, The classifier is specifically a bilinear classifier.

7. The multimodal feature fusion method based on semantic reasoning according to claim 1, characterized in that, The step of inputting the fused multimodal semantic representation into the classifier for classification determination specifically includes the following sub-steps: During the training phase, the model is optimized end-to-end using either symmetric cross-entropy or a hybrid loss function that combines contrastive loss, and further optimized through hyperparameter tuning.

8. The multimodal feature fusion method based on semantic reasoning according to claim 1, characterized in that, The multimodal feature fusion task specifically includes semantic retrieval task, image-text matching task, and question answering and multimodal classification task.

9. A multimodal feature fusion device based on semantic reasoning, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes the method as described in any one of claims 1 to 8.

Images