Multi-modal fusion method and system for incomplete multi-modal sentiment analysis

Abstract

The invention provides a multi-modal fusion method and system for incomplete multi-modal sentiment analysis. The method comprises the following steps: projecting original features of a text, an audio and an image to a unified potential space; in the potential space, the geometric distribution of the multi-modal features is aligned by minimizing the volume of a Gram matrix; dynamically shielding partial modal features, and inputting the partial modal features and other modal features into the double-branch network; in the first branch, other modal features are aligned to shielding features through a cross-modal attention mechanism, and a hypergraph structure convolution and multi-head self-attention are utilized to model a high-order structure dependency relationship in a modal to obtain a first feature; in the second branch, mode feature redundant information is captured through a soft attention matrix and suppressed, and a second feature is obtained; and fusing the first feature and the second feature based on a shared encoder to obtain a multi-modal fusion feature. The robustness and accuracy of multi-modal fusion under the condition of modal deficiency are improved by expanding the distribution expression ability of the features in the modals and compressing the geometric structure deviation between the modals.

Description

Technical Field

[0001] This invention relates to the field of incomplete multimodal sentiment analysis technology, and in particular to a multimodal fusion method and system for incomplete multimodal sentiment analysis. Background Technology

[0002] Emotion recognition, a core task in the field of multimodal artificial intelligence, achieves accurate analysis of human emotional states by integrating signals from multiple sources such as speech, text, and vision. It has wide applications in scenarios such as education, virtual assistants, and intelligent healthcare. Multimodal emotion recognition (MER) leverages the complementary advantages of multimodal signals, acquiring more comprehensive emotional semantic information compared to single-modal analysis. However, in practical applications, limitations such as sensor malfunctions, acquisition noise, network latency, and privacy protection often lead to the loss of some modal information, resulting in a significant decline in emotion recognition performance.

[0003] Current mainstream technologies for addressing the modality loss problem primarily employ two approaches: reconstruction-based and shared representation-based methods. The former recovers missing information from existing modalities using generative models, while the latter projects multimodal features onto a common space to extract general representations. While these approaches alleviate the impact of modality loss to some extent, they still face several technical bottlenecks: First, intramodal features often exhibit distribution shrinkage and representational collapse, resulting in insufficient modeling of modality-specific semantic differences and ineffective information compensation when modalities are missing. Second, multimodal features exhibit distribution bias and geometric inconsistencies in the common space, leading to insufficient modeling of cross-modal semantic relationships and reduced recognition accuracy. Third, the lack of a unified mechanism for modeling higher-order structures within modalities and complex cross-modal dependencies limits the ability to reconstruct missing modalities. Fourth, the failure to effectively suppress redundant or conflicting information between modalities results in semantic overlap and information conflicts during feature fusion, affecting the final recognition performance. These shortcomings make it difficult for existing methods to meet practical needs in modality loss scenarios in terms of robustness and accuracy. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a multimodal fusion method and system for incomplete multimodal sentiment analysis. By expanding the distributional representation capability of features within a modality and compressing the geometrical structural deviations between modalities, the robustness and accuracy of multimodal fusion under modality missing conditions are improved.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a multimodal fusion method for incomplete multimodal sentiment analysis, comprising: The preprocessed original features of text, audio, and images are projected into a unified latent space; In the latent space, the radial basis kernel function is used to calculate the Gram matrix between modal features, and the geometric distribution of multimodal features is aligned by minimizing its volume; During the testing phase, some modal features are dynamically masked. Based on the unmasked modal features, a missing modality encoder is used to reconstruct the masked modal features and model the high-order structural dependencies within the modality. The reconstructed modal features and the unmasked modal features are then input into a dual-branch network. In the first branch, the features are aligned and fused through a cross-modal attention mechanism to obtain the first feature. In the second branch, a soft attention matrix is ​​used to capture and suppress redundant information of the modal features and to widen the gap between similar features to increase modal diversity expression, thus obtaining the second feature. Multimodal fusion features are obtained by fusing the first and second features based on the shared encoder.

[0006] Secondly, the present invention provides a multimodal fusion system for incomplete multimodal sentiment analysis, comprising: The mapping module is configured to project the preprocessed raw features of text, audio, and images onto a unified latent space; The intermodal volume alignment module is configured to compute the Gram matrix between modal features using a radial basis kernel function in the latent space, and align the geometric distribution of multimodal features by minimizing its volume; The intramodal enhancement normalization module is configured to dynamically mask some modal features during the testing phase, reconstruct the masked modal features using a missing modality encoder based on the unmasked modal features, and model the high-order structural dependencies within the modality. The reconstructed modal features and the unmasked modal features are then input into a dual-branch network. In the first branch, the features are aligned and fused through a cross-modal attention mechanism to obtain the first feature. In the second branch, redundant information of the modal features is captured and suppressed through a soft attention matrix, and similar features are separated to increase modal diversity expression to obtain the second feature. The fusion module is configured to fuse the first and second features based on the shared encoder to obtain multimodal fusion features.

[0007] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal fusion method for incomplete multimodal sentiment analysis described in the first aspect.

[0008] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal fusion method for incomplete multimodal sentiment analysis described in the first aspect.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention projects multimodal features onto a unified latent space during model training and combines radial basis function kernel function to minimize Gram matrix volume to achieve geometrical alignment between modalities, effectively alleviating the problem of inconsistent modal distribution and laying the foundation for cross-modal semantic association. During the testing phase, when dynamically masking modalities, missing modalities are reconstructed using unmasked features and higher-order dependencies are modeled to enhance adaptability to modal missing scenarios. The dual-branch network achieves cross-modal attention alignment fusion and soft attention to suppress redundancy, taking into account modal complementarity and diversity, and finally obtains robust features through fusion via a shared encoder. This method can improve fusion accuracy whether the modalities are complete or missing, especially in practical scenarios such as sensor failure, effectively enhancing the stability and accuracy of multimodal tasks and providing reliable technical support for fields such as sentiment analysis.

[0010] (2) This invention proposes an innovative multimodal alignment method based on minimizing the volume of the kernel Gram matrix, which realizes the high-order geometric relationship consistency alignment of multimodal emotional features and enhances the robustness of fusion in the case of modality loss, significantly improving the fusion effect of intermodal information and the recognition performance of the model.

[0011] (3) The present invention designs a hypergraph enhanced self-attention mechanism that integrates hypergraph structure convolution and multi-head self-attention, which enriches the expression of single-modal high-order structures and enhances the discriminative power and robustness within the modality.

[0012] (4) The present invention develops an intramodal distribution normalization module, which makes the single modal features more evenly distributed in the latent space, thereby improving robustness in the case of modality loss, effectively alleviating the redundancy and over-aggregation of single modal expression, and improving the diversity and discriminability of features.

[0013] (5) This invention constructs a robust multimodal fusion strategy for modal missingness, ensuring that the system can still accurately identify emotional state under incomplete input conditions, thereby improving the practical application value of the model.

[0014] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0016] Figure 1 This is a flowchart illustrating the main steps of a multimodal fusion method for incomplete multimodal sentiment analysis provided in this embodiment of the invention. Figure 2This is a framework diagram of a multimodal fusion method for incomplete multimodal sentiment analysis provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of the cross-modal volume alignment mechanism provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the modal distribution normalization mechanism provided in an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] Example 1 like Figure 1 As shown in the figure, this embodiment discloses a multimodal fusion method for incomplete multimodal sentiment analysis, including the following steps: S1: Project the preprocessed original features of text, audio, and images into a unified latent space; S2: The radial basis kernel function is used to calculate the Gram matrix between modal features in the latent space, and the geometric distribution of multimodal features is aligned by minimizing its volume; S3: During the testing phase, some modal features are dynamically masked. Based on the unmasked modal features, the masked modal features are reconstructed using a missing modality encoder, and the high-order structural dependencies within the modality are modeled. The reconstructed modal features and the unmasked modal features are then input into a dual-branch network. In the first branch, the features are aligned and fused through a cross-modal attention mechanism to obtain the first feature. In the second branch, redundant information of the modal features is captured and suppressed through a soft attention matrix, and similar features are separated to increase modal diversity expression to obtain the second feature. S4: Based on the shared encoder, the first and second features are fused to obtain the multimodal fusion features.

[0019] Next, combined Figure 2 This embodiment provides a detailed description of a multimodal fusion method for incomplete multimodal sentiment analysis.

[0020] First, three modal data—text, audio, and image—are acquired and input into the first encoder, which extracts multimodal features from the input data based on the text encoder, audio encoder, and image encoder, respectively.

[0021] Furthermore, let the anchor point feature of a certain modality (such as text) be denoted as... Its corresponding other modal features (such as audio and video) are represented as All features are then projected into a shared embedding space and subjected to... Normalization is used to ensure scale consistency.

[0022] (a) Intermodal volume alignment To alleviate the problem of inconsistent geometric distributions of different modalities during training and to achieve robust multimodal alignment in scenarios with missing modalities, this embodiment proposes a kernel-based intermodal volume alignment mechanism: like Figure 3 As shown, the left image displays the volume before modality alignment, with a larger volume indicating insufficient modality alignment. The right image shows the reduced volume after the alignment module, indicating that the three modalities have achieved basic alignment. Specifically, the similarity between any two modal features is first calculated using the radial basis function (RBF) kernel to construct a cross-modal Gram matrix. Since the determinant of the Gram matrix can represent the geometric volume spanned by the features in the reproducing kernel Hilbert space (RKHS), and this volume can capture the higher-order geometric relationships of modal features, a contrastive loss is introduced. Using the Gram matrix determinant as the core, the RKHS volume is indirectly reduced by minimizing the negative logarithm of the determinant, thereby reducing the dispersion between modalities and making the latent spatial distribution of feature embedding more consistent. Even if a modality is missing, feature space alignment optimization can effectively improve the fusion effect and reconstruction stability.

[0023] This embodiment defines a positive definite kernel function. Radial basis function (RBF) kernel function is typically chosen:

[0024] in, , These represent feature vectors for two different modalities (e.g., text modal features and speech modal features). The bandwidth parameter of the RBF kernel controls the kernel function's sensitivity to distance. This kernel function measures the nonlinear similarity between vector pairs in a high-dimensional space.

[0025] Based on this kernel function, a feature similarity matrix Gram matrix is ​​constructed. Its elements are defined as:

[0026] in, Indicates anchor point features, Other modal features ( ), Indicates the first Each modal feature vector.

[0027] Kernel functions naturally induce the Reproducing Kernel Hilbert Space (RKHS) by defining the inner product between features. This space serves as a latent high-dimensional feature mapping domain, allowing direct similarity calculation of low-dimensional input features. To quantify the distribution consistency of modal features in the RKHS, the feature similarity of different modalities is first transformed into a Gram matrix using a kernel function. Since the determinant of the Gram matrix is ​​directly related to the geometric volume spanned by the features in the RKHS (the volume reflects the span of feature distribution), this volume is defined based on the determinant of the Gram matrix, as shown in the formula:

[0028] In the formula, Represents the determinant of the Gram matrix. It is a tiny constant used for numerical stability to avoid negative or zero values ​​when calculating the square root. This represents the volume calculation function. Indicates anchor point features, This represents the set of modal features other than the anchor point. The volume characterizes the distribution span of the modal features in the high-dimensional space: a smaller volume indicates more concentrated features and better consistency between modalities; a larger volume indicates more dispersed features.

[0029] Furthermore, for a containing For each training batch of samples, calculate the volume between each anchor point and all candidate modal feature groups, i.e., summarize the RKHS volumes of all samples to obtain a volume matrix. ,in:

[0030] To supervise cross-modal alignment, the following contrastive loss function is defined:

[0031] in, Represents cross-entropy loss, It is a learnable temperature parameter used to control the smoothness of the similarity distribution. It is the sample's identity tag, used to identify the first... The anchor point and the first Pair a positive sample with a positive sample.

[0032] During the model training and iterative optimization phase, the optimizer is used based on the contrastive loss function. Adjusting model parameters minimizes the volume corresponding to cross-modal features of the same sample in the volume matrix and suppresses the overlap of volumes of different samples, thereby minimizing the volume and promoting a more consistent distribution of modal features in the embedding space. Specifically, the optimization goal of volume is indirectly achieved by minimizing the contrastive loss function, so that the cross-modal features of the same sample form a more compact distribution in the embedding space and a more discriminative distribution among different samples.

[0033] To simulate modal loss in real-world scenarios, this embodiment introduces a modal loss simulation mechanism. For example... Figure 2 As shown, taking the missing text modality as an example, the modality is set as a blank feature (i.e., without any valid input information) to simulate its missing state. In order to recover the missing modality to support subsequent sentiment analysis prediction tasks, this embodiment utilizes other available modality information (such as video modality and audio modality), adds and fuses them, and then inputs them into the encoder of the text modality for reconstruction.

[0034] It should be noted that this text encoder has already received and processed the original text-related features during the training phase, thus possessing a certain degree of modality awareness. By combining video and audio modalities, this encoder can more effectively infer and reconstruct missing text features.

[0035] Furthermore, the above process can be flexibly extended to scenarios where multiple modalities are simultaneously missing. For example, when both the text and audio modalities are missing, the system will use the available modalities (such as the video modal) to sequentially input into the corresponding missing modal encoder for reconstruction. Compared to the case of single-modal missing features, this type of scenario does not require performing inter-modal feature fusion steps (such as adding video and audio), but directly utilizes the available modalities to complete the multiple reconstruction process.

[0036] (ii) Hypergraph-enhanced self-attention mechanism Hypergraph-enhanced self-attention mechanisms are applied to all modality-specific encoders and shared encoders to improve the interaction modeling ability between single-modality feature nodes and effectively reconstruct features in modality-missing scenarios. In this mechanism, the complete modality is first encoded by a hypergraph-enhanced self-attention encoder to extract high-quality modality representations. This encoder utilizes a hypergraph structure to build high-order relationships between single-modality feature nodes, connecting semantically related points between different features through hyperedges, thereby enhancing the attention mechanism's ability to model global modality dependencies.

[0037] For missing modalities, reconstruction is performed by inputting information from existing modalities into a dedicated encoder corresponding to the missing modality. The dedicated encoder can fuse contextual information from the complete modality and, guided by a hypergraph-enhanced attention mechanism, generate a reconstructed representation with semantic consistency and modal feature style.

[0038] To simultaneously capture fine-grained contextual information and high-order structural dependencies within each modality, a hypergraph-enhanced multi-head self-attention mechanism is proposed. This mechanism introduces a structure-aware path into each attention head to improve the model's ability to model the global structure.

[0039] Specifically, given single-modal input features First, project it onto a uniform dimension. And add position encoding. Then, each mode is independently processed. Modeling is performed using layer Transformer encoder blocks.

[0040] In each encoder block, standard multi-head self-attention (MSA) and feedforward network (FFN) are retained to model local contextual information. Simultaneously, to introduce structure-aware capabilities, for each attention head... Calculate the soft adjacency matrix based on token similarity Based on this, a potential hypergraph is constructed. On this basis, lightweight hypergraph convolution operations are introduced in parallel, enabling each attention head to perceive higher-order token interaction relationships.

[0041] For each attention head, three information streams are obtained: content-based self-attention output. Hypergraph Augmentation and residual input The three are softly fused using learnable weights:

[0042] in, .

[0043] The aforementioned weights are normalized using softmax and learned independently in each attention head.

[0044] Ultimately, all The fusion output of individual attention points To splice:

[0045] The output projection layer then maps back to the original dimension. This representation is subsequently fed into the FFN and residual connection branches, and the overall structure follows the standard Transformer architecture.

[0046] By embedding hypergraph structure modeling into each attention head, the model can simultaneously focus on local token details and long-distance high-order dependencies, thereby learning a more robust unimodal representation and improving the overall expressive power of multimodal fusion.

[0047] (iii) In-modal distribution normalization To enhance the diversity of representations within modalities and prevent representation collapse, an Intra-modal Distribution Normalization (IDN) mechanism, inspired by contrastive regularization methods, is introduced. IDN is applied to text, image, and audio modalities respectively, aiming to preserve the discriminative expressive features within each modality.

[0048] like Figure 4 As shown, the left image illustrates the state of a single modality's internal features before contrastive normalization, where the modal features are overly compact; the right image shows the more diverse representation of the modal features after contrastive normalization. Specifically, let the feature tensor of a single modality be... ,in Indicates batch size. Indicates the sequence length. This is the feature dimension. In this embodiment, the features are first analyzed along the token dimension. Normalization:

[0049] To stabilize subsequent similarity calculations, a soft attention matrix is ​​then calculated based on the cosine similarity between tokens within a modality. This attention mechanism is used to capture redundant information patterns in the sequence. In this embodiment, redundant representation components are aggregated using this attention mechanism. It is used to describe components with high repetition in a sequence.

[0050] To mitigate the over-smoothing problem and preserve the unique token representation, this embodiment improves feature discriminability by suppressing this redundant component:

[0051] in, The suppression coefficient can be set to a fixed value or obtained through learning. Finally, this embodiment modifies the features. Layer Normalization is performed to further stabilize its distribution.

[0052] By explicitly reducing redundancy within a modality, IDN can encourage token representations to be more dispersed in the embedding space, thereby improving modality-specific expressiveness and robustness, especially in cases where the modality is incomplete.

[0053] This embodiment employs a dual-branch fusion strategy to enhance intermodal complementarity and fusion expressive power, thereby improving the accuracy and robustness of sentiment prediction. This strategy includes two parallel branches, which perform differentiated processing and unified modeling of modal features respectively: the first branch takes each modal feature as input and enhances the modal representation through IDN contrastive normalization. The second branch uses a cross-modal attention mechanism, using the text modality as an anchor point to guide audio and image modalities to align with it. After completing their respective feature processing, the two branches concatenate the output features along the channel dimension and obtain a joint modal representation through feature weighted fusion. This fused feature is then used as input to a shared encoder for unified modeling and high-order representation learning. This shared encoder, like the modality-specific encoder, is also built based on a hypergraph-enhanced self-attention mechanism.

[0054] Example 1 To verify the effectiveness of this embodiment, the proposed ECI model was evaluated on three standard multimodal benchmark datasets: CMU-MOSI, CMU-MOSEI, and IEMOCAP. CMU-MOSI contains 2,199 video clips with sentiment scores, and the score range is... to CMU-MOSEI expands upon this to 22,856 segments, covering a wider range of topics. IEMOCAP consists of two-person dialogues labeled with emotion categories; assessments are conducted using 4-category and 6-category classifications, respectively.

[0055] Based on existing research, MOSI and MOSEI are considered binary sentiment analysis tasks (when scoring...). (When the result is positive), and reports the accuracy and F1 score. For IEMOCAP, this embodiment reports the weighted accuracy (WA) and the unweighted accuracy (UA).

[0056] To evaluate the robustness of the model in modality-missing scenarios, a fixed missing modality testing protocol is adopted: during the testing phase, certain modality combinations are systematically excluded. For example, {l,v}, {l,a}, and {v,a} represent a single missing modality; {l}, {v}, and {a} represent two missing modalities simultaneously.

[0057] All experiments were performed on an NVIDIA RTX 6000 Ada GPU (48GB VRAM). The number of training epochs was set to 300, the batch size to 32, and the initial learning rate to... .

[0058] Table 1. Comparison of the method in this embodiment with other algorithms;

[0059]

[0060]

[0061] Table 1 shows the performance comparison under the fixed missing modality protocol. For CMU-MOSI and CMU-MOSEI, the results are reported as accuracy / F1 score; while for IEMOCAP, weighted accuracy (WA) / unweighted accuracy (UA) is used.

[0062] This specific embodiment effectively addresses the problem of insufficient alignment of embedding spaces between modalities by projecting multimodal features onto a unified latent space and aligning the geometric distribution using radial basis function kernels, thereby improving cross-modal semantic relevance. The dynamic masking mechanism, combined with a dual-branch network design, can model intramodal high-order dependencies and complex cross-modal relationships through cross-modal attention and hypergraph convolution, while also suppressing redundant and conflicting information with soft attention, enhancing adaptability to scenarios with missing modalities. By expanding the expressive power of intramodal features and compressing intermodal geometric deviations, this method can significantly improve the robustness and accuracy of emotion recognition in practical applications such as sensor failure and information loss, providing more reliable technical support for the implementation of multimodal artificial intelligence in education, healthcare, and other scenarios.

[0063] Example 2 This embodiment provides a multimodal fusion system for incomplete multimodal sentiment analysis, including: The mapping module is configured to project the preprocessed raw features of text, audio, and images onto a unified latent space; The intermodal volume alignment module is configured to compute the Gram matrix between modal features using a radial basis kernel function in the latent space, and align the geometric distribution of multimodal features by minimizing its volume; The intramodal enhancement normalization module is configured to dynamically mask some modal features during the testing phase, reconstruct the masked modal features using a missing modality encoder based on the unmasked modal features, and model the high-order structural dependencies within the modality. The reconstructed modal features and the unmasked modal features are then input into a dual-branch network. In the first branch, the features are aligned and fused through a cross-modal attention mechanism to obtain the first feature. In the second branch, redundant information of the modal features is captured and suppressed through a soft attention matrix, and similar features are separated to increase modal diversity expression to obtain the second feature. The fusion module is configured to fuse the first and second features based on the shared encoder to obtain multimodal fusion features.

[0064] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the multimodal fusion method for incomplete multimodal sentiment analysis as described in Embodiment 1 above.

[0065] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the multimodal fusion method for incomplete multimodal sentiment analysis as described in Embodiment 1 above.

[0066] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0067] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal fusion method for incomplete multimodal sentiment analysis, characterized in that, include: The preprocessed original features of text, audio, and images are projected into a unified latent space; In the latent space, the radial basis kernel function is used to calculate the Gram matrix between modal features, and the geometric distribution of multimodal features is aligned by minimizing its volume; During the testing phase, some modal features are dynamically masked. Based on the unmasked modal features, a missing modality encoder is used to reconstruct the masked modal features and model the high-order structural dependencies within the modality. The reconstructed modal features and the unmasked modal features are then input into a dual-branch network. In the first branch, the features are aligned and fused through a cross-modal attention mechanism to obtain the first feature. In the second branch, a soft attention matrix is ​​used to capture and suppress redundant information of the modal features and to widen the gap between similar features to increase modal diversity expression, thus obtaining the second feature. Multimodal fusion features are obtained by fusing the first and second features based on the shared encoder.

2. The multimodal fusion method for incomplete multimodal sentiment analysis as described in claim 1, characterized in that, The step of projecting the preprocessed text, audio, and image original features into a unified latent space specifically involves: acquiring text, audio, and image data, extracting the original features of each modality through a first encoder, and projecting them into a unified latent space.

3. The multimodal fusion method for incomplete multimodal sentiment analysis as described in claim 1, characterized in that, The step of calculating the Gram matrix between modal features using a radial basis function in the latent space, and aligning the geometric distribution of multimodal features by minimizing its volume, specifically includes: In the latent space, the similarity between features of any two modes is calculated using the radial basis kernel function to construct a cross-modal Gram matrix; The Gram matrix volume is defined by the determinant of the Gram matrix and mapped onto the regenerating kernel Hilbert space. The Gram matrix volume is minimized by the contrastive loss function to ensure consistent alignment of the geometric distribution of multimodal features.

4. The multimodal fusion method for incomplete multimodal sentiment analysis as described in claim 1, characterized in that, The dynamic shielding modal features simulate the loss of any modal feature.

5. A multimodal fusion method for incomplete multimodal sentiment analysis as described in claim 1, characterized in that, The first branch is aligned and fused using a cross-modal attention mechanism to obtain the first feature, which specifically includes: The reconstructed modal features and the unmasked modal features are combined using a multi-head self-attention mechanism to extract the local contextual representation of the single modal features; For each attention head, a soft adjacency matrix is ​​constructed based on token similarity to generate a hypergraph convolutional augmented representation; The local context representation, hypergraph convolutional augmentation representation, and residual input are fused according to learnable weights, and all attention head outputs are concatenated and mapped to the original dimension to generate the reconstructed first feature.

6. The multimodal fusion method for incomplete multimodal sentiment analysis as described in claim 1, characterized in that, In the second branch, redundant information of modal features is captured and suppressed through a soft attention matrix, and similar features are distanced to increase modal diversity expression, resulting in a second feature, which specifically includes: Normalize the single-modal feature tensor; Based on the cosine similarity between tokens within a modality, a soft attention matrix is ​​calculated to capture redundant information patterns in the feature sequence, and redundant information patterns are aggregated to obtain redundant representation components. Redundant representation components are suppressed and feature normalization is performed to obtain the second feature.

7. A multimodal fusion method for incomplete multimodal sentiment analysis as described in claim 1, characterized in that, The shared encoder is constructed based on a hypergraph-enhanced self-attention mechanism.

8. A multimodal fusion system for incomplete multimodal sentiment analysis, characterized in that, include: The mapping module is configured to project the preprocessed raw features of text, audio, and images onto a unified latent space; The intermodal volume alignment module is configured to compute the Gram matrix between modal features using a radial basis kernel function in the latent space, and align the geometric distribution of multimodal features by minimizing its volume; The intramodal enhancement normalization module is configured to dynamically mask some modal features during the testing phase, reconstruct the masked modal features using a missing modality encoder based on the unmasked modal features, and model the high-order structural dependencies within the modality. The reconstructed modal features and the unmasked modal features are then input into a dual-branch network. In the first branch, the features are aligned and fused through a cross-modal attention mechanism to obtain the first feature. In the second branch, redundant information of the modal features is captured and suppressed through a soft attention matrix, and similar features are separated to increase modal diversity expression to obtain the second feature. The fusion module is configured to fuse the first and second features based on the shared encoder to obtain multimodal fusion features.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multimodal fusion method for incomplete multimodal sentiment analysis as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal fusion method for incomplete multimodal sentiment analysis as described in any one of claims 1-7.

Images