Semantic auxiliary multi-modal sentiment analysis method and system for incomplete data

By employing cross-modal semantic refinement and anchorless spectral alignment techniques, the problem of incomplete data in multimodal sentiment analysis is solved, achieving efficient sentiment feature compensation and stable prediction, thereby improving the accuracy and robustness of the analysis.

CN123048996APending Publication Date: 2026-11-06SHANDONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611236482.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-14
Publication Date
2026-11-06

AI Technical Summary

Technical Problem

Existing multimodal sentiment analysis methods struggle to recover high-level semantic information when faced with incomplete data, are prone to generating false features and prediction errors, and suffer from high computational costs and poor robustness.

Method used

A cross-modal semantic refinement module is used to generate latent semantics through a large language model, and a cross-modal spectrum alignment module is used for anchor-free nonlinear alignment to achieve high-level compensation and stable prediction of sentiment features.

Benefits of technology

Without reconstructing the original data, it improves the accuracy and robustness of multimodal sentiment analysis, reduces computational overhead, and maintains the discriminative ability between different samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123048996A_ABST
    Figure CN123048996A_ABST
Patent Text Reader

Abstract

This application discloses a semantic-assisted multimodal sentiment analysis method and system for incomplete data. The method includes: preprocessing the missing data of the multimodal data; inputting the multimodal data to be analyzed into a trained multimodal sentiment analysis model to obtain sentiment prediction results; wherein the trained multimodal sentiment analysis model is used to extract features from the multimodal data to obtain visual features, audio features, and text features; performing cross-modal semantic refinement on the visual features, audio features, and text features to obtain semantic features; performing cross-modal spectral alignment on the semantic features, visual features, audio features, and text features to obtain four spectral-aligned features; fusing the four spectral-aligned features to obtain a fused multimodal feature representation; and performing sentiment prediction on the fused multimodal feature representation to obtain sentiment prediction results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of sentiment analysis technology, and in particular to semantic-assisted multimodal sentiment analysis methods and systems for incomplete data. Background Technology

[0002] Currently, multimodal sentiment analysis typically predicts the sentiment tendency, sentiment category, or sentiment intensity of a target object by jointly modeling textual, visual, and audio information. Most existing methods assume that the data for each modality is complete and available. However, in practical applications, factors such as sensor failure, environmental noise, target occlusion, and unstable data transmission can lead to partial missing text terms, video frames, or audio segments, or even the complete absence of a particular modality. In such cases, models trained on complete data struggle to obtain sufficient and reliable sentiment cues, easily resulting in incomplete sentiment feature representation and degraded predictive performance.

[0003] To alleviate the problem of missing multimodal information, some existing technologies recover features of the missing modalities based on observed modalities through feature reconstruction. However, these methods mainly learn low-level statistical relationships between different modalities, making it difficult to recover high-level semantic information that is more crucial for sentiment judgment. Furthermore, the same text may correspond to different speech intonations and facial expressions; directly reconstructing missing visual or audio features can easily generate false features inconsistent with genuine sentiment evidence, thus misleading subsequent sentiment prediction.

[0004] Other existing technologies directly fuse currently observable modal features by designing complex cross-modal interaction or fusion networks. However, when the proportion of missing modalities is high, the available features themselves may contain noise and have limited information content. The fusion model tends to rely on modal co-occurrence relationships in the training data, making it difficult to obtain stable sentiment semantic representations. In addition, although large language models have strong contextual understanding and semantic abstraction capabilities, directly generating sentiment descriptions or inference text usually requires performing autoregressive text decoding, which has problems such as high computational cost, low inference efficiency, and potential introduction of text generation bias.

[0005] Furthermore, existing cross-modal alignment methods often employ pairwise alignment mechanisms anchored by text or other dominant modalities, constraining the consistency between visual, audio, and other features and the anchor features. These methods neglect the overall structural relationships between semantic, textual, visual, and audio representations, making it difficult to fully capture the global nonlinear dependencies between multimodal features. When the anchor modality is severely missing or noisy, the alignment direction may become unstable. Simultaneously, features from different samples may converge during alignment, leading to representation collapse and reduced distinguishability between different sentiment samples. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application provides a semantic-assisted multimodal sentiment analysis method and system for incomplete data; how to obtain reliable high-level sentiment semantic information from existing multimodal data when text, visual, or audio information is incomplete; how to achieve computationally low-cost latent semantic compensation without reconstructing the original missing data or performing explicit text decoding; and how to perform overall nonlinear alignment of semantic, text, visual, and audio representations without relying on preset anchor modalities, while maintaining the discriminative ability between different samples, thereby improving the accuracy, stability, and robustness of multimodal sentiment analysis under different data missing conditions.

[0007] On the one hand, it provides semantic-assisted multimodal sentiment analysis methods for incomplete data, including: Acquire the multimodal data to be analyzed, which includes visual modal data, audio modal data, and text modal data, and preprocess the missing data in the multimodal data; The multimodal data to be analyzed is input into the trained multimodal sentiment analysis model to obtain sentiment prediction results; The trained multimodal sentiment analysis model is used for: Feature extraction is performed on multimodal data to obtain visual features, audio features, and text features; Semantic features are obtained by cross-modal semantic refinement of visual features, audio features, and text features; Semantic features, visual features, audio features, and text features are subjected to cross-modal spectral alignment to obtain four types of spectral aligned features; The four spectral-aligned features are fused to obtain a fused multimodal feature representation; Sentiment prediction is performed on the fused multimodal feature representation to obtain the sentiment prediction results.

[0008] On the other hand, a semantically assisted multimodal sentiment analysis system for incomplete data is provided, including: The acquisition module is configured to: acquire multimodal data to be analyzed, including visual modal data, audio modal data and text modal data, and preprocess missing data in the multimodal data; The sentiment analysis module is configured to input the multimodal data to be analyzed into the trained multimodal sentiment analysis model and obtain the sentiment prediction results. The trained multimodal sentiment analysis model is used for: Feature extraction is performed on multimodal data to obtain visual features, audio features, and text features; Semantic features are obtained by cross-modal semantic refinement of visual features, audio features, and text features; Semantic features, visual features, audio features, and text features are subjected to cross-modal spectral alignment to obtain four types of spectral aligned features; The four spectral-aligned features are fused to obtain a fused multimodal feature representation; Sentiment prediction is performed on the fused multimodal feature representation to obtain the sentiment prediction results.

[0009] Furthermore, an electronic device is also provided, including: Memory, used for non-transitory storage of computer-readable instructions; and Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0010] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method described in the first aspect is performed.

[0011] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0012] The above technical solution has the following advantages or beneficial effects: 1) This application utilizes a frozen large language model to generate continuous latent semantics from incomplete text, visual and audio information, achieving high-level semantic compensation for sentiment analysis without the need to reconstruct the original missing data.

[0013] 2) This application proposes a cross-modal semantic refinement module, which constructs a unified multimodal prefix through a lightweight visual and audio adapter, and uses a hidden state iterative appending mechanism to generate compact emotion-related semantics, avoiding explicit text decoding.

[0014] 3) This application proposes a cross-modal spectrum alignment module, which performs anchor-free global nonlinear joint alignment of semantic, textual, visual and audio features through the kernel gram matrix and dominant spectral component enhancement mechanism.

[0015] 4) This application proposes instance-level spectrum separation constraints, which enhance cross-modal consistency of the same sample while maintaining semantic differences between different samples, reducing representation collapse and improving the discriminative ability of sentiment prediction. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0017] Figure 1 This is a schematic diagram of the internal structure of the multimodal sentiment analysis model in Example 1; Figure 2 This is a schematic diagram of the internal structure of the feature extraction module in Example 1; Figure 3 This is a schematic diagram of the cross-modal semantic refinement module in Example 1; Figure 4 This is a schematic diagram of the internal structure of the visual or audio adapter in Embodiment 1; Figure 5 This is a schematic diagram of the internal structure of the cross-modal spectrum alignment module in Embodiment 1; Figure 6 This is a schematic diagram of the internal structure of the modal feature unification unit in Example 1; Figure 7 This is a schematic diagram of the internal structure of the core Gram matrix construction unit in Example 1; Figure 8 This is a schematic diagram of the internal structure of the spectral enhancement unit in Example 1; Figures 9(a) to 9(h) show the confusion matrices of the LNLN method in Example 1 and the present application under different missing rates in the MOSI dataset. Detailed Implementation

[0018] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0019] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0020] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0021] Terminology Explanation: Incomplete multimodal sentiment analysis refers to the prediction of the sentiment tendency, category, or intensity of a target object by comprehensively utilizing observable multimodal information when there are partial missing segments, damaged information, or overall missing information in the text, visual, or audio modalities.

[0022] Latent semantic compensation: refers to using large language models to extract high-level semantic information related to emotion from incomplete multimodal features and supplement the missing information with continuous hidden vectors, without directly reconstructing the original text, image or audio data.

[0023] Cross-modal semantic refinement refers to mapping text, visual, and audio features to a unified embedding space of a large language model, and gradually obtaining semantic representations with sentiment discrimination capabilities by iteratively generating and appending latent semantic states.

[0024] Cross-modal spectral alignment: refers to constructing a kernel gram matrix based on semantic, textual, visual, and audio features, and enhancing the dominant spectral components of the matrix to align different modalities along a common semantic direction in a nonlinear feature space.

[0025] Instance-level spectral separation constraint: refers to imposing differential constraints on the dominant semantic direction of different samples, reducing the similarity of feature representations between samples, avoiding representation collapse, and maintaining the distinguishability between different sentiment samples.

[0026] Example 1 This embodiment provides a semantic-assisted multimodal sentiment analysis method for incomplete data; like Figure 1 As shown, semantic-assisted multimodal sentiment analysis methods for incomplete data include: S101: Obtain the multimodal data to be analyzed, including visual modal data, audio modal data and text modal data, and preprocess the missing data of the multimodal data; S102: Input the multimodal data to be analyzed into the trained multimodal sentiment analysis model to obtain the sentiment prediction results; The trained multimodal sentiment analysis model is used for: Feature extraction is performed on multimodal data to obtain visual features, audio features, and text features; Semantic features are obtained by cross-modal semantic refinement of visual features, audio features, and text features; Semantic features, visual features, audio features, and text features are subjected to cross-modal spectral alignment to obtain four types of spectral aligned features; The four spectral-aligned features are fused to obtain a fused multimodal feature representation; Sentiment prediction is performed on the fused multimodal feature representation to obtain the sentiment prediction results.

[0027] Further, step S101: acquiring the multimodal data to be analyzed, the multimodal data including visual modal data, audio modal data, and text modal data, specifically including: For any multimodal sample to be analyzed, its visual modality data, audio modality data, and text modality data are obtained and represented as follows:

[0028] in, , and These represent the visual modality, audio modality, and text modality, respectively.

[0029] Further, S101: preprocessing the missing data in the multimodal data, specifically including: Modal data may have partial data gaps or overall modal gaps; Among them, local data loss includes missing video frames, missing audio segments, and missing text terms; Overall modality loss refers to the unavailability of at least one of the visual, audio, or text modalities; For locally missing data, the missing status is recorded using a missing location mask.

[0030] The missing video frame features and audio segment features are set to preset fill values, and the missing text terms are set as unknown term tags.

[0031] The padding values ​​and unknown term tags are only used to identify missing locations, and this application is not intended to recover the original missing data.

[0032] Furthermore, such as Figure 2 As shown, the feature extraction from multimodal data to obtain visual features, audio features, and text features specifically includes: Using pre-trained modal encoders corresponding to each modality respectively Feature extraction is performed on the input data to obtain:

[0033] in, Indicates the first The sequence length of each modality, Indicates the first The feature dimensions of a modality.

[0034] In one implementation, a visual encoder is used to extract facial movements, poses, or scene change features from video frames; an audio encoder is used to extract acoustic features such as pitch, energy, speech rate, and spectrum; and a text encoder is used to extract word-level contextual semantic features.

[0035] The visual encoder, implemented using OpenFace, is used to extract facial behavior and visual representation features from video frames; the audio encoder, implemented using Librosa, is used to extract acoustic representation features from audio signals; and the text encoder, implemented using BERT, is used to extract contextual semantic features from text sequences.

[0036] Furthermore, such as Figure 3 As shown, the process of performing cross-modal semantic refinement on visual features, audio features, and text features to obtain semantic features specifically includes: A visual adapter is used to process the visual features to obtain compact cue features for the visual modality; an audio adapter is used to process the audio features to obtain compact cue features for the audio modality. The compact cue features of the visual modality are mapped and transformed to obtain visual prefix features; the compact cue features of the audio modality are mapped and transformed to obtain audio prefix features. Text features are processed using the lexical embedding layer of a large language model to obtain text embeddings; The visual prefix features, audio prefix features, and text embeddings are concatenated to obtain the initial multimodal prefix; The initial multimodal prefix is ​​input into the large language model, and the latent semantics are iteratively refined to obtain the latent semantic state sequence. The latent semantic state sequence is then processed to obtain semantic features.

[0037] The beneficial effect of the above technical solution is that, in order to obtain high-level semantics related to emotion from incomplete multimodal evidence, this application sets up a cross-modal semantic refinement module. The cross-modal semantic refinement module includes a visual adapter, an audio adapter, a multimodal prefix construction unit, and a latent semantic iteration unit.

[0038] Since visual features, audio features, and the word embeddings of large language models differ in feature dimensions and semantic space, this application sets up visual adapters and audio adapters respectively to map non-linguistic modal features to the frozen large language model embedding space.

[0039] Furthermore, such as Figure 4 As shown, the step of using a visual adapter to process visual features to obtain compact cue features for the visual modality and using an audio adapter to process audio features to obtain compact cue features for the audio modality includes: The visual adapter and the audio adapter have the same internal structure. The visual adapter includes one or more attention processing blocks. Each attention processing block includes: a position embedding layer, a cross attention mechanism layer, a self attention mechanism layer and a feedforward network connected in sequence; the output of the feedforward network is also connected to the input of the cross attention mechanism layer. For visual or audio modal ,set up One learnable cue vector:

[0040] To imbue visual frames or audio clips with temporal sequence information, the location embedding layer incorporates modal features. Add learnable positional encoding ,get:

[0041] The cross-attention mechanism layer uses learnable cue vectors As a query, based on modal sequence features Using these as keys and values, a cross-attention operation is performed to aggregate reliable information related to sentiment judgment from the incomplete modality sequence. The computation process is represented as follows:

[0042]

[0043]

[0044] in, , and For trainable mapping parameters, This represents the attention feature dimension.

[0045] The self-attention mechanism layer performs self-attention operations on the updated cue vectors, enabling information exchange between different cue vectors, thereby modeling the dependencies between different time positions within the same modality; After processing by multiple attention processing blocks, the visual adapter obtains compact cue features of the visual modality, and the audio adapter obtains compact cue features of the audio modality.

[0046] It should be understood that learnable cue vectors are a set of trainable continuous vector parameters set internally by the model, and are not manually written text prompts. They are optimized along with the model parameters during training and serve as query vectors in cross-attention, adaptively extracting relevant information for sentiment analysis from visual frame or audio segment features. M cue vectors are set for each modality, with the dimension of each vector consistent with the feature dimension of that modality; after cross-attention, self-attention, and linear mapping, compact modality prefix features are formed for input to a large language model.

[0047] Further, the mapping and transformation of the compact cue features of the visual modality to obtain visual prefix features; and the mapping and transformation of the compact cue features of the audio modality to obtain audio prefix features, includes: The visual adapter output is transformed to the embedding dimension of the large language model via a linear mapping. Obtain visual prefix features The audio adapter output is transformed to the embedding dimension of the large language model via a linear mapping. Obtain audio prefix features ; .

[0048] Furthermore, the text embedding is obtained by processing the text features using the lexical embedding layer of the large language model, including: The text modality is processed through the lexical embedding layer of a large language model to obtain the text embedding. :

[0049] in, The length of the text embedding sequence.

[0050] The text modality is embedded through the frozen lexical embedding layer of the large language model itself, mapping discrete text lexical units into continuous text embeddings consistent with the hidden dimensions of the large language model. In one specific implementation, the large language model employs Qwen3-1.7B, whose lexical embedding layer is used to obtain text embeddings.

[0051] Further, the concatenation of the visual prefix features, audio prefix features, and text embedding to obtain the initial multimodal prefix includes: The text embedding, visual prefix features, and audio prefix features are concatenated in a preset order to form the initial multimodal prefix:

[0052] Initial multimodal prefix As a continuous input sequence for a frozen large language model, textual, visual, and audio evidence reside within a unified semantic embedding space.

[0053] The large language model maintains its parameters frozen during the training of the sentiment analysis model of this application, primarily utilizing its pre-trained context modeling and semantic abstraction capabilities. By freezing the parameters of the large language model, the number of parameters that need to be updated can be reduced, and the destruction of the existing semantic capabilities of the large language model during training on limited sentiment data can be minimized.

[0054] Furthermore, the step of inputting the initial multimodal prefix into a large language model and performing latent semantic iterative refinement to obtain a latent semantic state sequence includes: (1) Let the frozen large language model be Its parameters are ; No. During secondary semantic refinement, the input to the large language model is: ; Among them, when At that time, the input to the large language model is the initial multimodal prefix. ; when At that time, the input of the large language model is the first... The updated multimodal prefix obtained after round semantic refinement ; Each round generates latent semantic states. Afterwards, Added to the The wheel prefix is ​​obtained ; For the input sequence The large language model outputs the last hidden state corresponding to each position in the current input sequence. , ; In one specific implementation, the frozen large language model uses Qwen3-1.7B. The model parameters are kept frozen and used to generate latent semantic hidden states based on multimodal prefixes. It should be noted that Qwen3-1.7B is a specific embodiment, and the large language model of this invention is not limited to this model in principle; other pre-trained large language models with corresponding hidden state output capabilities can also be used.

[0055] Each sequence position refers to the current multimodal prefix sequence input to the large language model. The multimodal prefix is ​​composed of multiple consecutive feature vectors arranged in sequence, including the positions corresponding to text word embeddings, visual prefix vectors, audio prefix vectors, and latent semantic states added during subsequent semantic refinement. The large language model outputs a final hidden state for each of these sequence positions, resulting in a single hidden state sequence as the overall output.

[0056] (2) Take the hidden state corresponding to the last position of the input sequence as the context semantic summary of the current multimodal prefix. :

[0057] This represents the context semantic summary extraction function, used to extract the hidden state corresponding to the last position of the input sequence from the last hidden state sequence output by the large language model; it is an operation that takes values ​​from the frozen output of the large language model.

[0058] For the current input sequence ,have:

[0059] in, This indicates that the large language model is frozen. This indicates the length of the current input sequence. Therefore, This represents the last hidden state corresponding to the last position of the current input sequence, and serves as a contextual semantic summary of the current multimodal prefix. The last position of the input sequence refers to the multimodal prefix of the current input large language model. The position of the last consecutive feature vector arranged in sequence.

[0060] During the initial semantic refinement, due to Its last position corresponds to the last audio prefix vector arranged according to the preset concatenation order; starting from the second semantic refinement, since the latent semantic states generated in the previous round have been added to the input sequence, the last position of the current input sequence corresponds to the latent semantic states added in the previous round. The hidden state at the last position of the large language model comprehensively encodes the contextual information of all preceding multimodal prefixes, and is therefore used to generate the latent semantic state for this round.

[0061] (3) in the In the next semantic refinement process, based on the first The multimodal prefix generation after the second update One potential semantic state:

[0062] in, This is the preset number of semantic refinements.

[0063] (4) Generate latent semantic states Appended as a continuous input unit to the current multimodal prefix, we get:

[0064] Due to latent semantic state It has the same dimension as the lexical embeddings of large language models, so there is no need to convert them into natural language lexical units before they can directly participate in the next hidden state calculation.

[0065] (5) Repeat steps (1) to (4) to obtain the latent semantic state sequence. :

[0066] Further, the processing of the latent semantic state sequence to obtain semantic features includes: Pooling and linear projection are performed on the latent semantic state sequence to obtain sentiment-related semantic features:

[0067] in, This represents average pooling, max pooling, attention pooling, or a combination thereof. and These represent the trainable projection matrix and the bias, respectively.

[0068] Furthermore, such as Figure 5As shown, the cross-modal spectral alignment processing of semantic features, visual features, audio features, and text features yields four types of spectrally aligned features, specifically including: Visual features, audio features, and text features are processed to obtain unified-dimensional visual feature modal representations, audio feature modal representations, and text feature modal representations; Normalize the semantic features, visual feature modal representations, audio feature modal representations, and text feature modal representations; The normalized semantic features, visual feature modal representations, audio feature modal representations, and text feature modal representations are arranged to obtain a shared representation matrix; Based on the shared representation matrix, a linear Galam matrix is ​​constructed; Each feature in the linear Gram matrix is ​​mapped to the regenerated kernel Hilbert space, and the kernel Gram matrix is ​​constructed. Eigenvalue decomposition is performed on the Kergram matrix to obtain eigenvalues; The eigenvalues ​​are used as spectral distribution scores, and the dominant spectral enhancement loss is calculated based on the spectral distribution scores. Calculate the dominant feature vector corresponding to the largest feature value of each sample, map the dominant vector back to the original shared representation space, and normalize it to obtain the dominant semantic direction of each sample; The squared similarity between the dominant semantic directions of different samples is calculated to obtain the instance-level spectral separation loss.

[0069] To ensure consistency between latent semantic features and textual, visual, and audio features, this application includes a cross-modal spectrum alignment module. This module comprises a modal feature unification unit, a kernel grammar matrix construction unit, a dominant spectrum enhancement unit, and an instance-level spectrum separation unit.

[0070] Furthermore, visual features, audio features, and text features are processed to obtain a unified-dimensional modal representation, specifically including: Visual sequence features, audio sequence features, and text sequence features are encoded, pooled, and linearly mapped respectively to obtain a modal representation with a unified dimension:

[0071] Each modality coding process can consist of a linear mapping layer and one or more Transformer coding layers.

[0072] Furthermore, the semantic features, visual feature modal representations, audio feature modal representations, and text feature modal representations are normalized, specifically including: Normalize semantic features and modal features:

[0073] Furthermore, such as Figure 6 As shown, the normalized semantic features, visual feature modal representations, audio feature modal representations, and text feature modal representations are arranged to obtain a shared representation matrix, which specifically includes: The normalized semantic, visual, audio, and text features are arranged column-wise to form a shared representation matrix:

[0074] Furthermore, the construction of the linear glam matrix based on the shared representation matrix specifically includes: First, a linear glam matrix can be constructed based on the shared representation matrix:

[0075] The Gram matrix is ​​used to describe the pairwise geometric relationships between different feature representations in the same sample.

[0076] Furthermore, such as Figure 7 As shown, the process of mapping each feature in the linear Gram matrix to the regenerating kernel Hilbert space and constructing the kernel Gram matrix specifically includes: To further capture the nonlinear semantic relationships between different modalities, a kernel function is used to map each feature to a regenerating kernel Hilbert space, and a kernel Gram matrix is ​​constructed. :

[0077] The kernel function is a radial basis function kernel:

[0078] in, This refers to the kernel bandwidth parameter.

[0079] Nuclear Gram Matrix It also incorporates non-linear similarity relationships between semantic, visual, audio, and textual features. When the four types of features of the same sample are highly consistent in the kernel space, the kernel Gram matrix tends to be a low-rank structure; in the ideal case, when the features are aligned along the same semantic direction, the kernel Gram matrix tends to be a rank-one matrix.

[0080] Furthermore, such as Figure 8 As shown, the eigenvalue decomposition of the kernel Gram matrix to obtain eigenvalues ​​specifically includes: Perform eigenvalue decomposition on the kernel-gram matrix:

[0081]

[0082] Maximum eigenvalue This corresponds to the dominant nonlinear semantic direction shared by semantic, visual, audio, and textual features within the same sample. The higher the proportion of the largest eigenvalue among all eigenvalues, the more concentrated the features of different modalities are in a common latent semantic subspace.

[0083] Furthermore, the step of using the eigenvalues ​​as spectral distribution scores and calculating the dominant spectral enhancement loss based on the spectral distribution scores specifically includes: Using the eigenvalues ​​of the nuclear Gram matrix as spectral distribution scores, first define the... The spectral probability of each sample is:

[0084] Dominant Spectrum Enhancement Loss :

[0085] in, This represents the number of samples in the batch. For temperature parameters, Indicates the first The first sample's kernel gram matrix 1 eigenvalue, This represents the spectral probability corresponding to the largest eigenvalue.

[0086] By minimizing This increases the proportion of the largest feature value relative to the other feature values, causing the four types of features of the same sample to cluster along a common semantic direction.

[0087] Enhancing only the dominant spectral components within the same sample may cause different samples to be mapped to similar semantic directions. To maintain the distinguishability between different sentiment samples, this application further sets instance-level spectral separation constraints.

[0088] Further, the process of calculating the dominant feature vector corresponding to the largest feature value of each sample, mapping the dominant vector back to the original shared representation space, and normalizing it to obtain the dominant semantic direction of each sample specifically includes: For the For each sample, obtain its maximum eigenvalue. Corresponding dominant feature vector Mapping the dominant feature vector back to the original shared representation space and normalizing it yields the first... The dominant semantic direction of each sample:

[0089] Furthermore, the calculation of the squared similarity between the dominant semantic directions of different samples to obtain the instance-level spectral separation loss specifically includes: Calculate the squared similarity between the dominant semantic directions of different samples to obtain the instance-level spectral separation loss. :

[0090] By minimizing This reduces the similarity between the dominant semantic directions of different samples and prevents the feature representations of different emotion samples from converging to the same position.

[0091] Furthermore, the fusion of the four spectral-aligned features to obtain the fused multimodal feature representation specifically includes: The normalized semantic features, visual feature modal representation, audio feature modal representation, and text feature modal representation are fused to obtain a multimodal feature representation.

[0092] The latent semantic features, visual features, audio features, and text features that have completed spectral alignment are fused to obtain the final multimodal representation:

[0093] Besides element-wise summation, weighted summation, linear projection after concatenation, or attention fusion can also be used. When using weighted summation, it can be expressed as:

[0094] in, , , and These are preset or learnable fusion weights.

[0095] Furthermore, the process of performing sentiment prediction on the fused multimodal feature representation to obtain the sentiment prediction result specifically includes: The fused multimodal representation is then input into the sentiment prediction module:

[0096] When the sentiment analysis task is a sentiment intensity regression task This represents the predicted sentiment intensity value. When the sentiment analysis task is a sentiment classification task, Softmax processing is performed on the output of the linear layer to obtain the predicted probability of each sentiment category.

[0097] For regression tasks, the mean squared error can be used to predict the loss:

[0098] in, For the first The true sentiment labels of each sample.

[0099] Further, in step S102: the multimodal data to be analyzed is input into the trained multimodal sentiment analysis model to obtain sentiment prediction results, wherein the trained multimodal sentiment analysis model includes: The module consists of a feature extraction module, a cross-modal semantic refinement module, a cross-modal spectrum alignment module, a multimodal feature fusion module, and a sentiment prediction module, which are connected in sequence.

[0100] The trained multimodal sentiment analysis model, and the total loss function during training, include: The overall training loss is:

[0101] in, and These are the weights for the cross-modal spectral alignment loss and the instance-level spectral separation loss, respectively. In one implementation, and All are set to 1.

[0102] When the task is a sentiment classification task, you can Replace it with cross-entropy loss.

[0103] During model training, it is preferable to freeze the pre-trained modal encoders and the large language model, and update the parameters of the visual adapter, audio adapter, modal feature unification network, semantic projection layer, and sentiment prediction layer. After the model training is completed, the incomplete multimodal data to be analyzed is input into the model, and the corresponding sentiment category or sentiment intensity can be output.

[0104] This application provides an incomplete multimodal sentiment analysis method and system based on latent semantic assistance and anchorless spectrum alignment. The method includes processes such as multimodal data preprocessing, cross-modal semantic refinement, cross-modal spectrum alignment, multimodal feature fusion, and sentiment prediction.

[0105] This application first utilizes a modal encoder to extract sequence features from incomplete text, visual, and audio data. Then, through lightweight visual and audio adapters, non-linguistic modal features are mapped to a frozen large language model embedding space and concatenated with text features to form a multimodal prefix. Sentiment-related latent semantic states are iteratively generated in the continuous hidden space of the large language model to achieve high-level semantic compensation for missing information. Furthermore, kernel gram matrices corresponding to semantic, text, visual, and audio features are constructed, and anchorless cross-modal joint alignment is achieved by enhancing dominant spectral components. Instance-level spectral separation constraints maintain discriminability between different samples. Finally, the features of each modality are fused, and sentiment prediction results are output.

[0106] This application was validated on three publicly available multimodal sentiment analysis datasets. MOSI and MOSEI are English multimodal sentiment analysis datasets, with sentiment annotation ranges of [missing information]. 3 to +3; SIMS is a Chinese multimodal sentiment analysis dataset, with sentiment annotation ranges from 3 to +3. 1 to +1. All three datasets contain text, visual, and audio information, which can be used to verify the applicability of this application under different languages, different data sizes, and different missing data conditions.

[0107] In the intramodal missing settings, missing locations of visual frames and audio segments are represented by zero vectors, while missing locations of text terms are represented by unknown term tags. During training, sample-level Bernoulli masks are used, retaining 50% of the complete samples. During testing, the missing rate is gradually increased from 0.0 to 0.9 with a step size of 0.1, and missing locations for each modality are sampled independently. The experimental results are the average of three experiments with random seeds 1111, 1112, and 1113. MOSI and MOSEI are evaluated using Acc-2, F1, Acc-5, Acc-7, MAE, and Pearson correlation coefficients; SIMS is evaluated using Acc-2, F1, Acc-3, Acc-5, MAE, and Pearson correlation coefficients.

[0108] The model in this application is trained in 200 epochs with a batch size of 64, using the AdamW optimizer and an initial learning rate of 1×10⁻⁶. -4 The algorithm incorporates learning rate warm-up, cosine annealing, and early stopping strategies. The input sequence length is set to 8, and the unified hidden dimension is set to 128. The frozen large language model uses Qwen3-1.7B. The visual and audio adapters each have 8 learnable cue vectors and employ 2 attention blocks; the number of latent semantic refinements is set to 4. The radial basis function kernel bandwidth σ is set to 1.0, and the spectral alignment temperature parameter τ is set to 0.1. Experiments were conducted on an NVIDIA RTX 6000 Ada GPU.

[0109] Under intramodal missing conditions with missing rates ranging from 0.0 to 0.9, this application achieves superior average results on the MOSI, MOSEI, and SIMS datasets. These results demonstrate that latent semantic compensation can reduce the loss of high-level sentiment information caused by missing local video frames, audio segments, or text terms; and that anchorless spectral alignment can avoid using a single modality with severe missing or noisy features as a fixed alignment benchmark.

[0110] On MOSI, the Acc-5 score of this application reached 40.93, an improvement of 2.43 percentage points compared to the highest baseline result of 38.50 in the prior art; the Acc-7 score reached 36.49, an improvement of 2.23 percentage points compared to the highest baseline result of 34.26; and the MAE decreased to 1.011. On MOSEI, the Acc-5 score of this application reached 48.12, an improvement of 1.48 percentage points compared to the highest baseline result of 46.64; the MAE decreased to 0.648, and the Pearson correlation coefficient increased to 0.601. On SIMS, the F1 score of this application reached 77.50, an improvement of 1.29 percentage points compared to the highest baseline result of 76.21; the Acc-3 score reached 58.84, an improvement of 1.52 percentage points compared to the highest baseline result of 57.32; and the Pearson correlation coefficient reached 0.501, an improvement of 0.087 compared to the highest baseline result of 0.414.

[0111] Table 1. Average comparison results of this application and existing methods on the MOSI dataset.

[0112] Note: The results are the average of missing rates from 0.0 to 0.9; the two values ​​in Acc-2 and F1 correspond to the two divisions of "negative / non-negative" and "negative / positive", respectively; the lower the MAE, the better, and the higher the other indicators, the better.

[0113] Table 2. Average comparison results of this application and existing methods on the MOSEI dataset.

[0114] Note: Results are the average of missing rates from 0.0 to 0.9; "—" indicates that the corresponding results were not reported in the original paper; the lower the MAE, the better, and the higher the other indicators, the better.

[0115] Table 3. Average comparison results of this application and existing methods on the SIMS dataset.

[0116] Note: Results are the average of missing rates from 0.0 to 0.9; lower MAE is better, and higher values ​​for other indicators are better.

[0117] Under the overall modality missing setting, this application further evaluated the results on MOSEI with single modality, two modalities, and all three modalities retained. The curly braces in Table 4 indicate available modality combinations, where l, a, and v represent text, audio, and visual modalities, respectively. Experimental results show that this application achieved the highest F1 score in all seven modality combinations, with an average F1 score of 77.89, which is 2.71 percentage points higher than the second-best method, CorrKD, which scored 75.18.

[0118] When only the text modality is retained, this application achieves an F1 score of 83.61; when only the audio or visual modality is retained, the F1 scores are 66.59 and 65.91, respectively; and when the text modality is unavailable and only the audio and visual modalities are retained, the F1 score reaches 72.68. These results demonstrate that the visual and audio adapters can extract emotion-related information from non-linguistic evidence and form high-level semantic representations usable for prediction through semantic refinement of the large language model's hidden space, thereby reducing dependence on a fixed dominant modality.

[0119] Table 4. F1 Comparison Results of this Application under Different Available Mode Combinations in MOSEI

[0120] Note: l, a, and v represent text, audio, and visual modalities, respectively; curly braces indicate modal combinations available during reasoning.

[0121] Figures 9(a) to 9(h) show the confusion matrices of the LNLN method and this application on the MOSI dataset under different missing rates, where the missing rates are set to 0.1, 0.5, 0.7, and 0.9, respectively. Categories 0 to 6 represent strong negative, negative, weak negative, neutral, weak positive, positive, and strong positive sentiment, respectively. As the missing rate increases from 0.1 to 0.9, the recognition performance of both methods decreases, but this application maintains a clearer diagonal structure at all missing rates, indicating that it has a more stable discrimination ability for different sentiment categories. Especially under high missing rate conditions, the prediction results of LNLN are significantly concentrated in the neutral category, while this application can effectively reduce this phenomenon and improve the recognition ratio of strong negative, negative, positive, and strong positive categories. For example, when the missing rate is 0.9, the correct recognition rate of the strong positive category of this application is increased from 22% for LNLN to 44%. These results demonstrate that cross-modal semantic refinement can supplement high-level sentiment semantics from remaining modalities, and cross-modal spectrum alignment can enhance the consistency of representations across different modalities and maintain differences between samples, thereby mitigating the degradation of classification boundaries caused by severe data gaps and improving the accuracy, stability, and robustness of incomplete multimodal sentiment analysis.

[0122] This application proposes an incomplete multimodal sentiment analysis method and system based on latent semantic assistance and anchorless spectrum alignment, aiming to address the problems existing in current technologies, such as inaccurate reconstruction of missing modal features, insufficient high-level sentiment semantics, high overhead of explicit text generation, insufficient cross-modal global alignment, and easy collapse of sample representations. By designing a cross-modal semantic refinement module and a cross-modal spectrum alignment module, sentiment-related continuous latent semantics are generated using a large language model without directly reconstructing missing data. Furthermore, anchorless global nonlinear alignment is performed on semantic, textual, visual, and audio features, thereby improving the accuracy, stability, and robustness of sentiment prediction under incomplete data conditions.

[0123] Specifically, this application first utilizes a text encoder, a visual encoder, and an audio encoder to extract features from text, visual, and audio data that are partially or entirely missing. Addressing the issue of dimensional and semantic differences between visual and audio features and the large language model embedding space, this application sets up lightweight visual and audio adapters. Through learnable cues, positional encoding, self-attention, and cross-attention, reliable information from each modality sequence is adaptively aggregated and mapped to the frozen large language model embedding space.

[0124] Building upon this foundation, this application proposes a cross-modal semantic refinement module. This module concatenates text embeddings, visual prefix features, and audio prefix features in a predetermined order to form a unified multimodal prefix, which is then input into a frozen large language model. Unlike existing methods for generating sentiment descriptions or inference text, this application directly extracts the hidden state at the last position of the large language model, appends it as a continuous latent semantic state to the current multimodal prefix, and repeats the hidden state extraction and appending operations. Through multiple iterations, sentiment-related semantics are progressively refined within the continuous hidden space of the large language model, resulting in a compact and discriminative latent semantic representation. This avoids the additional computational overhead and text generation bias caused by autoregressive text decoding.

[0125] Secondly, this application proposes a cross-modal spectral alignment module, which normalizes latent semantic features, text features, visual features, and audio features, and constructs a kernel Gram matrix using kernel functions to describe the global nonlinear relationships between the features. By performing eigenvalue decomposition on the kernel Gram matrix, the dominant spectral component corresponding to the largest eigenvalue is enhanced, enabling different modal representations of the same sample to cluster along a common latent semantic direction, thereby achieving anchor-free cross-modal joint alignment independent of text modality or other dominant modalities.

[0126] Furthermore, to prevent different samples from converging to the same semantic representation during cross-modal alignment, this application sets instance-level spectral separation constraints, maps the dominant feature vectors of the kernel Gram matrix back to the feature space, and reduces the similarity between the dominant semantic directions of different samples to maintain the distinguishability between different sentiment samples. Finally, the latent semantic, textual, visual, and audio features that have been aligned are fused and input into the prediction module to obtain the corresponding sentiment category or sentiment intensity prediction results.

[0127] This application can generate high-level sentiment semantics using existing modal evidence in cases of partial missing text terms, video frames, audio segments, and overall missing modalities, and achieves overall consistency alignment of multiple heterogeneous features. Related experimental results show that this application can improve the prediction performance and stability of multimodal sentiment analysis under different data missing ratios, while avoiding the high computational costs associated with missing feature reconstruction and explicit text decoding.

[0128] Example 2 This embodiment provides a semantic-assisted multimodal sentiment analysis system for incomplete data, including: The acquisition module is configured to: acquire multimodal data to be analyzed, including visual modal data, audio modal data and text modal data, and preprocess missing data in the multimodal data; The sentiment analysis module is configured to input the multimodal data to be analyzed into the trained multimodal sentiment analysis model and obtain the sentiment prediction results. The trained multimodal sentiment analysis model is used for: Feature extraction is performed on multimodal data to obtain visual features, audio features, and text features; Semantic features are obtained by cross-modal semantic refinement of visual features, audio features, and text features; Semantic features, visual features, audio features, and text features are subjected to cross-modal spectral alignment to obtain four types of spectral aligned features; The four spectral-aligned features are fused to obtain a fused multimodal feature representation; Sentiment prediction is performed on the fused multimodal feature representation to obtain the sentiment prediction results.

[0129] It should be noted that the acquisition module and sentiment analysis module mentioned above correspond to steps S101 to S102 in Embodiment 1. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0130] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0131] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0132] Example 3 This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0133] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0134] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0135] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0136] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0137] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] Example 4 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0139] Example 5 This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements the method in Embodiment 1.

[0140] This application also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0141] The computer program code used to implement the methods of this application may be written in one or more programming languages. This computer program code may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0142] In the context of this application, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0143] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A semantic aided multi-modal sentiment analysis method for incomplete data, characterized in that, include: Acquire the multimodal data to be analyzed, which includes visual modal data, audio modal data, and text modal data, and preprocess the missing data in the multimodal data; The multimodal data to be analyzed is input into the trained multimodal sentiment analysis model to obtain sentiment prediction results; The trained multimodal sentiment analysis model is used for: Feature extraction is performed on multimodal data to obtain visual features, audio features, and text features; Semantic features are obtained by cross-modal semantic refinement of visual features, audio features, and text features; Semantic features, visual features, audio features, and text features are subjected to cross-modal spectral alignment to obtain four types of spectral aligned features; The four spectral-aligned features are fused to obtain a fused multimodal feature representation; Sentiment prediction is performed on the fused multimodal feature representation to obtain the sentiment prediction results.

2. The semantic aided multi-modal sentiment analysis method for incomplete data according to claim 1, characterized in that, The process of performing cross-modal semantic refinement on visual features, audio features, and text features to obtain semantic features specifically includes: A visual adapter is used to process the visual features to obtain compact cue features for the visual modality; an audio adapter is used to process the audio features to obtain compact cue features for the audio modality. The compact cue features of the visual modality are mapped and transformed to obtain visual prefix features; the compact cue features of the audio modality are mapped and transformed to obtain audio prefix features. Text features are processed using the lexical embedding layer of a large language model to obtain text embeddings; The visual prefix features, audio prefix features, and text embeddings are concatenated to obtain the initial multimodal prefix; The initial multimodal prefix is ​​input into the large language model, and the latent semantics are iteratively refined to obtain the latent semantic state sequence. The latent semantic state sequence is then processed to obtain semantic features.

3. The semantic aided multi-modal sentiment analysis method for incomplete data as claimed in claim 2, wherein, The visual features are processed using a visual adapter to obtain compact cue features of the visual modality; An audio adapter is used to process the audio features to obtain compact cue features of the audio modality, including: The visual adapter and the audio adapter have the same internal structure. The visual adapter includes one or more attention processing blocks. Each attention processing block includes: a position embedding layer, a cross attention mechanism layer, a self attention mechanism layer and a feedforward network connected in sequence; the output of the feedforward network is also connected to the input of the cross attention mechanism layer. For the visual modality or the audio modality , set a learnable prompt vector: To give the visual frames or audio segments temporal order information, the positional embedding layer adds learnable position encodings to the modality features , which gives:​ Cross-attention mechanism layer with learnable cue vector As queries, with modal sequence features As keys and values, performing cross-attention operation to aggregate reliable information related to sentiment judgment from incomplete modal sequence, whose computation process is represented as: in, , and For trainable mapping parameters, For attention feature dimensions; The self-attention mechanism layer performs self-attention operations on the updated cue vectors, enabling information exchange between different cue vectors, thereby modeling the dependencies between different time positions within the same modality; After processing by multiple attention processing blocks, the visual adapter obtains compact cue features of the visual modality, and the audio adapter obtains compact cue features of the audio modality.

4. The semantic-assisted multimodal sentiment analysis method for incomplete data as described in claim 2, characterized in that, The process of inputting the initial multimodal prefix into a large language model and performing iterative refinement of latent semantics to obtain a sequence of latent semantic states includes: (1) Let the frozen large language model be Its parameters are ; No. During secondary semantic refinement, the input to the large language model is: ; Among them, when At that time, the input to the large language model is the initial multimodal prefix. ; when At that time, the input of the large language model is the first... The updated multimodal prefix obtained after round semantic refinement ; Each round generates latent semantic states. Afterwards, Added to the The wheel prefix is ​​obtained ; For the input sequence The large language model outputs the last hidden state corresponding to each position in the current input sequence. , ; (2) Take the hidden state corresponding to the last position of the input sequence as the context semantic summary of the current multimodal prefix. : This represents the context semantic summary extraction function, used to extract the hidden state corresponding to the last position of the input sequence from the last hidden state sequence output by the large language model; it is an operation that extracts values ​​from the frozen output of the large language model. For the current input sequence ,have: in, This indicates that the large language model is frozen. Indicates the length of the current input sequence; (3) in the In the next semantic refinement process, based on the first The multimodal prefix generation after the second update One potential semantic state: in, This is the preset number of semantic refinements; (4) Generate latent semantic states Appended as a continuous input unit to the current multimodal prefix, we get: (5) Repeat steps (1) to (4) to obtain the latent semantic state sequence. : 。 5. The semantic-assisted multimodal sentiment analysis method for incomplete data as described in claim 1, characterized in that, The process of performing cross-modal spectral alignment on semantic features, visual features, audio features, and text features to obtain four types of spectral aligned features specifically includes: Visual features, audio features, and text features are processed to obtain unified-dimensional visual feature modal representations, audio feature modal representations, and text feature modal representations; Normalize the semantic features, visual feature modal representations, audio feature modal representations, and text feature modal representations; The normalized semantic features, visual feature modal representations, audio feature modal representations, and text feature modal representations are arranged to obtain a shared representation matrix; Based on the shared representation matrix, a linear Galam matrix is ​​constructed; Each feature in the linear Gram matrix is ​​mapped to the regenerated kernel Hilbert space, and the kernel Gram matrix is ​​constructed. Eigenvalue decomposition is performed on the Kergram matrix to obtain eigenvalues; The eigenvalues ​​are used as spectral distribution scores, and the dominant spectral enhancement loss is calculated based on the spectral distribution scores. Calculate the dominant feature vector corresponding to the largest feature value of each sample, map the dominant vector back to the original shared representation space, and normalize it to obtain the dominant semantic direction of each sample; The squared similarity between the dominant semantic directions of different samples is calculated to obtain the instance-level spectral separation loss.

6. The semantic-assisted multimodal sentiment analysis method for incomplete data as described in claim 5, characterized in that, The eigenvalue decomposition of the kernel Gram matrix to obtain eigenvalues ​​specifically includes: Perform eigenvalue decomposition on the kernel-gram matrix: Maximum eigenvalue This corresponds to the dominant nonlinear semantic direction shared by semantic, visual, audio, and textual features within the same sample; or, The step of using eigenvalues ​​as spectral distribution scores and calculating the dominant spectral enhancement loss based on those scores specifically includes: Using the eigenvalues ​​of the nuclear Gram matrix as spectral distribution scores, first define the... The spectral probability of each sample is: Dominant Spectrum Enhancement Loss : in, This represents the number of samples in the batch. For temperature parameters, Indicates the first The first sample's kernel gram matrix 1 eigenvalue, This represents the spectral probability corresponding to the largest eigenvalue; or, The process of calculating the dominant feature vector corresponding to the largest feature value of each sample, mapping the dominant vector back to the original shared representation space, and normalizing it to obtain the dominant semantic direction of each sample specifically includes: For the For each sample, obtain its maximum eigenvalue. Corresponding dominant feature vector Mapping the dominant feature vector back to the original shared representation space and normalizing it yields the first... The dominant semantic direction of each sample: 。 7. The semantic-assisted multimodal sentiment analysis method for incomplete data as described in claim 5, characterized in that, The calculation of the squared similarity between the dominant semantic directions of different samples to obtain the instance-level spectral separation loss specifically includes: Calculate the squared similarity between the dominant semantic directions of different samples to obtain the instance-level spectral separation loss. : By minimizing This reduces the similarity between the dominant semantic directions of different samples and prevents the feature representations of different emotion samples from converging to the same position.

8. A semantically assisted multimodal sentiment analysis system for incomplete data, characterized by: include: The acquisition module is configured to: acquire multimodal data to be analyzed, including visual modal data, audio modal data and text modal data, and preprocess missing data in the multimodal data; The sentiment analysis module is configured to input the multimodal data to be analyzed into the trained multimodal sentiment analysis model and obtain the sentiment prediction results. The trained multimodal sentiment analysis model is used for: Feature extraction is performed on multimodal data to obtain visual features, audio features, and text features; Semantic features are obtained by cross-modal semantic refinement of visual features, audio features, and text features; Semantic features, visual features, audio features, and text features are subjected to cross-modal spectral alignment to obtain four types of spectral aligned features; The four spectral-aligned features are fused to obtain a fused multimodal feature representation; Sentiment prediction is performed on the fused multimodal feature representation to obtain the sentiment prediction results.

9. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-7.

10. A storage medium, characterized in that, Non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method of any one of claims 1-7 is performed.