Robust multi-modal sentiment analysis method and model based on conformal regression feature repair

Abstract

The application relates to the field of software and discloses a robust multi-modal sentiment analysis method and model based on conformal regression feature repair. First, through a conformal decomposition and anchor point selection module, the extracted noise features are decomposed into orthogonal latent variants, and a statistical trust region is constructed by using a conformal prediction theory to accurately screen out high-confidence semantic anchors from damaged data; subsequently, through an anchor point guided feature repair module, the anchors are used as navigation signals to align the global feature distribution, and a soft gating mechanism is combined to eliminate fine-grained variance fluctuations. The application not only effectively reverses semantic drift caused by data pollution, maximally retains intrinsic effective information, and supports zero-delay post-fusion inference in the model deployment stage, and significantly improves the sentiment prediction accuracy and robustness of the model in a complex real noise environment.

Description

Technical Field

[0001] This invention relates to the field of software, and more particularly to a robust multimodal sentiment analysis method and model based on conformal regression feature repair. Background Technology

[0002] With the rapid development of artificial intelligence and human-computer interaction technologies, multimedia data is experiencing explosive growth. Driven by this trend, academia and industry have raised new demands for more accurate and intelligent emotion understanding and intent capture. To more effectively extract target emotional features from complex and heterogeneous data, researchers have proposed the task of multimodal sentiment analysis. Specifically, the goal of multimodal sentiment analysis is to effectively integrate users' natural language text, visual facial expressions, and audio acoustic features to accurately predict the emotional tendency and intensity of the target object. This technology has a wide range of applications, such as intelligent medical emotion monitoring, intelligent customer service systems, and intelligent security emotion early warning. By deploying multimodal sentiment analysis technology, the naturalness of human-computer interaction can be effectively improved, the development cost of emotion intelligence-related applications can be reduced, and the efficiency of data analysis in complex scenarios can be increased.

[0003] Existing multimodal sentiment analysis methods can be broadly categorized into traditional methods for ideal data and robust enhancement methods for low-quality data. The former, the primary approach in early sentiment analysis, typically assumes that training and testing data are acquired under ideal laboratory conditions, completely free of noise contamination. By learning strong supervisory signals from high-quality, multimodal input information, this ideal data-based analysis method has achieved significant performance in this task and has been extensively studied. However, in most real-world multimedia applications, the acquired data is often highly susceptible to real-world noise contamination due to sensor failures, signal loss, or complex background interference. This contamination can induce severe semantic shifts in the feature space, thereby disrupting the reliable mapping between features and sentiment intensity and limiting the practical application of traditional analysis methods. Therefore, in recent years, researchers have proposed robust multimodal sentiment analysis methods for low-quality data. These methods attempt to mitigate the impact of noise by strengthening the dominant modality or decoupling features. However, existing robust methods often employ a coarse, overall suppression strategy, treating damaged modalities as globally unreliable data sources. This approach is highly prone to modality misalignment and strong encoder bias. Because these methods crudely discard the remaining valid information in the damaged modes, their performance often drops significantly when faced with high missing rates or complex noise.

[0004] Furthermore, both of these traditional approaches suffer from a critical flaw: the efficiency of model inference. To achieve noise resistance, most existing robust models heavily rely on extremely complex cross-modal interactions and online feature reconstruction computations during the inference phase. Much of this computation can only be performed after receiving user input, resulting in a massive online computational burden and a significant waste of time and computational resources. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a robust multimodal sentiment analysis method based on conformal regression feature repair, comprising the following steps: Obtain the original multimodal input data, and divide the dataset corresponding to the original multimodal input data into a calibration set and a training set; use a feature extraction network to extract the temporal feature representation sequences of the language modality, visual modality and audio modality in the original multimodal input data respectively; The temporal feature representation sequence is input into a temporal average pooling layer to extract a global feature representation vector; Using the global feature representation vector as the initial input, a projection mapping is performed through a multilayer perceptron to obtain a candidate set containing multiple potential variants; The single-modal predictor from the previous training round is used to make predictions on the calibration set. The prediction residuals are calculated as inconsistency scores, and all the inconsistency scores are sorted in ascending order. The corresponding global quantile threshold is determined from the ascending order results based on the set significance level parameter. Based on the global quantile threshold, a corresponding conformal prediction interval is constructed for each potential variant in the candidate set; The absolute deviation between the point prediction value of each latent variant and the set reference target is calculated. The absolute deviation is input into a pre-built piecewise function for calculation to obtain the score of each latent variant. The reference target includes the predicted values ​​of the true sentiment label and the text-dominant modality. Based on the scores, the potential variants in the candidate set are sorted in descending order, a preset number of potential variants at the top of the sort are extracted, and the extracted potential variants are input into the average pooling layer for pooling operation, and semantic anchors are output. The semantic anchor is subtracted from the corresponding global feature representation vector to obtain the center correction vector. The center correction vector is then extended along the temporal dimension, and the extended vector is superimposed on the temporal feature representation sequence to obtain the aligned feature sequence. Calculate the feature variance of multiple potential variants in the candidate set across multiple dimensions, generate gating weights using the feature variances, and perform element-wise multiplication of the gating weights with the aligned feature sequence to obtain the repaired feature representation. The repaired feature representations of the language modality, visual modality, and audio modality are concatenated along the feature dimension to construct a multimodal joint representation. The multimodal joint representation is then input into a multimodal predictor for prediction. The overall objective function, which is composed of the multimodal main task prediction loss, the single-modal auxiliary task regression loss, and the latent variant orthogonality loss, is calculated. The model parameters are then updated using the backpropagation algorithm. Extract the temporal feature representation sequence of each modality in the multimodal data to be tested, and directly concatenate the extracted temporal feature representation sequence of each modality along the feature dimension to construct a joint multimodal test representation. Input the constructed joint multimodal test representation into the trained multimodal predictor for a single forward inference and output the sentiment analysis prediction score.

[0006] The projection mapping using a multilayer perceptron yields a candidate set containing multiple potential variants, including: During the projection mapping process, the cosine similarity matrix between each potential variant in the normalized potential variant candidate set is calculated. Calculate the Frobenius norm of the difference between the cosine similarity matrix and the identity matrix, and add the calculated Frobenius norm as the potential variant orthogonality loss to the overall objective function.

[0007] The step of determining the corresponding global quantile threshold in the ascending order results based on the set significance level parameter includes: For each sample in the calibration set, the absolute value of the difference between its true sentiment label and the prediction value of the single-modal predictor for that sample in the previous training round is calculated as the inconsistency score of that sample. Calculate the target quantile index k according to the formula: ; in, The total number of samples in the calibration set. For the set significance level, The index label for the target quantile. This indicates the rounding up operation; After sorting all the inconsistent scores in ascending order, the inconsistent score at the k-th position is determined as the current global quantile threshold for the corresponding modality.

[0008] The step of constructing a corresponding conformal prediction interval for each potential variant in the candidate set based on the global quantile threshold includes: For any potential variant, the lower bound of the conformal prediction interval is obtained by subtracting the global quantile threshold from the point prediction value mapped by the multilayer perceptron. The value obtained by adding the predicted point value to the global quantile threshold is used as the upper bound of the conformal prediction interval; The lower and upper bounds constitute the conformal prediction interval corresponding to the potential variant.

[0009] The absolute deviation between the point prediction value of each latent variant and the set reference target is calculated, and the absolute deviation is input into a pre-constructed piecewise function for calculation to obtain the score of each latent variant, including: Calculate the absolute L1 norm deviation between the point predictions of the potential variants and the set reference target; Construct the piecewise function; wherein, when the absolute deviation of the L1 norm is less than or equal to the global quantile threshold, the output value of the piecewise function is a constant 1; when the absolute deviation of the L1 norm is greater than the global quantile threshold, the output value of the piecewise function is a calculated value with the natural constant e as the base and the negative exponent of the quotient obtained by dividing the difference between the absolute deviation of the L1 norm and the global quantile threshold by the global quantile threshold. Calculate two sets of L1 norm absolute deviations between the point prediction value and the true sentiment label, and between the point prediction value and the prediction value of the dominant text modality, respectively, and substitute them into the piecewise function to obtain the fidelity score and cross-modal consistency score. The score of the latent variant is calculated by weighting and summing the fidelity score and the cross-modal consistency score using the set fidelity and consistency weights.

[0010] The step of weighting and summing the fidelity score and the cross-modal consistency score using the set weights for fidelity and consistency terms includes: When calculating the score for the language modality as the dominant text modality, the weight of the consistency item is set to zero; When calculating scores for visual and audio modalities, independent weights are set for the fidelity item and the consistency item, and both the fidelity item weight and the consistency item weight are non-zero.

[0011] The step of expanding the center correction vector along the temporal dimension and superimposing the expanded vector onto the temporal feature representation sequence to obtain the aligned feature sequence includes: The center correction vector is broadcast and extended along the feature temporal dimension to obtain an extended correction vector with the same length as the temporal feature representation sequence. The elements in the extended correction vector are added to the temporal feature representation sequence according to the corresponding time steps, and the aligned feature sequence is output.

[0012] The step of calculating the feature variance of multiple latent variants in the candidate set across multiple dimensions and generating gating weights using the feature variance includes: Based on multiple potential variants within the candidate set, calculate the cross-variant feature variance corresponding to each feature dimension; Take the negative number of the variance of the cross-variant feature and input the negative number into the Sigmoid activation function for mapping operation to calculate the gating weight of the corresponding feature dimension.

[0013] This invention also relates to a robust multimodal sentiment analysis model based on conformal regression feature repair, applying a robust multimodal sentiment analysis method based on conformal regression feature repair, the model comprising: The feature extraction network module is configured to acquire multimodal raw input data and divide it into a calibration set and a training set; and extract temporal feature representation sequences of language modality, visual modality and audio modality from the multimodal raw input data respectively; The conformal decomposition and anchor selection module is configured to, during the model training phase, extract a global feature representation vector from the temporal feature representation sequence for each modality through a temporal average pooling layer, and then perform projection mapping through a multilayer perceptron to obtain a candidate set containing multiple latent variants; use the single-modality predictor from the previous training round to make predictions on the calibration set to obtain prediction residuals as inconsistency scores, and calculate and determine a global quantile threshold based on a set significance level parameter and the ascending order of the inconsistency scores; construct a conformal prediction interval for the latent variants based on the global quantile threshold; input the absolute deviation of the point prediction value of the latent variant from the set reference target into a pre-constructed piecewise function to obtain a score, sort the latent variants in descending order according to the score, and input the top preset number of latent variants into an average pooling layer to synthesize semantic anchors; The feature alignment and repair module is configured to, during the model training phase, for each modality, extend the center correction vector obtained by subtracting the semantic anchor point from the global feature representation vector along the temporal dimension and superimpose it onto the temporal feature representation sequence to obtain an aligned feature sequence; calculate the feature variance of the latent variant in multiple dimensions to generate gating weights, and perform element-wise multiplication of the gating weights with the aligned feature sequence to obtain the repaired feature representation; The multimodal evaluation and prediction module is configured to, during the model training phase, concatenate the repaired feature representations of the language modality, visual modality, and audio modality along the feature dimension to construct a multimodal joint representation, and input it into the multimodal predictor for prediction, updating the model parameters according to the calculated overall objective function; during the model deployment and testing phase, it receives the temporal feature representation sequences of each single modality of the multimodal data to be tested extracted by the feature extraction network module, directly concatenates them along the feature dimension to construct a multimodal test joint representation, and directly inputs it into the multimodal predictor for a single forward inference, outputting a sentiment analysis prediction score; during the model deployment and testing phase, the conformal decomposition and anchor selection module and the feature alignment and repair module are both configured to be disabled.

[0014] The present invention also relates to an electronic device, including a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements various steps of a robust multimodal sentiment analysis method based on conformal regression feature repair.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention addresses the technical challenges of existing multimodal sentiment analysis in three aspects: noise robustness, semantic information retention rate, and model inference efficiency. This invention provides a robust multimodal sentiment analysis model based on conformal regression feature repair, which maintains high-precision sentiment prediction when dealing with extreme noise interference in reality, while significantly reducing the computational complexity of the model in actual deployment.

[0016] This invention innovatively introduces conformal prediction theory and constructs a statistically supported conformal decomposition and anchor selection mechanism. This mechanism effectively overcomes the semantic loss and representational entanglement defects caused by the coarse "total suppression" of traditional methods, enabling the model to accurately select high-confidence semantic residual fragments as repair anchors in an incomplete and noisy damaged feature space. This invention proposes an anchor-guided feature repair mechanism that uses high-confidence anchors as navigation signals to align the global feature distribution and combines soft gating to eliminate fine-grained variance fluctuations. This mechanism actively corrects damaged data, effectively reverses semantic drift caused by data contamination, and preserves the intrinsic effective semantic information remaining in the noise to the greatest extent possible. This invention designs an asymmetric training and inference architecture that completes complex feature space calibration and repair during the training phase, thereby avoiding cross-modal interaction and online repair calculations after model deployment. This greatly reduces the time overhead of the model in sentiment analysis in real application scenarios, achieves low-latency post-fusion inference, and significantly reduces computational costs. The invention has been tested on two benchmark datasets, CMU-MOSI and CMU-MOSEI, under the stringent conditions of missing rates for various modalities (0.0 to 0.9), and its core indicators such as classification accuracy and regression mean absolute error have comprehensively surpassed the existing best robust multimodal sentiment analysis algorithms, demonstrating excellent noise resistance and robustness. In addition, the model also has a significant speed advantage in the inference process.

[0017] This invention can be widely applied to multimodal content analysis, understanding, and emotion computing in various complex real-world environments, such as smart medical emotion monitoring, intelligent customer service systems, intelligent security emotion early warning, and artificial intelligence software development. It can effectively reduce the system misjudgment rate caused by sensor failure or background noise, improve the human-computer interaction experience, and increase the work efficiency of staff in related fields. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0019] Figure 2 This is a specific framework diagram of the present invention. Detailed Implementation

[0020] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] This invention proposes a method that simultaneously addresses three issues: noise robustness, semantic information preservation rate, and model inference efficiency, aiming to achieve a more practically meaningful robust multimodal sentiment analysis algorithm. Specifically, we innovatively introduce conformal prediction theory to establish a statistically supported reliability assessment mechanism, replacing the original coarse-grained global suppression paradigm. This mechanism does not impose distributional assumptions and can accurately select high-confidence residual semantic fragments as repair anchors in a damaged feature space filled with uncertainty. This anchor-guided fine-grained differentiated repair method can significantly preserve intrinsic effective semantic information, reversing semantic drift caused by data contamination to the greatest extent, and achieving performance superior to existing robust sentiment analysis algorithms under various modality missing rates. Furthermore, by completely confining conformal regression and feature repair processes to the model training stage, this invention effectively bypasses all complex repair calculations during model inference, enabling online inference of multimodal data to be performed directly through a post-fusion method with extremely low latency. This significantly improves the computational efficiency of the model during deployment and saves a significant amount of computational resources. This technology has broad application prospects. For example, in the development of intelligent medical monitoring systems, traditional sentiment analysis technology plays a crucial role in identifying patient emotions. However, it heavily relies on high-quality sensor audio and visual data, and is prone to misjudgment if the equipment is interfered with, resulting in high maintenance costs. By applying this invention, the system can still perform semantic self-repair and maintain high prediction accuracy even when receiving damaged medical multimodal data. Furthermore, due to the significant advantage of this invention in model inference speed, the real-time response capability and computational cost-effectiveness of the entire sentiment monitoring system will be greatly improved.

[0022] The following is combined Figure 1 The present invention will be described in detail below.

[0023] Example 1

[0024] Step 1: Obtain multimodal raw input data containing text, images, and audio. Use a pre-trained deep learning network as a single-modal feature extractor to extract the temporal feature representation sequences of the corresponding language modality, visual modality, and audio modality, respectively, where and represent the sequence length and feature dimension, respectively. In order to meet the statistical requirements of conformal prediction, the original training dataset is divided into a non-overlapping calibration set and a training set. Step 2: Input the feature representation sequences of language, vision and audio modalities obtained in Step 1 into the temporal average pooling layer for dimensionality reduction and smoothing, thereby extracting a fixed-length global feature representation vector for each modality. Step 3: Construct a multilayer perceptron for a specific modality, using the global feature representation vector as the initial semantic seed, and perform progressive nonlinear projection of latent variants through the multilayer perceptron to obtain a candidate set containing latent variants, and introduce orthogonality constraints during the projection process; Step 4: At the beginning of the first round of training of the current model, freeze and use the single-modal predictor solidified from the previous round of training to perform forward inference prediction on the calibration set, calculate the prediction residuals corresponding to all calibration samples as inconsistency scores, sort the inconsistency scores in ascending order, calculate the probability quantiles according to the preset confidence significance level, and determine the global quantile threshold for the current round for this specific modality. Step 5: Based on the global quantile threshold obtained in Step 4, construct the corresponding conformal prediction interval for each potential variant obtained in Step 3, thereby establishing a statistical confidence region in the uncertain feature space; Step 6: Construct a segmented reliability evaluation function to calculate the alignment confidence between the point prediction value of each potential variant and the set reference target; wherein, the reference target includes the real sentiment label and the real-time prediction value of the text-dominant modality; Step 7: Calculate the fidelity score and cross-modal consistency score for each potential variant using the reliability evaluation function. Weight the two scores using hyperparameters to obtain the final reliability score for each potential variant. Sort the candidate potential variants in descending order of their final reliability scores, and extract the highest-ranking, high-confidence candidate variants for input into an average pooling layer for unbiased synthesis. The final output is the synthesized robust semantic anchor. Step 8: Calculate the algebraic deviation value between the robust semantic anchor point and the global feature representation vector obtained in Step 2, and use the deviation value as the global center correction vector. Then, broadcast and expand the center correction vector along the temporal dimension, and superimpose it onto the original feature representation sequence in Step 1 to obtain the aligned feature sequence after global spatial distribution calibration and centroid alignment. Step 9: Calculate the multidimensional feature variance across variants using the latent variant candidate set in Step 3. Use the feature variance as a noise sensitivity index to construct a soft suppression gate. Then, perform fine-grained variance fluctuation elimination on the aligned feature sequence through element-level multiplication to obtain the final repaired feature representation. Step 10: Concatenate and stitch together the final repaired feature representations of language, vision, and audio modalities along the feature channel dimension to construct a unified multimodal joint representation. Feed the multimodal joint representation into the multimodal predictor for the final sentiment tendency intensity regression prediction. Simultaneously, calculate the overall objective function composed of the multimodal main task prediction loss, the single-modal auxiliary task regression loss, and the latent variant orthogonality loss. Use the backpropagation algorithm for end-to-end iterative updates of the parameters of the entire analysis model.

[0025] Step 11: During the model deployment and testing phase, do not enable the conformal decomposition and anchor selection modules and the feature repair module; only retain the single-modal feature extractor and the trained multimodal predictor. Step 12: Receive multimodal input data from the test set or real-world application scenarios, and use the corresponding pre-trained models to extract the original feature representation sequences of language, vision, and audio modalities respectively; Step 13: Using a post-fusion mechanism, bypassing the feature alignment and variance suppression process, the extracted original feature representations of each single modality are directly spliced ​​along the feature dimension to construct a basic multimodal joint representation; Step 14: Input the basic multimodal joint representation directly into the trained multimodal predictor, and output the final sentiment analysis intensity score directly through a single forward inference.

[0026] The working principle / process of this invention is as follows: In the data preparation stage, the original temporal feature sequences of language, vision and audio modalities in the input multimodal data are first extracted using a pre-trained model. Then, the original training dataset is randomly divided into a calibration set and a training set that do not overlap, and a random mask is introduced to simulate random noise pollution in real-world environments.

[0027] During the training phase, the model combats damaged data by constructing a semantic self-repairing closed-loop flow. First, the original feature sequences of each modality are temporally averaged and pooled to extract global feature vectors, which are then projected into a latent variant candidate set with orthogonality constraints using a multilayer perceptron. Next, the inconsistency score is calculated on the calibration set using the single-modality predictor solidified in the previous round, and a global quantile threshold is determined based on a set significance level, thus constructing a conformal prediction interval for each latent variant. Then, based on the absolute deviation between the variant point prediction value and the reference target, a piecewise reliability evaluation function is constructed using the conformal prediction interval, thereby selecting high-confidence variants to synthesize robust semantic anchors. Finally, the deviation between the anchor and the global feature vector is calculated as a correction vector, which is superimposed on the original sequence to complete global distribution alignment; simultaneously, feature variance is calculated across variants to measure noise sensitivity, and a soft suppression gating is generated to perform fine-grained variance fluctuation elimination on the aligned sequence, obtaining the repaired feature representation, which is then concatenated and input into the multimodal predictor to output sentiment prediction. End-to-end parameter iterative updates are performed through a composite overall objective function.

[0028] During the verification and testing phase, this invention constructs an asymmetric, lightweight inference architecture. In this phase, the system completely shuts down and bypasses complex conformal decomposition, anchor selection, and feature repair modules, retaining only the underlying feature extractor and the trained multimodal predictor during field deployment. When receiving multimodal input data from a test set or a real-world application scenario, it directly extracts the original temporal feature sequences of each single modality, employs a post-fusion mechanism to directly concatenate them along the feature dimension, quickly constructing a basic multimodal joint representation, which is then fed into the multimodal predictor for a single forward inference, directly outputting the final sentiment analysis intensity prediction score. This design successfully confines noise resistance and uncertainty repair to the training phase, enabling zero online repair latency and extremely low computational overhead during deployment and testing, significantly improving inference efficiency in real-world scenarios.

[0029] Example 2

[0030] A robust multimodal sentiment analysis model based on conformal regression feature repair, combined with Figure 2 In a preferred embodiment of the present invention, step 3 further includes, in detail, steps 3.1 to 3.2 of the calculation process for the orthogonality constraint of the potential variants. Step 3.1: Calculate the cosine similarity matrix between the normalized latent variant candidate sets. .

[0031] Step 3.2: Calculate the orthogonality loss by minimizing the deviation between the cosine similarity matrix and the identity matrix to constrain variant divergence:

[0032] in, For orthogonality loss function, The cosine similarity matrix between the normalized candidate sets of potential variants is... It is the identity matrix. It is the Frobenius norm.

[0033] As a preferred technical solution, steps 4 and 5 are further characterized in that the process of constructing the conformal prediction interval includes steps 4.1 to 4.3: Step 4.1: Using the calibration set The first in Taking a sample as an example, calculate its non-consistency score. :

[0034] in, For the first The non-consistency score of each sample This is the true sentiment label for the sample. The single-modal predictor retained from the previous training round. This is the global feature representation vector for the corresponding sample.

[0035] Step 4.2: Set the calibration set size to Sort all inconsistent scores in ascending order and select the first... The score is used as the global quantile threshold. And satisfy:

[0036] in, The total number of samples in the calibration set. For the set significance level, The index label for the target quantile. For modality The global quantile threshold corresponding to the current round.

[0037] Step 4.3: For any variant and its point prediction values Generate the corresponding conformal prediction interval:

[0038] in, For potential variants The conformal prediction interval This is the point prediction value for this variant after mapping in a multilayer perceptron.

[0039] As a preferred technical solution, steps 6 and 7 are characterized in that the derivation process of segmented reliability assessment and anchor point synthesis includes steps 6.1 to 6.2: Step 6.1: Let the absolute deviation between the variant prediction and the reference target be . Calculate the reliability assessment function in the form of piecewise exponential decay. :

[0040] in, Point prediction value Compared with reference target The absolute deviation of the L1 norm between them.

[0041] Step 6.2: Calculate the final reliability score for the variant. and select the first High-confidence candidate variants synthesize semantic anchors :

[0042] in, The true sentiment labels of the samples The predicted value for the dominant modality of the text; and These are the weight hyperparameters for the fidelity and consistency terms, respectively; for the text-dominant modality, the hyperparameters are set as follows: For audio and visual assist modalities, set independent hyperparameters. .

[0043] As a preferred technical solution, steps 8 and 9 are further characterized in that the execution process of the feature repair module includes steps 8.1 to 8.3: Step 8.1: Calculate the global center correction vector This is then superimposed onto the original input sequence to obtain the aligned feature sequence. :

[0044] in, The global center correction vector, As the semantic anchor point for synthesis, This is the global feature representation vector. The original feature representation sequence, To complete the feature sequence after global distribution calibration and alignment.

[0045] Step 8.2: According to The characteristic variance of each latent variant across various dimensions Computational soft suppression gating :

[0046] in, For soft suppression gating weights, For candidate variant set The multidimensional feature variance is calculated across variants.

[0047] Step 8.3: Perform fine-grained variance fluctuation elimination on the aligned feature sequence through element-wise multiplication to obtain the final repaired complete feature representation. :

[0048] in, To align feature sequences, These are soft-suppression gating weights.

[0049] Two scenario examples are given below: The first scenario case study evaluates the overall performance and noise robustness on two major benchmark datasets: CMU-MOSI and CMU-MOSEI. The CMU-MOSI dataset contains a series of short and long video clips collected from internet streaming websites; the CMU-MOSEI dataset is currently the largest publicly available benchmark dataset in the field of multimodal language analysis and affective intelligence, containing a large number of high-density, open-domain affective multimedia video samples. Both datasets underwent fine-grained affective sentiment strength regression annotation manually (scores ranging from -3 to +3). To rigorously verify the noise robustness of this invention, a random masking mechanism was introduced into the temporal feature representation sequences of language, visual, and audio modalities to simulate extreme application scenarios with high noise pollution and incomplete data in the real world. This invention uses pre-trained BERT, OpenFace, and Librosa as the underlying feature extraction networks for the three modalities, respectively. Based on the aforementioned unified noise injection settings, this invention underwent a comprehensive quantitative performance comparison with eight existing cutting-edge multimodal sentiment analysis methods. Evaluation metrics included Acc-7 / Acc-5 / Acc-2 classification accuracy, F1-Score, and mean absolute error (MAE) and Pearson correlation coefficient (Corr) for the regression task. Specific experimental comparison data for this invention on the two datasets are shown in Table 1. Table 1. Performance comparison on CMU-MOSI and CMU-MOSEI datasets when using damaged data.

[0050] The second scenario case study: Systematic ablation experiments and internal mechanism evaluations were conducted on two benchmark datasets targeting the two core functional modules of this invention. This case study aims to ablate and test the substantial contributions of each component to blocking feature entanglement and correcting semantic drift. Test variants include: a baseline model that completely removes the self-healing loop and only uses the basic post-fusion architecture; a comparative model that removes the conformal prediction mechanism and directly uses the original feature mean to replace the semantic anchor; and a comparative model that retains the conformal anchor but removes subsequent global alignment and soft suppression feature recovery operations. A detailed summary of the comprehensive ablation experiment results is shown in Table 2. Table 2 Comparison of the system ablation experimental performance of the core components of this invention on dual datasets.

[0051] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robust multimodal sentiment analysis method based on conformal regression feature repair, characterized in that, Includes the following steps: Obtain the original multimodal input data, and divide the dataset corresponding to the original multimodal input data into a calibration set and a training set; The temporal feature representation sequences of the language modality, visual modality, and audio modality in the original multimodal input data are extracted using a feature extraction network. The temporal feature representation sequence is input into a temporal average pooling layer to extract a global feature representation vector; Using the global feature representation vector as the initial input, a projection mapping is performed through a multilayer perceptron to obtain a candidate set containing multiple potential variants; The single-modal predictor from the previous training round is used to make predictions on the calibration set. The prediction residuals are calculated as inconsistency scores, and all the inconsistency scores are sorted in ascending order. The corresponding global quantile threshold is determined from the ascending order results based on the set significance level parameter. Based on the global quantile threshold, a corresponding conformal prediction interval is constructed for each potential variant in the candidate set; The absolute deviation between the point prediction value of each latent variant and the set reference target is calculated. The absolute deviation is input into a pre-built piecewise function for calculation to obtain the score of each latent variant. The reference target includes the predicted values ​​of the true sentiment label and the text-dominant modality. Based on the scores, the potential variants in the candidate set are sorted in descending order, a preset number of potential variants at the top of the sort are extracted, and the extracted potential variants are input into the average pooling layer for pooling operation, and semantic anchors are output. The semantic anchor is subtracted from the corresponding global feature representation vector to obtain the center correction vector. The center correction vector is then extended along the temporal dimension, and the extended vector is superimposed on the temporal feature representation sequence to obtain the aligned feature sequence. Calculate the feature variance of multiple potential variants in the candidate set across multiple dimensions, generate gating weights using the feature variances, and perform element-wise multiplication of the gating weights with the aligned feature sequence to obtain the repaired feature representation. The repaired feature representations of the language modality, visual modality, and audio modality are concatenated along the feature dimension to construct a multimodal joint representation. The multimodal joint representation is then input into a multimodal predictor for prediction. The overall objective function, which is composed of the multimodal main task prediction loss, the single-modal auxiliary task regression loss, and the latent variant orthogonality loss, is calculated. The model parameters are then updated using the backpropagation algorithm.

2. The robust multimodal sentiment analysis method based on conformal regression feature repair according to claim 1, characterized in that, The projection mapping using a multilayer perceptron yields a candidate set containing multiple potential variants, including: During the projection mapping process, the cosine similarity matrix between each potential variant in the normalized potential variant candidate set is calculated. Calculate the Frobenius norm of the difference between the cosine similarity matrix and the identity matrix, and add the calculated Frobenius norm as the potential variant orthogonality loss to the overall objective function.

3. The robust multimodal sentiment analysis method based on conformal regression feature repair according to claim 1, characterized in that, The step of determining the corresponding global quantile threshold in the ascending order results based on the set significance level parameter includes: For each sample in the calibration set, the absolute value of the difference between its true sentiment label and the prediction value of the single-modal predictor for that sample in the previous training round is calculated as the inconsistency score of that sample. Calculate the target quantile index k according to the formula: ; in, The total number of samples in the calibration set. For the set significance level, The index label for the target quantile. This indicates the rounding up operation; After sorting all the inconsistent scores in ascending order, the inconsistent score at the k-th position is determined as the current global quantile threshold for the corresponding modality.

4. The robust multimodal sentiment analysis method based on conformal regression feature repair according to claim 3, characterized in that, The step of constructing a corresponding conformal prediction interval for each potential variant in the candidate set based on the global quantile threshold includes: For any potential variant, the lower bound of the conformal prediction interval is obtained by subtracting the global quantile threshold from the point prediction value mapped by the multilayer perceptron. The value obtained by adding the predicted point value to the global quantile threshold is used as the upper bound of the conformal prediction interval; The lower and upper bounds constitute the conformal prediction interval corresponding to the potential variant.

5. The robust multimodal sentiment analysis method based on conformal regression feature repair according to claim 1, characterized in that, The absolute deviation between the point prediction value of each latent variant and the set reference target is calculated, and the absolute deviation is input into a pre-constructed piecewise function for calculation to obtain the score of each latent variant, including: Calculate the absolute L1 norm deviation between the point predictions of the potential variants and the set reference target; Construct the piecewise function; wherein, when the absolute deviation of the L1 norm is less than or equal to the global quantile threshold, the output value of the piecewise function is a constant 1; when the absolute deviation of the L1 norm is greater than the global quantile threshold, the output value of the piecewise function is a calculated value with the natural constant $e$ as the base and the negative exponent of the quotient obtained by dividing the difference between the absolute deviation of the L1 norm and the global quantile threshold by the global quantile threshold. Calculate two sets of L1 norm absolute deviations between the point prediction value and the true sentiment label, and between the point prediction value and the prediction value of the dominant text modality, respectively, and substitute them into the piecewise function to obtain the fidelity score and cross-modal consistency score. The score of the latent variant is calculated by weighting and summing the fidelity score and the cross-modal consistency score using the set fidelity and consistency weights.

6. The robust multimodal sentiment analysis method based on conformal regression feature repair according to claim 5, characterized in that, The step of weighting and summing the fidelity score and the cross-modal consistency score using the set weights for fidelity and consistency terms includes: When calculating the score for the language modality as the dominant text modality, the weight of the consistency item is set to zero; When calculating scores for visual and audio modalities, independent weights are set for the fidelity item and the consistency item, and both the fidelity item weight and the consistency item weight are non-zero.

7. The robust multimodal sentiment analysis method based on conformal regression feature repair according to claim 1, characterized in that, The step of expanding the center correction vector along the temporal dimension and superimposing the expanded vector onto the temporal feature representation sequence to obtain the aligned feature sequence includes: The center correction vector is broadcast and extended along the feature temporal dimension to obtain an extended correction vector with the same length as the temporal feature representation sequence. The elements in the extended correction vector are added to the temporal feature representation sequence according to the corresponding time steps, and the aligned feature sequence is output.

8. The robust multimodal sentiment analysis method based on conformal regression feature repair according to claim 1, characterized in that, The step of calculating the feature variance of multiple latent variants in the candidate set across multiple dimensions and generating gating weights using the feature variance includes: Based on multiple potential variants within the candidate set, calculate the cross-variant feature variance corresponding to each feature dimension; Take the negative number of the variance of the cross-variant feature and input the negative number into the Sigmoid activation function for mapping operation to calculate the gating weight of the corresponding feature dimension.

9. A robust multimodal sentiment analysis model based on conformal regression feature repair, characterized in that, The robust multimodal sentiment analysis method based on conformal regression feature repair as described in any one of claims 1 to 8, wherein the model comprises: The feature extraction network module is configured to acquire multimodal raw input data and divide it into a calibration set and a training set; and extract temporal feature representation sequences of language modality, visual modality and audio modality from the multimodal raw input data respectively; The conformal decomposition and anchor selection module is configured to, during the model training phase, extract a global feature representation vector from the temporal feature representation sequence for each modality through a temporal average pooling layer, and then perform projection mapping through a multilayer perceptron to obtain a candidate set containing multiple latent variants; use the single-modality predictor from the previous training round to make predictions on the calibration set to obtain prediction residuals as inconsistency scores, and calculate and determine a global quantile threshold based on a set significance level parameter and the ascending order of the inconsistency scores; construct a conformal prediction interval for the latent variants based on the global quantile threshold; input the absolute deviation of the point prediction value of the latent variant from the set reference target into a pre-constructed piecewise function to obtain a score, sort the latent variants in descending order according to the score, and input the top preset number of latent variants into an average pooling layer to synthesize semantic anchors; The feature alignment and repair module is configured to, during the model training phase, for each modality, extend the center correction vector obtained by subtracting the semantic anchor point from the global feature representation vector along the temporal dimension and superimpose it onto the temporal feature representation sequence to obtain an aligned feature sequence; calculate the feature variance of the latent variant in multiple dimensions to generate gating weights, and perform element-wise multiplication of the gating weights with the aligned feature sequence to obtain the repaired feature representation; The multimodal evaluation and prediction module is configured to, during the model training phase, concatenate the repaired feature representations of the language modality, visual modality, and audio modality along the feature dimension to construct a multimodal joint representation, and input it into the multimodal predictor for prediction, updating the model parameters according to the calculated overall objective function; during the model deployment and testing phase, it receives the temporal feature representation sequences of each single modality of the multimodal data to be tested extracted by the feature extraction network module, directly concatenates them along the feature dimension to construct a multimodal test joint representation, and directly inputs it into the multimodal predictor for a single forward inference, outputting a sentiment analysis prediction score; during the model deployment and testing phase, the conformal decomposition and anchor selection module and the feature alignment and repair module are both configured to be disabled.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the various steps of the robust multimodal sentiment analysis method based on conformal regression feature repair as described in any one of claims 1 to 8.

Images