Multi-modal emotional tendency analysis method for natural language processing

Abstract

The invention relates to the technical field of computers, in particular to a multi-modal emotional tendency analysis method for natural language processing, which comprises the following steps: acquiring multi-modal original data; obtaining each feature vector; obtaining a multi-modal discrimination probability vector according to each feature vector; obtaining an interpretation text sequence according to each obtained feature vector; obtaining an interpretation discrimination probability vector according to the interpretation text sequence; determining an emotional tendency category label according to the interpretation discrimination probability vector; calculating classification loss according to the interpretation discrimination probability vector and the real emotion label; and the consistency loss and the total loss are explained according to calculation. According to the method, a multi-modal fusion sentiment classification main branch based on modal self-adaptive gating is constructed, an explanation branch for generating an explanation text and alignment loss between probability distributions of the explanation branch and the explanation branch are added, and a sentiment judgment result and natural language explanation are restrained at the same time under a unified end-to-end training framework; therefore, the technical problem of multi-modal emotion recognition precision and'interpretation-decision 'consistency is considered.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a multimodal sentiment analysis method for natural language processing. Background Technology

[0002] With the rapid development of applications such as online customer service, social media, online education, and intelligent voice assistants, the automatic recognition of user emotional states has gradually become an important research direction in the field of natural language processing. Currently, text-based sentiment analysis methods are widely used. These methods typically extract features and classify only the user-input text, achieving some success in scenarios where the corpus is relatively well-structured and emotional expression is direct. However, in actual human-computer interaction, user emotions are often reflected in multiple modalities, including text content, speech tone, and facial expressions. Relying solely on a single text modality easily overlooks key information such as stress, pauses, and speech rate changes in speech, as well as facial expressions and postures in videos, leading to inaccurate sentiment judgment and insensitivity to emotional changes.

[0003] To overcome the limitations of single-modal methods, existing technologies have proposed multimodal sentiment analysis methods. These methods cascade or simply weight and fuse features from multiple modalities such as text, speech, and video before inputting them into a classification network for sentiment determination. While these methods improve overall sentiment classification accuracy to some extent, they generally suffer from two problems: First, most methods employ fixed weights or simple concatenation, failing to dynamically adjust the contribution of each modality based on differences in signal quality, noise levels, and semantic information across different samples. When a modality is heavily affected by noise or has incomplete data collection, it is still assigned a high weight, leading to instability in the multimodal fusion features and fluctuating classification results. Second, most existing multimodal methods only output sentiment category labels and their probability distributions, lacking natural language explanations tightly coupled with the multimodal decision-making process. Alternatively, they only provide a rough visualization of the model through post-hoc explanations, failing to ensure consistency between the "explanation content" and the "model's true decision-making basis" at the probabilistic level from the model's internal mechanisms.

[0004] In scenarios where explanations need to be provided to users or where model outputs require manual review, providing only the probability of sentiment categories without reliable explanations can lead to two problems: First, business personnel cannot determine whether the sentiment categories given by the model are sufficiently supported, making it difficult to formulate differentiated service strategies accordingly. Second, in existing methods that introduce natural language explanations, the explanations are often provided by independent text generation models, trained separately from the multimodal classification models. The lack of clear alignment constraints between the two can easily lead to inconsistencies between the "evidence of the emotions described in the explanatory text" and the "evidence actually used in the multimodal classification branches," affecting the credibility and reusability of the system.

[0005] Therefore, in the existing technology, how to construct a multimodal sentiment analysis method that has both high sentiment classification accuracy and can ensure that the natural language interpretation and multimodal discrimination results are consistent in probability distribution through joint training mechanism, so as to balance the robustness of multimodal fusion and the interpretability of sentiment judgment results, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] To overcome the aforementioned technical deficiencies, the present invention aims to provide a multimodal sentiment analysis method for natural language processing. By constructing a multimodal fusion sentiment classification main branch based on modality adaptive gating and adding an interpretation branch for generating explanatory text and an alignment loss between the probability distributions of the two, the method simultaneously constrains the sentiment judgment results and natural language interpretation under a unified end-to-end training framework, thereby addressing the technical problem of balancing the accuracy of multimodal sentiment recognition with the consistency of "interpretation-decision".

[0007] This invention discloses a multimodal sentiment analysis method for natural language processing, comprising the following steps: Step S1: Obtain the multimodal raw data of the object to be analyzed. The multimodal raw data includes text data, speech data and video data aligned on the same time axis. Perform word segmentation and sub-word segmentation on the text data, denoise removal and frame segmentation on the speech data, and face detection and key frame selection on the video data to obtain the text input sequence, speech input sequence and video input sequence. Step S2: Input the text input sequence into the text encoding sub-network to obtain the text modality feature vector. The speech input sequence is input into the speech coding subnetwork to obtain the speech modal feature vector. The video input sequence is fed into the video coding subnetwork to obtain the video modal feature vector. Among them, text modal feature vectors Speech modal feature vectors Video modal feature vectors It is a column vector; Step S3, during the training phase, the text modality feature vector Speech modal feature vectors and video modal feature vectors Inputting the multimodal direct-connect discriminant subnetwork yields the multimodal fused feature vector. And calculate the multimodal discrimination probability vector according to the following formula. :

[0008] in, For multimodal discrimination weight matrix, It is a column vector; Step S4, convert the text modality feature vector Speech modal feature vectors and video modal feature vectors Input a modality interpretation generation subnetwork, and generate text modality interpretation subsequences within the modality interpretation generation subnetwork. Speech modal interpretation subsequence and video modality interpretation subsequence and interpret subsequences in text modality Speech modal interpretation subsequence and video modality interpretation subsequence Modal markers are added before and after, and the three are connected in a preset order to obtain the explanatory text sequence. ; Step S5 will explain the text sequence. Input the interpretive encoding subnetwork to obtain the interpretive feature vector. This will explain the feature vectors. Input the interpretation and discrimination subnetwork, and calculate the interpretation and discrimination probability vector according to the following formula. :

[0009] in, To explain the discriminant weight matrix, It is a column vector; Step S6: Based on the interpretation of the probability vector Determine the emotional tendency category label Sentiment category tags Corresponding to the interpretation discriminant probability vector The category index with the highest probability; Step S7, during the training phase, determine the discriminant probability vector based on the interpretation. With genuine emotion tags Calculate classification loss Authentic emotional tags Pre-labeled sentiment category tags; Step S8: During the training phase, based on the multimodal discrimination probability vector... With the interpretation of the discriminant probability vector Calculate the interpretation consistency loss Calculate the total loss according to the following formula. :

[0010]

[0011] in, Represents the L2 norm, The coefficients are real numbers greater than zero, representing the total loss. This is used to jointly update the trainable parameters in the text coding subnetwork, speech coding subnetwork, video coding subnetwork, modal interpretation generation subnetwork, interpretation coding subnetwork, and interpretation discriminant subnetwork. During the inference phase, only steps S1, S2, S4, S5, and S6 are executed.

[0012] Compared with existing technologies, the above technical solution has the following advantages: 1. This invention extracts text modality feature vectors, speech modality feature vectors, and video modality feature vectors in steps S2 and S3, respectively. In the multimodal fusion stage, a modality adaptive gating mechanism is introduced to adaptively assign gating weights to each modality based on the modality quality and feature contribution of the current sample. Then, the multimodal features are weighted and fused to obtain a multimodal fusion feature vector. This allows the contribution of each modality to be dynamically adjusted in scenarios with low signal-to-noise ratio, missing modalities, or unbalanced modality quality, thereby improving the robustness of the multimodal sentiment analysis method in complex real-world scenarios and the overall accuracy of sentiment classification.

[0013] 2. In step S4, this invention constructs an interpretation generation subnetwork and an interpretation discrimination branch, generating explanatory text using text modal features, speech modal features, video modal features, and multimodal fusion features. The explanatory text is further encoded into an interpretation feature vector, outputting an interpretation discrimination probability vector. Simultaneously, the interpretation alignment loss constrains the multimodal discrimination probability vector and the interpretation discrimination probability vector to maintain consistency in the probability distribution space. This improves the interpretability of sentiment judgment results while ensuring consistency between natural language interpretation and multimodal decision results, enabling the explanatory text to accurately reflect the model's judgment criteria and facilitating understanding and review of model output by business personnel.

[0014] 3. In step S5 of this invention, the classification loss of the multimodal direct-connection discriminant branch, the classification loss of the interpretation discriminant branch, and the interpretation alignment loss are all incorporated into a single end-to-end total loss function. Optimization methods such as mini-batch gradient descent are used to jointly update all trainable parameters, achieving integrated training of data acquisition, feature encoding, multimodal fusion, sentiment classification, and interpretation generation. This avoids the inconsistency problem caused by multi-stage independent training, enabling multimodal feature extraction and fusion, as well as its own service to both sentiment classification and interpretation tasks, to improve the overall convergence speed and training stability of the model, resulting in higher comprehensive performance. Attached Figure Description

[0015] Figure 1 The graphs show the performance curves of multimodal sentiment classification under different signal-to-noise ratios. Figure 2A bar / curve diagram illustrating the impact of multimodal feature encoding dimensions on overall sentiment classification performance; Figure 3 A schematic diagram showing the comparison of sentiment classification accuracy curves during the training process for different multimodal fusion methods; Figure 4 A schematic diagram illustrating the curve of alignment loss as a function of the number of training epochs; Figure 5 A bar chart comparing the overall sentiment classification accuracy of different sentiment analysis methods across multiple datasets; Figure 6 A graphical representation of the impact of the number of user feedback samples on the overall accuracy and interpretation accuracy of online sentiment classification; Figure 7 This is a schematic diagram illustrating the impact of the number of concurrent requests on the system's average response time and average throughput. Figure 8 A schematic diagram illustrating the trade-off between overall sentiment classification accuracy and member inference attack success rate under different levels of privacy noise. Figure 9 A schematic diagram illustrating the impact of the number of adaptive iteration rounds on the accuracy of target domain sentiment classification and the domain difference index; Figure 10 A schematic diagram illustrating the impact of different emotional risk escalation thresholds on the escalation processing ratio and user complaint rate; Figure 11 This is a schematic diagram illustrating the steps of the multimodal sentiment analysis method for natural language processing according to the present invention. Detailed Implementation

[0016] See Figure 11 As shown in this embodiment, a multimodal sentiment analysis method for natural language processing is provided, including the following steps: acquiring multimodal raw data; acquiring each feature vector; acquiring a multimodal discrimination probability vector based on each feature vector; acquiring an explanatory text sequence based on the acquired feature vectors; acquiring an explanation discrimination probability vector based on the explanation text sequence; determining the sentiment category label based on the explanation discrimination probability vector; calculating the classification loss based on the explanation discrimination probability vector and the true sentiment label; and calculating the explanation consistency loss and the total loss. This invention constructs a multimodal fusion sentiment classification main branch based on modality adaptive gating, and adds an explanation branch that generates the explanation text and an alignment loss between their probability distributions. Under a unified end-to-end training framework, it simultaneously constrains the sentiment judgment result and the natural language explanation, thereby addressing the technical problem of balancing multimodal sentiment recognition accuracy and "explanation-decision" consistency.

[0017] Step S1 is the multimodal raw data acquisition and preprocessing step, used to provide high-quality, time-aligned text input sequences, speech input sequences, and video input sequences closely related to emotional expression for subsequent multimodal feature encoding, modality interpretation generation, and sentiment tendency determination. Specifically, step S1 first collects raw text data, raw speech data, and raw video data related to emotional expression for the interaction scenario of the object to be analyzed, and adds unified timestamp information to the three types of raw data, so that all raw data establish a unified time reference during the acquisition stage. Raw text data can come from keyboard-input chat content, text messages in instant messaging systems, subtitle streams corresponding to video or audio content, or initial transcribed text obtained by the speech recognition module from raw speech data; raw speech data can be continuously acquired through a single microphone or microphone array at a preset sampling rate, such as 16kHz; raw video data can be acquired through a monocular camera or multi-camera device at a preset frame rate, such as 25 frames per second. During the acquisition process, all raw text data records the timestamp when the text is generated or transcribed, all raw audio data records the sampling time in units of sampling points or audio frames, and all raw video data records the shooting time in units of frames, thereby ensuring that the three types of raw data can be compared and aligned on the timeline.

[0018] After obtaining the multimodal raw data, step S1 standardizes and structures the raw text data to form the text input sequence used later. Specifically, the raw text data can first be unified by character encoding, converting text content from different terminals, operating systems, and input methods into the same character set format, filtering out invisible control characters and obviously abnormal characters to ensure that the text content is parseable; then, emoticons, special marks, and redundant whitespace in the raw text data are standardized, and emoticons with the same function or similar emotional tendencies are uniformly replaced with placeholder marks with clear meanings, such as uniformly representing several emotion-related symbols such as "laughing," "crying," and "angry," thereby avoiding the introduction of meaningless noise in the subsequent word segmentation stage; then, according to the characteristics of the target language, the word segmentation tool is called to segment the raw text data, dividing the continuous text stream into word units, and combining the preset sub-word segmentation rules, low-frequency words, compound words, and out-of-vocabulary proper nouns are further split into sub-word units to reduce the vocabulary size and alleviate the out-of-vocabulary word problem. In some embodiments, a sentiment dictionary, a negation dictionary, and a degree adverb dictionary can be combined to perform sentiment tagging on the word segmentation results. Words with obvious sentiment tendencies are marked as sentiment words, words with reversed sentiment polarities are marked as negation words, and words that change sentiment intensity are marked as degree adverbs. Through the above processing, richer text-side sentiment cues can be provided for the subsequent modality interpretation and generation module while preserving the original semantics. Finally, the word units are organized according to sentence boundaries, line breaks, and timestamp information to form a text input sequence arranged in chronological order. Each text input sequence has a clear temporal correspondence with the subsequent speech input sequence and video input sequence.

[0019] For the raw speech data, step S1 performs noise suppression, quality assessment, and acoustic feature extraction for the emotion analysis scenario to form a speech input sequence. Specifically, silence detection and endpoint detection can be performed on the entire raw speech data first, automatically dividing the raw speech data into one or more speech segments, and filtering out time intervals containing only environmental noise or no effective speech activity to reduce the interference of meaningless data on subsequent modeling. On this basis, noise suppression is performed on each speech segment, which can be done using spectral subtraction, Wiener filtering, or a deep learning-based speech enhancement model to reduce noise in the speech segments, making the energy contrast between the speech components and background noise in the speech segments more obvious. Subsequently, short-time analysis is performed on the noise-suppressed speech segments, dividing the speech segments into frames according to a preset frame length and frame shift and windowing them, for example, using a parameter configuration of 20ms frame length and 10ms frame shift, to obtain a series of short-time speech analysis frames. For each short-time speech analysis frame, multi-dimensional acoustic features such as Mel frequency cepstral coefficients, short-time energy, fundamental frequency, spectral flatness, and formant positions can be extracted. These acoustic features are then spliced ​​together in chronological order to form a speech feature frame sequence.

[0020] In some preferred embodiments, step S1 further calculates a speech quality score for each short-time speech analysis frame or each speech segment to mark the reliability of the speech input data. The speech quality score can be calculated by comprehensively considering factors such as short-time signal-to-noise ratio, speech activity probability, and speech energy stability, and speech segments are divided into several levels: high quality, medium quality, and low quality. For speech feature frames or speech segments marked as low quality, weighting, interpolation, or direct removal can be performed when constructing the speech input sequence, thereby suppressing the negative impact of abnormal speech such as extreme noise and severe distortion on the overall model. Through this strategy, under the same dataset and model structure conditions, compared with the baseline scheme that does not perform speech noise suppression and speech quality screening, the multimodal sentiment analysis method significantly improves the sentiment classification accuracy on both the validation set and the test set. For example, when conducting a comparative experiment on a set of speech data containing various noise environments, the average signal-to-noise ratio of the speech segments is used as the grouping criterion, and the sentiment classification accuracy under two configurations—without the above speech preprocessing strategy and with the above speech preprocessing strategy—is statistically analyzed in different signal-to-noise ratio intervals.

[0021] For the raw video data, step S1 mainly extracts emotion-related information from the face and expression regions to construct the video input sequence. Specifically, face detection can be performed on each frame of the raw video data to obtain the position and size of candidate face regions. If multiple faces exist within the same frame, the face region corresponding to the main speaker can be determined by combining speech source localization or historical trajectory information. To reduce the impact of redundant frames on subsequent encoding networks, step S1 can select keyframes from the raw video data based on pixel differences, optical flow, or pose changes between adjacent frames. Image frames with significant expression changes, obvious head pose changes, or obvious eye and mouth movements are marked as keyframes. Image frames with stable and highly repetitive expression changes are downsized by a preset factor. For the retained keyframes, brightness normalization, contrast adjustment, and geometric alignment can be performed to unify face images under different lighting conditions and shooting angles to a standard scale and pose, thereby reducing the interference of external environmental factors on expression analysis. Furthermore, a video quality score can be calculated for each keyframe. For example, a quality score can be calculated by comprehensively considering indicators such as face detection confidence, the number of consecutive frames for facial landmark tracking, and image blur. High-quality keyframes are marked as priority retention frames, while low-quality keyframes are marked as candidate frames. High-quality keyframes are prioritized when constructing the video input sequence. Comparative experiments on datasets containing various camera devices and lighting conditions show that after adopting the above keyframe selection and quality scoring strategy, the facial expression features extracted from the video input sequence are more stable, and the fluctuation of emotion classification results on the cross-scene test set is significantly reduced. This indicates that the preprocessing operation on the original video data in step S1 can improve the overall method's adaptability to changes in the shooting environment.

[0022] To ensure that the text input sequence, speech input sequence, and video input sequence correspond to the same emotional event within the same time frame, step S1 further includes multimodal time alignment and emotional event segmentation operations. In one implementation, based on the endpoint detection results of the raw speech data, a long-term speech recording can first be divided into several speech sentence segments. Each speech sentence segment is used as a basic time slice, and the raw text data segments and raw video data frames located within that time slice are categorized to form preliminary speech sentence units. Based on this, the word-level or phrase-level time alignment information output by the speech recognition module can be used to finely align the raw text data within the speech sentence unit, obtaining the corresponding text subsequence; using frame timestamps and keyframe position information, the raw video data within the speech sentence unit is aligned to obtain the corresponding set of video keyframes. Through these operations, each text subsequence in the text input sequence, each speech feature frame sequence in the speech input sequence, and each set of video keyframes in the video input sequence have a unified time frame and together constitute a multimodal sentence unit, providing a time-synchronized input basis for subsequent multimodal feature encoding and modal interpretation generation.

[0023] In summary, step S1 ensures the temporal synchronization and integrity of raw text, raw speech, and raw video data during the multimodal raw data acquisition stage. In the preprocessing stage, character unification, emoji normalization, word segmentation, and sentiment word annotation are performed on the raw text data; silence detection, endpoint detection, noise suppression, quality scoring, and abnormal segment removal are performed on the raw speech data; and face detection, keyframe selection, image standardization, and quality scoring are performed on the raw video data. Furthermore, multimodal temporal alignment and dynamic segmentation of sentiment events are used to construct text input sequences, speech input sequences, and video input sequences corresponding to sentiment events. This ensures that the subsequent multimodal sentiment analysis method maintains high sentiment classification accuracy and good weak sentiment recognition capabilities even in noisy environments, complex lighting conditions, and long dialogues.

[0024] like Figure 1 As shown, under different speech signal-to-noise ratio threshold settings, the sentiment classification accuracy curve using the speech preprocessing strategy is significantly higher than the sentiment classification accuracy curve without the speech preprocessing strategy across the entire range, indicating that the speech preprocessing design in step S1 can effectively improve the performance of the multimodal sentiment analysis method in noisy environments.

[0025] After the above-mentioned multimodal raw data acquisition and preprocessing are completed, step S2 is the multimodal feature encoding step, which is used to map the text input sequence, speech input sequence and video input sequence into text modality feature vectors respectively. Speech modal feature vectors and video modal feature vectors This serves as the foundational representation for subsequent multimodal fusion, modality interpretation generation, and sentiment analysis. (Step S2: Text Modality Feature Vector) Speech modal feature vectors and video modal feature vectors All are column vectors, and each column vector numerically encodes the global sentiment information and local context information of the corresponding modality in a given emotional event segment.

[0026] For the text input sequence, step S2 constructs a text encoding sub-network to map the text input sequence obtained in step S1 into text modal feature vectors. Specifically, each word unit in the text input sequence can first be assigned a word vector representation. The word vectors are then added to the positional encoding to obtain the text embedding sequence. This text embedding sequence is then input into a context encoding module consisting of a multi-layered self-attention structure and a feedforward network. Through multi-layer stacking, long-distance dependencies within and across sentences are captured, resulting in a text latent state sequence containing contextual information. Building upon this, to highlight sentiment-related positions, the text encoding sub-network can introduce a sentiment attention mechanism. Based on the positions of sentiment words, negation words, and degree adverbs labeled in step S1, different attention weights are assigned to the text latent state sequence, giving higher weights to sentiment-related positions and lower weights to sentiment-irrelevant positions. Finally, through weighted pooling or dedicated classification labeling, the weighted text latent state sequence is compressed into a fixed-dimensional text modality feature vector. The text modal feature vector It contains both overall semantic information and explicitly reinforces key emotional information.

[0027] In some preferred embodiments, to improve the transferability and expressiveness of the text modality feature vector t, the text encoding sub-network can be pre-trained on large-scale general corpora and sentiment-annotated corpora, and then fine-tuned on the target multimodal sentiment dataset. In a set of comparative experiments, comparing the text modality feature vector scheme obtained by simply using word vector average pooling with the text modality feature vector scheme obtained by using the above-mentioned context encoding and sentiment attention mechanisms, a significant difference in overall sentiment classification accuracy can be observed.

[0028] For the speech input sequence, step S2 constructs a speech coding sub-network to map the speech feature frame sequence obtained from the preprocessing in step S1 into speech modal feature vectors. Specifically, the speech feature frame sequence can first be organized into a two-dimensional feature matrix in the time dimension, with the time frame index as one dimension and the acoustic feature dimension as the other. This feature matrix is ​​then input into a convolutional neural network to extract local time-frequency patterns, capturing short-term pitch variations, formant variations, and energy variations. Following the convolutional neural network, a bidirectional recurrent neural network or a temporal self-attention encoding module can be connected to model long-range dependencies in the speech feature frame sequence, thereby extracting emotion-related pitch contours, speech rate variations, and pause patterns. Based on this, the temporal latent states are aggregated through attention-based weighted pooling or by explicitly introducing speech event tags to obtain a fixed-dimensional speech modal feature vector. Speech modal feature vector Each dimension element reflects the synthesis of emotion-related acoustic features of the speech signal in different time ranges, and can effectively encode the speaker's emotional intensity, tension level, and tone changes.

[0029] To verify the impact of the speech coding subnetwork design on overall performance, we can compare the performance differences between the simple average pooling speech feature scheme and the convolutional temporal coding scheme by changing only the structure of the speech coding subnetwork under the same text coding subnetwork and video coding subnetwork configurations.

[0030] For the video input sequence, step S2 constructs a video coding sub-network to map the video keyframe sequence obtained in step S1 into video modal feature vectors. Specifically, spatial expression features can first be extracted from the facial region of each video keyframe using a two-dimensional convolutional neural network, encoding expression information from aspects such as facial texture changes, eye and mouth shape changes. After obtaining the spatial feature vector of each video keyframe, the temporally ordered sequence of video keyframe features can be input into a temporal coding module, such as a one-dimensional convolutional temporal network, a recurrent neural network, or a self-attention temporal coding module, to characterize the trajectory of expression changes over time. For video keyframe feature sequences containing multiple frames, an attention-based temporal aggregation strategy can be adopted, assigning higher weights to time positions with significant expression changes. Based on this, weighted pooling of the temporal features is performed to obtain a fixed-dimensional video modality feature vector. Video modal feature vectors It can express the comprehensive pattern of facial expressions, head posture, and eye movements in a given emotional event fragment, providing visual emotional basis for subsequent multimodal fusion and modal interpretation generation.

[0031] Through the above design, step S2 effectively maps the text input sequence, speech input sequence, and video input sequence obtained in step S1 into text modality feature vectors. Speech modal feature vector s and video modal feature vector This ensures that each modality possesses good separability and fusionability within a unified vector space. Further implementation details in subsequent methods show that these modality feature vectors are input into a multimodal direct-connect discriminant subnetwork and a modality interpretation and generation subnetwork. Through alignment training between interpretation encoding and multimodal discrimination, the final sentiment judgment achieves both high classification accuracy and traceable natural language interpretation capabilities.

[0032] like Figure 2 As shown, under different feature dimension configurations, the overall sentiment classification accuracy curve using the hierarchical context coding feature generation method is significantly higher than that using the simple average pooling feature generation method across the entire range. The difference is greatest when the feature dimensions are 128 and 256, indicating that the multi-layer context coding structure used in step S2 has a significant performance advantage.

[0033] The text modality feature vector is obtained in step S2 above. Speech modal feature vectors and video modal feature vectors Next, step S3 is the multimodal direct connection discrimination step, used to construct a multimodal direct connection discrimination sub-network, which converts the text modality feature vectors... Speech modal feature vectors and video modal feature vectors Fuse into a multimodal fusion feature vector And based on multimodal fusion feature vectors Calculate the multimodal discrimination probability vector This allows for sentiment analysis based directly on multimodal features without relying on natural language interpretation. (Multimodal discrimination probability vector) It can also be used as a reference during the training phase to constrain consistency with sentiment judgment results based on natural language interpretation, thereby improving the stability and discriminative ability of the overall model.

[0034] In one implementation, in order to effectively fuse features from different modalities in the same vector space, step S3 first processes the text modal feature vectors... Speech modal feature vectors and video modal feature vectors Dimension alignment is performed. Specifically, three linear transformation layers can be set up to align the text modality feature vectors. Speech modal feature vectors and video modal feature vectors Perform an affine transformation to map the three to the same feature dimension. The text alignment feature vector is obtained. Speech alignment feature vector Aligning feature vectors with video The weight and bias parameters of the aforementioned linear transformation layer are jointly trained with the subsequent multimodal direct-connect discriminant subnetwork on a multimodal sentiment dataset, thereby improving the text alignment feature vector. Speech alignment feature vector Aligning feature vectors with video It is better suited to the distribution characteristics of sentiment judgment tasks.

[0035] To avoid modal redundancy and modal conflicts caused by simple splicing, in a preferred embodiment, step S3 introduces a modality adaptive gating mechanism based on the aligned feature vectors. This mechanism dynamically adjusts the text alignment feature vectors according to the specific circumstances of different emotional event fragments. Speech alignment feature vector Aligning feature vectors with video Multimodal fusion feature vector The modality adaptive gating mechanism can use a small feedforward neural network to align text feature vectors. Speech alignment feature vector Aligning feature vectors with video Perform joint modeling and output three non-negative real weights. , and And through normalization, the three weights are made to satisfy: Then, calculate the multimodal fusion feature vector according to the following formula. : .in, This represents a multimodal fusion feature vector. This represents the text alignment feature vector. Represents the speech alignment feature vector. Represents the video alignment feature vector. , , These represent the adaptive weights for the text, speech, and video modalities, respectively. In actual training, the modality adaptive gating mechanism automatically learns the importance of each modality in different scenarios through backpropagation; for example, it reduces the weights when the speech signal quality is poor. The value decreases when the video frame quality is low. The value of the low-quality mode is thus avoided. It will have adverse effects.

[0036] After obtaining the multimodal fusion feature vector Next, step S3 uses a multi-layer fully connected network to fuse the feature vectors from multiple modalities. A nonlinear mapping is performed to obtain the multimodal discriminative hidden representation vector, and the multimodal discriminative probability vector is calculated in the output layer. In a common implementation, a multi-layer fully connected network can be simplified to a single affine transformation layer and a single nonlinear activation function layer, and then the unnormalized score vector corresponding to the sentiment category is calculated through a single linear output layer. Then use the softmax function to normalize the unnormalized score vector. Perform normalization. If the number of sentiment categories is... The parameters of the output layer include the weight matrix. and bias vector ,in The size is , The length is At this point, the multimodal discrimination probability vector The calculation process can be formally written as: , , .in, For length is The unnormalized score vector, Indicates the first Unnormalized scores for each sentiment category, Represents the multimodal discrimination probability vector The middle corresponds to the first The probability components of each sentiment category.

[0037] To facilitate understanding of the above calculation process, a specific numerical calculation example is given below. This example can be derived from the intermediate calculation results of a multimodal direct-connect discriminant subnetwork trained on a publicly available multimodal sentiment dataset on a certain validation sample. In this example, it is assumed that after feature encoding in step S2 and modality adaptive gating fusion, the multimodal fused feature vector... It is a 4-dimensional column vector: Assuming the number of emotion categories For each of the three emotion categories, "positive," "neutral," and "negative," the output layer weight matrix is ​​defined. With bias vector for: ,

[0038] according to The unnormalized score vector can be calculated. The three component values. For the first sentiment category, we have: Calculate each item individually: , , , .

[0039] Adding the above product to the bias gives: .

[0040] For the second emotion category, we have: .

[0041] Calculate item by item: , , , .

[0042] Adding them together, we get: .

[0043] For the third emotion category, we have: .

[0044] Calculate item by item: , , , .

[0045] Adding them together, we get: .

[0046] Therefore, the unnormalized score vector for: ; After obtaining the unnormalized score vector Next, step S3 converts it into a multimodal discriminant probability vector using the softmax function. First, calculate the exponent value for each component: , , Then calculate the exponent sum: .

[0047] Based on this, we can obtain the multimodal discrimination probabilities corresponding to the three sentiment categories: , , .

[0048] Therefore, in this specific example, the multimodal direct-connect discriminant subnetwork gives a multimodal discrimination probability of approximately 0.576 for the "positive" sentiment category, approximately 0.264 for the "neutral" sentiment category, and approximately 0.160 for the "negative" sentiment category. The above specific values ​​fully illustrate the calculation process of matrix multiplication and softmax normalization, helping to understand the multimodal discrimination probability vector. The way it is formed.

[0049] To verify the superiority of the modality-adaptive gating fusion method over the simple concatenation fusion method, under the same configuration of text coding subnetwork, speech coding subnetwork, and video coding subnetwork, only the fusion strategy in the multimodal direct-connect discriminant subnetwork was changed. The differences in overall sentiment classification accuracy when using the simple concatenation fusion method, the additive attention fusion method, and the modality-adaptive gating fusion method were compared. (See also...) Figure 3 As shown, in the early stage of training, the accuracy of the modal adaptive gating fusion method increases faster. In the later stage of training, the accuracy difference between the two methods stabilizes at more than 4 percentage points, further verifying the advantages of the multimodal fusion structure in step S3 in terms of convergence speed and final performance.

[0050] Through the above design, step S3 will convert the text modality feature vector Speech modal feature vectors and video modal feature vectors Mapped to a multimodal fusion feature vector with strong discriminative power And based on multimodal fusion feature vectors Calculate the multimodal discrimination probability vector In subsequent steps, the multimodal discrimination probability vector The sentiment discrimination probability vector obtained from the branch generated based on natural language interpretation is used together to construct the alignment loss, which further improves the discrimination accuracy of the overall sentiment tendency analysis method while taking into account interpretability.

[0051] The multimodal fusion feature vector is obtained in step S3 above. and multimodal discrimination probability vector Next, step S4 is the interpretation generation and alignment training step, used to construct the interpretation generation sub-network and fuse multimodal feature vectors. Modeling is performed on the features of each modality to generate natural language explanation text consistent with the sentiment judgment results. Alignment loss is used to jointly constrain the multimodal direct connection discrimination branch and the explanation generation branch, thereby improving the interpretability of sentiment judgment results while ensuring the performance of sentiment classification.

[0052] In one implementation, step S4 first constructs an interpretation-generating subnetwork, used to interpret text modality feature vectors. Speech modal feature vectors Video modal feature vectors and multimodal fusion feature vectors The explanation generation subnetwork generates a sequence of explanatory text. The explanation generation subnetwork can employ an encoder-decoder structure. Its encoder side can share the text encoding subnetwork from step S2, or it can have a separate explanation encoding subnetwork to re-encode the original text input sequence or the implicit states of intermediate text. The decoder side can employ an autoregressive text generation structure, progressively outputting the explanatory text at the word or sub-word level. To ensure the explanatory text covers multimodal information, the explanation generation subnetwork can simultaneously access text modality feature vectors during the decoding process. Speech modal feature vectors Video modal feature vectors and multimodal fusion feature vector It selectively focuses on intonation changes in speech, facial expressions in video, and emotional words in text through an attention mechanism, thereby generating explanatory sentences that contain multimodal evidence.

[0053] After the explanatory text is generated, step S4, in order to measure the correspondence between the explanatory text and the sentiment category, encodes the explanatory text to obtain an explanatory feature vector. And based on the interpretation of feature vectors Construct the interpretation and discrimination branch, and output the interpretation and discrimination probability vector. Specifically, the text encoding structure generated for the explanation can be set to be the same as or similar to that in step S2. The words or subwords in the explanation text are mapped to vectors and context-encoded to obtain the hidden state sequence of the explanation text. Then, a fixed-dimensional explanation feature vector is obtained through average pooling or attention aggregation. In the interpretation and discriminant output layer, the weight matrix is ​​set. and bias vector ,in, The size is , To explain the eigenvectors , For the number of emotion categories, The length is Explain the unnormalized score vector of the discriminant branch. and Explanation of the Discriminant Probability Vector It can be calculated in the following form: , , .in, For length is The unnormalized score vector, Indicates the first Unnormalized scores for each sentiment category, Represents the discriminant probability vector The middle corresponds to the first The probability components of each sentiment category.

[0054] To illustrate the specific numerical calculation process, the following example from step S3 explains the multimodal discrimination probability vector. With the interpretation of the discriminant probability vector Alignment calculation. Assuming for a given validation sample, the multimodal discrimination probability vector obtained through step S3... for: The first component corresponds to the "positive" category, the second component to the "neutral" category, and the third component to the "negative" category. Further assuming that step S4 obtains the interpretation discrimination probability vector through the interpretation generation subnetwork and the interpretation discrimination branch... for: In this example, both the multimodal discrimination branch and the interpretation discrimination branch consider the sample to be more inclined towards the "positive" category, but their specific probability distributions differ somewhat. To ensure that the outputs of the interpretation branch and the multimodal direct-connection discrimination branch are consistent, step S4 can construct the interpretation alignment loss to measure... and The difference between them. In one implementation, the alignment loss can be interpreted as defined in the form of mean squared error: .

[0055] For the numerical examples above, we have: , , ; Then the square of each term is: , , ; Add the three items together and divide by the number of emotion categories. ,get:

[0056] Therefore, on this specific sample, the multimodal discrimination probability vector With the interpretation of the discriminant probability vector The mean squared error between them is approximately 0.00069, indicating that their distributions are quite similar. For other samples, if and If the difference is large, then the corresponding The numerical value will increase significantly. During the training process, backpropagation can be used to drive the interpretation generator subnetwork and interpretation discriminant branch to adjust their parameters, so that their output probability distribution gradually approaches the multimodal direct connection discriminant branch.

[0057] In the overall training process, step S4 can further weight and combine the classification loss of the multimodal direct connection discriminant branch, the classification loss of the multimodal interpretation discriminant branch, and the interpretation alignment loss to form the final training objective function. Let the classification loss of the multimodal direct connection discriminant branch be denoted as... The classification loss of the explanation discriminant branch is denoted as Let the explained alignment loss be denoted as The total loss function can then be defined as: ,in, and is a non-negative weighting coefficient used to balance classification performance and interpretation alignment. In an exemplary experiment, if set... , And assuming that for a given training sample, the probability of the multimodal discriminant branch when the true sentiment category is "positive" is... The corresponding multimodal classification loss can be regarded as the cross-entropy loss: Meanwhile, if the probability of the discriminant branch in interpreting the true sentiment category as "positive" is... The classification loss can then be interpreted as: In combination with the above The total loss for this sample can be obtained as follows: .

[0058] As can be seen from the numerical examples above, in the actual training process, the multimodal direct connection discrimination branch, the interpretation discrimination branch, and the interpretation alignment loss work together to ensure that the model optimizes the accuracy of sentiment classification while constraining the sentiment tendency expressed by the interpretation text to be consistent with the multimodal direct connection discrimination results.

[0059] Figure 4 The horizontal axis represents the number of training epochs, and the vertical axis represents the average value on the validation set. Numerical value Figure 4 The middle curve represents the alignment loss. The change in the average value on the validation set with increasing training rounds (monotonically decreasing) indicates that the interpretation alignment training in step S4 can effectively improve the interpretation discrimination probability vector. With multimodal discrimination probability vector The training process gradually approaches the target, thereby enhancing the consistency between the interpreted text and the sentiment judgment results.

[0060] In summary, step S4 generates explanatory text containing multimodal evidence based on different modal features through the explanation generation sub-network and the explanation discrimination branch, and uses the explanation alignment loss to adjust the explanation discrimination probability vector. With multimodal discrimination probability vector By imposing constraints in the probability space, the final sentiment assessment results are not only competitive in classification accuracy, but also have a high level of interpretability.

[0061] After completing the multimodal raw data acquisition and preprocessing, multimodal feature encoding, multimodal direct connection discrimination, and interpretation generation and alignment training design in steps S1 to S4, step S5 is the end-to-end training and inference application step, which is used to jointly optimize the entire multimodal sentiment analysis model under a unified training framework, and deploy the model to online or offline application scenarios after training is completed, perform sentiment analysis on newly input multimodal data and output the corresponding natural language interpretation text.

[0062] In one implementation, step S5 first partitions the multimodal sentiment dataset into a training set, a validation set, and a test set. The training set is used to update model parameters, the validation set is used to monitor performance changes during training and prevent overfitting, and the test set is used to evaluate the generalization ability of the final model. Assume the training set contains... Each sample consists of raw text data, raw audio data, and raw video data, along with real-world sentiment category tags and manually annotated explanatory text. After processing in steps S1 and S2, the resulting multimodal samples can be used to... The text modal feature vector is obtained from each sample. Speech modal feature vectors Video modal feature vectors and multimodal fusion feature vectors After calculations in steps S3 and S4, the multimodal discrimination probability vectors can be obtained respectively. Explain the eigenvectors and Explanation of the Discriminant Probability Vector .

[0063] During the training phase, step S5 integrates the classification loss of the multimodal direct-connection discriminant branch, the classification loss of the multimodal interpretation discriminant branch, and the interpretation alignment loss into the overall loss function for joint optimization. For the first... For each training sample, the classification loss of the multimodal direct connection discriminant branch can be calculated based on the real sentiment category labels. The explanation classification loss is calculated based on the relationship between the output of the explanation discrimination branch and the true sentiment category label. And calculate the interpretation alignment loss according to the formula defined in step S4. Based on this, the first... Total loss for each sample Defined as ,in, and These are non-negative weighting coefficients used to balance multimodal classification accuracy, interpreted classification accuracy, and interpreted alignment. In actual training, and This can be determined through a grid search on the validation set. For example, in an exemplary experiment, it can be... and When the values ​​were set to 0.5, the experimental results showed that a high accuracy in sentiment classification and good consistency in interpretation could be achieved under this configuration.

[0064] For the entire training set, the total loss can be averaged across the sample dimensions to obtain the average total loss over the training set. Defined as: .

[0065] in, Let be the total number of samples in the training set. To achieve end-to-end joint training, step S5 can use an optimization method based on stochastic gradient descent to iteratively update the model parameters. Assume the set of all trainable parameters of the model is denoted as . The learning rate is denoted as Therefore, during each parameter update, the parameters can be updated using gradient descent as follows: .

[0066] In practice, to improve training efficiency and reduce gradient estimation variance, step S5 typically employs mini-batch training, dividing the training set into several sub-sets containing... Mini-batches of samples are used, and the average total loss is calculated and parameters are updated only for samples within each mini-batch. If the... The mini-batch used in the next iteration contains a set of sample indices. Its size is The corresponding average total loss for small batches is... It can be written as: At this point, the parameter update rule becomes: .

[0067] By repeating the mini-batch gradient update process over multiple training epochs, the average total loss on the training set can be gradually reduced. The changes in sentiment classification accuracy, explanation quality metric, and explanation alignment loss on the validation set were observed to select an appropriate number of training epochs and model parameter snapshots for the final deployment model version. In an exemplary experiment, as the number of training epochs gradually increased from 5 to 30, the average explanation alignment loss on the validation set decreased from approximately 0.020 to approximately 0.003, consistent with the values ​​plotted in step S4. Figure 4 The consistent curves indicate that the generative branch and the multimodal direct-connect discriminative branch gradually align at the probability distribution level during training.

[0068] After completing the training process, step S5 proceeds to the inference application phase. In the inference application phase, when a new multimodal input sample arrives, the system first executes step S1 to collect and preprocess the raw text, speech, and video data of that sample, obtaining the corresponding text input sequence, speech input sequence, and video input sequence; then, step S2 is executed to map the above input sequences into text modality feature vectors. Speech modal feature vectors and video modal feature vectors And generate multimodal fusion feature vectors. Next, step S3 is executed, where the multimodal discrimination probability vector is calculated through the multimodal direct-connect discriminant subnetwork. The system obtains the predicted sentiment category of the sample; simultaneously, it executes step S4, which generates an explanatory text corresponding to the current sample through the explanation generation subnetwork, and obtains the explanation discrimination probability vector through the explanation discrimination branch. Used for internal consistency checks.

[0069] In inference applications, the system can determine the probability vector based on the multimodal discrimination. The sentiment category corresponding to the largest component is used as the final sentiment tendency output, and the explanatory text generated by the explanation generation subnetwork is also output to the user. In some implementations, to further improve the reliability of the system, a threshold-based confidence judgment mechanism can be set, for example, when... When the maximum component value is lower than a preset threshold (e.g., 0.4), the sample is marked as a low-confidence sample, and an additional review process or simplified explanation text is triggered. In other implementations, comparisons can also be made. and If the numerical difference in the main class exceeds the preset tolerance (e.g., 0.2), the sample is marked as an inconsistent sample so that it can be recorded in the log system and used for subsequent model retraining or manual review.

[0070] See Figure 5 As shown, Figure 5 The horizontal axis represents different datasets, and the vertical axis represents the overall sentiment classification accuracy. Each group of bars represents the performance of the three methods on that dataset (leftmost bar: overall sentiment classification accuracy of the text unimodal sentiment analysis method; middle bar: overall sentiment classification accuracy of the simple multimodal concatenation and fusion method; rightmost bar: overall sentiment classification accuracy of the multimodal sentiment tendency analysis method of this invention). It can be seen from the figure that the method of this invention has achieved the highest sentiment classification accuracy on each dataset.

[0071] In summary, step S5 constructs a unified end-to-end training framework, integrating the multimodal direct-connection discriminative branch, the interpretation generation sub-network, and the interpretation alignment loss into the same objective function for joint optimization. After training, the complete model is deployed to real-world application scenarios to perform sentiment analysis on real-time input text, speech, and video data and generate natural language interpretation text.

[0072] After completing the end-to-end training and inference application in step S5, step S6 is the user feedback collection and model adaptive update step, which is used to continuously collect user feedback information on sentiment judgment results and explanatory text during actual application, and incrementally update or retrain the multimodal sentiment tendency analysis model based on the feedback information, thereby improving the model's adaptability and long-term performance in the target application scenario while ensuring system stability.

[0073] In one implementation, step S6 first records a complete inference log for each processed multimodal sample during the inference application phase. The inference log includes at least: identification information of the original text data, original speech data, and original video data corresponding to the sample, the text input sequence, speech input sequence, video input sequence obtained in steps S1 and S2, and the corresponding text modality feature vectors. Speech modal feature vectors Video modal feature vectors The multimodal fusion feature vector obtained in step S3 and multimodal discrimination probability vector The explanatory text and explanatory feature vector generated in step S4 And the interpretation of the discriminant probability vector This includes the final output of the sentiment category prediction. In this way, offline analysis and relearning of historical samples based on inference logs can be performed without repeating large-scale forward inference computations.

[0074] Based on the aforementioned inference log, step S6 further deploys a feedback collection interface on the user side to collect user feedback on the sentiment assessment results and explanatory text. Specifically, for each output sample, a sentiment assessment result confirmation option and an explanatory text evaluation option can be provided. The sentiment assessment result confirmation option allows the user to mark the sentiment category given by the system as "correct" or "incorrect." When the user selects "incorrect," they can further select one or more correct sentiment categories as correction labels. The explanatory text evaluation option allows the user to score the correctness and comprehensibility of the explanatory text, for example, using a rating scale of 1 to 5, where 1 point represents "completely inconsistent" and 5 points represent "highly consistent." In some implementations, the user can also make partial modifications to the explanatory text or add or remove key information to obtain an explanation that is closer to the user's understanding.

[0075] To facilitate the use of the above feedback information during the training phase, step S6 transforms user feedback into formalized feedback labels and feedback weights. Specifically, for the first... For a sample with user feedback, the sentiment category after user confirmation or correction can be recorded as the feedback sentiment label. The user's rating of the correctness of the explanation text is recorded as the explanation score. Based on this, step S6 calculates the feedback weight for each feedback sample according to the consistency between the feedback sentiment label and the original label, the magnitude of the user's explanation rating, and the system's own confidence information. Feedback weight This is a non-negative real number used to represent the importance of the feedback sample in subsequent incremental training. For example, the feedback weight can be calculated in the following form: .

[0076] in, , , For non-negative hyperparameter coefficients, This indicates the original output sentiment category and the feedback sentiment label. Whether the consistency index (e.g., consistency) is consistent When inconsistent ), Rate the explanatory text for the user (with a value ranging from 1 to 5). This represents the confidence index of the system on this sample (e.g., taking the multimodal discriminant probability vector). The maximum component value). In this example, when the system output sentiment category is inconsistent with the user feedback, the explanation score is high, and the system confidence is low, the corresponding feedback weight is... They will be assigned a large value to be used to correct the judgment behavior of such samples in the subsequent adaptive update process.

[0077] After constructing the feedback label and feedback weights After obtaining the feedback sample set, step S6 can construct an incremental training set containing the feedback samples based on the original training set, which is used for adaptive updating of the model. In one implementation, step S6 introduces a feedback loss term while keeping the original training set unchanged. This is used to measure the model's prediction results on the feedback sample set and the feedback sentiment label. The difference lies in the weighted cross-entropy loss. The feedback loss can be defined as the weighted cross-entropy loss, in the following form: ,in, This indicates the number of samples in the feedback sample set. Indicates the first The cross-entropy loss value corresponding to each feedback sample The aforementioned feedback weights. The feedback loss... When introducing the total loss function, a weighting coefficient can be added to the total loss defined in step S5. Adjusting the feedback loss, the new total loss function is expressed as: .

[0078] in, , , The meaning is consistent with that in steps S4 and S5. , , These are non-negative weighting coefficients. In practical implementation, appropriate coefficients can be selected based on performance on the validation set. Numerical value. For example, in an exemplary experiment, when When the value is 0.2, it can maintain the stability of the original training set while introducing user feedback correction capabilities.

[0079] To further monitor the model's calibration performance and interpretability consistency in real-world applications, step S6 can calculate the calibration error index and interpretability consistency index within a rolling time window. Assuming that in the most recent... In a given online sample, the multimodal discrimination probability vector Using the largest component as the confidence value, the confidence interval can be divided into... For each interval of equal width, the difference between the actual accuracy and the average confidence level is calculated, and the calibration error index is calculated accordingly. Specifically, the first The number of samples in each interval is denoted as . The average accuracy of the samples within this interval is denoted as The average confidence level is denoted as Then the calibration error index It can be defined as: .in, This represents the total number of online samples within the most recent window. A large value indicates a significant discrepancy between the model's confidence level and the actual accuracy. Calibration performance can be improved by increasing the collection of feedback samples and the adaptive update frequency. Simultaneously, the interpretability consistency index can be used to statistically analyze the multimodal discrimination probability vector. With the interpretation of the discriminant probability vector Consistency across the main class. For example, the most recent... In a sample set, the number of samples that satisfy the condition that the main class of the multimodal discrimination branch and the interpretation discrimination branch are the same and the difference in the main class probability does not exceed a preset threshold is denoted as . Then the consistency rate of explanation It can be defined as: .when When the value is low, it indicates that there are many inconsistencies between the explanatory text and the multimodal discrimination result. Step S6 can trigger the retraining process for the explanation generation sub-network and the explanation discrimination branch, so that the explanation judgment result is re-aligned with the multimodal direct connection discrimination result.

[0080] See Figure 6 As shown, from Figure 6 As the number of feedback samples increases, both curves show a gradual upward trend. Moreover, after the number of feedback samples reaches a certain scale, the performance improvement tends to stabilize. This indicates that the feedback weight calculation and adaptive update mechanism designed in step S6 can effectively utilize a limited number of feedback samples to improve model performance, while avoiding instability caused by over-reliance on single feedback.

[0081] In summary, step S6 involves collecting user feedback on the sentiment assessment results and explanatory text during the online inference phase, and then converting this feedback information into data with feedback tags. and feedback weights The feedback sample set is used, and a feedback loss term is introduced under a unified loss function framework. This enables continuous adaptive updates to the multimodal sentiment analysis model; simultaneously, it incorporates calibration error indicators. Consistency rate of explanation By monitoring metrics such as these, the long-term performance of the model in real-world application scenarios is evaluated and optimized, ensuring that the invention not only has high sentiment classification accuracy and interpretability during initial deployment, but also continuously improves performance and reliability through user feedback during ongoing operation.

[0082] After completing the multimodal data processing flow, model training, and adaptive update in steps S1 to S6, step S7 is the system deployment and operation monitoring step. It is used to integrate the multimodal sentiment analysis method of natural language processing into the specific application system, ensure the real-time performance, stability, and scalability of the multimodal sentiment analysis service in the actual operating environment, and continuously evaluate the system performance through operation monitoring indicators, thereby supporting long-term online service.

[0083] In one implementation, step S7 first packages and manages the version of the model parameters trained based on steps S5 and S6. Specifically, this includes packaging and version management of the multimodal feature encoding subnetwork, the multimodal direct-connect discriminant subnetwork, the interpretation and generation subnetwork, and related preprocessing configurations, feature dimension configurations, and loss function weight parameters. , , Each model version is uniformly packaged into a model version package, and a unique version identifier is assigned to each model version. When the system is deployed, the current model version identifier can be specified, and when a rollback or upgrade is required, the model replacement process is completed by switching the model version identifier, so as to ensure the controllability and traceability of online services.

[0084] Based on the aforementioned model version management, step S7 designs the overall system deployment architecture. Specifically, a layered architecture can be adopted, consisting of a front-end data acquisition component, a service gateway component, and a back-end multimodal sentiment analysis service component. The front-end data acquisition component is responsible for collecting raw text, voice, and video data from user terminals, adding timestamps or segmentation identifiers according to the format required in step S1, and packaging the processed multimodal requests into a unified data structure. The service gateway component is responsible for receiving multimodal requests from multiple front-end data acquisition components, performing identity verification, rate limiting, and load balancing on the requests, and then distributing the requests to multiple back-end multimodal sentiment analysis service instances. The back-end multimodal sentiment analysis service component instantiates multiple model inference processes in a single-machine or multi-machine environment. Each inference process executes the forward computation logic in steps S1 to S4 sequentially to obtain sentiment category prediction results and explanatory text, and returns the results to the front-end data acquisition component through the service gateway. This layered architecture allows the system to maintain overall performance by horizontally scaling the number of back-end service instances when increasing the number of front-end accesses or increasing the concurrent request load.

[0085] In the actual deployment implementation, step S7 also requires a quantitative evaluation of the system's response time and throughput. Assume that within a certain evaluation time window, the system processed a total of... There are 1 multimodal request, and the end-to-end response time for each request is denoted as follows: The average response time within that time window It can be defined as: Average throughput within this time window It can be defined as: .in, This indicates the length of the evaluation time window. In actual deployment, by adjusting parameters such as the number of backend service instances, the number of concurrent threads per instance, and the batch size, the average response time and throughput under different concurrent request conditions can be measured, thereby selecting an appropriate system configuration.

[0086] See Figure 7 As shown, Figure 7 The horizontal axis represents the number of concurrent requests, and the vertical axis represents the average response time and average throughput. The solid line curve represents the system's average response time. The trend of changes with the increase in the number of concurrent requests; the dashed line represents the average throughput of the system. The trend of (requests processed per second) as the number of concurrent requests increases. From Figure 7 It can be intuitively seen that when the number of concurrent requests is low, the average throughput increases rapidly with the increase of the number of concurrent requests, while the average response time remains at a low level. When the number of concurrent requests exceeds a certain threshold, the increase in average throughput tends to saturate, while the average response time shows a significant increase, indicating that within this range the system is approaching the hardware resource limit or needs to further increase the number of service instances.

[0087] In terms of operation monitoring, step S7 can collect and visualize various operational status indicators in real time, enabling operations and maintenance personnel to promptly identify potential bottlenecks or anomalies. Operation monitoring indicators may include: average response time. Average throughput This includes metrics such as request queue length, CPU utilization, GPU utilization, memory usage, disk read / write speed, and network bandwidth usage. In multimodal sentiment analysis scenarios, model-related monitoring metrics can also be introduced, such as real-time monitoring of the average confidence score of online samples and the interpretability consistency rate of online samples. This includes explaining the sample estimates of alignment loss. When a certain monitoring metric exceeds a preset threshold, step S7 can trigger an alarm mechanism or an automatic scaling strategy, such as automatically increasing or decreasing the number of backend service instances, adjusting batch size, or request distribution strategy to maintain the stability of the overall system performance.

[0088] In some implementations, to ensure system compatibility across different application scenarios, step S7 can also provide multiple access methods, such as HTTP-based REST interfaces, message queue-based asynchronous interfaces, and function call interfaces for local applications. For REST interfaces, standardized request and response field formats can be defined. Requests may include identification information for text, voice, and video, or directly contain encoded multimodal data. Responses may include sentiment category prediction results and multimodal discrimination probability vectors. The model provides the main class probability value, explanatory text, and necessary metadata fields. For asynchronous interfaces, batch sentiment annotation of large-scale offline data can be achieved through message queues, and results can be returned through callback queues or log files. For local application interfaces, preprocessed multimodal features can be directly input into the loaded model instance through local function calls to obtain sentiment results and explanatory text.

[0089] In summary, step S7, through the design of the deployment architecture, optimization of resource allocation, and the introduction of a runtime monitoring mechanism for the multimodal sentiment analysis model, enables the present invention to not only possess high sentiment classification accuracy and interpretability at the algorithm level, but also to have good real-time performance, scalability, and maintainability at the engineering implementation level.

[0090] After completing steps S1 to S7 of multimodal data processing, model training, adaptive updates, and system deployment and operation monitoring, step S8 is a data security and privacy protection control step. It is used to protect the raw text data, raw voice data, raw video data, and intermediate features involved in the entire process of multimodal sentiment analysis method, reduce the risk of leakage of user identity, sensitive attributes and other information during collection, transmission, storage and training, and achieve a balance between privacy protection and sentiment classification performance.

[0091] In one implementation, step S8 first performs desensitization processing on the raw text data, raw voice data, and raw video data at the data acquisition end. Specifically, for the raw text data, explicit identity information such as names, contact information, addresses, and account identifiers can be identified and replaced with general placeholders, such as using a unified marker to replace specific names, using a generalized region to replace detailed addresses, and using a mask to replace specific accounts. For the raw voice data, the speaker feature weakening module can be enabled depending on the application scenario. In this module, voiceprint transformation or vocal tract feature mapping can be used to map the speaker features in the raw voice to standardized pseudo-speaker features, thereby weakening the voiceprint features related to individual identity while retaining the tone and rhythm changes related to emotion. For the raw video data, facial areas or other sensitive areas in the image can be blurred or occluded on the local end. When the emotion analysis task does not require the identification of specific facial features, only the structured features related to facial expression changes are retained, avoiding the direct uploading of raw video footage that can identify the person. After the above desensitization process, the front-end data collection component organizes the desensitized multimodal data into multimodal requests according to the requirements of step S1 and sends them to the back-end multimodal sentiment analysis service.

[0092] In another implementation, to further reduce the sensitivity of the raw data during transmission and storage, step S8 can deploy the multimodal feature encoding portion of step S2 on an edge device close to the data acquisition end, so that the text modal feature vector Speech modal feature vectors and video modal feature vectors In the local generation phase, only the multimodal feature vectors and necessary timestamps and sample identification information are uploaded to the central server for subsequent processing in steps S3 to S6. At this point, the central server does not need to access the original text, audio, and video content; instead, it directly performs sentiment analysis and interpretation based on the multimodal feature vectors, thus shortening the transmission path of the original data across the network from an architectural perspective. For training data storage that must be performed at the central end, step S8 can encrypt the storage medium, persistently storing the de-identified multimodal data and multimodal feature vectors in encrypted form, and restricting access to only authorized service processes through access control policies.

[0093] During the training phase, step S8 can introduce a differential privacy-style noise injection mechanism during gradient update to reduce the sensitivity of individual training samples to the final model parameters. Specifically, in a mini-batch training scenario, assuming that in a certain training iteration, the mini-batch sample set is... Its size is For sets For each sample in the dataset, the corresponding parameter gradient vector can be calculated, and the gradient vector of the first sample can be used to calculate the gradient vector of the second sample. The gradient vector corresponding to each sample is denoted as . Step S8 can be performed on each gradient vector. Perform norm clipping to constrain its Euclidean norm to a preset threshold. Within this range, the clipped gradient vector is obtained. The cutting process can be formally represented as:

[0094] in, Represents the gradient vector The 2-norm, The threshold value is used for cropping. Then, the cropped gradient vectors are averaged over a mini-batch to obtain the average cropped gradient vector. : .

[0095] Based on this, step S8 can introduce a Gaussian noise vector for each gradient update. Each component of the noise vector follows a constant with a mean of 0 and a variance of 0. The normal distribution, where Privacy noise intensity parameters. Privacy gradient vector used to update parameters after noise injection. Defined as: .

[0096] Assume the set of all trainable parameters of the model is The learning rate is Then, the parameter update rule in this iteration is: .

[0097] Through the gradient pruning and noise injection processes described above, the influence of a single training sample on the model parameters can be limited to a certain extent, thereby reducing the success rate of member inference attacks or reconstruction attacks based on model parameters.

[0098] To facilitate understanding of the above calculation process, a simplified two-dimensional gradient example is used below to illustrate the specific numerical calculations of gradient clipping and noise injection. Assume that in a minibatch of a certain training iteration... For each of the samples, their original gradient vectors are as follows: , Set the cropping threshold First, calculate the L2 norm of the two gradient vectors:

[0099] .

[0100] For the first sample, since Therefore, the gradient vector remains unchanged after clipping, so:

[0101] For the second sample, since Therefore, it needs to be scaled proportionally: ; Then calculate the average clipping gradient vector. : ; Assuming privacy noise vector for: ; Then the privacy gradient vector for: ; If learning rate Then the parameter update amount is The numerical examples above fully demonstrate the specific calculation process of gradient clipping and noise injection.

[0102] See Figure 8 As mentioned above, in Figure 8 The two curves in the middle follow The increase of the value shows the opposite trend: the overall sentiment classification accuracy decreases slowly, while the success rate of member inference attacks decreases significantly and gradually flattens out. This indicates that the privacy noise injection mechanism introduced in step S8 has a limited impact on performance within a controllable range, but can significantly reduce the risk of the model being used for privacy attacks.

[0103] Furthermore, step S8 can introduce privacy audit logs and access control policies during the system deployment and operation monitoring phase. Specifically, audit logs can be generated for each access operation to training data, model update operation, and feedback sample writing operation. These audit logs include the operation type, timestamp, operator identification, and the scope of data involved. Centralized management and periodic auditing can detect any abnormal access behavior. Regarding access control, role-based access control policies can restrict access permissions for different service components or operations personnel to raw data, multimodal feature vectors, and model parameters. For example, only the training service can be allowed to access encrypted multimodal feature vectors, and only the model deployment service can be allowed to access the model weight files required for inference, thereby further reducing the risk of data leakage at the system level.

[0104] In summary, step S8 protects sensitive information in the multimodal sentiment analysis method from multiple stages and levels by performing desensitization processing on text, voice and video at the data acquisition end, deploying edge coding and feature uploading mechanisms at the feature level, introducing gradient pruning and privacy noise injection during the training phase, and combining privacy auditing and access control strategies during the operation phase.

[0105] After completing steps S1 to S8, which involve multimodal data processing, model training, adaptive updates, system deployment, and data security and privacy protection controls, step S9 is a cross-scenario migration and domain adaptive optimization step. This step is used to migrate the multimodal sentiment analysis method of natural language processing from one source domain to multiple target application scenarios. By using a domain adaptive training strategy, the performance degradation caused by differences in data distribution between different scenarios is reduced, enabling the method to maintain stable sentiment classification performance and interpretation effects in various application domains such as customer service dialogue, online education, and public opinion analysis.

[0106] In one implementation, step S9 first divides the different application scenarios into domains, defining the data domain used for the original model training as the source domain and the business scenario to be migrated as the target domain. Assuming there exists a set of source domains and one or more sets of target domains, the source domain is denoted as... Let the target domain set be denoted as , ,… each of these fields This includes multimodal samples and their sentiment labels collected within this domain. In the initial training phase of step S5, the multimodal sentiment analysis model primarily focuses on the source domain. When trained on the target domain, the model parameters are more adapted to the data distribution of the source domain; when the model is deployed to a new business scenario, the target domain... The text expression habits, speech intonation patterns, and video shooting environment of the target domain may differ significantly from those of the source domain. Without domain-adaptive optimization, the accuracy of sentiment classification in the target domain may decline.

[0107] To address the aforementioned issues, step S9 can introduce a domain-adaptive loss between the source and target domains to simultaneously optimize the classification performance of the source domain and the consistency of the source-target domain feature distributions during training. In one implementation, step S9 is for each domain... Set a domain weight coefficient. ,in And satisfy .

[0108] During joint training, the classification losses from each domain can be summed by domain weights to obtain the domain-weighted classification loss. Its form can be written as: .

[0109] in, Indicates in the field The classification loss calculated above can be achieved using cross-entropy loss or other suitable classification loss methods. In practical applications, the domain weight coefficients can be adjusted based on the sample size, business importance, and label quality of different domains. Configure settings, such as assigning relatively larger weights to target areas with fewer samples but higher business importance, to accelerate performance improvement in those areas.

[0110] In addition to the domain-weighted classification loss, step S9 can further reduce inter-domain differences by aligning the feature distributions of the source and target domains. Specifically, this can be achieved by fusing feature vectors across multiple modalities. In the feature space, a domain difference metric is introduced to measure the difference in feature distribution between the source and target domains. Let be the set of multimodal fused feature vectors of a mini-batch of samples from the source domain in a given training iteration. target area The set of multimodal fusion feature vectors for a small batch of samples is denoted as Based on these feature vectors, the difference in feature mean or other statistical differences between the source and target domains can be estimated, and a domain-adaptive loss can be constructed accordingly. In a simplified implementation, the mean difference can be used as the basis for constructing the domain adaptive loss, and the mean vector of the source domain features can be denoted as... target area The eigenmean vector is denoted as Then the domain adaptive loss can be defined as: ,in, These are non-negative weighting coefficients used to adjust the source-target domain. Alignment strength between them This represents the sum of squares of the differences in feature means. It is achieved by minimizing this sum during training. This can make the multimodal fusion feature vector of the target domain statistically closer to the source domain, thereby mitigating the impact of data distribution offset on sentiment classification performance.

[0111] When incorporating the domain-adaptive loss into the overall training, step S9 can add a domain-adaptive loss term to the total loss function in step S5 and introduce new weight coefficients. This makes the new total loss function express as: .in, , , , The meaning is consistent with that in steps S4 to S6. , , The aforementioned weighting coefficients, These are non-negative weighting coefficients used to balance the relationship between the domain adaptive loss and other losses. They are minimized during training. This allows for gradual improvement in sentiment classification performance in the target domain while maintaining performance in the source domain.

[0112] See Figure 9 As shown, Figure 9 The horizontal axis represents the number of domain-adaptive iteration rounds, and the vertical axis represents the target domain sentiment classification accuracy and domain difference index. Figure 9 The solid line curve represents the change in target domain sentiment classification accuracy with increasing number of domain-adaptive training iterations; the dashed line curve represents the change in domain difference metrics (e.g., source-target feature mean difference) with increasing number of iterations. Figure 9 It can be intuitively seen that as the number of iterations of domain adaptation increases, the accuracy of the target domain sentiment classification continues to rise and gradually approaches saturation, while the domain difference index continues to decline and gradually approaches stability. This indicates that after several rounds of domain adaptation training, the feature distribution of the target domain gradually approaches the feature distribution of the source domain, and the inconsistency between domains is alleviated.

[0113] In some implementations, step S9 can also be combined with the user feedback mechanism of step S6 to construct a joint sample set with domain labels and feedback labels, which can be used to further improve the adaptability of the target domain. Specifically, samples from the target domain can be used to... The feedback samples are given higher feedback weights. and in feedback loss The introduction of a domain-weighted factor allows samples with incorrect predictions but reliable feedback in the target domain to receive greater adjustment during adaptive updates, thereby accelerating the convergence of performance in the target domain. Furthermore, domain-specific input can be introduced into the explanation generation subnetwork, enabling the explanation text to be tailored to the language style and business needs of different domains. For example, in the customer service dialogue domain, explanations closer to customer service terminology can be generated, while in the online education domain, explanations more focused on teaching feedback can be generated, thus improving practical usability across multiple scenarios.

[0114] In summary, step S9 introduces domain-weighted classification loss and domain-adaptive loss between the source domain and multiple target domains, and incorporates domain-adaptive weight coefficients into the overall loss function. This study demonstrates the transfer and optimization of multimodal sentiment analysis methods in natural language processing across multiple scenarios and domains. While maintaining performance in the source domain, it significantly improves the accuracy of sentiment classification in target domains such as customer service dialogues, online education, and public opinion commentary, and reduces the risk of performance degradation caused by differences in domain distribution.

[0115] After completing steps S1 to S9—multimodal data processing, model training and adaptive updates, system deployment, data security control, and cross-scenario migration—step S10 is the emotion-driven business strategy generation and effect evaluation step. This step is used to structurally map the results of multimodal sentiment analysis and their natural language interpretation into executable business action strategies, and to quantitatively evaluate the effects of various strategies on actual business indicators, thereby achieving a closed loop from "emotion recognition" to "emotion-driven business decision-making."

[0116] In one implementation, step S10 first uses the multimodal discrimination probability vector output in step S3 as a basis. Construct sentiment decision scores to drive different types of business actions. Assume the sentiment category set includes three categories: "positive," "neutral," and "negative," with the corresponding multimodal discriminant probability components denoted as follows: , , And satisfy: To quantitatively characterize the risk level of a user's current emotions, an emotional risk score can be defined. An example of this is: ,in, and This is a non-negative weighting coefficient used to balance the contribution of "negative" and "positive" sentiment probabilities to overall risk. For example, in a customer service scenario, it can be... Set it to 1.0, and We set it to 0.5 to reflect the greater need to focus on the impact of "negative emotions".

[0117] Based on the aforementioned emotional risk score, step S10 can define multiple trigger thresholds for business actions. Assuming three action levels are defined for a customer service scenario: normal handling, priority handling, and escalation handling, a normal-priority threshold can be set. and priority – upgrade threshold ,in For a given request sample, if If so, the sample is assigned to the regular processing queue; if If so, the sample is assigned to the priority processing queue; if If the sample is not found to be in the required state, it will be assigned to the escalation processing queue for handling by more experienced personnel or higher-level processes. This strategy can be formally expressed as: if... Then the business action type ;like Then the business action type ;like Then the business action type .in, This indicates the type of business action corresponding to the current sample. , , These represent three types of actions: normal processing, priority processing, and escalation processing.

[0118] To illustrate the above calculation process, a specific numerical example is given below. Assume that for a given dialogue sample, the multimodal discriminant probability vector is:

[0119] Set weights , Substituting into the emotional risk score formula, we get:

[0120] If a threshold is configured , Then the emotional risk score of the sample Between and Between these, the corresponding business action type is prioritized. Similarly, for another example sample, if , , Then we have: .

[0121] at this time, Therefore, the sample was assigned to the regular processing queue. .

[0122] In another implementation, step S10 can further structure the natural language explanation text output in step S4 into explanation features to assist in fine-tuning business strategies. Specifically, explanation labels can be defined for key phrases appearing in the explanation text (such as "expressing dissatisfaction in multiple rounds," "frequently mentioning refunds," "repeatedly mentioning praise," etc.), and explanation feature vectors can be set. Each dimension of the algorithm represents whether a particular explanatory label is triggered or the intensity of the trigger. At the business decision-making level, emotional risk scores can be used... Explanation of eigenvectors Combined into a decision input vector It calculates a comprehensive business decision score using a set of configurable business rules or a simple linear scoring function. For example, it can be defined as: .in, , , , These are configurable business parameters. This indicates whether there are explanatory tags related to "refund" in the explanatory text. This indicates whether there are explanatory tags related to "complaint" in the explanatory text. By analyzing... Setting multi-level thresholds allows for more detailed action types at the business strategy level, such as triggering coupon distribution, triggering manual follow-up, or triggering alert work orders.

[0123] See Figure 10 As shown, Figure 10 The horizontal axis represents the upgrade threshold. The vertical axis represents the upgrade processing rate and the user complaint rate. Figure 10 The solid line curve in the graph represents the escalation processing ratio (the proportion of requests allocated to the escalation processing queue) as a function of the emotional risk escalation threshold. The trend of change; the dashed curve represents the user complaint rate as the emotional risk escalates to a certain threshold. The trend of change. From Figure 10 It can be seen intuitively that, with As the cost increases, the proportion of upgrade processing decreases significantly, while user complaints initially drop rapidly but then rebound slightly after exceeding a certain threshold, indicating the existence of a threshold range where performance and cost are relatively balanced.

[0124] In some implementations, step S10 can also combine the operation monitoring mechanism of step S7 and the user feedback mechanism of step S6 to construct an automatic strategy adjustment module. This module can periodically perform statistical analysis on business indicators over a recent period and automatically adjust the weighting of the emotional risk score based on indicators such as complaint rate, satisfaction score, escalation processing ratio, and system load. , and threshold , Parameters such as these can be adjusted. For example, when a significant increase in the user complaint rate is detected over a period of time, the percentage can be appropriately reduced. and The value of allows more samples with moderate emotional risk to enter the priority processing or escalation processing queue; when the overall system load pressure is high and the complaint rate is low, the threshold can be appropriately increased to reduce the proportion of escalation processing, thereby reducing resource consumption while ensuring user experience.

[0125] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A multimodal sentiment analysis method for natural language processing, characterized in that, Includes the following steps: Step S1: Obtain the multimodal raw data of the object to be analyzed. The multimodal raw data includes text data, speech data and video data aligned on the same time axis. Perform word segmentation and sub-word segmentation on the text data, perform noise reduction and frame segmentation on the speech data, and perform face detection and key frame selection on the video data to obtain the text input sequence, speech input sequence and video input sequence. Step S2: Input the text input sequence into the text encoding sub-network to obtain the text modality feature vector. The speech input sequence is input into the speech coding subnetwork to obtain the speech modal feature vector. The video input sequence is input into the video coding subnetwork to obtain the video modal feature vector. ; Step S3, during the training phase, the text modality feature vector The speech modal feature vector and the video modal feature vector Inputting the multimodal direct-connect discriminant subnetwork yields the multimodal fused feature vector. And calculate the multimodal discrimination probability vector according to the following formula. : ， in, For multimodal discrimination weight matrix, It is a column vector; Step S4, convert the text modality feature vector The speech modal feature vector and the video modal feature vector Input a modal interpretation generation subnetwork, and generate text modal interpretation subsequences within the modal interpretation generation subnetwork. Speech modal interpretation subsequence and video modality interpretation subsequence and interpret the subsequence in the text modality. The speech modality interpretation subsequence and the video modality interpretation subsequence Modal markers are added before and after, and the three are connected in a preset order to obtain the explanatory text sequence. ; Step S5, the explanatory text sequence Input the interpretive encoding subnetwork to obtain the interpretive feature vector. The explanatory feature vector Input the interpretation and discrimination subnetwork, and calculate the interpretation and discrimination probability vector according to the following formula. : ， in, To explain the discriminant weight matrix, It is a column vector; Step S6, based on the interpretation and discrimination probability vector Determine the emotional tendency category label The sentiment tendency category label Corresponding to the interpretation and discrimination probability vector The category index with the highest probability; Step S7, during the training phase, the probability vector is determined based on the interpretation. With genuine emotion tags Calculate classification loss The true emotion tags Pre-labeled sentiment category tags; Step S8, during the training phase, based on the multimodal discrimination probability vector With the explained discriminant probability vector Calculate the interpretation consistency loss Calculate the total loss according to the following formula. : ， ， in, Represents the L2 norm, The coefficients are real numbers greater than zero. Total loss The trainable parameters in the text encoding subnetwork, the speech encoding subnetwork, the video encoding subnetwork, the modal interpretation generation subnetwork, the interpretation encoding subnetwork, and the interpretation discrimination subnetwork are jointly updated, and only steps S1, S2, S4, S5, and S6 are executed during the inference phase.

2. The multimodal sentiment analysis method for natural language processing according to claim 1, characterized in that, In step S4, the modality interpretation generation subnetwork generates the text modality interpretation subsequence. The speech modality interpretation subsequence and the video modality interpretation subsequence A unified template is used, which includes: modality initiation markers, sentiment direction phrases, sentiment intensity phrases, and evidence description phrases, and is applied in the text modality interpretation subsequence. Insert copy markers corresponding to high attention weight words in the text input sequence into the speech modality interpretation subsequence. Insert index markers corresponding to high-energy or high-pitched frames in the speech input sequence into the video modality interpretation subsequence. Insert index markers corresponding to keyframes in the video input sequence.

3. The multimodal sentiment analysis method for natural language processing according to claim 1, characterized in that, In step S5, the interpretation encoding subnetwork processes the interpretation text sequence. During encoding, modal position encoding and temporal position encoding are added to each word vector. The modal position encoding is determined based on the modal marker, and the temporal position encoding is determined based on the index marker. The temporal position encoding is calculated using the following formula: ， in, The time step sequence corresponding to the index marker. The preset maximum number of time steps, The integer position index in the time position encoding.

4. The multimodal sentiment analysis method for natural language processing according to claim 1, characterized in that, The multimodal sentiment analysis method for natural language processing further includes step S9 during the training phase. Step S9 is performed between steps S4 and S8, and step S9 includes: Dual samples are constructed based on the original samples, and at least one type of modality data in the dual samples is replaced with modality data that is different from that in the original samples; Steps S1 to S8 are performed on the original sample and the dual sample respectively to obtain the explanatory text sequence of the original sample. Explanatory text sequence of the dual sample And calculate the corresponding interpretation and discrimination probability vectors respectively. and ; The duality consistency loss is calculated using the following formula. : ， The duality consistency loss Add to the total loss, so that the total loss is updated to: ， in, The coefficients are real numbers greater than zero.

5. The multimodal sentiment analysis method for natural language processing according to claim 1, characterized in that, The multimodal sentiment analysis method for natural language processing further includes step S10 during the training phase. Step S10 is performed between steps S5 and S8, and step S10 includes: The explanatory feature vector Input the reconstructed subnetwork to obtain the reconstructed multimodal feature vector. ; The reconstruction loss is calculated using the following formula. : ， The reconstruction loss Add to the total loss, so that the total loss is updated to: ， in, The coefficients are real numbers greater than zero.

6. The multimodal sentiment analysis method for natural language processing according to claim 1, characterized in that, In step S4, the modality interpretation and generation subnetwork uses the text modality feature vector... The speech modal feature vector and the video modal feature vector Calculate the modal confidence scalar , and And calculate the mode-generated gating system according to the following formula. , and : ， ， ， in, , , , , , The coefficients are real numbers. For the Sigmoid function, , , For those located in the interval When the modality generation gating coefficient is lower than a preset threshold, the modality interpretation generation subnetwork does not generate the corresponding modality interpretation subsequence.

7. The multimodal sentiment analysis method for natural language processing according to claim 1, characterized in that, The multimodal sentiment analysis method for natural language processing employs a sliding time window mechanism when processing long-term multimodal data streams. The multimodal raw data is divided into multiple window segments according to a fixed window length and window step size. Steps S1 to S6 are performed on each window segment to obtain a window-level explanatory text sequence and a window-level sentiment category label. Based on the window start time sequence, the explanatory text sequences of adjacent windows are connected in chronological order, and window markers are inserted at the window boundaries to obtain the global explanatory text sequence.

8. The multimodal sentiment analysis method for natural language processing according to claim 1, characterized in that, In step S4, the modality interpretation generation subnetwork processes the interpretation text sequence. Each word position in the code employs a pointer-vocabulary hybrid output mechanism, which includes: Calculate the attention distribution vector for the text input sequence. ; Calculate the vocabulary distribution vector for the explanatory vocabulary. ; Calculate the copy factor The copy coefficient is located in the interval real numbers; The final output distribution is calculated using the following formula: ， in, This is the final output distribution vector.

9. The multimodal sentiment analysis method for natural language processing according to claim 1, characterized in that, In step S5, the interpretation discrimination subnetwork employs an attention mechanism with an interpretation mask, the interpretation mask being based on the interpretation text sequence. The generation of modal marker identifiers and index markers makes it possible to correspond to the text sequence not described in the explanation text sequence. The implicit units at modal or temporal locations appearing in the calculation of the explanatory feature vector It was blocked at the time.

10. The multimodal sentiment analysis method for natural language processing according to claim 1, characterized in that, The multimodal sentiment analysis method in natural language processing assigns weights to the interpretation consistency loss coefficients during the training phase. A phased update strategy is adopted, which includes: In the initial training phase, the weight coefficients of the interpretation consistency loss are set to initial values. ; After each training cycle, the weight coefficients of the interpretation consistency loss are updated according to the following formula: ， in, The maximum weighting coefficient, For the weight growth step size, This represents the number of training cycles that have been completed. , , It is a real number greater than zero.

Images