A Multimodal Sentiment Analysis Method and System Based on Dynamic Noise Estimation and KAN

Abstract

This application discloses a multimodal sentiment analysis method and system based on dynamic noise estimation and KAN, belonging to the field of multimodal large language model technology. The method includes the following steps: inputting the extracted multimodal feature vectors into the DNE of the corresponding modal features of the machine learning model to generate noise confidence scores for the corresponding modal features; calculating the regularization coefficient corresponding to the noise confidence score for each feature dimension; calculating the loss value of the loss function of the machine learning model based on the regularization coefficient; updating the weights and spline control coefficients of the NIB-KAN based on the loss value; and generating the first sentiment analysis result of the video to be analyzed based on the feature encoder and NIB-KAN of the trained machine learning model, as well as the sentiment analyzer. This application improves the robustness of the machine learning model in performing sentiment analysis on videos in complex environments.

Description

Technical Field

[0001] This application belongs to the field of multimodal large language model technology, specifically involving a multimodal sentiment analysis method and system based on dynamic noise estimation and KAN. Background Technology

[0002] Currently, multimodal large language models typically use linear projection layers as modality alignment layers to achieve a unified mapping of visual, auditory, and textual features when performing sentiment analysis on videos.

[0003] However, this technical solution has the following obvious shortcomings:

[0004] First, linear alignment layers can only perform global linear transformations. When video or audio data is affected by physical noise, the noise after passing through the upstream nonlinear feature extraction network causes a complex nonlinear distribution shift in the feature space, resulting in noisy features being directly passed downstream. Second, using fixed weights for all inputs means that when the signal quality of a particular modality is extremely poor, high-noise features will still interfere with overall inference. Furthermore, traditional techniques typically perform selection on a modal-by-modality basis, leading to the loss of some effective semantic information. Summary of the Invention

[0005] The purpose of this application is to provide a multimodal sentiment analysis method and system based on dynamic noise estimation and KAN, which can improve the robustness of machine learning models when performing sentiment analysis on videos in complex environments.

[0006] To solve the above-mentioned technical problems, this application is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a multimodal sentiment analysis method based on dynamic noise estimation and KAN, the method comprising:

[0008] During the forward propagation of the training phase of the machine learning model, after the feature encoder based on the machine learning model extracts features from the training samples, the extracted multimodal feature vectors are input into the dynamic noise estimator (DNE) of the corresponding modal features of the machine learning model to generate noise confidence scores for the corresponding modal features.

[0009] Calculate the regularization coefficient corresponding to the noise confidence score for each feature dimension;

[0010] The loss value of the loss function of the machine learning model is calculated based on the regularization coefficient;

[0011] During backpropagation in the training phase, the weights and spline control coefficients of the Noise-Aware Information Bottleneck – Kolmogorov-Arnold Network (NIB-KAN) for the corresponding modal features of the machine learning model are updated based on the loss value; wherein, the NIB-KAN is used to perform modal alignment of multimodal features, and the updated weights and updated spline control coefficients are used for the next round of training of the machine learning model.

[0012] After training the machine learning model is completed, in response to receiving the video to be analyzed, a first sentiment analysis result of the video to be analyzed is generated based on the feature encoder and NIB-KAN of the trained machine learning model, as well as the sentiment analyzer.

[0013] Secondly, embodiments of this application provide a multimodal sentiment analysis system based on dynamic noise estimation and KAN, the multimodal sentiment analysis system based on dynamic noise estimation and KAN includes:

[0014] The first generation module is used to generate noise confidence scores for the corresponding modal features by inputting the extracted multimodal feature vectors into the dynamic noise estimator (DNE) of the corresponding modal features of the machine learning model after feature extraction of the training samples based on the feature encoder of the machine learning model during the forward propagation of the training phase.

[0015] The first calculation module is used to calculate the regularization coefficient corresponding to the noise confidence score for each feature dimension.

[0016] The second calculation module is used to calculate the loss value of the loss function of the machine learning model based on the regularization coefficient;

[0017] An update module is used to update the weights and spline control coefficients of the Noise-Aware Information Bottleneck-Kolmogorov-Arnold Network (NIB-KAN) of the corresponding modal features of the machine learning model based on the loss value during backpropagation in the training phase; wherein, the NIB-KAN is used to perform modal alignment of multimodal features, and the updated weights and updated spline control coefficients are used for the next round of training of the machine learning model.

[0018] The second generation module is used to generate a first sentiment analysis result of the video to be analyzed based on the feature encoder and NIB-KAN of the trained machine learning model and the sentiment analyzer after the training of the machine learning model is completed, in response to receiving the video to be analyzed.

[0019] Thirdly, embodiments of this application provide a computer device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0021] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0022] In this embodiment, firstly, by employing NIB-KAN as a modality alignment layer, the nonlinear mapping constructed using the B-spline basis functions of KAN can fit the complex distribution shifts caused by noise in the feature space, realigning noisy features to a clean feature distribution. This overcomes the limitation of linear projection layers in handling nonlinear distortions, thus preventing noisy features from being directly propagated downstream. Secondly, a noise confidence score is generated during training using DNE, and adaptive regularization is applied to the weights and spline control coefficients of NIB-KAN based on this score. During backpropagation, this forces the parameters of channels corresponding to high-noise feature dimensions to zero, thereby solidifying the noise blocking capability after training. This achieves fine-grained screening of noisy channels, preventing high-noise features from interfering with subsequent inference. Furthermore, NIB-KAN allows some noisy features to be repaired before high-noise features are blocked, enabling the machine learning model to adapt to different modal signal quality. This allows the machine learning model to accurately suppress noise while preserving effective semantic information. Finally, during the inference stage of the machine learning model, highly robust sentiment analysis can be achieved solely based on the solidified parameters, significantly improving performance and stability in real-world noisy scenarios. Attached Figure Description

[0023] Figure 1 This is one of the flowcharts illustrating a multimodal sentiment analysis method based on dynamic noise estimation and KAN provided in some embodiments of this application;

[0024] Figure 2 This is a schematic diagram of the structure of a multimodal sentiment analysis system based on dynamic noise estimation and KAN provided in some embodiments of this application;

[0025] Figure 3 This is a schematic diagram of the connection between the NIB-KAN projector and the dynamic noise estimator provided in some embodiments of this application;

[0026] Figure 4This is one of the flowcharts illustrating a multimodal sentiment analysis method based on dynamic noise estimation and KAN provided in some embodiments of this application;

[0027] Figure 5 This is a block diagram of a multimodal sentiment analysis system based on dynamic noise estimation and KAN provided in some embodiments of this application;

[0028] Figure 6 These are internal structural diagrams of a computer device provided in some embodiments of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] The following description, in conjunction with the accompanying drawings, details the multimodal sentiment analysis method based on dynamic noise estimation and KAN provided in this application through specific embodiments and application scenarios.

[0032] In one exemplary embodiment, reference is made to Figure 1 The method includes steps 102-106. Wherein:

[0033] Step 102: During the forward propagation of the training phase of the machine learning model, after the feature encoder based on the machine learning model extracts features from the training samples, the extracted multimodal feature vectors are input into the dynamic noise estimator (DNE) of the corresponding modal features of the machine learning model to generate noise confidence scores for the corresponding modal features.

[0034] In this embodiment, the machine learning model includes a feature encoder, a projector, a dynamic noise estimator (DNE), and a sentiment analyzer. The projector, used for modal alignment of the multimodal features output by the feature encoder, is built upon the Noise-aware Information Bottleneck-Kolmogorov-Arnold Network (NIB-KAN).

[0035] NIB-KAN is built upon KAN. NIB-KAN refers to a KAN with noise suppression capabilities after adjusting the weights and spline control coefficients of KAN through DNE. KAN is a nonlinear mapping network based on B-spline functions.

[0036] DNE can be a multilayer perceptron (MLP) containing two fully connected layers.

[0037] In the application phase, the machine learning model does not apply this DNE.

[0038] In some embodiments, the sentiment analyzer may be constructed using, but is not limited to, a Large Language Model (LLM) or a classifier. The LLM may be the Qwen2.5-7B-Instruct model.

[0039] In some embodiments, a feature encoder may include an audio feature extraction unit, a temporal dynamic feature extraction unit, a global context feature extraction unit, and a local detail feature extraction unit.

[0040] Specifically, the audio feature extraction unit inputs audio data from the video data to obtain an audio feature vector; the temporal dynamic feature extraction unit inputs aligned face image sequences from the video data to obtain a temporal dynamic feature vector; the global context feature extraction unit inputs peak emotion frames from the video data to obtain a global context feature vector; and the local detail feature extraction unit inputs aligned face image sequences from the video data frame by frame to obtain a local detail feature vector.

[0041] In this embodiment, the feature encoder includes four feature extraction units, the projector includes four NIB-KANs, and four DNEs serve as bypass control units for the NIB-KANs. During the machine learning model training process, the four feature vectors output by the feature encoder are input into their respective independent trainable NIB-KANs and DNEs, and the NIB-KANs output modally aligned token vectors. Finally, the sentiment analyzer concatenates the four token sequences with the text instruction vectors and inputs them into an LLM or classifier to generate a second sentiment analysis result.

[0042] Among them, the text instruction vector refers to the vector corresponding to the sentiment analysis instruction input by the user.

[0043] In some embodiments, the audio feature extraction unit may use a pre-trained WavLM encoder; the temporal dynamic feature extraction unit may use a pre-trained VideoMAE V2 encoder; and the global context feature extraction unit and the local detail feature extraction unit may reuse a pre-trained DINOv2 encoder.

[0044] In this embodiment, for audio feature extraction, the original audio waveform (audio data) can be resampled to 16kHz and normalized, input into the WavLM model, and the output sequence of the last layer of the WavLM model can be extracted and average pooled to obtain the audio feature vector. For temporal dynamic feature extraction: 16 frames of aligned face image sequence are obtained from the video data, the image size is adjusted to 224x224 pixels, input into the VideoMAE V2 model, segmented into a spatiotemporal block embedding sequence through Cube Embedding, and after Transformer encoding, the vector corresponding to the CLS (Classification) Token is extracted as the temporal dynamic feature vector. For global context feature extraction: the original peak emotion frames are obtained from the video data, the frame size is adjusted to 518x518 pixels, input into the DINOv2 model, segmented into an image patch embedding sequence through Patch Embedding, and after Transformer encoding, the vector corresponding to the CLS Token is extracted as the global context feature vector. For local detail feature extraction: 16 frames of aligned face image sequence are obtained from video data, the size is adjusted to 224x224 pixels frame by frame and input into the DINOv2 model independently. First, spatial average pooling is performed on the output of each frame to obtain frame-level features, and then temporal average pooling is performed on all frame-level features to obtain local detail feature vectors.

[0045] In this embodiment, the extraction process for each modal feature is as follows:

[0046] First, the audio feature extraction process is as follows:

[0047] I. Input preprocessing.

[0048] The original audio waveform segment is ,in, This refers to the original number of sampling points. Preprocessing is performed using the feature processor of the WavLM model (resampling to 16kHz, normalization, etc.) to obtain a waveform tensor that matches the model input. , This is the number of sampling points after preprocessing. This preprocessing is achieved using the following formula (1):

[0049] (1)

[0050] in, It is a waveform tensor; It is a segment of the original audio waveform; It is the feature processor for the WavLM model.

[0051] II. WavLM model encoding.

[0052] The WavLM model consists of a Convolutional Neural Network (CNN) feature extractor and an N-layer Transformer encoder. composition.

[0053] First, the CNN feature extractor extracts waveform tensors. Downsampling is performed to convert the acoustic feature sequences into low frame rate sequences. ,in, It is the length of the feature sequence, i.e., the number of frames. , It is the feature dimension output by the CNN feature extractor. This downsampling is achieved by the following formula (2).

[0054] (2)

[0055] in, It is an acoustic feature sequence; It is a CNN feature extractor.

[0056] Then, the feature sequence Passing in sequence layer The expression is as shown in formula (3):

[0057] (3)

[0058] in, It is the first layer The output hidden state sequence, It is the first layer The output hidden state sequence; yes The hidden layer dimension.

[0059] III. Feature aggregation and output.

[0060] The last layer can be used Output hidden state sequence As a high-level representation of the original audio waveform segment.

[0061] To obtain a fixed-dimensional audio feature vector representing the entire original audio waveform segment. , can be Average pooling is performed using the following formula (4).

[0062] (4)

[0063] in, yes The Middle Feature vectors of each time step (frame); yes The length.

[0064] Second, the extraction process for visual modal features (including temporal dynamic features, global context features, and local detail features) is as follows:

[0065] (1) Data preparation stage.

[0066] OpenFace is an open-source facial behavior analysis toolkit, primarily used for data preprocessing in this embodiment. It uses keypoint localization and affine transformation techniques to correct faces in video data into standardized, aligned images; simultaneously, it utilizes the intensity quantization function of facial action units (AUs) to calculate emotion intensity curves to accurately locate peak emotion frames.

[0067] Process raw video clips using OpenFace Identify peak emotion frame index and from The aligned face images are extracted and saved, forming a file sequence containing multiple frames of aligned face images. A segmented uniform sampling strategy is used to select 16 frames of aligned face images from the file sequence and store them in a file path. China. (The last part appears to be a fragment and doesn't translate directly.) and Stored in the data annotation file.

[0068] (2) Temporal dynamic feature extraction.

[0069] I. Input loading and preprocessing.

[0070] Load file path from data annotation file The 16 frames of aligned face images form an image sequence. .

[0071] Using the processor that comes with the VideoMAE V2 model The following preprocessing is performed: each frame is resized to 224x224 pixels and pixel normalized to obtain the video frame tensor used as input to the VideoMAE V2 model. This preprocessing is achieved through the following formula (5).

[0072] (5)

[0073] in, It is the processor that comes with the VideoMAE V2 model; Load file path from data annotation file The image sequence is formed by aligning 16 frames of face images; It is the video frame tensor input to the VideoMAE V2 model.

[0074] II. Temporal coding and feature output.

[0075] First, The data is fed into the pre-trained VideoMAE V2 model and embedded through the spatiotemporal block embedding module. The input is divided into spatial and temporal dimensions. N The spatiotemporal blocks are non-overlapping, and each spatiotemporal block is then flattened and mapped to a 1024-dimensional embedding vector through a shared linear layer, forming a spatiotemporal block embedding sequence. .Should The expression is shown in formula (6).

[0076] (6)

[0077] in, It is a spatiotemporal block embedding sequence; It is in the pre-trained VideoMAE V2 model .

[0078] Then, using the following formula (7), in Add a learnable 1024-dimensional element at the beginning. Embedding yields a new sequence; add the corresponding spatiotemporal location code to the new sequence. This yields the first input sequence that is ultimately fed into the Transformer encoder. .

[0079] (7)

[0080] in, This is the first input sequence of the Transformer encoder; For feature splicing operations, Special classification tags used for aggregating visual semantics of images or videos; It is used for spatiotemporal location encoding.

[0081] Will Input into the pre-trained ViT-Large encoder of the VideoMAE V2 model, starting from the last layer of the encoder. Extract from the output hidden state sequence The corresponding embedding vector serves as the temporal dynamic feature vector. .

[0082] (3) Global context feature extraction.

[0083] I. Input loading and preprocessing.

[0084] The peak sentiment frame index is loaded from the data annotation file using the following formula (8). This allows the loading of the corresponding original peak emotion frames. .

[0085] (8)

[0086] in, This is the original peak sentiment frame; Indicates a loading operation; This is the index for peak sentiment frames.

[0087] The peak frame was processed using the processor that accompanies the DINOv2 model: its size was resized to 518x518 pixels and pixel normalization was performed to obtain the image tensor used as input to the DINOv2 model. This process is achieved through the following formula (9).

[0088] (9)

[0089] in, For image tensors; A processor designed for the DINOv2 model.

[0090] II. Global encoding and feature output.

[0091] The preprocessed result The image is fed into the pre-trained DINOv2 model and processed by the image patch embedding module based on the following formula (10). The input is divided into spatial dimensions. N The image patches are then divided into three non-overlapping blocks. Each block is then flattened and mapped to a 1024-dimensional embedding vector through a shared linear layer, forming the first image patch embedding sequence. .

[0092] (10)

[0093] in, Embed the sequence for the first image patch; This is the image patch embedding module in the pre-trained DINOv2 model.

[0094] Based on the following formula (11), Start by adding a learnable 1024-dimensional element. The embedding yields a new sequence. The corresponding first spatial location code is then added to the new sequence. This yields the second input sequence that is ultimately fed into the Transformer encoder. .

[0095] (11)

[0096] in, This is the second input sequence of the Transformer encoder; Encodes the first spatial location.

[0097] Will Input into the DINOv2 pre-trained ViT-Large encoder, starting from the last layer of the encoder. Extract from the output hidden state sequence The corresponding embedding vector serves as the global context feature vector. .

[0098] (4) Extraction of local detail features.

[0099] I. Input loading and preprocessing.

[0100] Load file path from data annotation file The 16 frames of aligned face images in the image are based on the following formula (12), for the first frame... i (0≤) i ≤15) frames of images Preprocessing was performed independently using the processor that comes with the DINOv2 model: the image was resized to 224x224 pixels and pixel normalized to obtain the image tensor used as input to the DINOv2 model. .

[0101] (12)

[0102] in, The image tensor is input to the DINOv2 model. Align the first of 16 frames of face images Frame image.

[0103] II. Local coding and feature output.

[0104] The 16 image tensors obtained after preprocessing The images are fed into the pre-trained DINOv2 model, and based on the following formula (13), they are processed through the image patch embedding module. The input is divided into spatial dimensions. N Each image patch is then flattened and mapped to a 1024-dimensional embedding vector through a shared linear layer, forming the second image patch embedding sequence. .

[0105] (13)

[0106] in, The second image block is embedded with a sequence.

[0107] Based on the following formula (14), Add the corresponding second spatial location code This yields the third input sequence that is ultimately fed into the Transformer encoder. .

[0108] (14)

[0109] in, This is the third input sequence of the Transformer encoder; Encodes the second spatial location.

[0110] Will Input into the DINOv2 pre-trained ViT-Large encoder, and process the last layer of the encoder. The embedding vectors corresponding to all blocks in the output hidden state sequence are spatially averaged to obtain frame-level features. After processing all frames and obtaining 16 frame-level feature vectors, these 16 frame-level feature vectors are temporally averaged to obtain the final local detail features. .

[0111] The above describes an example of feature extraction from training samples.

[0112] The multimodal features include audio feature vectors, temporal dynamic feature vectors, global context feature vectors, and local detail feature vectors.

[0113] It should be noted that different modal features generate noise confidence scores through corresponding DNEs. For example, if there are 4 modal features, there are 4 corresponding DNEs.

[0114] In some embodiments, the step of inputting the extracted multimodal feature vectors into the dynamic noise estimator (DNE) of the corresponding modal feature of the machine learning model to generate a noise confidence score for the corresponding modal feature includes:

[0115] Based on the DNE of the corresponding modal features of the machine learning model, feature compression and activation, probability mapping and normalization are performed on the vectors of the corresponding modal features in the extracted multimodal feature vectors to generate noise confidence scores of the corresponding modal features.

[0116] In some embodiments, the feature compression and activation are achieved by the following formula (15):

[0117] (15)

[0118] Among them, the For corresponding modal features The complete input feature vector; This is the learnable weight matrix for the first stage, where the first stage is the training stage using coarse-grained samples; The bias vector for the first stage; The activation function; To The latent space feature vector is obtained by performing feature compression and activation.

[0119] The probability mapping and normalization are achieved through the following formula (16):

[0120] (16)

[0121] Among them, the This is the learnable weight matrix for the second stage, which is a fine-tuning stage using fine-grained samples; The bias scalar for the second stage; The normalization function; This represents the noise confidence score.

[0122] Step 104: Calculate the regularization coefficient corresponding to the noise confidence score for each feature dimension.

[0123] Regularization coefficient It can be calculated using the following formula (17):

[0124] (17)

[0125] in, For corresponding modal features The regularization coefficient; The basic regularization coefficient; This is the noise sensitivity adjustment coefficient. Both are adjustable hyperparameters and can be adjusted according to the noise distribution in the actual application scenario.

[0126] Furthermore, in some embodiments, NIB-KAN is also used to process input features such as... Mapped to output token sequence .

[0127] In some embodiments, during forward propagation in the training phase of the machine learning model, the method further includes:

[0128] Based on the NIB-KAN, modality alignment is performed on the multimodal features, and corresponding word sequences are generated;

[0129] The word sequence and the received text instruction vector are concatenated to obtain a multimodal input sequence;

[0130] The sentiment analyzer based on the machine learning model generates a second sentiment analysis result corresponding to the multimodal input sequence.

[0131] for The Each dimension, its value The result is obtained by superposition calculation of univariate nonlinear functions, that is, based on the NIB-KAN, the multimodal features are modally aligned and the corresponding word sequence is generated, which is achieved by the following formula (18):

[0132] (18)

[0133] in, The word sequence in the output is the first... Feature values ​​of each word element; For the first The total feature dimension of each modality; Feature dimension index for multimodal features; For the first The modal feature of the first Feature values ​​of each feature dimension; For the first In the modality, the first The feature dimension to the first The mapping function based on B-spline function for each word is expressed as shown in formula (19):

[0134] (19)

[0135] in, For the first A B-spline basis function in Output value at; For the first The modality, the first The feature dimension, the first Learnable scaling weights for each word unit; For the first The modality, the first The feature dimension, the first The word element, the first Learnable control coefficients corresponding to each B-spline basis function; The number of B-spline meshes; It is the Sigmoid linear unit activation function.

[0136] It should be noted that by introducing B-spline functions for modal alignment, the machine learning model can learn different response slopes for different intervals of the input value range. When the input features... When the signal falls into a noisy region, the machine learning model can flatten the function curve (the gradient approaches 0) by adjusting the coefficients, thereby physically blocking the propagation of the noise signal.

[0137] It's important to note that when video or audio data is affected by physical noise, the feature space undergoes a non-linear distribution shift. This is because the upstream feature extraction network contains numerous non-linear activation functions. In this case, the noise is not simply added to the features. The noise and signal undergo a complex transformation as they pass through layers of non-linear functions—that is, F(signal + noise) ≠ F(signal) + F(noise).

[0138] Traditional linear alignment layers cannot handle nonlinear noise interference. This is due to the limitations of their mathematical definition; they cannot correct for such nonlinear distribution shifts, resulting in noise features being preserved and passed on to subsequent processing stages. Specifically, the operation of a linear alignment layer is essentially an affine transformation (matrix multiplication plus bias). Its geometric effect is limited to global rotation, scaling, and translation of the feature space. In other words, a linear alignment layer can only perform globally consistent linear transformations on the input features. For nonlinear distortions (such as curve deformation) occurring in the feature space, the linear alignment layer cannot construct a corresponding inverse nonlinear transformation function for fitting or restoration. Because the linear alignment layer lacks the ability to change the curvature of the feature manifold, it cannot map the distorted noise distribution back to the normal signal distribution. Therefore, noise features are preserved intact and passed on to subsequent stages.

[0139] It should be emphasized that this embodiment, by introducing B-spline basis functions, can overcome the defects caused by the linear alignment layer, that is, it can correct the nonlinear distribution offset and prevent noise features from being retained and passed on to subsequent processing stages.

[0140] First, the noisy feature F(signal + noise) is restored to the ideal clean feature F(signal). A high-degree-of-freedom nonlinear mapping function, constructed using B-splines from NIB-KAN, replaces the traditional linear mapping. This function can accurately fit the complex curve transformation relationships in the feature space, thereby realigning features that have undergone nonlinear shifts to the correct data distribution.

[0141] Secondly, noise propagation is prevented: The NIB mechanism is used to calculate regularization coefficients via a dynamic noise estimator. When noise offset in certain feature channels is detected to be too severe and cannot be effectively corrected, the NIB mechanism forces the scaling weights (w) and spline control coefficients (c) corresponding to that feature channel to zero during the training phase using a loss function. Zeroing these parameters ensures the output of that feature channel remains constant at 0, thus completely severing the path of noise signal propagation to subsequent processing stages at the physical level.

[0142] B-spline basis functions possess piecewise fitting capabilities. They divide the input range into multiple small intervals, each with independent control coefficients. This means machine learning models can learn different processing strategies for different input ranges. Theoretically, through weighted combinations, B-spline basis functions can approximate arbitrarily complex continuous functions. Therefore, they can construct accurate nonlinear inverse transform relationships, thereby remapping the shifted feature space onto the target distribution.

[0143] Step 106: Calculate the loss value of the loss function of the machine learning model based on the regularization coefficient.

[0144] Step 108: During backpropagation in the training phase, update the weights and spline control coefficients of the NIB-KAN for the corresponding modal features of the machine learning model based on the loss value; wherein, the NIB-KAN is used for modal alignment of multimodal features, and the updated weights and updated spline control coefficients are used for the next round of training of the machine learning model.

[0145] Step 110: After completing the training of the machine learning model, in response to receiving the video to be analyzed, a first sentiment analysis result of the video to be analyzed is generated based on the feature encoder and NIB-KAN of the trained machine learning model and the sentiment analyzer.

[0146] In some embodiments, calculating the loss value of the loss function of the machine learning model based on the regularization coefficient includes:

[0147] The structured sparsity loss is calculated based on the regularization coefficient; the structured sparsity loss is used to adjust the structured sparsity loss using the regularization coefficient. and Apply L1 regularization constraints.

[0148] In some embodiments, the loss value calculated based on the regularization coefficient is... This can be achieved through the following formula (20):

[0149] (20)

[0150] in, This represents the loss value for structured sparse loss; For corresponding modal features The regularization coefficient; This indicates the calculation of the L1 norm.

[0151] In this embodiment, NIB-KAN and DNE are deeply coupled through an adaptive structured sparse loss function as shown in Equation (20) during training. The noise confidence score output by DNE is converted into a dynamic regularization coefficient and introduced as a weight term into the penalty terms of the scaling weight (w) and spline control coefficient (c) in the mapping function of NIB-KAN. This mechanism establishes a gradient-based mathematical constraint between the two: when DNE detects that the noise level of a certain feature channel is too high, it will significantly increase the regularization penalty corresponding to that feature channel. This forces the optimization algorithm to prioritize reducing the parameter value of the high-noise channel in the process of minimizing the total loss, thereby transforming the physical noise interference into a mathematical optimization constraint.

[0152] During the parameter update phase of backpropagation, NIB-KAN adaptively adjusts its internal mapping function parameters in response to this dynamic penalty. For channels marked as high-noise by DNE, the significant regularization pressure forces the scaling weights (w) and spline control coefficients (c) in NIB-KAN's mapping function to converge and become zero. This parameter zeroing means the channel's output is constantly set to zero, thus physically severing the transmission path of noise features to subsequent networks, achieving the filtering function of a "noise-aware information bottleneck." For clean channels, the parameters are preserved, and the spline function is used to correct nonlinear offsets.

[0153] It should be noted that through this collaborative training, the final machine learning model does not require DNE to participate in inference in real time. It can achieve non-linear alignment of features and noise blocking (structured pruning) based solely on the parameter structure solidified by training.

[0154] It should be noted that the training process of the machine learning model in this embodiment is modeled as a multi-objective optimization problem, and the total loss value of the total loss function of this multi-objective optimization problem is... It is defined as follows (21).

[0155] (twenty one)

[0156] in, For hyperparameters, For semantic consistency loss, Predict losses for the task; This represents the total loss value.

[0157] in, It is calculated using the following formula (22).

[0158] (twenty two)

[0159] in, Specifically, a comparison strategy is used to calculate the original input features. Inject random Gaussian noise to generate perturbed samples .Will and Modal alignment is performed separately to obtain the token sequence. and Formula (22) calculates... and Euclidean distance in the embedded space.

[0160] It should be noted that this loss is used to penalize mapping functions that cause drastic changes in output due to small perturbations in the input, forcing the model to learn modal invariance.

[0161] in, It is calculated using the following formula (23).

[0162] (twenty three)

[0163] in, This represents the task prediction loss value of the model; The total length of the target sentiment text sequence. This is the time step index for the sequence. Indicates in model parameters Given a multimodal unified input sequence and the Historical texts before the march At that time, the model for the first Step-by-step real text markup The loss function is calculated by taking the natural logarithm of the conditional predicted probability for each time step of the sequence and summing the results for the entire sequence. Adding a negative sign transforms maximizing the true predicted probability into minimizing the loss objective, which is optimizable by gradient descent. Finally, it is divided by the total sequence length. After average normalization, a stable task prediction loss is finally obtained to guide the updating of model parameters.

[0164] This embodiment first employs NIB-KAN as a modality alignment layer. Utilizing the nonlinear mapping constructed by KAN's B-spline basis functions, it can fit the complex distribution shifts in the feature space caused by noise, realigning noisy features to a clean feature distribution. This overcomes the limitation of linear projection layers in handling nonlinear distortions, thus preventing noisy features from being directly propagated downstream. Secondly, DNE generates noise confidence scores during training, and adaptive regularization is applied to the weights and spline control coefficients of NIB-KAN based on these scores. During backpropagation, this forces the parameters of channels corresponding to high-noise feature dimensions to zero, thereby solidifying the noise blocking capability after training. This achieves fine-grained screening of noisy channels, preventing high-noise features from interfering with subsequent inference. Furthermore, NIB-KAN allows for the repair of some noisy features before blocking high-noise features, enabling the machine learning model to adapt to different modal signal quality. This allows the machine learning model to accurately suppress noise while preserving effective semantic information. Finally, during the inference stage of the machine learning model, highly robust sentiment analysis can be achieved solely based on the solidified parameters, significantly improving performance and stability in real-world noisy scenarios.

[0165] It is important to emphasize that, in this embodiment, after the upstream nonlinear feature extractor outputs noisy features, these noisy features are transmitted in parallel to NIB-KAN and DNE. At this stage, NIB-KAN, as the primary feature processing unit, uses B-spline basis functions to construct a nonlinear mapping to prepare for feature alignment; while DNE, as a parallel sensing unit, calculates a quantified noise confidence score by analyzing the distribution statistics of the input features. This noise confidence score accurately reflects the severity of physical noise contamination in each feature channel. It is crucial to emphasize that DNE does not directly modify feature values, but rather provides guidance signals for subsequent optimization processes by outputting a noise score.

[0166] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0167] For ease of understanding, a specific embodiment will be used as an example:

[0168] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a multimodal sentiment analysis system based on dynamic noise estimation and KAN provided in some embodiments of this application. In this embodiment, the multimodal sentiment analysis system based on dynamic noise estimation and KAN includes an audio feature extraction unit, a temporal feature extraction unit, a global and local feature extraction unit, a dynamic noise estimator, a NIB-KAN projector, and a large language model.

[0169] The audio feature extraction unit uses the WavLM model, the temporal feature extraction unit uses the VideoMAE V2 model, and the global and local feature extraction units use the DINOv2 model. The outputs of the audio feature extraction unit, temporal feature extraction unit, and global and local feature extraction units are all connected to the inputs of the dynamic noise estimator and the NIB-KAN projector. The token sequence output from the four parallel NIB-KAN projectors is concatenated with the text instruction vector to form the input sequence, which is then connected to the input of the large language model. The large language model uses the Qwen2.5-7B-Instruct model.

[0170] It should be noted that, Figure 2 In this model, different colored "circles" represent feature vectors of different modalities. "Squares" represent text tokens, with "pink" representing visual feature tokens, "magenta" representing audio feature tokens, and "brown" representing instruction text tokens. During the concatenation process, the two types of tokens are replaced by the corresponding visual and audio feature vectors, forming a unified multimodal sequence for input into a large language model.

[0171] See Figure 3 , Figure 3 This is a schematic diagram illustrating the connection principle between the NIB-KAN projector and the dynamic noise estimator provided in some embodiments of this application. The specific connection principle between the NIB-KAN projector and the dynamic noise estimator is as follows:

[0172] The dynamic noise estimator and the NIB-KAN projector are configured in parallel. The feature vector is simultaneously input to both the dynamic noise estimator and the NIB-KAN projector. The output of the dynamic noise estimator is connected to the control parameter interface of the NIB-KAN projector.

[0173] See Figure 4 , Figure 4 This is one of the flowcharts illustrating a multimodal sentiment analysis method based on dynamic noise estimation and KAN provided in some embodiments of this application. The method includes the following steps:

[0174] ① The audio feature extraction unit, the temporal feature extraction unit, and the global and local feature extraction units extract features from the input video data and output four feature vectors.

[0175] ② The dynamic noise estimator receives the feature vector, calculates the noise control signal, and transmits the signal to the NIB-KAN projector.

[0176] ③ The NIB-KAN projector receives the feature vector and the noise control signal from the dynamic noise estimator as a regularization parameter (i.e., regularization coefficient). It prunes the internal parameters according to the regularization parameter and outputs the aligned token vector (i.e., token sequence).

[0177] ④ The token sequences output by the four NIB-KAN projectors are concatenated with the text instruction vector to form a multimodal input sequence.

[0178] ⑤ The large language model receives the multimodal input sequence, performs inference, and generates a second sentiment analysis result.

[0179] It should be noted that, to ensure the feasibility of this method, the specific training strategy, training dataset, and parameter settings are as follows:

[0180] Training strategy:

[0181] This embodiment employs a two-stage training paradigm, combined with a dynamic dual-task sampling strategy (i.e., sampling emotion recognition instructions and emotion inference instructions with equal probability) and noise enhancement:

[0182] The first stage uses coarse-grained samples for training. This stage utilizes only basic modal descriptions and sentiment labels, aiming to allow the model to learn the mapping from multimodal features to basic semantic representations and establish basic sentiment recognition capabilities.

[0183] The second stage involves fine-tuning using fine-grained samples. This stage introduces high-quality reasoning text containing causal logical chains and trains the model under continuous online noise enhancement. In this stage, the model evolves from simple cue listing to true logical integration, learning to handle audiovisual conflicts and perform causal reasoning from subtle cues.

[0184] To accurately simulate the specific interference faced by different modalities in real-world scenarios, differentiated online noise enhancement strategies were designed for the four defined feature perspectives. The specific noise enhancement parameters are shown in the parameter settings.

[0185] Training dataset:

[0186] The training data in this embodiment is primarily constructed based on the Multimodal Emotion Recognition and Reasoning (MERR) dataset. This dataset contains 28,618 coarse-grained samples and 4,487 fine-grained samples, covering eight core emotion categories.

[0187] To compensate for the lack of a "Disgust" category in the original dataset, this embodiment additionally supplements the coarse-grained set with 350 samples selected from the Multimodal Emotion Lines Dataset (MELD) and the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP), and the fine-grained set with 50 samples. All new samples were strictly labeled according to the MERR standard protocol, ultimately constructing an enhanced training set that strengthens the "Disgust" category and covers 9 emotion categories, containing a total of 28,968 coarse-grained samples and 4,537 fine-grained samples.

[0188] Parameter settings:

[0189] Network structure parameters: The latent space dimension in the dynamic noise estimator is set to 64. The feature extraction uses WavLM-Large as the audio encoder and VideoMAE V2 and DINOv2-Large / 14 as the visual encoder.

[0190] Training hyperparameters: The AdamW optimizer was used throughout training, with weight decay set to 0.01, and dynamically adjusted using a cosine annealing learning rate strategy with a warm-up step of 500. For different training phases, the first phase had a global batch size of 32, an initial learning rate of 5e-5, a minimum learning rate decay to 1e-6, and a training duration of 3 epochs; the second phase had a global batch size of 16, an initial learning rate of 1e-5, a minimum learning rate of 1e-7, and a training duration of 2 epochs.

[0191] Large model fine-tuning parameters: The base model Qwen2.5-7B-Instruct adopts the LoRA (Low-Rank Adaptation) fine-tuning strategy, in which the rank and scaling factor (alpha) are both set to 64, and the dropout is set to 0.05.

[0192] Joint optimization weights: In the joint optimization objective, the task prediction loss weight is set to 1.0, the structured sparsity loss weight is set to 0.1 to avoid over-pruning, and the semantic consistency loss weight is set to 0.5 to ensure the stability of the feature space.

[0193] Noise enhancement parameters: An online injection strategy is used during training, with each viewpoint of each sample triggered independently with a 40% probability. Additive white Gaussian noise with an intensity of 0.1 is applied to the audio modality. Frame-level and feature-level masks with a scale of 0.5 are applied to the temporal visual modality and the local detail modality, respectively. Gaussian noise with an intensity of 0.03 or Gaussian blur with an intensity of 0.3 is randomly applied to the global visual modality.

[0194] Based on the same inventive concept, this application also provides a multimodal sentiment analysis system based on dynamic noise estimation and KAN for implementing the aforementioned multimodal sentiment analysis method based on dynamic noise estimation and KAN. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal sentiment analysis system based on dynamic noise estimation and KAN provided below can be found in the limitations of the multimodal sentiment analysis method based on dynamic noise estimation and KAN described above, and will not be repeated here.

[0195] In one exemplary embodiment, such as Figure 5As shown, a multimodal sentiment analysis system based on dynamic noise estimation and KAN is provided, including: a first generation module 100, a first calculation module 200, a second calculation module 300, an update module 400, and a second generation module 500, wherein:

[0196] The first generation module 100 is used to, during the forward propagation of the training phase of the machine learning model, after extracting features from the training samples based on the feature encoder of the machine learning model, input the extracted multimodal feature vectors into the dynamic noise estimator (DNE) of the corresponding modal features of the machine learning model to generate noise confidence scores for the corresponding modal features.

[0197] The first calculation module 200 is used to calculate the regularization coefficient corresponding to the noise confidence score for each feature dimension.

[0198] The second calculation module 300 is used to calculate the loss value of the loss function of the machine learning model based on the regularization coefficient;

[0199] The update module 400 is used to update the weights and spline control coefficients of the corresponding modal features of the machine learning model based on the loss value during backpropagation in the training phase; wherein, the NIB-KAN is used to perform modal alignment of multimodal features, and the updated weights and updated spline control coefficients are used for the next round of training of the machine learning model.

[0200] The second generation module 500 is configured to, after completing the training of the machine learning model, generate a first sentiment analysis result of the video to be analyzed in response to receiving the video to be analyzed, based on the feature encoder and NIB-KAN of the trained machine learning model and the sentiment analyzer.

[0201] In some embodiments, the first generation module 100 is specifically used for:

[0202] Based on the DNE of the corresponding modal features of the machine learning model, feature compression and activation, probability mapping and normalization are performed on the vectors of the corresponding modal features in the extracted multimodal feature vectors to generate noise confidence scores of the corresponding modal features.

[0203] In some embodiments, the feature compression and activation are achieved by the following formula:

[0204]

[0205] Among them, the For corresponding modal features The complete input feature vector; This is the learnable weight matrix for the first stage, where the first stage is the training stage using coarse-grained samples; The bias vector for the first stage; The activation function; To The latent space feature vector obtained by performing feature compression and activation;

[0206] The probability mapping and normalization are achieved through the following formula:

[0207]

[0208] Among them, the This is the learnable weight matrix for the second stage, which is a fine-tuning stage using fine-grained samples; The bias scalar for the second stage; The normalization function; This represents the noise confidence score.

[0209] In some embodiments, the system further includes:

[0210] The third generation module is used to perform modal alignment on the multimodal features based on the NIB-KAN and generate corresponding word sequences;

[0211] The concatenation module is used to concatenate the word sequence and the received text instruction vector to obtain a multimodal input sequence;

[0212] The fourth generation module is used to generate a second sentiment analysis result corresponding to the multimodal input sequence based on the sentiment analyzer of the machine learning model.

[0213] In some embodiments, the third generation module is implemented using the following formula:

[0214]

[0215] in, The word sequence in the output is the first... Feature values ​​of each word element; For the first The total feature dimension of each modality; Feature dimension index for multimodal features; For the first The modal feature of the first Feature values ​​of each feature dimension; For the first In the modality, the first The feature dimension to the first The B-spline-based mapping function for each word is expressed as follows:

[0216]

[0217] in, For the first A B-spline basis function in Output value at; For the first The modality, the first The feature dimension, the first Learnable scaling weights for each word unit; For the first The modality, the first The feature dimension, the first The word element, the first Learnable control coefficients corresponding to each B-spline basis function; The number of B-spline meshes; It is the Sigmoid linear unit activation function.

[0218] In some embodiments, the second computing module 300 is specifically used for:

[0219] The structured sparsity loss is calculated based on the regularization coefficient; the structured sparsity loss is used to adjust the structured sparsity loss using the regularization coefficient. and Apply L1 regularization constraints.

[0220] In some embodiments, the second calculation module 300 is implemented using the following formula:

[0221]

[0222] Among them, the For corresponding modal features The regularization coefficient; This indicates the calculation of the L1 norm.

[0223] In some embodiments, the DNE is a multilayer perceptron (MLP) containing two fully connected layers.

[0224] The modules in the aforementioned multimodal sentiment analysis system based on dynamic noise estimation and KAN can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, allowing the processor to call and execute the corresponding operations of each module.

[0225] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal sentiment analysis method based on dynamic noise estimation and KAN.

[0226] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0227] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0228] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0229] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multimodal sentiment analysis method based on dynamic noise estimation and KAN, characterized in that, The multimodal sentiment analysis method based on dynamic noise estimation and KAN includes: During the forward propagation of the training phase of the machine learning model, after the feature encoder based on the machine learning model extracts features from the training samples, the extracted multimodal feature vectors are input into the dynamic noise estimator (DNE) of the corresponding modal features of the machine learning model to generate noise confidence scores for the corresponding modal features. Calculate the regularization coefficient corresponding to the noise confidence score for each feature dimension; The loss value of the loss function of the machine learning model is calculated based on the regularization coefficient; During backpropagation in the training phase, the weights and spline control coefficients of the Noise-Aware Information Bottleneck – Kolmogorov-Arnold Network (NIB-KAN) for the corresponding modal features of the machine learning model are updated based on the loss value; wherein, the NIB-KAN is used to perform modal alignment of multimodal features, and the updated weights and updated spline control coefficients are used for the next round of training of the machine learning model. After the machine learning model is trained, in response to receiving the video to be analyzed, a first sentiment analysis result of the video to be analyzed is generated based on the feature encoder and NIB-KAN of the trained machine learning model and the sentiment analyzer. During the forward propagation phase of the machine learning model's training, the method further includes: Based on the NIB-KAN, modality alignment is performed on the multimodal features, and corresponding word sequences are generated; The word sequence and the received text instruction vector are concatenated to obtain a multimodal input sequence; Based on the sentiment analyzer of the machine learning model, a second sentiment analysis result corresponding to the multimodal input sequence is generated; Based on the NIB-KAN, modal alignment is performed on the multimodal features, and corresponding word sequences are generated, which is achieved through the following formula: in, The word sequence in the output is the first... Feature values ​​of each word element; For the first The total feature dimension of each modality; Feature dimension index for multimodal features; For the first The modal feature of the first Feature values ​​of each feature dimension; For the first In the modality, the first The feature dimension to the first The B-spline-based mapping function for each word is expressed as follows: in, For the first A B-spline basis function in Output value at; For the first The modality, the first The feature dimension, the first Learnable scaling weights for each word element; For the first The modality, the first The feature dimension, the first The word element, the first Learnable control coefficients corresponding to each B-spline basis function; The number of B-spline meshes; It is the Sigmoid linear unit activation function; The calculation of the loss value of the loss function of the machine learning model based on the regularization coefficient includes: The structured sparsity loss is calculated based on the regularization coefficient; the structured sparsity loss is used to adjust the structured sparsity loss using the regularization coefficient. and Apply L1 regularization constraints; The loss value calculated based on the regularization coefficient for structured sparse loss. This can be achieved through the following formula: Among them, the For corresponding modal features The regularization coefficient; This indicates the calculation of the L1 norm.

2. The multimodal sentiment analysis method based on dynamic noise estimation and KAN according to claim 1, characterized in that, The step of inputting the extracted multimodal feature vectors into the dynamic noise estimator (DNE) of the corresponding modal feature of the machine learning model to generate noise confidence scores for the corresponding modal features includes: Based on the DNE of the corresponding modal features of the machine learning model, feature compression and activation, probability mapping and normalization are performed on the vectors of the corresponding modal features in the extracted multimodal feature vectors to generate noise confidence scores of the corresponding modal features.

3. The multimodal sentiment analysis method based on dynamic noise estimation and KAN according to claim 2, characterized in that, The feature compression and activation are achieved through the following formula: Among them, the For corresponding modal features The complete input feature vector; This is the learnable weight matrix for the first stage, where the first stage is the training stage using coarse-grained samples; The bias vector for the first stage; The activation function; To The latent space feature vector obtained by performing feature compression and activation; The probability mapping and normalization are achieved through the following formula: Among them, the This is the learnable weight matrix for the second stage, which is a fine-tuning stage using fine-grained samples; The bias scalar for the second stage; The normalization function; This represents the noise confidence score.

4. The multimodal sentiment analysis method based on dynamic noise estimation and KAN according to claim 1, characterized in that, The DNE is a multilayer perceptron (MLP) containing two fully connected layers.

5. A multimodal sentiment analysis system based on dynamic noise estimation and KAN, characterized in that, The multimodal sentiment analysis system based on dynamic noise estimation and KAN includes: The first generation module is used to generate noise confidence scores for the corresponding modal features by inputting the extracted multimodal feature vectors into the dynamic noise estimator (DNE) of the corresponding modal features of the machine learning model after feature extraction of the training samples based on the feature encoder of the machine learning model during the forward propagation of the training phase. The first calculation module is used to calculate the regularization coefficient corresponding to the noise confidence score for each feature dimension. The second calculation module is used to calculate the loss value of the loss function of the machine learning model based on the regularization coefficient; An update module is used to update the weights and spline control coefficients of the Noise-Aware Information Bottleneck-Kolmogorov-Arnold Network (NIB-KAN) of the corresponding modal features of the machine learning model based on the loss value during backpropagation in the training phase; wherein, the NIB-KAN is used to perform modal alignment of the multimodal features, and the updated weights and updated spline control coefficients are used for the next round of training of the machine learning model; The second generation module is used to generate a first sentiment analysis result of the video to be analyzed based on the feature encoder and NIB-KAN of the trained machine learning model and the sentiment analyzer after the training of the machine learning model is completed, in response to receiving the video to be analyzed. The system also includes: The third generation module is used to perform modal alignment on the multimodal features based on the NIB-KAN and generate corresponding word sequences; The concatenation module is used to concatenate the word sequence and the received text instruction vector to obtain a multimodal input sequence; The fourth generation module is used to generate a second sentiment analysis result corresponding to the multimodal input sequence based on the sentiment analyzer of the machine learning model. The third generation module is implemented using the following formula: in, The word sequence in the output is the first... Feature values ​​of each word element; For the first The total feature dimension of each modality; Feature dimension index for multimodal features; For the first The modal feature of the first Feature values ​​of each feature dimension; For the first In the modality, the first The feature dimension to the first The B-spline-based mapping function for each word is expressed as follows: in, For the first A B-spline basis function in Output value at; For the first The modality, the first The feature dimension, the first Learnable scaling weights for each word element; For the first The modality, the first The feature dimension, the first The word element, the first Learnable control coefficients corresponding to each B-spline basis function; The number of B-spline meshes; It is the Sigmoid linear unit activation function; The second calculation module is specifically used for: The structured sparsity loss is calculated based on the regularization coefficient; the structured sparsity loss is used to adjust the structured sparsity loss using the regularization coefficient. and Apply L1 regularization constraints; The second calculation module is implemented using the following formula: Among them, the For corresponding modal features The regularization coefficient; This indicates the calculation of the L1 norm.

6. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the multimodal sentiment analysis method based on dynamic noise estimation and KAN as described in any one of claims 1-4.

Images