A multimodal sentiment classification method based on dynamic game strategy
By optimizing the multimodal sentiment classification method through dynamic game strategy, the problem of insufficient utilization of inter-modal synergy effects in traditional methods is solved, and a more efficient sentiment classification effect is achieved.
Patent Information
- Application Number
- CN202510928879.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-07
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2045-07-07
AI Technical Summary
Traditional multimodal sentiment classification methods fail to fully utilize the synergistic effect between modalities, resulting in insufficient information fusion and affecting the accuracy of sentiment classification. In addition, the prisoner's dilemma model is insufficiently applied in multimodal sentiment classification.
A multimodal sentiment classification method based on dynamic game strategy is adopted. Through feature extraction, redundancy elimination, tensor decomposition and high-order synergy effect compensation mechanism, feature weights and screening thresholds are dynamically adjusted, feature subsets are optimized, and finally input into the multimodal sentiment classification model for prediction.
It significantly improves the accuracy and performance of multimodal sentiment classification, can better adapt to the complexity of multimodal data, and improve the accuracy of sentiment classification.
Smart Images
Figure CN120429602B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of multimodal sentiment classification, and in particular to a multimodal sentiment classification method based on dynamic game strategy. Background Art
[0002] Traditional multimodal sentiment classification methods often fail to fully exploit intermodal synergies during feature selection, resulting in insufficient information fusion and, in turn, impacting sentiment classification accuracy. While traditional game models—the prisoner's dilemma and non-cooperative games—provide a theoretical foundation, their application to multimodal sentiment classification remains insufficient. The prisoner's dilemma model assumes a lack of cooperation between participants, with each modality making independent decisions. This makes it difficult to capture intermodal synergies in multimodal data. Furthermore, the heterogeneous nature of modal data makes effective modeling within a unified framework challenging. Summary of the Invention
[0003] Based on this, it is necessary to provide a multimodal sentiment classification method based on dynamic game strategy, which includes:
[0004] S1: Obtain multimodal data to be sentimentally classified, including images, text, and speech, and perform feature extraction to obtain multimodal features; perform redundancy elimination on the multimodal features to obtain a preliminary feature subset;
[0005] S2: In each round of the dynamic game strategy, the contribution of each feature in the preliminary feature subset is compensated through a high-order synergy compensation mechanism based on tensor decomposition, and the compensation contribution of each feature is obtained;
[0006] The weight of each feature is updated based on the compensation contribution of each feature to obtain the updated feature weight of each feature;
[0007] The preliminary feature subset is screened based on the updated feature weights to obtain the optimized feature subset;
[0008] S3: Perform multiple rounds of game until the convergence condition is reached and the final feature subset is obtained;
[0009] S4: Input the final feature subset into the multimodal sentiment classification model and output the predicted sentiment category.
[0010] Preferably, extracting features from multimodal data includes:
[0011] Input the image data into the ResNet pre-training model to obtain image modality features;
[0012] Input text data into the pre-trained BERT model to obtain text modality features;
[0013] Input the speech data into the Wav2Vec2.0 model to obtain speech modal features;
[0014] Combine multiple image modality features, text modality features, and speech modality features to obtain multimodal features.
[0015] Preferably, the redundancy elimination process for the multimodal features includes:
[0016] Calculate the covariance between any two features in the multimodal features and the standard deviation of each feature;
[0017] Calculate the product of the standard deviations of the two features to obtain the first product;
[0018] Divide the covariance between the two features by the first product to get the correlation between the two features;
[0019] Traverse all features in the multimodal features, calculate the correlation between all two features, and use them as elements of the matrix to construct the correlation matrix;
[0020] Compare the modulus of any element in the correlation matrix with the correlation threshold. When the modulus of the element is greater than the correlation threshold, compare the importance, average weight and variance between the two features in the element in turn.
[0021] Among them, the feature with less importance between the two features is eliminated;
[0022] When the importance of two features is the same, the feature with the smaller average weight between the two features will be eliminated;
[0023] If the average weights of the two features are still the same, the feature with the smaller variance between the two features will be eliminated;
[0024] The importance of a feature is obtained by adding the product of the initial weight of the feature and the balance coefficient, and the product of the updated feature weight of the feature in the game and the equilibrium number of the balance coefficient with respect to 1;
[0025] The average weight of the feature is obtained by averaging the feature weights updated in multiple rounds of the game;
[0026] All elements in the correlation matrix are traversed, and redundant multimodal features are eliminated. The obtained multimodal features after redundant elimination are the preliminary feature subset.
[0027] Preferably, the method further includes dynamically adjusting the correlation threshold according to the off-diagonal elements of the correlation matrix:
[0028] Calculate the mean and standard deviation of the off-diagonal elements in the correlation matrix;
[0029] Multiplying the standard deviation of the off-diagonal elements by a preset adjustment parameter to obtain a second product;
[0030] The second product is added to the mean of the off-diagonal elements, and the correlation threshold is dynamically updated using the addition result.
[0031] Preferably, compensating the contribution of each feature in the preliminary feature subset by a high-order synergy effect compensation mechanism based on tensor decomposition includes:
[0032] Represent the preliminary feature subset as a three-dimensional tensor, and perform CP decomposition on the three-dimensional tensor to decompose the weight of each interaction mode;
[0033] For any feature in the preliminary feature subset,
[0034] Calculate the product of the weight of the interaction mode in any interaction mode and the participation of the feature in the corresponding interaction mode, and sum the corresponding products of all interaction modes; multiply the sum by the synergy adjustment coefficient and add it to the basic contribution of the feature to obtain the compensation contribution corresponding to the feature;
[0035] Traverse all features in the preliminary feature subset to obtain the compensation contribution corresponding to each feature.
[0036] Preferably, the participation degree of a feature in the interaction mode is determined by whether the feature participates in the interaction mode. When the feature participates in the interaction mode, the participation degree of the feature in the interaction mode is 1; otherwise, the participation degree of the feature in the interaction mode is 0.
[0037] Preferably, updating the weight of each feature based on the compensation contribution of each feature includes:
[0038] For any feature in the preliminary feature subset,
[0039] Multiply the weight of the feature in the tth round of the game by the preset smoothing factor to obtain the third product;
[0040] Calculate the ratio of the compensation contribution corresponding to the feature to the compensation contribution of all features;
[0041] multiplying the ratio by a balance of the smoothing factor about 1 to obtain a fourth product;
[0042] Add the third product and the fourth product to get the weight of the feature in the t+1th round of the game;
[0043] Traverse all features in the preliminary feature subset and obtain the updated feature weights of each feature.
[0044] Preferably, screening the preliminary feature subset based on the updated feature weights includes:
[0045] For any feature in the preliminary feature subset,
[0046] Compare the updated feature weight with the screening threshold. When the updated feature weight is greater than the screening threshold, retain the corresponding feature; otherwise, discard the corresponding feature.
[0047] Traverse all features in the preliminary feature subset and construct an optimized feature subset based on the retained features.
[0048] Preferably, dynamically adjusting the screening threshold includes:
[0049] Calculate the difference between the model performance and the target performance after the tth round of game;
[0050] Multiply the difference by the learning rate to get the fifth product;
[0051] Add the fifth product to the screening threshold in the t-th round of the game to obtain the screening threshold in the t+1-th round of the game.
[0052] Preferably, the convergence condition includes:
[0053] Or when the model performance in any round of the game is greater than or equal to the performance threshold, the iteration is stopped;
[0054] Or when the change in model performance in any round of the game is less than the change threshold, the iteration is stopped;
[0055] Or the game rounds reach the maximum preset number and the iteration stops.
[0056] Beneficial effects: This method adopts a dynamic game strategy that can better adapt to the complexity of multimodal data; through this strategy, the system can more effectively select features, thereby improving the accuracy of sentiment classification and significantly improving the performance of multimodal sentiment classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0058] Figure 1 This is a flowchart of a multimodal emotion classification method based on dynamic game strategy in an embodiment of the present application. DETAILED DESCRIPTION
[0059] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.
[0060] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.
[0061] like Figure 1 As shown, this embodiment provides a multimodal sentiment classification method based on a dynamic game strategy, the method comprising:
[0062] S1: Obtain multimodal data to be sentimentally classified, including images, text, and speech, and perform feature extraction to obtain multimodal features; perform redundancy elimination on the multimodal features to obtain a preliminary feature subset.
[0063] Specifically, feature extraction of multimodal data includes:
[0064] Input the image data into the ResNet pre-training model to obtain image modality features;
[0065] Input text data into the pre-trained BERT model to obtain text modality features;
[0066] Input the speech data into the Wav2Vec2.0 model to obtain speech modal features;
[0067] Combine multiple image modality features, text modality features, and speech modality features to obtain multimodal features.
[0068] Furthermore, the redundant elimination process for the multimodal features includes:
[0069] Calculate the covariance between any two features in the multimodal features and the standard deviation of each feature;
[0070] Calculate the product of the standard deviations of the two features to obtain the first product;
[0071] Divide the covariance between the two features by the first product to get the correlation between the two features;
[0072] Traverse all features in the multimodal features, calculate the correlation between all two features, and use them as elements of the matrix to construct the correlation matrix;
[0073] Compare the modulus of any element in the correlation matrix with the correlation threshold. When the modulus of the element is greater than the correlation threshold, compare the importance, average weight, and variance between the two features in the element in turn.
[0074] Among them, the feature with less importance between the two features is eliminated;
[0075] When the importance of two features is the same, the feature with the smaller average weight between the two features will be eliminated;
[0076] If the average weights of the two features are still the same, the feature with the smaller variance between the two features will be eliminated;
[0077] The importance of a feature is obtained by adding the product of the initial weight of the feature and the balance coefficient, and the product of the updated feature weight of the feature in the game and the equilibrium number of the balance coefficient with respect to 1;
[0078] The average weight of the feature is obtained by averaging the feature weights updated in multiple rounds of the game;
[0079] All elements in the correlation matrix are traversed, and redundant multimodal features are eliminated. The obtained multimodal features after redundant elimination are the preliminary feature subset.
[0080] In this embodiment, the initial weights of the image modality features are calculated based on the channel attention weights of the ResNet pre-trained model;
[0081] The initial weights of text modality features are calculated based on the L2 norm of the pre-trained BERT model;
[0082] The initial weights of speech modality features are calculated based on the frame-level energy statistics of the Wav2Vec2.0 model.
[0083] Furthermore, if the feature dimension needs to be kept consistent, after the removal, the removed position is filled with the mean of other features belonging to the same mode as the removed feature; and the filled features need to be re-updated in the next round of iteration.
[0084] In this embodiment, the correlation threshold is also dynamically adjusted according to the off-diagonal elements of the correlation matrix:
[0085] Calculate the mean and standard deviation of the off-diagonal elements in the correlation matrix;
[0086] Multiplying the standard deviation of the off-diagonal elements by a preset adjustment parameter to obtain a second product;
[0087] The second product is added to the mean of the off-diagonal elements, and the correlation threshold is dynamically updated using the addition result.
[0088] S2: In each round of the dynamic game strategy, the contribution of each feature in the preliminary feature subset is compensated by a high-order synergy compensation mechanism based on tensor decomposition to obtain the compensated contribution of each feature.
[0089] The composition of dynamic game strategy includes:
[0090] 1. Basic framework of the game model:
[0091] Participants: Each feature in the preliminary feature subset is considered as a participant in the game;
[0092] Strategy: The strategy for each feature is to retain the feature if its weight is greater than the screening threshold, otherwise discard the feature;
[0093] Reward function (replaced by compensation contribution in this embodiment): The reward function of the corresponding feature is constructed by calculating the difference between the performance of the sentiment classification model using a feature in the preliminary feature subset (the performance can be accuracy, F1 score, etc.) and the performance of the sentiment classification model using the preliminary feature subset.
[0094] 2. Update rules:
[0095] Update strategy based on reward function and dynamic adjustment of screening threshold.
[0096] 3. Algorithm process:
[0097] Initialization, initialization strategy, screening threshold and target performance;
[0098] Iterative game, for each round of the game: 1) calculate the reward function for each feature, 2) update the strategy, 3) generate a new feature subset based on the updated strategy.
[0099] Convergence conditions include:
[0100] Or when the model performance in any round of the game is greater than or equal to the performance threshold, the iteration is stopped;
[0101] Or when the change in model performance in any round of the game is less than the change threshold, the iteration is stopped;
[0102] Or the game rounds reach the maximum preset number and the iteration stops.
[0103] Specifically, compensating the contribution of each feature in the preliminary feature subset through the high-order synergy compensation mechanism based on tensor decomposition includes:
[0104] The preliminary feature subset is represented as a three-dimensional tensor, and the three-dimensional tensor is decomposed by CP to decompose the weight of each interaction mode and the factor vectors of image modality, text modality, and speech modality in each interaction mode;
[0105] For any feature in the preliminary feature subset,
[0106] Calculate the product of the interaction mode weight in any interaction mode and the feature's participation in the corresponding interaction mode, and sum the corresponding products of all interaction modes; multiply the sum by the synergy adjustment coefficient and add it to the basic contribution of the feature (determined by the independent performance of the feature) to obtain the corresponding compensation contribution of the feature;
[0107] Traverse all features in the preliminary feature subset to obtain the compensation contribution corresponding to each feature.
[0108] In this embodiment, the weights of each interaction mode and the factor vectors of the image modality, text modality, and speech modality in each interaction mode are used to quantify the synergistic effect of the interaction modes; this synergistic effect of the interaction modes is used to represent the global contribution of the corresponding interaction mode to the preliminary feature subset. The specific calculation process is to perform a tensor product operation on the factor vectors of the image modality, text modality, and speech modality in the same interaction mode, and multiply the result by the weight of the same interaction mode to obtain the synergistic effect of the corresponding interaction mode.
[0109] In this embodiment, the participation degree of a feature in the interaction mode is determined by whether the feature participates in the interaction mode. When the feature participates in the interaction mode, the participation degree of the feature in the interaction mode is 1; otherwise, the participation degree of the feature in the interaction mode is 0.
[0110] The weight of each feature is updated based on the compensation contribution of each feature to obtain the updated feature weight of each feature.
[0111] Specifically, updating the weight of each feature based on the compensation contribution of each feature includes:
[0112] For any feature in the preliminary feature subset,
[0113] Multiply the weight of the feature in the tth round of the game by the preset smoothing factor to obtain the third product;
[0114] Calculate the ratio of the compensation contribution corresponding to the feature to the compensation contribution of all features;
[0115] multiplying the ratio by a balance of the smoothing factor about 1 to obtain a fourth product;
[0116] Add the third product and the fourth product to get the weight of the feature in the t+1th round of the game;
[0117] Traverse all features in the preliminary feature subset and obtain the updated feature weights of each feature.
[0118] In this embodiment, an adaptive weighting mechanism based on emotional state is adopted for the features, including:
[0119] 1. Dynamic adjustment of modal weights:
[0120] Modal weight calculation, each mode (such as voice, text, image) in emotional state The weight under For: Modal In emotional state The ratio of the contribution of the modal to the contribution of all modalities in the emotional state.
[0121] Modal contribution calculation: Modal Contribution To: Use the include modal The performance of emotion recognition model using feature subsets of The difference between the performance of emotion recognition models based on the feature subsets.
[0122] 2. Weight update driven by emotional state:
[0123] Weight update, modal weight According to emotional state Dynamic Updates:
[0124] Modal weight Multiplying by a preset smoothing factor to obtain a sixth product;
[0125] Calculate the mode Contribution The second proportion among all modal contributions;
[0126] multiplying the second ratio by a balance of the smoothing factor about 1 to obtain a seventh product;
[0127] Add the sixth product and the seventh product to get the modal weight .
[0128] The smoothing factor is used to control the balance between historical weight and current feedback.
[0129] 3. Inter-modal consistency detection:
[0130] Inter-modality consistency score calculation: Modality and modal The consistency score between the emotional state representations Defined as: ;
[0131] in, is modal emotional state representation; is modal Emotional categories Confidence score of is the number of sentiment categories.
[0132] Modal weight adjustment
[0133] If their emotional state indicates and Consistency score Below a certain threshold When the modal and modal There is a conflict between:
[0134] ;
[0135] in, is the conflict detection threshold.
[0136] For conflicting modalities, the weights are dynamically adjusted according to the confidence scores of their emotional state representations:
[0137] ;
[0138] in, is the learning rate for weight adjustment; is modal Confidence score for the most likely sentiment class.
[0139] 4. Weight normalization and feature representation:
[0140] Modal weight normalization: normalize the modal weight Normalization.
[0141] Weighted feature representation: The product of the normalized modality weight and the emotional state representation in each modality is summed to calculate the weighted feature representation.
[0142] The preliminary feature subset is screened based on the updated feature weights to obtain the optimized feature subset.
[0143] Specifically, screening the preliminary feature subset based on the updated feature weights includes:
[0144] For any feature in the preliminary feature subset,
[0145] Compare the updated feature weight with the screening threshold. When the updated feature weight is greater than the screening threshold, retain the corresponding feature; otherwise, discard the corresponding feature.
[0146] Traverse all features in the preliminary feature subset and construct an optimized feature subset based on the retained features.
[0147] In this embodiment, the screening threshold is also dynamically adjusted:
[0148] Calculate the difference between the model performance and the target performance after the tth round of game;
[0149] Multiply the difference by the learning rate to get the fifth product;
[0150] Add the fifth product to the screening threshold in the t-th round of the game to obtain the screening threshold in the t+1-th round of the game.
[0151] In this embodiment, the learning rate is obtained by multiplying the initial learning rate by a numerical value; the base of the numerical value is a natural base, and the exponent is the negative of the ratio of the current game round to the maximum preset number.
[0152] S3: Conduct multiple rounds of game until the convergence condition is reached and the final feature subset is obtained.
[0153] S4: Input the final feature subset into the multimodal sentiment classification model and output the predicted sentiment category.
[0154] Specifically, this step includes:
[0155] Convert the final feature subset into a feature vector:
[0156] Each feature in the final feature subset is Z-score normalized to eliminate the dimensional differences between modalities;
[0157] The standardized features of each modality are concatenated into a joint vector according to the dimensions of the modality;
[0158] When the joint vector has a high dimension or is heterogeneous, it is mapped to a low-dimensional space (meeting the dimension of the multimodal sentiment classification model) through a fully connected layer to obtain an embedded vector;
[0159] The embedding vector is L2 normalized to ensure the consistency of the dimensions of the features of different modalities and obtain the feature vector.
[0160] A multimodal sentiment classification model is constructed, using a deep neural network (DNN) as the basic classifier. Its network structure includes the following core modules:
[0161] The feature embedding layer maps the feature vectors to a high-dimensional embedding space to capture nonlinear relationships.
[0162] The attention interaction layer introduces a multi-head self-attention mechanism to enhance the interactive representation between multimodal features.
[0163] The classification output layer outputs the probability distribution of emotion categories through the softmax function.
[0164] Model training and optimization: End-to-end training is performed by calculating the cross-entropy loss function based on the output probability distribution of sentiment categories. During training, the feedback mechanism of the dynamic game strategy is combined with backpropagation of feature weights to ensure the classifier's sensitivity to high-contribution features.
[0165] Sentiment classification performance verification and feedback loop:
[0166] Classification performance evaluation: Verify the model performance on an independent test set. Evaluation indicators include classification accuracy and macro-average F1 score.
[0167] The feedback closed-loop mechanism feeds the classification performance results back into the dynamic game strategy, forming an optimization closed-loop:
[0168] If the classification accuracy does not reach the preset threshold, the re-optimization process of the optimized feature subset is triggered and multiple rounds of game are re-executed; otherwise, the model performance is considered excellent and no further optimization is required.
[0169] b. If the fluctuation of the macro-average F1 score exceeds the tolerance range, dynamically adjust the modal weight and synergy effect adjustment coefficient to suppress the influence of the noise mode.
[0170] Sentiment classification result output:
[0171] For the input multimodal data, the multimodal sentiment classification model outputs the probability distribution of sentiment categories. The probability distribution of sentiment categories of each modality is used as the confidence score of the corresponding modality for the sentiment category. The confidence scores of any modality for all sentiment categories are integrated to obtain the corresponding sentiment state representation.
[0172] Sum the product of the normalized modality weight and the emotional state representation in each modality to calculate the weighted feature representation;
[0173] The emotion category with the highest confidence in the weighted feature representation is selected as the predicted emotion category.
[0174] The multimodal sentiment classification method based on dynamic game strategy provided in this embodiment has the following beneficial effects: the method adopts a dynamic game strategy, which can better adapt to the complexity of multimodal data; through this strategy, the system can more effectively select features, thereby improving the accuracy of sentiment classification and significantly improving the performance of multimodal sentiment classification; this innovation provides strong support for achieving more accurate and reliable sentiment classification.
[0175] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0176] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.
Claims
1. A multimodal sentiment classification method based on dynamic game strategy, characterized in that: include: S1: Obtain multimodal data to be sentimentally classified, including images, text, and speech, and perform feature extraction to obtain multimodal features; Perform redundancy elimination on multimodal features to obtain a preliminary feature subset; S2: In each round of the dynamic game strategy, the contribution of each feature in the preliminary feature subset is compensated through a high-order synergy compensation mechanism based on tensor decomposition, including: Represent the preliminary feature subset as a three-dimensional tensor, and perform CP decomposition on the three-dimensional tensor to decompose the weight of each interaction mode; For any feature in the preliminary feature subset, Calculate the product of the weight of the interaction mode in any interaction mode and the participation of the feature in the corresponding interaction mode, and sum the corresponding products of all interaction modes; multiply the sum by the synergy adjustment coefficient and add it to the basic contribution of the feature to obtain the compensation contribution corresponding to the feature; Traverse all features in the preliminary feature subset and obtain the compensation contribution corresponding to each feature; The weight of each feature is updated based on its compensation contribution, including: For any feature in the preliminary feature subset, Multiply the weight of the feature in the tth round of the game by the preset smoothing factor to obtain the third product; Calculate the ratio of the compensation contribution corresponding to the feature to the compensation contribution of all features; multiplying the ratio by a balance of the smoothing factor about 1 to obtain a fourth product; Add the third product and the fourth product to get the weight of the feature in the t+1th round of the game; Traverse all features in the preliminary feature subset and obtain the updated feature weights of each feature; The preliminary feature subset is screened based on the updated feature weights to obtain the optimized feature subset; S3: Perform multiple rounds of game until the convergence condition is reached and the final feature subset is obtained; S4: Input the final feature subset into the multimodal sentiment classification model and output the predicted sentiment category.
2. The multimodal emotion classification method based on dynamic game strategy according to claim 1 is characterized in that: Feature extraction of multimodal data includes: Input the image data into the ResNet pre-training model to obtain image modality features; Input text data into the pre-trained BERT model to obtain text modality features; Input the speech data into the Wav2Vec2.0 model to obtain speech modal features; Combine multiple image modality features, text modality features, and speech modality features to obtain multimodal features.
3. The multimodal emotion classification method based on dynamic game strategy according to claim 2 is characterized in that: The redundant elimination process for the multimodal features includes: Calculate the covariance between any two features in the multimodal features and the standard deviation of each feature; Calculate the product of the standard deviations of the two features to obtain the first product; Divide the covariance between the two features by the first product to get the correlation between the two features; Traverse all features in the multimodal features, calculate the correlation between all two features, and use them as elements of the matrix to construct the correlation matrix; Compare the modulus of any element in the correlation matrix with the correlation threshold. When the modulus of the element is greater than the correlation threshold, compare the importance, average weight, and variance between the two features in the element in turn. Among them, the feature with less importance between the two features is eliminated; When the importance of two features is the same, the feature with the smallest average weight between the two features will be eliminated; If the average weights of the two features are still the same, the feature with the smallest variance between the two features will be eliminated; The importance of a feature is obtained by adding the product of the initial weight of the feature and the balance coefficient, and the product of the updated feature weight of the feature in the game and the equilibrium number of the balance coefficient with respect to 1; The average weight of the feature is obtained by averaging the feature weights updated in multiple rounds of the game; All elements in the correlation matrix are traversed, and redundant multimodal features are eliminated. The obtained multimodal features after redundant elimination are the preliminary feature subset.
4. The multimodal sentiment classification method based on dynamic game strategy according to claim 3 is characterized in that: It also includes dynamically adjusting the correlation threshold according to the off-diagonal elements of the correlation matrix: Calculate the mean and standard deviation of the off-diagonal elements in the correlation matrix; Multiplying the standard deviation of the off-diagonal elements by a preset adjustment parameter to obtain a second product; The second product is added to the mean of the off-diagonal elements, and the correlation threshold is dynamically updated using the addition result.
5. The multimodal sentiment classification method based on dynamic game strategy according to claim 1 is characterized in that: The participation degree of a feature in the interaction mode is determined by whether the feature participates in the interaction mode. When the feature participates in the interaction mode, the participation degree of the feature in the interaction mode is 1; otherwise, the participation degree of the feature in the interaction mode is 0.
6. The multimodal sentiment classification method based on dynamic game strategy according to claim 1 is characterized in that: The screening of the preliminary feature subset based on the updated feature weights includes: For any feature in the preliminary feature subset, Compare the updated feature weight with the screening threshold. When the updated feature weight is greater than the screening threshold, retain the corresponding feature; otherwise, discard the corresponding feature. Traverse all features in the preliminary feature subset and construct an optimized feature subset based on the retained features.
7. The multimodal sentiment classification method based on dynamic game strategy according to claim 6 is characterized in that: Dynamically adjusting the screening threshold includes: Calculate the difference between the model performance and the target performance after the tth round of game; Multiply the difference by the learning rate to get the fifth product; Add the fifth product to the screening threshold in the t-th round of the game to obtain the screening threshold in the t+1-th round of the game.
8. The multimodal sentiment classification method based on dynamic game strategy according to claim 1 is characterized in that: The convergence conditions include: Or when the model performance in any round of the game is greater than or equal to the performance threshold, the iteration is stopped; Or when the change in model performance in any round of the game is less than the change threshold, the iteration is stopped; Or the game rounds reach the maximum preset number and the iteration stops.