An adaptive optimization method and related device for multimodal fine-grained emotion recognition

By optimizing the speech-to-text quality using a large language model and combining it with the FST-PSO algorithm to optimize the class weights and hyperparameters of the XGBoost model, the problems of poor speech-to-text quality and class imbalance in multimodal emotion recognition are solved, thus improving the accuracy and stability of fine-grained emotion recognition in short videos.

CN123069154APending Publication Date: 2026-11-10TIANJIN NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611305028.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-26
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing multimodal emotion recognition technologies suffer from poor speech-to-text quality, insufficient class imbalance adaptability, low efficiency in model parameter optimization, and a lack of systematic collaborative optimization schemes in fine-grained emotion recognition scenarios for short videos, resulting in insufficient recognition accuracy and stability.

Method used

We employ a large language model to optimize speech-to-text quality, and use the FST-PSO algorithm to jointly optimize the class weights and hyperparameters of the XGBoost model. By combining multimodal feature fusion, we achieve adaptive optimization of multimodal fine-grained emotion recognition.

Benefits of technology

It significantly improves the utilization rate of text modal information, suppresses the learning bias caused by class imbalance, enhances the accuracy and generalization ability of fine-grained emotion recognition, and forms a complete adaptive optimization solution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069154A_ABST
    Figure CN123069154A_ABST
Patent Text Reader

Abstract

This invention belongs to the field of multimodal emotion recognition technology, and discloses an adaptive optimization method and related apparatus for multimodal fine-grained emotion recognition. It utilizes a large language model to semantically optimize the initial text obtained from speech recognition and expands minority class texts to balance the sample distribution. Subsequently, visual, audio, and optimized text features are extracted and fused into multimodal features, which are input into an XGBoost model with class weights. Simultaneously, the FST-PSO algorithm is used to jointly adaptively optimize the class weights and model hyperparameters. This method significantly improves the information utilization rate of text modalities, effectively suppresses learning bias caused by class imbalance, avoids the local optimum dilemma of traditional parameter tuning, and ultimately greatly improves the accuracy and generalization ability of fine-grained emotion recognition in complex short video scenarios, forming a complete adaptive optimization solution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of multimodal emotion recognition technology, and particularly relates to an adaptive optimization method and related apparatus for multimodal fine-grained emotion recognition. Background Technology

[0002] With the rapid development of short video platforms, users are increasingly expressing their emotions through multimodal information such as video, audio, and text. Multimodal emotion recognition technology has gradually become an important research direction in the field of emotion analysis. This technology achieves automatic identification of users' emotional tendencies by integrating multi-source information such as visual, audio, and text, and has been widely applied in short video content analysis, public opinion monitoring, and social media sentiment analysis. Existing multimodal emotion recognition methods mainly revolve around single-modal feature extraction, multimodal information fusion, and the construction of emotion classification models: text modality typically uses pre-trained language models such as BERT and RoBERTa for semantic feature extraction; visual modality mainly uses deep learning models such as ResNet, ViT, and VideoMAE to extract visual representations; audio modality mainly uses models such as VGGish, HuBERT, and Wav2Vec2 for acoustic feature learning, and speech information in videos is usually converted into text by automatic speech recognition models before subsequent analysis; at the classification model level, gradient boosting ensemble learning models such as XGBoost have been widely used due to their good performance in structured data processing, high-dimensional feature adaptation, and few-sample classification tasks.

[0003] However, existing multimodal emotion recognition technologies still have many shortcomings when applied to fine-grained emotion recognition scenarios in short videos. First, insufficient speech-to-text quality leads to low utilization of text modal information. Speech in short video scenarios generally has characteristics such as background noise, colloquial expressions, dialectal mixing, and variable speech rates. Automatic speech recognition results are prone to text errors, incoherent sentences, and missing emotional keywords. Existing methods have not fully utilized the text correction and semantic completion capabilities of large language models to optimize text quality for fine-grained emotion recognition tasks, directly affecting the subsequent text feature extraction and emotion classification results. Second, the problem of imbalance in fine-grained emotion categories is prominent. In real short video scenarios, the number of fine-grained negative emotion samples such as sadness, anger, and worry is far less than that of positive and neutral samples. Model training is prone to category bias, and existing simple oversampling, undersampling, or fixed category weight adjustment strategies cannot dynamically optimize the degree of category attention according to the model training state, resulting in insufficient learning of a few categories of emotion features. Secondly, XGBoost model parameter optimization lacks an adaptive adjustment mechanism. Its predictive performance is highly dependent on key hyperparameters such as learning rate, tree depth, and subsampling ratio. Traditional methods such as manual parameter tuning and grid search suffer from low search efficiency, insufficient parameter combination coverage, and difficulty in obtaining the global optimum. Furthermore, there is a lack of a collaborative optimization mechanism between model parameters and class weights, making it difficult to specifically improve the recognition performance of a few classes. Finally, existing methods lack a comprehensive optimization framework for fine-grained multimodal emotion recognition. Most studies focus on coarse-grained emotion classification tasks, failing to provide systematic solutions to the combined problems of insufficient speech and text quality, imbalanced sample classes, and difficulties in model parameter optimization in short video scenarios. This prevents the full utilization of the complementary advantages of multimodal information, resulting in insufficient recognition accuracy and generalization ability in complex real-world application environments.

[0004] It is evident that existing multimodal emotion recognition technologies suffer from several drawbacks in fine-grained emotion recognition scenarios for short videos. These include poor speech-to-text quality, insufficient class imbalance adaptability, low efficiency in model parameter optimization, and a lack of systematic collaborative optimization solutions. Consequently, these technologies struggle to guarantee the accuracy and stability of fine-grained emotion recognition. Summary of the Invention

[0005] This invention provides an adaptive optimization method and related apparatus for multimodal fine-grained emotion recognition. This method can effectively solve the problems of poor speech-to-text quality, insufficient class imbalance adaptation ability, low efficiency of model parameter optimization and lack of systematic collaborative optimization scheme in the context of fine-grained emotion recognition in short videos in existing multimodal emotion recognition technologies. It can ensure the accuracy and stability of fine-grained emotion recognition.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: An adaptive optimization method for multimodal fine-grained emotion recognition includes: Obtain the multimodal fusion features corresponding to the short video to be identified; The multimodal fusion features corresponding to the short video to be identified are input into the pre-trained classification model to obtain the probability results of various emotions, so as to determine the final fine-grained emotion category according to the principle of maximum probability. The training process of the classification model is as follows: A labeled dataset containing multiple fine-grained sentiments is constructed based on original short videos; the speech information of the original short videos is used for speech recognition to obtain the initial text; the initial text is semantically understood and optimized based on a large language model to obtain optimized text data; the minority class texts in the labeled dataset are expanded to obtain a training text set containing optimized text data and expanded enhanced text. Feature extraction is performed on the visual frame sequence, audio signal and training text set of the original short video respectively. The feature mapping network constructed based on each modality feature is then used to concatenate and fuse the extracted modality features to obtain the expanded multimodal fusion feature. The expanded multimodal fusion features are input into the XGBoost model with class weights; during training, the class weights of the corresponding class samples are adjusted according to the difference in the number of samples of different emotion categories. The FST-PSO algorithm is used to jointly optimize the class weights and XGBoost model hyperparameters, and the optimal class weights and XGBoost model hyperparameters are output to obtain the trained classification model.

[0007] Furthermore, the semantic understanding and optimization processing of the initial text based on the large language model specifically includes text correction, redundant information removal, semantic completion, and sentiment enhancement processing. A large language model is used to augment minority texts. During the augmentation process, enhanced text is generated through text reconstruction while maintaining the consistency of the sentiment category and core semantics of the original text. The generated enhanced text reuses the visual modal information and audio modal information of the original short video as non-text modal features.

[0008] Furthermore, when extracting features for each modality, a video self-supervised coding model is used to encode visual features for visual frame sequences, an audio pre-trained coding model is used to encode audio features for audio signals, and a pre-trained language model is used to encode text semantic features for training text sets. The feature mapping network is set independently for the three modalities of vision, audio and text, and is used to map the features of each modality in different dimensions to a unified low-dimensional feature space. After feature concatenation, standardization and normalization are performed to obtain multimodal fusion features.

[0009] Furthermore, the hyperparameters to be optimized in the XGBoost model are divided into three categories: model structure parameters, learning parameters, and regularization parameters. The model structure parameters include the number of trees and the maximum tree depth. The learning parameters include the learning rate, the sample sampling ratio, and the feature sampling ratio; The regularization parameters include the minimum leaf node weight, L1 regularization parameter, and L2 regularization parameter.

[0010] Furthermore, in the process of jointly optimizing the class weights and XGBoost model hyperparameters using the FST-PSO algorithm, the expanded multimodal fusion feature dataset is divided into a training subset and a validation subset. All class weights and XGBoost model hyperparameters are encoded together into a particle position vector, where each particle corresponds to a combination of class weights and XGBoost model hyperparameters. The position and velocity of the particle swarm are randomly initialized, and the initial individual optimal position and the initial global optimal position of the swarm are recorded.

[0011] Furthermore, the specific process of joint parameter optimization is as follows: The FST-PSO algorithm uses the weighted sum of classification accuracy and macro-average F1 value as the fitness function of particles. During each iteration, the Euclidean distance between the current position of each particle and the global optimal position is calculated, and the fitness difference between the current iteration and the previous iteration is used as a fitness change factor, which is used to characterize the distance of the particle from the optimal solution region and the search effect of the particle, respectively. Input the Euclidean distance and fitness change factor into the preset fuzzy rule module, and dynamically adjust the inertial weight, individual learning factor and social learning factor of the corresponding particles according to the preset fuzzy rules. Inertia weight, individual learning factor, and social learning factor are used to control the particle's ability to maintain its current motion trend, the degree to which the particle learns from its own historical best position, and the degree to which the particle learns from the global best position, respectively. Specifically, when a particle is far from the global optimum or its fitness is not sufficiently improved, the particle's global exploration capability is enhanced; when a particle is close to the global optimum and its fitness tends to stabilize, the particle's local search capability is enhanced; the particle's velocity and position are updated based on the adjusted parameters, and parameter boundary constraints are executed. After iterating to the termination condition, the global optimum parameter combination is output to obtain the optimal class weights and XGBoost model hyperparameters.

[0012] Furthermore, the particle velocity and position update formulas for joint parameter optimization are as follows: V i (t+1)=w i Vi (t)+c soc,i r1(G(t) X i (t))+c cog,i r2(P i (t) X i (t)) X i (t+1)=X i (t)+V i (t+1) In the formula, V i (t), V i (t+1) represent the velocity vectors of the i-th particle at the t-th and t+1-th iterations, respectively; w i Indicates inertia weight; c soc,i c represents the social learning factor; cog,i G(t) represents the individual learning factor; G(t) represents the global optimal position at the t-th iteration; X ... i (t), X i (t+1) represent the positions of the i-th particle in the t-th and t+1-th iterations, respectively; P i (t) represents the optimal position of the i-th particle obtained during the historical search process in the t-th iteration; r1 and r2 represent the random number control factors, respectively.

[0013] An adaptive optimization multimodal fine-grained emotion recognition system includes: The acquisition module is used to acquire the multimodal fusion features corresponding to the short video to be identified; The recognition module is used to input the multimodal fusion features corresponding to the short video to be recognized into the pre-trained classification model to obtain the probability results of various emotions, so as to determine the final fine-grained emotion category according to the principle of maximum probability. The training process of the classification model is specifically implemented as follows: A labeled dataset containing multiple fine-grained sentiments is constructed based on original short videos; the speech information of the original short videos is used for speech recognition to obtain the initial text; the initial text is semantically understood and optimized based on a large language model to obtain optimized text data; the minority class texts in the labeled dataset are expanded to obtain a training text set containing optimized text data and expanded enhanced text. Feature extraction is performed on the visual frame sequence, audio signal and training text set of the original short video respectively. The feature mapping network constructed based on each modality feature is then used to concatenate and fuse the extracted modality features to obtain the expanded multimodal fusion feature. The expanded multimodal fusion features are input into the XGBoost model with class weights; during training, the class weights of the corresponding class samples are adjusted according to the difference in the number of samples of different emotion categories. The FST-PSO algorithm is used to jointly optimize the class weights and XGBoost model hyperparameters, and the optimal class weights and XGBoost model hyperparameters are output to obtain the trained classification model.

[0014] An adaptive optimization multimodal fine-grained emotion recognition device includes: Memory, used to store computer programs; A processor is used to implement the above-described adaptive optimized multimodal fine-grained emotion recognition method when executing the computer program.

[0015] A computer-readable storage medium storing a computer program, which, when executed by a processor, is used to implement the above-described adaptive optimization multimodal fine-grained emotion recognition method.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention provides an adaptive optimization method for multimodal fine-grained emotion recognition. It utilizes a large language model to semantically optimize the initial text obtained from speech recognition and expands minority class texts to balance the sample distribution. Subsequently, visual, audio, and optimized text features are extracted and fused into multimodal features, which are input into an XGBoost model with class weights. Simultaneously, the FST-PSO algorithm is used to jointly and adaptively optimize the class weights and model hyperparameters. The large language model corrects semantic errors and missing emotion keywords in speech-to-text conversion; text expansion dynamically alleviates class bias caused by insufficient fine-grained negative emotion samples; FST-PSO replaces inefficient manual parameter tuning and collaboratively optimizes weights and parameters to specifically improve minority class recognition; and the multimodal fusion framework integrates the complementary advantages of each modality. This method significantly improves the information utilization rate of text modalities, effectively suppresses learning bias caused by class imbalance, avoids the local optimum dilemma of traditional parameter tuning, and ultimately greatly improves the accuracy and generalization ability of fine-grained emotion recognition in complex short video scenarios, forming a complete adaptive optimization solution. Attached Figure Description

[0017] Figure 1 The following is a flowchart illustrating the implementation of an adaptive optimization multimodal fine-grained emotion recognition method provided in this embodiment of the invention. Figure 2 Macro-average ROC curves of large language model enhancement and FST-PSO optimization mechanism ablation provided in embodiments of the present invention; Figure 3Macro-average ROC curves for different modal combinations provided in embodiments of the present invention; Figure 4 Macro-average ROC curves of large language model enhancement and traditional category imbalance processing strategies provided in embodiments of the present invention; Figure 5 The macro-average ROC curves of different classification models provided in the embodiments of the present invention under enhanced multimodal features; Figure 6 The main flowchart of an adaptive optimization multimodal fine-grained emotion recognition method provided in this embodiment of the invention; Figure 7 This is a schematic diagram of the structure of an adaptive optimization multimodal fine-grained emotion recognition system provided in an embodiment of the present invention. Detailed Implementation

[0018] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0019] As described in the background section, existing multimodal recognition methods for short video scenarios have the following technical shortcomings: First, insufficient speech-to-text quality leads to low utilization of text modal information: Existing multimodal sentiment recognition methods for short videos typically employ automatic speech recognition models to generate text information and then extract semantic features based on the generated text. However, due to the characteristics of short video speech, such as colloquial expressions, background noise, and diverse linguistic environments, automatic speech recognition results are prone to text errors, incoherent sentences, and missing sentiment keywords, leading to incomplete semantic information and affecting subsequent text feature extraction and sentiment classification. Although Large Language Models (LLMs) have gained capabilities in text correction, semantic completion, and content enhancement in recent years, existing methods have not fully utilized LLMs to improve speech-to-text quality and have not yet been optimized for fine-grained sentiment recognition tasks.

[0020] Second, the imbalance of fine-grained sentiment categories leads to insufficient recognition ability for minority categories: Existing emotion recognition methods typically rely on the original data distribution for model training. However, in real-world short video scenarios, the number of fine-grained negative emotion samples such as sadness, anger, and worry is significantly less than that of positive or neutral samples, making it easy for class bias to occur during model training. Existing methods often employ simple oversampling, undersampling, or fixed class weight adjustment strategies, making it difficult to dynamically optimize class attention based on the model's training state, resulting in insufficient learning of a few class emotion features.

[0021] Third, XGBoost model parameter optimization lacks an adaptive adjustment mechanism: XGBoost possesses strong data modeling capabilities, but its performance is significantly affected by the combination of model hyperparameters. Existing methods typically determine parameters through manual tuning, grid search, or random search, which suffers from low search efficiency and insufficient adaptability of parameter combinations. Furthermore, for fine-grained imbalanced sentiment scenarios, there is a lack of a collaborative optimization mechanism between model parameters and class weights, making it difficult to fully improve the sentiment recognition capabilities of a few classes.

[0022] Fourth, existing methods lack a comprehensive optimization framework for fine-grained multimodal emotion recognition: Existing research on multimodal emotion recognition primarily focuses on coarse-grained emotion classification tasks. For fine-grained negative emotion recognition, existing classification frameworks are often directly applied, lacking comprehensive solutions to problems such as insufficient audio-text quality, imbalanced sample classes, and difficulties in optimizing model parameters in short video scenarios. Existing methods struggle to fully leverage the complementary advantages of multimodal information from video, audio, and text, and still suffer from insufficient recognition accuracy and generalization ability in complex real-world application environments.

[0023] To address the problems of insufficient text modality information quality, imbalanced distribution of minority class samples, reliance on manual adjustment of classification model parameters, and insufficient model adaptability in existing fine-grained emotion recognition technologies, this embodiment provides an adaptive optimization multimodal fine-grained emotion recognition method. This method uses short video data as input, optimizes the quality of speech-to-text conversion through LLM, and performs semantic enhancement on minority class emotion samples. Furthermore, it integrates video, audio, and text multimodal features to construct a multimodal fusion feature representation. Finally, it employs the FST-PSO algorithm to jointly optimize the key hyperparameters and class weights of the XGBoost classification model, achieving adaptive adjustment of model parameters and thus improving the fine-grained emotion recognition capability in complex short video scenarios.

[0024] For example, such as Figure 6 As shown, this embodiment provides an adaptive optimization method for multimodal fine-grained emotion recognition, including: Obtain the multimodal fusion features corresponding to the short video to be identified; The multimodal fusion features corresponding to the short video to be identified are input into the pre-trained classification model to obtain the probability results of various emotions, so as to determine the final fine-grained emotion category according to the principle of maximum probability. The training process of the classification model is as follows: A labeled dataset containing multiple fine-grained sentiments is constructed based on original short videos; the speech information of the original short videos is used for speech recognition to obtain the initial text; the initial text is semantically understood and optimized based on a large language model to obtain optimized text data; the minority class texts in the labeled dataset are expanded to obtain a training text set containing optimized text data and expanded enhanced text. Feature extraction is performed on the visual frame sequence, audio signal and training text set of the original short video respectively. The feature mapping network constructed based on each modality feature is then used to concatenate and fuse the extracted modality features to obtain the expanded multimodal fusion feature. The expanded multimodal fusion features are input into the XGBoost model with class weights; during training, the class weights of the corresponding class samples are adjusted according to the difference in the number of samples of different emotion categories. The FST-PSO algorithm is used to jointly optimize the class weights and XGBoost model hyperparameters, and the optimal class weights and XGBoost model hyperparameters are output to obtain the trained classification model.

[0025] The adaptive optimization multimodal fine-grained emotion recognition method provided in this embodiment will be further explained below with reference to the accompanying drawings: Step 1: Short video data preprocessing and LLM text optimization enhancement: First, short video data was collected, and the raw data was preprocessed by deduplication and outlier filtering. Then, a fine-grained sentiment dataset was established through manual annotation, with sentiment categories including positive, neutral, sad, angry, and worried.

[0026] For audio information in short videos, a Whisper-large speech recognition model is used for speech-to-text processing to obtain initial text data. Since automatic speech recognition results may contain typos, incomplete sentences, colloquial redundancy, and lack of emotional expression, the initial text is input into an LLM (Large Language Model). The text undergoes semantic understanding and optimization processing using preset prompt templates, including text correction, redundancy removal, semantic completion, and emotional expression enhancement, resulting in optimized text data.

[0027] Furthermore, for emotion categories with fewer samples, such as sadness, anger, and worry, a dynamic enhancement strategy is set according to the sample size of different categories. The optimized minority class text is input into LLM, and new samples with different expressions are generated through text reconstruction under semantic preservation, thereby expanding the minority class text data and improving the balance of category distribution. Since the enhancement process maintains the consistency of the original text's emotion category and semantics, and only expands the text expression, the corresponding video and audio modal information is retained as non-textual modal features of the enhanced samples. The optimized original text and enhanced text are used together for subsequent feature extraction.

[0028] Step 2, Multimodal Feature Extraction and Fusion: Feature extraction is performed for different modalities in short videos. Specifically, the VideoMAE model is used to encode the video frame sequence to obtain video visual features; the Wav2Vec2 model is used to encode the audio signal to obtain audio features; and the RoBERTa model is used to encode the LLM-optimized text and the enhanced text to obtain text semantic features.

[0029] Furthermore, to reduce the dimensionality differences between different modal features, visual, audio, and text feature mapping networks are constructed separately to map each modal feature to a unified low-dimensional space, and feature concatenation is performed to form a multimodal fusion representation. Subsequently, the fusion features are standardized and normalized to obtain a unified feature representation for training the classification model.

[0030] For the enhanced text samples generated by LLM, since they maintain the same sentiment category and semantic information as the original video samples, the text features corresponding to the enhanced text are associated and fused with the video features and audio features of the original short video to form the expanded multimodal fusion features.

[0031] Step 3: Building the XGBoost classification model: The multimodal fusion features obtained in step 2 are input into the XGBoost classification model to achieve five-category fine-grained sentiment classification.

[0032] The XGBoost model is trained using a multi-class objective function and incorporates a class weight mechanism. This mechanism adjusts the importance of each class during training based on the difference in the number of samples from different emotion categories, enhancing the model's ability to learn a few emotion categories such as sadness, anger, and worry. The class weights, along with the XGBoost model hyperparameters, serve as search variables for the FST-PSO algorithm, enabling joint optimization of model structure parameters and class attention levels.

[0033] The parameters to be optimized in the XGBoost model include model structure parameters, learning parameters, and regularization parameters.

[0034] Among them, the model structure parameters include the number of trees parameter n_estimators and the maximum tree depth parameter max_depth; The learning parameters include the learning rate, the subsample sampling ratio, and the colsample bytree feature sampling ratio. The regularization parameters include the minimum leaf node weight min_child_weight, the L1 regularization parameter reg_alpha, and the L2 regularization parameter reg_lambda; The category weight parameters include positive category weight, neutral category weight, sadness category weight, anger category weight, and worry category weight.

[0035] Step 4: Perform joint parameter optimization using the FST-PSO algorithm: The multimodal fusion features are further divided into training and validation subsets. The FST-PSO algorithm (fuzzy self-adjusting particle swarm optimization algorithm) is used to jointly optimize the hyperparameters and class weights of the XGBoost model, as follows: First, the parameters to be optimized (XGBoost model hyperparameters and class weights) are encoded as particle position vectors, with each particle representing an XGBoost model configuration.

[0036] The particle position vector includes: (1) The weight parameters of the five emotion categories are represented as follows: W=(ω pos ω neu ω sad ω ang ω wor ) ω pos ω neu ω sad ω ang ω wor These correspond to five emotional weights: positive, neutral, sad, angry, and worried.

[0037] (2) XGBoost model parameter representation: θ=(n_estimators, max_depth, learning_rate, subsample, colsample_bytree, min_child_weight, reg_alpha, reg_lambda) Joint optimization of W and θ is performed, and each particle is represented as: X i =(W, θ) Where Xi represents the current particle position, i.e., the current combination of XGBoost hyperparameters and class weights; W represents the class weights; and θ represents the XGBoost model parameters. Therefore, each particle contains 5 emotion class weight parameters and 8 XGboost parameters, for a total of 5 + 8 = 13 dimensions.

[0038] (3) Particle initialization: The particle swarm positions and velocities are randomly initialized based on the search range of each optimization parameter. For each particle: Calculate the XGBoost model performance corresponding to the current position; Record the best position in the particle's history; Update the global optimal position.

[0039] (4) Fitness function calculation: Using the classification performance of the XGBoost model as the fitness function for particles: Fitness = αAccuracy + βMacro F1 Where Accuracy represents the overall classification accuracy; Macro-F1 represents the average F1 score of the five sentiment categories; α and β are the corresponding weight coefficients; and α+β=1.

[0040] By introducing Macro-F1, the optimization process can simultaneously consider the recognition performance of both the majority and minority classes.

[0041] (5) FST fuzzy self-adjustment mechanism: Unlike traditional particle swarm optimization algorithms that use fixed or uniform parameter settings, this invention uses FST-PSO to dynamically adjust particle parameters based on the search state of each particle.

[0042] For the i-th particle, calculate the following during each iteration: 1. Distance δ between the particle and the global optimal position i : δ i =‖X i (t)-G(t)‖ Where, ‖•‖ represents the Euclidean distance, and G(t) represents the globally optimal position at the t-th iteration. δ i Used to determine the distance of particle i from the optimal solution region.

[0043] 2. Particle fitness change factor: Φ is calculated based on the change in fitness between the current iteration and the previous iteration. i (t), used to evaluate the particle search performance: Φ i (t)=F i (t)-F i (t-1) Among them, F i F(t) represents the fitness value of particle i in the t-th iteration. i (t-1) represents the fitness value of particle i in the (t-1)th iteration, Φ i (t) determines whether the current search for particle i is valid, when Φi When (t) is large, it indicates that the search direction of particle i is valid; when Φ i When (t) approaches 0 or decreases, it indicates that particle i may be stuck and the search strategy needs to be adjusted.

[0044] 3. Adjusting parameters for fuzzy inference: Distance variable δ i and fitness change variable Φ i (t) Input the fuzzy rules module to dynamically adjust the particle's search parameters according to preset fuzzy rules. According to the preset fuzzy rules, when a particle is far from the global optimum or its fitness improvement is insufficient, the particle's exploration ability is increased; when a particle is close to the global optimum and its fitness tends to stabilize, its local search ability is increased. Through this method, different particles can adaptively adjust the following parameters according to their own search state: The inertial weight w of the i-th particle at the t-th iteration i This refers to the ability of particles to maintain their current motion trend. The individual learning factor c of the i-th particle at the t-th iteration cog,i This is used to control the degree to which a particle learns from its own historical best position; The social learning factor c of the i-th particle at the t-th iteration soc,i This is used to control the degree to which a particle learns towards the global optimal position.

[0045] 4. Particle velocity and position update: V i (t+1)=w i V i (t)+c soc,i r1(G(t) X i (t))+c cog,i r2(P i (t) X i (t)) V i (t): The velocity vector of the i-th particle at the t-th iteration, used to represent the direction and magnitude of the particle's movement in the parameter search space; P i (t): The best position of the i-th particle during the historical search process at the t-th iteration, that is, the position of the particle when it has the highest fitness value; r1 and r2 are random number control factors used to enhance the randomness of the particle search process, taking random numbers in the range [0, 1].

[0046] Then update the particle position: X i(t+1)=X i (t)+V i (t+1) Boundary constraints are applied to ensure that the parameters are within the valid search range.

[0047] 5. Output the optimal parameter combination: The following steps are executed iteratively: fitness calculation, fuzzy inference to adjust parameters, particle velocity update, particle position update, and optimal position update.

[0048] After reaching the maximum number of iterations or the pre-set classification metric (i.e., reaching the termination condition), the globally optimal particle is output, which is the optimal class weight and the optimal XGBoost parameter combination, for model training.

[0049] Step 5: Model Training and Emotion Recognition Based on the optimal class weights and XGBoost hyperparameters obtained in step 4, the final model is retrained on the complete training set to obtain the classification model.

[0050] The training samples, optimized and enhanced by LLM, are input into the model. The original samples are used to maintain the distribution of the real data, while the LLM-enhanced samples are used to improve the learning ability of the minority classes.

[0051] The contribution ratio of different emotion category samples during model training is adjusted using the category weights obtained in step 4.

[0052] After the model is trained, the visual, audio and text fusion features corresponding to the short video to be identified are input into the classification model. The model outputs the probabilities of five types of emotions: positive, neutral, sad, angry and worried. Finally, the final emotion category is determined according to the principle of maximum probability.

[0053] Step 6, Performance Evaluation: Accuracy, Macro-F1, and Macro-AUC are used to evaluate the overall recognition performance. These evaluation metrics are used to comprehensively verify the effectiveness of the proposed method in fine-grained multimodal emotion recognition tasks.

[0054] For example, the identification method provided in the above embodiments has been specifically applied and implemented, and the implementation process is as follows: This embodiment selects relevant short videos published on the Douyin platform during a certain period as input data. It identifies users' emotional states during the pandemic by utilizing multimodal information including video footage, audio information, and user-expressed text. First, LLM is used to optimize the speech-to-text results, and enhancement is applied to a few emotional categories. Then, features from the video, audio, and text modalities are extracted and fused. Finally, the FST-PSO algorithm is used to jointly optimize the XGBoost classification model parameters and category weights, achieving the recognition of five fine-grained emotions: positive, neutral, sadness, anger, and worry.

[0055] The specific implementation steps are as follows: Step 1: Short video data collection and preprocessing: First, short video data related to a certain event is collected from the Douyin platform; the data includes video image information, audio information, and text information corresponding to the video.

[0056] The collected data is preprocessed, including duplicate video removal, invalid video filtering, and missing data deletion, to obtain valid short video samples.

[0057] Subsequently, the samples are manually labeled with sentiment based on video content, audio information, and user-expressed semantics to create a fine-grained sentiment classification dataset. This embodiment sets five sentiment labels: positive, neutral, sad, angry, and worried, as supervision labels for subsequent model training and testing.

[0058] Step 2: LLM-based speech-text optimization and minority class sample augmentation: To address the issues of colloquial speech, background noise interference, and insufficient text quality in automatic speech recognition (ARK) in short videos, this paper first employs a Whisper-large model to perform speech recognition on the short video audio, converting the audio information into initial text. Subsequently, the generated initial text is input into the LLM (Low- ... (1) Delete colloquial and redundant content that does not contribute to actual semantic meaning; (2) Correcting typos and sentence errors generated by speech recognition; (3) Supplement missing semantic information; (4) Optimize emotional expression to make the text more in line with natural language expression habits.

[0059] After the above processing, the optimized text is obtained.

[0060] Furthermore, to address the issue of insufficient sample numbers for a few categories such as sadness, anger, and worry in short video data, the text input LLM will be optimized. By maintaining the emotion category constraint, multiple new text samples with consistent semantics but different expressions will be generated, thereby expanding the text for a few categories.

[0061] Since the enhancement process maintains consistency with the original text's sentiment category and semantics, and only expands the text's expression, it retains the video and audio features of the corresponding short video as non-textual modal information of the enhanced sample.

[0062] Optimized text and enhanced text are used together for subsequent feature extraction.

[0063] Step 3: Multimodal single feature extraction: For the processed short video data, features of three modalities—visual, audio, and text—are extracted. Specifically: (1) Visual feature extraction: The VideoMAE model is used to encode the video frame sequence, extract scene, character actions and visual expression information in the video, and obtain visual feature vectors.

[0064] (2) Audio feature extraction: The Wav2Vec2 model is used to encode the audio signal of short video and extract the acoustic information feature vector in the speech.

[0065] (3) Text feature extraction: The RoBERTa model was used to encode the LLM-optimized text and the enhanced text respectively, and the semantic feature vectors of the text were extracted.

[0066] For the enhanced text samples generated by LLM, their text features are associated and fused with the video and audio features of the corresponding original video to form enhanced multimodal training samples.

[0067] Step 4: Multimodal feature fusion: Three feature mapping networks—visual, audio, and text—are constructed separately to map features from different modalities to a unified feature space. Then, the mapped visual, audio, and text features are concatenated to form a multimodal fusion feature. This fusion feature is further standardized and normalized to obtain a unified feature representation for training the classification model.

[0068] Step 5: Construct an XGBoost fine-grained sentiment classification model: The multimodal fusion features obtained in step 4 are input into the XGBoost classification model to achieve five-category fine-grained sentiment classification.

[0069] The XGBoost model is trained using a multi-class objective function and incorporates a class weighting mechanism to adjust the contribution of different class samples to the model training process based on the difference in the number of samples in different emotion categories.

[0070] In this embodiment, the XGBoost parameters to be optimized include: (1) Model structure parameters: The parameter for the number of trees is n_estimators; The maximum tree depth parameter is max_depth; (2) Learning parameters: learning rate; Subsample; Feature sampling ratio colsample_bytree; (3) Regularization parameters: Minimum leaf node weight: min_child_weight; L1 regularization parameter reg_alpha; L2 regularization parameter reg_lambda; Meanwhile, the weights of five emotion categories—positive, neutral, sadness, anger, and worry—are used as optimization variables.

[0071] The XGBoost parameters and class weights mentioned above together constitute the search space of the FST-PSO algorithm.

[0072] Step 6: Jointly optimize XGBoost parameters and class weights based on the FST-PSO algorithm The fuzzy self-adjusting particle swarm optimization algorithm FST-PSO is used to jointly optimize the XGBoost model parameters and class weights.

[0073] First, the optimization variables are encoded as particle position vectors: X i = (W,θ) Where W represents the weights of the five emotion categories: W = (ω pos , ω neu , ω sad , ω ang , ω wor ) θ represents the XGBoost model parameters: θ=(n_estimators, max_depth, learning_rate, subsample, colsample_bytree, min_child_weight, reg_alpha, reg_lambda) Therefore, in this embodiment, each particle has 13 dimensions.

[0074] The particle swarm is initialized according to a preset parameter range, with each particle corresponding to a specific XGBoost model configuration. For each particle, a classification model is constructed based on the class weights corresponding to its current position and the XGBoost parameters, and the model fitness is calculated. The fitness function is defined as: Fitness = αAccuracy + βMacro F1 Here, Accuracy represents the overall classification accuracy; Macro-F1 represents the average F1 score across multiple classes.

[0075] During particle iteration, the FST fuzzy self-adjustment mechanism is used to dynamically adjust the particle search parameters. For the i-th particle, calculate: δ i = || X i (t)-G(t) ‖ Where δ represents the distance between the particle's current position and the global optimal position.

[0076] Simultaneously calculate the particle fitness change factor: Φ i (t) = F i (t)-F i (t-1) Here, Φ represents the fitness change during the particle search process. δ and Φ are input into the fuzzy rule control module, which dynamically adjusts the inertia weight w according to preset fuzzy rules. i Individual learning factor c cog,i Social learning factor c soc,i .

[0077] Update particle velocity based on the adjusted parameters: V i (t+1)=w i V i (t)+c soc,i r1(G(t) X i (t))+c cog,i r2(P i (t) X i (t)) Then update the particle position: X i (t+1)=X i (t)+V i (t+1) After multiple iterations, the particle position with the highest fitness is obtained, which is the optimal combination of class weights and the optimal combination of XGBoost parameters.

[0078] Step 7: Model Training and Fine-Grained Emotion Recognition The XGBoost classification model is reconstructed based on the optimal parameter combination obtained in step 6.

[0079] First, the model determination process is completed on the training and validation sets using optimized parameters; then, the final classification model is retrained on the complete training data using the obtained optimal parameters.

[0080] The video, audio, and text fusion features corresponding to the short video to be identified are input into the trained XGBoost model, which outputs five emotion probabilities: positive, neutral, sad, angry, and worried. The final emotion category is determined based on the principle of maximizing probability.

[0081] Step 8: Ablation and Comparison Experiment: To verify the impact of key technologies such as large language model enhancement, multimodal feature fusion, and FST-PSO adaptive optimization on the performance of multimodal fine-grained emotion recognition in this embodiment, core mechanism ablation experiments, comparative experiments on contributions from different modalities, comparative experiments on class imbalance handling strategies, and comparative experiments on different classification models were conducted. Accuracy, Macro-F1, and Macro-AUC were used to evaluate the overall recognition performance of the model, and ROC curves were used to analyze the comprehensive discriminative ability of different schemes under different classification thresholds.

[0082] (1) Ablation experiment of core mechanism: First, using XGBoost as the base classifier, the large language model enhancement, FST-PSO parameter optimization, and sample weight optimization mechanisms in this embodiment are progressively ablated. Original feature and default parameter model, enhanced feature and default parameter model, enhanced feature and FST-PSO parameter optimization model, and enhanced feature and FST-PSO parameter and sample weight joint optimization model are set up respectively. The experimental results are shown in Table 1.

[0083] Table 1 shows the experimental results of ablation of the core mechanism.

[0084] Table 1 shows that, under the condition of using enhanced features, FST-PSO parameter optimization improved the model's accuracy from 0.7585 to 0.7806 and the Macro-F1 score from 0.7942 to 0.8087. Further joint optimization of sample weights resulted in an accuracy of 0.7840, a Macro-F1 score of 0.8107, and a Macro-AUC of 0.926 for the complete model. The results indicate that large language model enhancement and the FST-PSO adaptive optimization mechanism can jointly improve the model's overall recognition performance.

[0085] Further comparisons of the core mechanisms were made using macro-average ROC curves, such as... Figure 2 As shown, the complete model "LLM enhanced features + FST-PSO parameter and sample weight joint optimization" has a Macro-AUC of 0.926, the highest among the four schemes; the Macro-AUC of LLM enhanced features + XGBoost default parameters is 0.917, the LLM enhanced features + FST-PSO parameter optimized model is 0.908, and the original features + XGBoost default parameter model is 0.734. This demonstrates that the complete technical scheme exhibits good overall discriminative ability under different discrimination thresholds. It should be noted that the Macro-AUC does not monotonically increase when only FST-PSO parameter optimization is performed; therefore, this embodiment combines Accuracy, Macro-F1, and Macro-AUC to comprehensively evaluate the model performance.

[0086] (2) Comparison experiment of contributions from different modes: To verify the role of visual, audio, and textual modal information in fine-grained emotion recognition, recognition experiments were conducted using text monomodality, audio monomodality, visual monomodality, pairwise modal combinations, and text-audio-visual trimodal combinations. The results are shown in Table 2.

[0087] Table 2 shows the recognition results for different modality combinations.

[0088] Based on the existing experimental results, the text modality and audio modality have strong emotional representation capabilities, while the visual modality has relatively low performance when used alone. The trimodal complete model achieved the highest comprehensive evaluation index, indicating that the joint representation of text, audio and visual information can take advantage of the complementarity of different information sources.

[0089] like Figure 3 As shown, the macro-average ROC curves for different modal combinations are presented. The trimodal T+A+V model achieves a Macro-AUC of 0.926, which is higher than that of the text monomodal model (0.757), text + audio bimodal model (0.749), text + visual bimodal model (0.732), audio + visual bimodal model (0.727), audio monomodal model (0.723), and visual monomodal model (0.517). The overall position of the ROC curves further indicates that the complete trimodal scheme has good comprehensive discrimination ability.

[0090] (3) Comparative experiment on different categories of imbalance treatment strategies: To verify the applicability of the large language model sample augmentation strategy adopted in this embodiment under class imbalance conditions, the large language model augmentation strategy of this invention is compared with traditional class imbalance processing methods such as SMOTE and SMOTEENN using XGBoost as the base classifier. The results are shown in Table 3.

[0091] Table 3 shows the comparison results of different types of imbalance handling strategies.

[0092] Table 3 shows that the large language model enhancement scheme used in this embodiment achieved a combined accuracy of 0.7840, a macro-F1 score of 0.8107, and a macro-AUC of 0.926 in the current experiment. Further comparisons were made using macro-average ROC curves to evaluate different class imbalance handling strategies, such as... Figure 4 As shown, compared with traditional data processing methods based on feature space interpolation or sample cleaning, this invention utilizes a large language model to semantically enhance training samples of a few categories, so that newly added samples maintain the semantic constraints of the corresponding sentiment category, thereby providing richer training information for fine-grained sentiment recognition under class imbalance conditions.

[0093] (4) Comparison experiment of different classification models: To verify the applicability of the classifier used in this embodiment, under the same enhanced multimodal feature conditions, the XGBoost optimized by FST-PSO was compared with the RF, CNN, MLP and SVM classification models. Among them, RF, CNN, MLP and SVM used GridSearch for parameter optimization. The results are shown in Table 4.

[0094] Table 4 shows the recognition results of different classification models.

[0095] As shown in Table 4, the XGBoost and FST-PSO combination used in this embodiment achieved high results in the three comprehensive indicators of Accuracy, Macro-F1 and Macro-AUC, with Accuracy at 0.7840, Macro-F1 at 0.8107 and Macro-AUC at 0.926, indicating that the classifier and optimization mechanism used have good adaptability to the enhanced multimodal features.

[0096] To further compare different classification models and traditional sample processing strategies, a macro-average ROC curve was plotted, such as... Figure 5As shown, the macro-AUC of the complete method is 0.926; under enhanced feature conditions, the macro-AUCs of RF, SVM, MLP, and CNN are 0.903, 0.894, 0.871, and 0.870, respectively. After processing the original features with SMOTE or SMOTEENN, the overall macro-AUC of each model is lower than that of the complete method, with XGBoost+SMOTE and XGBoost+SMOTEENN having macro-AUCs of 0.771 and 0.698, respectively. These results further illustrate the comprehensive recognition capability of the complete technical solution of this invention through its overall performance under different discrimination thresholds.

[0097] Based on the above experimental results, this embodiment improves the fine-grained emotion recognition performance by enhancing the large language model, fusing multimodal features, and jointly optimizing FST-PSO parameters and sample weights. This is achieved through training sample representation, cross-modal information utilization, and adaptive optimization of the classification model. The complete method achieves a comprehensive recognition result of Accuracy of 0.7840, Macro-F1 of 0.8107, and Macro-AUC of 0.926.

[0098] For example, such as Figure 7 As shown, this embodiment also provides an adaptive optimization multimodal fine-grained emotion recognition system, including: an acquisition module for acquiring multimodal fusion features corresponding to the short video to be recognized; and a recognition module for inputting the multimodal fusion features corresponding to the short video to be recognized into a pre-trained classification model to obtain probability results of various emotions, so as to determine the final fine-grained emotion category according to the maximum probability principle. The training process of the classification model is specifically implemented as follows: constructing a labeled dataset containing multiple fine-grained emotions based on the original short video; performing speech recognition on the speech information of the original short video to obtain initial text; performing semantic understanding and optimization processing on the initial text based on a large language model to obtain optimized text data; and performing further processing on the minority class text of the labeled dataset. The process involves expanding the data to obtain a training text set containing optimized text data and enhanced text. Features are extracted from the visual frame sequence, audio signal, and training text set of the original short video. A feature mapping network built based on these modal features is then used to concatenate and fuse the extracted modal features, resulting in expanded multimodal fusion features. These expanded multimodal fusion features are then input into an XGBoost model with class weights. During training, the class weights of samples in different emotion categories are adjusted according to the difference in the number of samples in each category. The FST-PSO algorithm is used to jointly optimize the class weights and XGBoost model hyperparameters, outputting the optimal class weights and XGBoost model hyperparameters to obtain the trained classification model.

[0099] The present invention also provides an adaptive optimization multimodal fine-grained emotion recognition device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the adaptive optimization multimodal fine-grained emotion recognition method.

[0100] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the adaptive optimization multimodal fine-grained emotion recognition method.

[0101] When the processor executes the computer program, it implements the aforementioned steps of adaptive optimization of multimodal fine-grained emotion recognition, for example: acquiring the multimodal fusion features corresponding to the short video to be recognized; inputting the multimodal fusion features corresponding to the short video to be recognized into a pre-trained classification model to obtain the probability results of various emotions, so as to determine the final fine-grained emotion category according to the maximum probability principle; wherein, the training process of the classification model is as follows: constructing a labeled dataset containing multiple fine-grained emotions based on the original short video; performing speech recognition on the speech information of the original short video to obtain initial text; performing semantic understanding and optimization processing on the initial text based on a large language model to obtain optimized text data; expanding the minority class text of the labeled dataset to obtain... The training text set includes optimized text data and augmented text. Features are extracted from the visual frame sequence, audio signal, and training text set of the original short video. A feature mapping network based on each modality feature is used to concatenate and fuse the extracted modality features to obtain the augmented multimodal fusion features. The augmented multimodal fusion features are then input into an XGBoost model with class weights. During training, the class weights of the corresponding class samples are adjusted according to the difference in the number of samples of different emotion categories. The FST-PSO algorithm is used to jointly optimize the class weights and XGBoost model hyperparameters, and the optimal class weights and XGBoost model hyperparameters are output to obtain the trained classification model.

[0102] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing preset functions, the instruction segments describing the execution process of the computer program in the adaptive optimization multimodal fine-grained emotion recognition device. For example, the computer program can be divided into an acquisition module and a recognition module; the specific functions are as follows: the acquisition module is used to acquire the multimodal fusion features corresponding to the short video to be recognized; the recognition module is used to input the multimodal fusion features corresponding to the short video to be recognized into a pre-trained classification model to obtain the probability results of various emotions, so as to determine the final fine-grained emotion category according to the maximum probability principle; wherein, the training process of the classification model is specifically implemented as follows: constructing a labeled dataset containing multiple fine-grained emotions based on the original short video; performing speech recognition on the speech information of the original short video to obtain initial text; performing semantic understanding and optimization processing on the initial text based on a large language model to obtain optimized text data; and performing semantic understanding and optimization processing on the minority class text of the labeled dataset. The process involves expanding the data to obtain a training text set containing optimized text data and enhanced text. Features are extracted from the visual frame sequence, audio signal, and training text set of the original short video. A feature mapping network built based on these modal features is then used to concatenate and fuse the extracted modal features, resulting in expanded multimodal fusion features. These expanded multimodal fusion features are then input into an XGBoost model with class weights. During training, the class weights of samples in different emotion categories are adjusted according to the difference in the number of samples in each category. The FST-PSO algorithm is used to jointly optimize the class weights and XGBoost model hyperparameters, outputting the optimal class weights and XGBoost model hyperparameters to obtain the trained classification model.

[0103] The adaptive optimization multimodal fine-grained emotion recognition device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The adaptive optimization multimodal fine-grained emotion recognition device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above are examples of adaptive optimization multimodal fine-grained emotion recognition devices and do not constitute a limitation on such devices. The device may include more components than described above, or combine certain components, or use different components. For example, the adaptive optimization multimodal fine-grained emotion recognition device may also include input / output devices, network access devices, buses, etc.

[0104] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or any conventional processor. This processor is the control center of the adaptive optimized multimodal fine-grained emotion recognition system, connecting various parts of the entire adaptive optimized multimodal fine-grained emotion recognition device through various interfaces and lines.

[0105] The memory can be used to store the computer program and / or modules. The processor implements various functions of the adaptive optimization multimodal fine-grained emotion recognition device by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory.

[0106] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, image playback, etc.). The data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0107] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the adaptive optimization multimodal fine-grained emotion recognition method.

[0108] If the modules / units integrated in the adaptive optimization multimodal fine-grained emotion recognition system are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0109] Based on this understanding, the present invention can implement all or part of the processes in the above-described adaptive optimization multimodal fine-grained emotion recognition method, or it can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the above-described adaptive optimization multimodal fine-grained emotion recognition method. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or a preset intermediate form, etc.

[0110] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0111] It should be noted that the content contained in the computer-readable storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0112] Compared with existing multimodal emotion recognition methods, this invention provides an adaptive optimization method for fine-grained multimodal emotion recognition, which has the following advantages: First, improve the quality of text modal features and increase the efficiency of multimodal sentiment information utilization: This invention utilizes LLM to optimize the speech-to-text results generated by Whisper-large. Through preset prompt templates, it achieves text error correction, redundant information removal, semantic completion, and enhanced emotional expression, reducing text noise and semantic gaps in automatic speech recognition and improving the completeness and emotional expressiveness of the text information. Feature extraction is performed based on the optimized text input RoBERTa model, enabling the text modality to more accurately represent fine-grained emotional features and enhancing the complementary effect of different modalities in multimodal emotion recognition.

[0113] Second, achieve joint adaptive optimization of XGBoost model hyperparameters and class weights: This invention applies the FST-PSO algorithm to the XGBoost model optimization process. Through a fuzzy control mechanism, it dynamically adjusts the inertia weights and learning factors based on the iteration state, improving the adaptability of the particle swarm search process. Simultaneously, it uses key XGBoost hyperparameters and weights for different sentiment categories as optimization objectives, enabling the model to automatically adjust its learning strategy based on the distribution of fine-grained sentiment data. Compared to traditional manual parameter tuning or fixed-parameter search methods, this invention reduces dependence on parameter configuration, improves the adaptability of model parameter selection, and makes the model more suitable for fine-grained sentiment recognition tasks under imbalanced class conditions.

[0114] Third, improve the ability to recognize emotions in minority categories and mitigate the effects of class imbalance: To address the issue of limited sample sizes for fine-grained negative emotions such as sadness, anger, and worry, which can lead to model bias towards the majority class, this invention expands the minority class sample size through LLM text augmentation and optimizes class weights using FST-PSO. This allows the model to focus more on minority class emotional features during training. Experimental results show that, compared to traditional XGBoost and classification methods such as SVM, CNN, MLP, and Random Forest, the proposed method exhibits better overall recognition performance in fine-grained emotion classification tasks, particularly improving the accuracy and recall of minority class emotions.

[0115] Fourth, it has good generalization ability and engineering application value: This invention employs a combined approach of pre-trained models, LLM (Limited Learning Model), and machine learning models. It is not dependent on specific application scenarios or fixed data patterns; simply adjusting the training data according to the target task allows for transfer to different fine-grained emotion recognition scenarios. Furthermore, all models used have mature algorithmic foundations and favorable engineering implementation conditions, facilitating deployment and application.

[0116] Fifth, it has good scalability: The LLM text optimization strategy, multimodal feature fusion method, and FST-PSO model parameter optimization method proposed in this invention are not only applicable to fine-grained sentiment recognition tasks in short videos, but can also be extended to related fields such as online public opinion analysis, user feedback analysis, intelligent customer service, and social media sentiment computing, and have good technical promotion value.

[0117] The above embodiments are merely one of the implementation methods for achieving the technical solution of the present invention. The scope of protection claimed by the present invention is not limited to this embodiment, but also includes any variations, substitutions and other implementation methods that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. An adaptive optimization multimodal fine-grained emotion recognition method, characterized in that, include: Obtain the multimodal fusion features corresponding to the short video to be identified; The multimodal fusion features corresponding to the short video to be identified are input into the pre-trained classification model to obtain the probability results of various emotions, so as to determine the final fine-grained emotion category according to the principle of maximum probability. The training process of the classification model is as follows: A labeled dataset containing multiple fine-grained sentiments is constructed based on original short videos; the speech information of the original short videos is used for speech recognition to obtain the initial text; the initial text is semantically understood and optimized based on a large language model to obtain optimized text data; the minority class texts in the labeled dataset are expanded to obtain a training text set containing optimized text data and expanded enhanced text. Feature extraction is performed on the visual frame sequence, audio signal and training text set of the original short video respectively. The feature mapping network constructed based on each modality feature is then used to concatenate and fuse the extracted modality features to obtain the expanded multimodal fusion feature. The expanded multimodal fusion features are input into the XGBoost model with class weights; during training, the class weights of the corresponding class samples are adjusted according to the difference in the number of samples of different emotion categories. The FST-PSO algorithm is used to jointly optimize the class weights and XGBoost model hyperparameters, and the optimal class weights and XGBoost model hyperparameters are output to obtain the trained classification model.

2. The adaptive optimization multimodal fine-grained emotion recognition method according to claim 1, characterized in that, The semantic understanding and optimization processing of the initial text based on the large language model specifically includes text correction, redundancy removal, semantic completion, and sentiment enhancement. A large language model is used to augment minority texts. During the augmentation process, enhanced text is generated through text reconstruction while maintaining the consistency of the sentiment category and core semantics of the original text. The generated enhanced text reuses the visual modal information and audio modal information of the original short video as non-text modal features.

3. The adaptive optimization multimodal fine-grained emotion recognition method according to claim 1, characterized in that, When extracting features for each modality, a video self-supervised coding model is used to encode visual features for visual frame sequences, an audio pre-trained coding model is used to encode audio features for audio signals, and a pre-trained language model is used to encode text semantic features for training text sets. The feature mapping network is set independently for the three modalities of vision, audio and text, and is used to map the features of each modality in different dimensions to a unified low-dimensional feature space. After feature concatenation, standardization and normalization are performed to obtain multimodal fusion features.

4. The adaptive optimization multimodal fine-grained emotion recognition method according to claim 1, characterized in that, The hyperparameters to be optimized in the XGBoost model are divided into three categories: model structure parameters, learning parameters, and regularization parameters. The model structure parameters include the number of trees and the maximum tree depth. The learning parameters include the learning rate, the sample sampling ratio, and the feature sampling ratio; The regularization parameters include the minimum leaf node weight, L1 regularization parameter, and L2 regularization parameter.

5. The adaptive optimization multimodal fine-grained emotion recognition method according to claim 1, characterized in that, In the process of jointly optimizing the class weights and XGBoost model hyperparameters using the FST-PSO algorithm, the expanded multimodal fusion feature dataset is divided into a training subset and a validation subset. All class weights and XGBoost model hyperparameters are encoded together into a particle position vector, where each particle corresponds to a combination of class weights and XGBoost model hyperparameters. The position and velocity of the particle swarm are randomly initialized, and the initial individual optimal position and the initial global optimal position of the swarm are recorded.

6. The adaptive optimization multimodal fine-grained emotion recognition method according to claim 5, characterized in that, The specific process of joint parameter optimization is as follows: The FST-PSO algorithm uses the weighted sum of classification accuracy and macro-average F1 value as the fitness function of particles. During each iteration, the Euclidean distance between the current position of each particle and the global optimal position is calculated, and the fitness difference between the current iteration and the previous iteration is used as a fitness change factor, which is used to characterize the distance of the particle from the optimal solution region and the search effect of the particle, respectively. Input the Euclidean distance and fitness change factor into the preset fuzzy rule module, and dynamically adjust the inertial weight, individual learning factor and social learning factor of the corresponding particles according to the preset fuzzy rules. Inertia weight, individual learning factor, and social learning factor are used to control the particle's ability to maintain its current motion trend, the degree to which the particle learns from its own historical best position, and the degree to which the particle learns from the global best position, respectively. Specifically, when a particle is far from the global optimum or its fitness is not sufficiently improved, the particle's global exploration capability is enhanced; when a particle is close to the global optimum and its fitness tends to stabilize, the particle's local search capability is enhanced; the particle's velocity and position are updated based on the adjusted parameters, and parameter boundary constraints are executed. After iterating to the termination condition, the global optimum parameter combination is output to obtain the optimal class weights and XGBoost model hyperparameters.

7. The adaptive optimization multimodal fine-grained emotion recognition method according to claim 6, characterized in that, The particle velocity and position update formulas for joint parameter optimization are as follows: V i (t+1)=w i V i (t)+c soc,i r1(G(t) X i (t))+c cog,i r2(P i (t) X i (t)) X i (t+1)=X i (t)+V i (t+1) In the formula, V i (t), V i (t+1) represent the velocity vectors of the i-th particle at the t-th and t+1-th iterations, respectively; w i Indicates inertia weight; c soc,i c represents the social learning factor; cog,i G(t) represents the individual learning factor; G(t) represents the global optimal position at the t-th iteration; X ... i (t), X i (t+1) represent the positions of the i-th particle in the t-th and t+1-th iterations, respectively; P i (t) represents the optimal position of the i-th particle obtained during the historical search process in the t-th iteration; r1 and r2 represent the random number control factors, respectively.

8. An adaptive optimization multimodal fine-grained emotion recognition system, characterized in that, include: The acquisition module is used to acquire the multimodal fusion features corresponding to the short video to be identified; The recognition module is used to input the multimodal fusion features corresponding to the short video to be recognized into the pre-trained classification model to obtain the probability results of various emotions, so as to determine the final fine-grained emotion category according to the principle of maximum probability. The training process of the classification model is specifically implemented as follows: A labeled dataset containing multiple fine-grained sentiments is constructed based on original short videos; the speech information of the original short videos is used for speech recognition to obtain the initial text; the initial text is semantically understood and optimized based on a large language model to obtain optimized text data; the minority class texts in the labeled dataset are expanded to obtain a training text set containing optimized text data and expanded enhanced text. Feature extraction is performed on the visual frame sequence, audio signal and training text set of the original short video respectively. The feature mapping network constructed based on each modality feature is then used to concatenate and fuse the extracted modality features to obtain the expanded multimodal fusion feature. The expanded multimodal fusion features are input into the XGBoost model with class weights; during training, the class weights of the corresponding class samples are adjusted according to the difference in the number of samples of different emotion categories. The FST-PSO algorithm is used to jointly optimize the class weights and XGBoost model hyperparameters, and the optimal class weights and XGBoost model hyperparameters are output to obtain the trained classification model.

9. An adaptive optimization multimodal fine-grained emotion recognition device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the adaptive optimization multimodal fine-grained emotion recognition method according to any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it is used to implement the adaptive optimization multimodal fine-grained emotion recognition method according to any one of claims 1-7.