Emotion recognition method and device, computer equipment and storage medium

Abstract

The invention provides an emotion recognition method and device, computer equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining multi-modal data of a pet; performing feature extraction on the multi-modal data to obtain a feature vector of each modal; determining relevancy among the modals by using the feature vectors, and performing feature aggregation on the feature vectors by using a pre-trained graph neural network and the relevancy to obtain enhanced feature vectors of the modals; and performing attention fusion on each enhanced feature vector to obtain a fusion feature vector, and performing emotion recognition according to the fusion feature vector to obtain a target emotion type of the pet. Therefore, the relevancy between the modals can be determined by utilizing the feature vectors of the modals extracted from the multi-modal data, and feature enhancement is performed on the feature vectors of the modals according to the relevancy, so that the relevancy between the pet multi-modal data is fully mined, and the accuracy of pet emotion recognition is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to an emotion recognition method, apparatus, computer device, and storage medium. Background Technology

[0002] As more and more families choose to keep pets, how to understand their pets' emotions more conveniently and quickly has gradually become a focus for pet owners.

[0003] In related technologies, the identification of pet emotions fails to fully utilize the multimodal data collected from pets, which is not conducive to improving the accuracy of pet emotion identification. Summary of the Invention

[0004] The purpose of this invention is to provide an emotion recognition method, apparatus, computer device, and storage medium that can determine the correlation between modalities by using feature vectors of each modality extracted from multimodal data, and perform feature aggregation enhancement on the feature vectors of each modality based on the correlation, so as to fully explore the correlation between pet multimodal data and thus improve the accuracy of pet emotion recognition.

[0005] To address the aforementioned technical problems, this invention provides an emotion recognition method, comprising:

[0006] Acquire multimodal data about pets;

[0007] Feature extraction is performed on the multimodal data to obtain feature vectors for each modality;

[0008] The correlation between each modality is determined using the feature vectors, and the feature vectors are aggregated using a pre-trained graph neural network and the correlation to obtain the enhanced feature vectors of each modality.

[0009] Attention fusion is performed on each of the enhanced feature vectors to obtain a fused feature vector, and emotion recognition is performed based on the fused feature vector to obtain the target emotion type of the pet.

[0010] Optionally, the correlation between modalities is determined using the feature vectors, including:

[0011] The feature vectors are used to set the feature vector pairs between the modes;

[0012] The feature vectors are input into a pre-trained attention network to obtain attention scores between different modalities.

[0013] For each target mode, the attention scores between the target mode and other modes are processed using a normalized exponential function to obtain the correlation between the target mode and other modes.

[0014] Optionally, each graph node in the graph neural network corresponds to a different mode, and the graph nodes are interconnected.

[0015] By using a pre-trained graph neural network and the aforementioned relevance, feature aggregation is performed on each feature vector to obtain enhanced feature vectors for each modality, including:

[0016] An adjacency matrix is ​​set according to the relevance; wherein the adjacency matrix contains the weight values ​​of the edges between each graph node, and the weight values ​​are the relevance.

[0017] The feature vectors of each modality are input to the graph nodes corresponding to each modality, and the graph neural network is used to aggregate the features of each graph node based on the adjacency matrix to obtain the enhanced feature vectors of each modality.

[0018] Optionally, the multimodal data includes audio data, physiological data, behavioral data, and environmental data, and the feature vector includes audio features corresponding to the audio data, fluctuation pattern features corresponding to the physiological data, motion features corresponding to the behavioral data, and scene features corresponding to the environmental data.

[0019] Optionally, attention fusion is performed on each of the enhanced feature vectors to obtain a fused feature vector, and emotion recognition is performed based on the fused feature vector to obtain the target emotion type of the pet, including:

[0020] The attention weights of each modality are obtained by using a pre-trained multi-head attention module to process each of the enhanced feature vectors through an attention mechanism.

[0021] The enhanced feature vectors of each modality are weighted and fused using the attention weights of each modality to obtain the fused feature vector;

[0022] The predicted probabilities of the pet in various emotion types are determined by using a pre-trained classifier and the fused feature vector, and the emotion type corresponding to the highest predicted probability is set as the target emotion type.

[0023] Optionally, it also includes:

[0024] The first variance is determined using the attention weights of each modality;

[0025] When the maximum predicted probability and the first variance meet the first preset condition, the target emotion type is labeled for the multimodal data to obtain training data; wherein, the first preset condition is that the maximum predicted probability is greater than the first preset threshold and the variance is less than or equal to the second preset threshold.

[0026] When the maximum prediction probability and the first variance meet the second preset condition, the prediction probability and the attention weight are used to generate confirmation information and output it for display. The input user-labeled emotion type is received, and the user-labeled emotion type is labeled for the multimodal data to obtain training data. The second preset condition is that the maximum prediction probability is greater than the third preset threshold and less than or equal to the first preset threshold, or the variance is greater than the second preset threshold and the third preset threshold is less than the first preset threshold.

[0027] When the amount of training data reaches a preset amount, incremental training of the model is performed using the training data.

[0028] Optionally, incremental model training is performed using the training data, including:

[0029] Feature extraction is performed on the multimodal data in the training data to obtain the training enhancement feature vectors for each modality;

[0030] The multi-head attention module is used to process each of the trained enhanced feature vectors using an attention mechanism to obtain the initial attention weights for each modality;

[0031] The environmental data, physiological data, and behavioral data are used to generate a context state. The state-action value function is used to determine the attention weight adjustment value of each modality based on the context state. The initial attention weight is then adjusted using the attention weight adjustment value to obtain the adjusted attention weight of each modality.

[0032] The training enhancement feature vectors of each modality are weighted and fused using the adjusted attention weights to obtain the training fused feature vector;

[0033] The predicted emotion type of the pet is determined based on the training fusion feature vector. An incremental loss value and a reward value are determined based on the predicted emotion type and the emotion type labeled by the multimodal data. The multi-head attention module is updated using the incremental loss value, and the state-action value function is updated using the reward value.

[0034] Optionally, when the amount of training data reaches a preset amount, incremental model training is performed using the training data, including:

[0035] The training data is uploaded to the cloud device so that when the cloud device determines that the amount of training data has reached a preset amount, it uses the training data to perform incremental training on the teacher model, distills the trained teacher model into a student model, and distributes the student model.

[0036] Receive the student model and update the local model using the student model.

[0037] Optionally, the process of distilling the trained teacher model into a student model includes:

[0038] The training data is input into the teacher model and the student model respectively to obtain the teacher model output data and the student model output data.

[0039] The incremental loss is determined based on the student model output data, and the distribution difference value between the teacher model output data and the student model output data is determined.

[0040] The distillation loss is determined using the incremental loss and the distribution difference value, and the student model is updated using the distillation loss.

[0041] Optionally, feature extraction is performed on the multimodal data to obtain feature vectors for each modality, including:

[0042] The audio data is converted into Mel spectrum, and the physiological data, behavioral data, and environmental data are normalized.

[0043] The audio features are obtained by extracting features from the Mel spectrum using a pre-trained audio feature extraction model.

[0044] The physiological data are used to extract features using a pre-trained temporal convolutional model to obtain the fluctuation pattern features;

[0045] The motion features are obtained by extracting features from the behavioral data using a pre-trained temporal convolutional model.

[0046] The environmental data is used to extract features using a pre-trained fully connected layer to obtain the scene features.

[0047] Optionally, it also includes:

[0048] Determine whether the multimodal data meets the labeling conditions;

[0049] If the labeling conditions are met, the multimodal data is used to generate and output labeling prompts. User labeling information is accepted as input, and the user labeling information is added to the multimodal data to obtain transfer training data.

[0050] When it is determined that the amount of transfer training data has reached a preset amount, the pre-trained model is transferred and trained using the transfer training data.

[0051] Based on the pre-trained model that has completed transfer training, the step proceeds to extract features from the multimodal data to obtain feature vectors for each modality.

[0052] Optionally, determining whether the multimodal data meets the labeling conditions includes:

[0053] The second variance is determined using the physiological data, the behavioral data, and the environmental data, and then mapped to the 0~1 interval to obtain the recommendation label score;

[0054] When the recommended labeling score is determined to be greater than a preset value, the multimodal data is deemed to have met the labeling conditions.

[0055] The present invention also provides an emotion recognition device, comprising:

[0056] The acquisition module is used to acquire multimodal data about pets.

[0057] The feature extraction module is used to extract features from the multimodal data to obtain feature vectors for each modality;

[0058] The feature aggregation module is used to determine the correlation between each modality using the feature vectors, and to perform feature aggregation on each feature vector using a pre-trained graph neural network and the correlation to obtain the enhanced feature vectors of each modality.

[0059] The emotion recognition module is used to perform attention fusion on each of the enhanced feature vectors to obtain a fused feature vector, and to perform emotion recognition based on the fused feature vector to obtain the target emotion type of the pet.

[0060] The present invention also provides a computer device, comprising:

[0061] Memory, used to store computer programs;

[0062] A processor for implementing the emotion recognition method as described above when executing the computer program.

[0063] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the emotion recognition method described above.

[0064] This invention provides an emotion recognition method, comprising: acquiring multimodal data of a pet; extracting features from the multimodal data to obtain feature vectors for each modality; determining the correlation between each modality using the feature vectors, and aggregating the features of each feature vector using a pre-trained graph neural network and the correlation to obtain enhanced feature vectors for each modality; performing attention fusion on each enhanced feature vector to obtain a fused feature vector, and performing emotion recognition based on the fused feature vector to obtain the target emotion type of the pet.

[0065] The beneficial effects of this invention are as follows: First, it can acquire multimodal data of pets and extract features from the multimodal data to obtain feature vectors for each modality. Then, it can use the feature vectors to determine the correlation between each modality, and use a pre-trained graph neural network and the correlation to aggregate features from each feature vector, obtaining enhanced feature vectors for each modality. This means it can dynamically determine the correlation between each modality and use a graph neural network to aggregate features from each feature vector based on this correlation, thereby fully exploring the correlation between pet multimodal data. Subsequently, attention fusion can be performed on each enhanced feature vector to obtain a fused feature vector, and the target emotion type of the pet can be determined based on the fused feature vector, thus effectively improving the accuracy of pet emotion recognition.

[0066] The present invention also provides an emotion recognition device, a computer device, and a computer-readable storage medium, which have the above-mentioned beneficial effects. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0068] Figure 1 A flowchart of an emotion recognition method provided in an embodiment of the present invention;

[0069] Figure 2 A flowchart of another emotion recognition method provided in an embodiment of the present invention;

[0070] Figure 3 This is a structural block diagram of an emotion recognition device provided in an embodiment of the present invention;

[0071] Figure 4 This is a structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] As more and more families choose to keep pets, understanding their pets' emotions more conveniently and quickly has become a major concern for pet owners. However, current technologies for pet emotion recognition fail to fully utilize multimodal data collected from pets, hindering the improvement of accuracy in this area.

[0074] In view of this, to address the technical problem of how to improve the accuracy of pet emotion recognition, the present invention provides an emotion recognition method that can determine the correlation between modalities by using feature vectors of each modality extracted from multimodal data, and perform feature aggregation enhancement on the feature vectors of each modality based on the correlation, so as to fully explore the correlation between pet multimodal data and thus improve the accuracy of pet emotion recognition.

[0075] It should be noted that this embodiment does not limit the execution device of this method. For example, it can be a mobile terminal (such as a mobile phone or tablet), a computer, a server, or a wearable device (such as a smart collar for a pet). In addition, this method can also be completed by multiple devices working together. For example, a wearable device can collect data, and a mobile terminal can perform emotion recognition on the data collected by the wearable device to obtain the target emotion category of the pet.

[0076] For easier understanding, please refer to Figure 1 , Figure 1 A flowchart of an emotion recognition method provided in an embodiment of the present invention, the method may include:

[0077] S11. Obtain multimodal data of the pet.

[0078] In this step, multimodal data about the pet can be collected first to identify the pet's emotions from a multimodal perspective. This multimodal data may include, for example, audio data (44.1kHz sampling, capturing sound signals such as barking and whimpering), physiological data (such as heart rate and respiratory rate, 1Hz sampling, reflecting the pet's physiological state), behavioral data (such as accelerometer-generated motion sequences, such as running and standing, reflecting movement patterns), and environmental data (such as temperature, humidity, light intensity, and air quality, collected at low frequencies to provide context). This multimodal data can be collected by wearable devices worn by the pet (such as smart collars).

[0079] S12. Perform feature extraction on the multimodal data to obtain the feature vectors of each modality.

[0080] In this step, the features of each multimodal data will be extracted to obtain the feature vector of each modality. This feature vector may include, for example, audio features corresponding to audio data, wave pattern features corresponding to physiological data, motion features corresponding to behavioral data, and scene features corresponding to environmental data.

[0081] To achieve feature extraction, multimodal data must first be preprocessed. For example, for audio data, time-frequency conversion is required to obtain the corresponding Mel spectrum, which can then be used for feature extraction using a neural network model. For physiological, behavioral, and environmental data, normalization is necessary to eliminate dimensional differences between different modalities and different pets, mapping their values ​​to the [0,1] range to ensure cross-modal and cross-individual consistency. Additionally, timestamp synchronization can be performed on the preprocessed multimodal data to ensure the accuracy of subsequent feature extraction and fusion.

[0082] Furthermore, this embodiment can use a pre-trained feature extraction model to extract features from various multimodal data. For example, for audio data, it can be processed using a Convolutional Neural Network (CNN) or Transformer model to generate vectors that capture emotional features such as pitch and rhythm. For physiological data (heart rate, respiratory rate) and behavioral data (acceleration sequences), it can be processed using a Temporal Convolutional Network (TCN) to extract time-related fluctuation patterns and action features, such as heart rate acceleration or rapid running reflecting an excited state. For environmental data, scene features can be generated through fully connected layers to capture contextual information such as temperature, humidity, and lighting.

[0083] In one implementation, feature extraction of multimodal data to obtain feature vectors for each modality may include:

[0084] Step 121: Convert the audio data into Mel spectrum and normalize the physiological, behavioral, and environmental data.

[0085] The conversion process of the Mel spectrum can be represented as:

[0086] ;

[0087] in, This is the time-domain representation of audio data, measured in amplitude, reflecting the characteristics of a pet's voice. FFT (Fast Fourier Transform) is used to convert audio signals into a frequency-domain representation. This represents the weights (matrix) of the Mel filter bank, used to map frequencies to the Mel scale, which can simulate the auditory characteristics of animals. This represents a Mel spectrogram, which can have dimensions of 128×100, used to capture emotion-related features such as pitch and rhythm.

[0088] Normalization can be expressed as:

[0089] ;

[0090] in, The numbers represent the raw signal. H represents heart rate data in bpm. R represents respiratory data in breaths per minute. A represents acceleration data in m / s². E represents environmental data, such as temperature, humidity, and other physical quantities. This represents the minimum / maximum value in the dataset (e.g., heart rate 50-200 bpm), used for standardization. This represents a normalized sequence with a value range of [0,1] and a dimension of 60 (1Hz × 60 seconds), ensuring consistency across pets.

[0091] Step 122: Use a pre-trained audio feature extraction model to extract features from the Mel spectrum to obtain audio features.

[0092] Step 123: Use a pre-trained temporal convolutional model to extract features from physiological data to obtain fluctuation pattern features;

[0093] Step 124: Use a pre-trained temporal convolutional model to extract features from the behavioral data to obtain motion features;

[0094] Step 125: Use a pre-trained fully connected layer to extract features from the environmental data to obtain scene features.

[0095] The extraction process of each feature vector can be represented as follows:

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] in, This represents a Mel spectrogram with dimensions of 128×100. Features are extracted using an audio feature extraction model, which can be a CNN model or a Transformer model. Normalized sequences of heart rate data, respiratory data, acceleration data, and environmental data, with a dimension of 60, were used for feature extraction using TCN or fully connected layers. These represent the pre-trained model parameters for the audio feature extraction model, the temporal convolutional model, and the fully connected layer, respectively, used to optimize the feature extraction process. These represent audio features, fluctuation pattern features, motion features, and scene features, respectively, with a dimension of up to 128, capturing emotion-related patterns.

[0101] S13. Use the feature vectors to determine the correlation between each modality, and use the pre-trained graph neural network and the correlation to perform feature aggregation on each feature vector to obtain the enhanced feature vector of each modality.

[0102] After extracting the features of each multimodal data, considering that the pet's physiological state, movement state, and emotional state are constantly changing, and therefore the correlation between the pet's various modal data is also constantly changing, in order to fully explore the correlation between multimodal data, this step will dynamically calculate the correlation between each modality, and perform feature aggregation on each feature vector based on the correlation, so as to enhance the expressive power of the feature vector and improve the accuracy of pet emotion recognition.

[0103] The following section first introduces how to determine the correlation between modes:

[0104] In one implementation, determining the correlation between modes using feature vectors may include:

[0105] Step 1311: Use eigenvectors to set up eigenvector pairs between modes.

[0106] In this step, feature vector pairs can first be set using the feature vectors of each modality. For example, for audio data + behavioral data (audio, behavior), the corresponding feature vector pair can be: .

[0107] Step 1312: Input the feature vectors into the pre-trained attention network to obtain the attention scores between each modality.

[0108] In this step, for each pair of modalities, this embodiment can input its corresponding feature vectors into a pre-trained attention network to extract the correlation between the feature vectors and obtain the attention score between the modalities. For example, for (audio, behavior), its attention score can be expressed as:

[0109] ;

[0110] in, This represents the attention score between audio and behavioral data, where tanh represents the hyperbolic tangent function. This represents an attention network.

[0111] Similarly, the attention score between audio data and physiological data can be calculated. Attention score between audio data and environmental data Attention scores between audio data and the user's own attention level All scores.

[0112] Step 1313: For each target mode, use the normalized exponential function to process the attention scores between the target mode and other modes to obtain the correlation between the target mode and other modes.

[0113] In this step, for each of the four target modalities—audio, physiological, behavioral, and environmental—a normalized exponential function can be used to process the attention scores between the target modality and the other modalities, yielding the correlation between the target modality and the other modalities. Taking audio data as an example, its correlation with behavioral data can be expressed as:

[0114] ;

[0115] in, exp represents the natural exponential function.

[0116] Similarly, the correlation between audio data and physiological data can be calculated. Correlation between audio data and environmental data The correlation between audio data and itself The correlation calculation method for other modalities is similar.

[0117] Furthermore, this embodiment can introduce a Graph Neural Network (GNN) to aggregate features from the feature vectors of each modality based on the correlation between them, thereby enhancing feature representation capabilities. To this end, this embodiment can set up multiple graph nodes in the GNN, each corresponding to a different modality. These graph nodes are interconnected to enable feature interaction between modalities. The GNN includes an adjacency matrix to record the weights of the edges between graph nodes. These weights indicate the correlation between graph nodes, and in this embodiment, they can be set as the correlation between modalities. Since this embodiment dynamically calculates the correlation based on the multimodal data, it can adjust the adjacency matrix of the GNN after completing the correlation calculation before performing feature aggregation.

[0118] In one implementation, a pre-trained graph neural network and relevance are used to perform feature aggregation on each feature vector to obtain enhanced feature vectors for each modality, including:

[0119] Step 1321: Set the adjacency matrix according to the relevance; where the adjacency matrix contains the weight values ​​of the edges between each graph node, and the weight values ​​are the relevance.

[0120] Step 1322: Input the feature vectors of each modality into the graph nodes corresponding to each modality, and use the graph neural network to perform feature aggregation on each graph node based on the adjacency matrix to obtain the enhanced feature vectors of each modality.

[0121] Specifically, the process of a graph neural network on feature vectors can be represented as follows:

[0122] ;

[0123] in, This represents the modal features of the l-th layer of the graph neural network, and the initial state of the 0th layer is the feature extraction output (e.g., ...). wait). This represents the set of neighboring nodes of the i-th graph node. In this embodiment, it can be understood as the set of neighboring nodes containing other modes besides the i-th mode. Represents the adjacency matrix, which is based on the correlation between modes (e.g. (This indicates a high correlation between audio and behavior) Dynamic settings. These represent the weights and biases of the graph neural network, respectively, used to optimize the modeling of intermodal relationships. This represents the ReLU activation function (Rectified Linear Unit), which enhances expressiveness by introducing nonlinearity. This indicates that the enhanced feature vector can incorporate intermodal relationship information, thereby improving the feature representation capability.

[0124] S14. Attention fusion is performed on each enhanced feature vector to obtain a fused feature vector, and emotion recognition is performed based on the fused feature vector to obtain the target emotion type of the pet.

[0125] In this step, after the feature vector enhancement is completed, an attention mechanism can be used to fuse the enhanced feature vectors to obtain a fused feature vector. Then, emotion recognition can be performed based on the fused feature vector to determine the pet's target emotion type from a variety of preset emotion types.

[0126] To improve the attention fusion effect, this embodiment can introduce a multi-head attention module (MultiHead) to calculate the attention weights of each modality, and use these attention weights to fuse the enhanced feature vectors of each modality. Additionally, this embodiment can also introduce a classifier to perform emotion recognition based on the fused feature vectors to obtain the target emotion type of the pet.

[0127] In one implementation, attention fusion is performed on each enhanced feature vector to obtain a fused feature vector, and emotion recognition is performed based on the fused feature vector to obtain the target emotion type of the pet, which may include:

[0128] Step 141: Use a pre-trained multi-head attention module to process each enhanced feature vector using an attention mechanism to obtain the attention weights for each modality.

[0129] Attention weights can be represented as:

[0130] ;

[0131] in, Represents a query, key, and value vector, by Generated through a linear transformation, with a dimension of 128. MultiHead represents a multi-head attention mechanism used to capture complex relationships between modalities. Represents modal weights (e.g.) The attention weights of all modalities sum to 1, representing the contribution of each modality to the fusion.

[0132] Step 142: Use the attention weights to perform weighted fusion of the enhanced feature vectors of each modality to obtain the fused feature vector.

[0133] The fused feature vector can be represented as:

[0134] ;

[0135] in, This represents the fusion feature, which can have 128 dimensions and integrates multimodal information. The graph neural network represents the enhanced feature vector generated for the i-th modality, and its dimension can be 128. This represents the attention weight for the i-th modality.

[0136] Step 143: Use a pre-trained classifier and fused feature vectors to determine the predicted probability of the pet in multiple emotion types, and set the emotion type corresponding to the highest predicted probability as the target emotion type.

[0137] The process of determining the probability distribution of emotions can be represented as follows:

[0138] ;

[0139] in, This represents the weights and biases of the fully connected layers in the classifier, used to... Mapped to specific emotion categories (such as happiness, fear). This represents the probability distribution of emotions corresponding to various preset emotion types (e.g., "happy: 85%").

[0140] Based on the above embodiments, the present invention first acquires multimodal data of pets and extracts features from the multimodal data to obtain feature vectors for each modality. Subsequently, the present invention uses the feature vectors to determine the correlation between each modality, and uses a pre-trained graph neural network and the correlation to perform feature aggregation on each feature vector, obtaining enhanced feature vectors for each modality. That is, it can dynamically determine the correlation between each modality and use a graph neural network to perform feature aggregation on each feature vector based on this correlation, thereby fully exploring the correlation between pet multimodal data. Then, attention fusion can be performed on each enhanced feature vector to obtain a fused feature vector, and the target emotion type of the pet can be determined based on the fused feature vector, thereby effectively improving the accuracy of pet emotion recognition.

[0141] Based on the above embodiments, considering that a pet's physiological, behavioral, and emotional states will slowly change over time, leading to model aging and an inability to dynamically adapt to changes in the pet's state, thus causing a decrease in model accuracy, this embodiment can also acquire some high-quality multimodal data from the pet emotion recognition process, or acquire some manually labeled multimodal data that the model struggles to distinguish, as training data, and periodically use this training data for incremental model training, thereby effectively maintaining model performance. The process of collecting training data is described below.

[0142] In one embodiment, the method may further include:

[0143] S21. Determine the first variance using the attention weights of each modality.

[0144] In this step, after the multi-head attention module outputs the attention weights for each modality, this embodiment can use the attention weights of each modality to determine the first variance. The value of the first variance reveals various information. For example, a small first variance indicates that the contributions of each modality are equal, and the correlation between modalities can be fully explored; conversely, a large first variance indicates that the contributions of each modality are uneven, which may indicate insufficient exploration of the correlation between modalities or improper allocation of attention weights.

[0145] S22. When the maximum prediction probability and the first variance meet the first preset condition, the target emotion type is labeled for the multimodal data to obtain training data; wherein, the first preset condition is that the maximum prediction probability is greater than the first preset threshold and the variance is less than or equal to the second preset threshold.

[0146] In this embodiment, the maximum predicted probability output by the classifier and the first variance calculated above are used to filter the training data. In this step, if it is determined that the maximum predicted probability is greater than a first preset threshold and the variance is less than or equal to a second preset threshold, for example when... and When the model is accurate in detecting the pet's emotional type, it can be determined that the target emotional type identified by the model can be labeled for multimodal data to obtain training data.

[0147] Of course, the first and second preset thresholds mentioned above can also be set to other values, which can be set according to actual application needs.

[0148] S23. When the maximum prediction probability and the first variance meet the second preset condition, the confirmation information is generated and displayed using the prediction probability and attention weight. The user-labeled emotion type is received and the user-labeled emotion type is labeled for the multimodal data to obtain training data. The second preset condition is that the maximum prediction probability is greater than the third preset threshold and less than or equal to the first preset threshold, or the variance is greater than the second preset threshold and the third preset threshold is less than the first preset threshold.

[0149] In this step, if the maximum predicted probability is determined to be greater than the third preset threshold and less than or equal to the first preset threshold, or the variance is greater than the second preset threshold, for example when or At this point, the model struggles to identify pet emotion types. For such data, this embodiment can request confirmation from the user. For example, confirmation information can be generated using predicted probabilities and attention weights, and sent to the user via a mobile application, prompting the user to verify or correct. Users can annotate emotions via voice, text, or gestures. Furthermore, upon receiving user-annotated emotion types, the multimodal data can be labeled with user-labeled emotion types to obtain training data.

[0150] Of course, the third preset threshold can also be set to other values, as long as the third preset threshold is less than the first preset threshold. It can be set according to the actual application requirements.

[0151] In simple terms, in generating the probability distribution of emotions Then, the confidence level can be calculated. Attention weight variance If satisfied or Then set a conditional push notification identifier. The system pushes a confirmation request to the user via the mobile app. The push notification includes the current emotion probability (e.g., "System judges 'fear' 85%) and a visualization of modality weights (e.g., "Audio contributes 60%, physiological 20%)", prompting the user to verify or correct. The user responds with the target emotion via voice, text, or gesture. The system receives this as a temporary tag. Push notifications are only triggered in modal conflict or medium-confidence scenarios. In high-confidence (>90%) or low-confidence scenarios, No user interaction is required. This step ensures real-time performance and accuracy through conditional push notifications and visual feedback, enhancing the closed-loop interaction capability between the system and the user.

[0152] The conditional push notification setting can be represented as:

[0153] ;

[0154] in, This indicates a conditional push notification flag (1 = push notification, 0 = no push notification). It represents the maximum emotional probability and measures confidence level. This represents the variance of attention weights, which measures modal consistency (high variance indicates conflict).

[0155] Furthermore, in Scene, As high-quality user tags, they are preferentially stored in the buffer (with confidence levels considered to be 100%) and participate in incremental model learning. If Only when And the modes are consistent ( )hour, Turn to Store in the buffer.

[0156] S24. When the amount of training data reaches the preset amount, incremental training of the model is performed using the training data.

[0157] In this step, when the amount of training data reaches a preset amount (e.g., 100), incremental training of the model can be performed using the training data to maintain model performance. Incremental training can be performed during the model's idle time, such as at night.

[0158] Based on the above embodiments, to improve the allocation effect of attention weights, reinforcement learning can be introduced in this embodiment to guide the multi-head attention module to update parameters during model training. The training process of the multi-head attention module is described below. In one implementation, incremental model training using training data may include:

[0159] S31. Extract features from the multimodal data in the training data to obtain the training enhancement feature vectors for each modality.

[0160] The description of step S31 is similar to that of steps S12 to S13, and will not be repeated here.

[0161] S32. Use a multi-head attention module to process each training enhancement feature vector using an attention mechanism to obtain the initial attention weights for each modality.

[0162] In this step, unlike the above embodiments, the attention weights output by the multi-head attention module will be used as initial attention weights, waiting for adjustment by the reinforcement learning module, instead of being directly used to fuse training feature vectors.

[0163] S33. Generate contextual states using environmental data, physiological data, and behavioral data. Use the state-action value function to determine the attention weight adjustment value for each modality based on the contextual state. Adjust the initial attention weights using the attention weight adjustment value to obtain the adjusted attention weights for each modality.

[0164] In this step, a reinforcement learning module can be introduced to adjust attention weights based on the contextual state comprised of environmental, physiological, and behavioral data, thereby guiding the parameter updates of the multi-head attention module. The reinforcement learning (Q-learning) module can dynamically optimize attention weights based on real-time context (such as environmental noise levels, heart rate fluctuations, and behavioral activity). For example, in a high-noise environment, the system reduces the weight of the audio modality and increases the importance of physiological and behavioral modalities to adapt to scene changes.

[0165] Specifically, the core of the reinforcement learning module is the state-action value function. This function evaluates the expected reward of taking various actions in the current context state and selects the action with the highest expected reward as the final target action. In this embodiment, the action is a weight allocation action, which provides the attention weight adjustment value corresponding to each modality. Furthermore, this step can use the attention weight adjustment value to adjust the initial attention weight of each modality to obtain the adjusted attention weight of each modality.

[0166] S34. Use the adjusted attention weights to weight and fuse the training enhancement feature vectors of each modality to obtain the training fusion feature vector.

[0167] In this step, the training enhancement feature vectors of each modality will be weighted and fused using the adjusted attention weights to obtain the training fused feature vector.

[0168] The trained fusion feature vector can be represented as:

[0169] ;

[0170] in, The optimized weights are adjusted by reinforcement learning.

[0171] S35. Determine the predicted emotion type of the pet based on the training fusion feature vector, determine the incremental loss value and reward value based on the predicted emotion type and the emotion type labeled by the multimodal data, update the multi-head attention module using the incremental loss value, and update the state-action value function using the reward value.

[0172] In this step, the multi-head attention module and the state-action value function will be updated separately. For the multi-head attention module, since the state-action value function has already adjusted the attention weights assigned to it, and this adjustment will ultimately be reflected in the model's prediction results, the incremental loss value can be determined based on the predicted emotion type given by the model and the emotion type labeled by the multimodal data. The incremental loss value is then used to update the parameters of the multi-head attention module, so as to achieve the effect of guiding the multi-head attention module to learn using reinforcement learning.

[0173] Specifically, the incremental loss value can be expressed as:

[0174] ;

[0175] in, The label represents the true emotion, derived from user feedback or high-confidence predictions (such as "happy"). This represents the probability predicted by the model. N represents the amount of training data (e.g., 100), and represents the number of samples used for updates.

[0176] For the state-action value function, this embodiment will use reinforcement learning to update it. Specifically, the update process of the state-action value function can be represented as follows:

[0177] ;

[0178] in, State-action value function, guiding Optimization. 's' represents the context state, determined by... The data is spliced ​​together to reflect environmental noise, heart rate fluctuations, etc. 'a' represents a weighting action (such as increasing...). Adjust modal importance. R represents the reward, based on prediction accuracy (e.g., consistency with user feedback). ). This represents the learning rate (0.01), which controls the optimization speed. This represents a discount factor (0.9) to balance long-term rewards.

[0179] Based on the above embodiments, this embodiment can also implement online incremental learning, using cloud devices to incrementally train the model, and through a model distillation mechanism, compress the large teacher model into a smaller student model, and then distribute the student model to local devices (such as to wearable devices for pets), thereby saving computing resources on edge devices and achieving better incremental training results. The process of online incremental learning and knowledge distillation is described below.

[0180] In one implementation, when the amount of training data reaches a preset amount, incremental model training using the training data may include:

[0181] S41. Upload the training data to the cloud device so that when the amount of training data reaches the preset amount, the cloud device uses the training data to incrementally train the teacher model, distills the trained teacher model into a student model, and distributes the student model.

[0182] In this embodiment, after collecting training data, the local device can upload the training data to the cloud device. When the cloud device determines that the amount of training data has reached a preset amount, it can first use the training data to incrementally train the teacher model. Subsequently, the cloud device can distill the trained teacher model into a student model, which allows knowledge to be transferred from the large cloud model (teacher) to the small edge model (student), compressing the model size by about 50%, reducing the computational overhead of the local device (such as a smart collar), while retaining high accuracy. Finally, the cloud device can distribute the student model to the local device.

[0183] In one implementation, distilling the trained teacher model into a student model may include:

[0184] Step 411: Input the training data into the teacher model and the student model respectively to obtain the output data of the teacher model and the output data of the student model;

[0185] Step 412: Determine the incremental loss based on the student model output data, and determine the distribution difference between the teacher model output data and the student model output data;

[0186] Step 413: Determine the distillation loss using the incremental loss and distribution difference value, and update the student model using the distillation loss.

[0187] Specifically, distillation loss can be expressed as:

[0188] ;

[0189] in, Indicates distillation loss, This represents the output of the teacher model, which is... , indicating a high-precision predicted distribution. This represents the student model output, aiming to approximate the teacher model distribution. This represents the KL divergence, which measures the difference between the output distributions of the teacher and student models. This represents the balance factor (e.g., 0.7), which weighs distillation loss against classification loss. This represents the incremental learning loss, ensuring classification accuracy.

[0190] After obtaining the distillation loss, the parameters of the student model can be updated in the following manner in this embodiment:

[0191] ;

[0192] in, The parameters represent the edge model, which may include the feature extraction model parameters. and classifier parameters . This represents the learning rate (0.0001), controlling the step size for parameter updates. The gradient represents the knowledge distillation loss and drives the optimization of the edge model.

[0193] S42. Receive the student model and update the local model using the student model.

[0194] Based on the above embodiments, due to the differences in physiological, motor, and emotional expressions among various pets, and the difficulty of adapting general pre-trained models to these differences, directly using general pre-trained models for emotion recognition of specific pets often results in low accuracy and poor performance. To address this, this embodiment can also guide users to annotate a certain amount of data during initial use and utilize this data for transfer training of the general pre-trained model to ensure the model is adapted to the user's pet. The transfer training process is described below. In one possible implementation, this method may further include:

[0195] S51. Determine whether the multimodal data meets the labeling conditions.

[0196] In this embodiment, after collecting multimodal data, this step can determine whether the data meets the annotation conditions to guide the user to annotate at the optimal time. This embodiment can recommend the optimal annotation time by analyzing the fluctuations of real-time physiological and behavioral data. For example, when high heart rate and rapid movement are detected, the system prompts the user to annotate "excitement" to ensure that the sample captures significant emotional features.

[0197] Specifically, this embodiment can calculate the variance (i.e., the second variance) of physiological data, behavioral data, and environmental data, and map these variances to the 0-1 interval to eliminate the influence of dimensions. Subsequently, this embodiment can use the mapped value as a recommended labeling score and determine whether the score is greater than a preset value. If so, it can be determined that the pet has significant emotional characteristics, and the multimodal data meets the labeling conditions.

[0198] In one implementation, determining whether multimodal data meets the labeling criteria may include:

[0199] Step 511: Determine the second variance using physiological, behavioral, and environmental data, and map the second variance to the 0-1 interval to obtain the recommended label score;

[0200] Step 512: When the recommended labeling score is determined to be greater than the preset value, the multimodal data is deemed to have met the labeling conditions.

[0201] Specifically, the recommended rating score can be expressed as:

[0202] ;

[0203] in, This indicates the recommended rating score. For example, if... If the value is greater than 0.8, the app will prompt the user to label it (e.g., "Excitement detected, it is recommended to label it 'happy'"). Normalized sequences representing heart rate, respiration, and acceleration reflect physiological and behavioral fluctuations. It represents variance and quantifies the intensity of signal fluctuations (e.g., high heart rate and rapid movement indicate excitement). This represents the Sigmoid activation function, used to map variance to the range [0,1].

[0204] S52. If the annotation conditions are met, use the multimodal data to generate annotation prompts and output them. Accept the input user annotation information, add user annotation information to the multimodal data, and obtain transfer training data.

[0205] S53. When the amount of transfer training data reaches the preset amount, use the transfer training data to perform transfer training on the pre-trained model.

[0206] This step involves users labeling 10-20 emotion samples (including audio clips and behavioral data) on the mobile app, and then using transfer learning to generate a personalized initial model that can be quickly adapted to new users' pets.

[0207] Furthermore, during transfer learning, the backbone network of the feature extraction module can be frozen, with only the classifier and attention modules fine-tuned, reducing data requirements and improving initialization efficiency. The optimized model parameters are then fed back to the feature extraction and fusion modules, ensuring personalized effects are maintained throughout the system. This reduces the initial false recognition rate.

[0208] Specifically, the transfer learning loss can be expressed as:

[0209] ;

[0210] in, This represents the user's actual emotional label (such as "happy"), generated based on app interaction. Represents the probability distribution of emotions , representing the probability predicted by the model. This indicates the number of labeled samples (10-20), representing the amount of personalized data provided by the user. This represents the fine-tuned model parameters, including the classifier and attention module parameters. This represents the parameters of the pre-trained model, which can include audio feature extraction models, temporal convolutional models, and fully connected layers. and classifier . This represents the regularization weight (0.01), which prevents overfitting and maintains generalization ability.

[0211] S54. Based on the pre-trained model that has completed transfer training, proceed to the step of extracting features from multimodal data to obtain feature vectors for each modality.

[0212] For a better understanding of the complete process of this invention, please refer to [link / reference]. Figure 2 , Figure 2 A flowchart illustrating another emotion recognition method provided in an embodiment of the present invention. The core of the present invention lies in:

[0213] 1. Behavioral data and GNN modeling: Add behavioral data (such as acceleration sequences) and use GNN to capture the dynamic relationships between modalities to enhance feature expressiveness.

[0214] 2. Enhanced learning weight optimization: Q-learning is used to dynamically adjust attention weights based on context, improving fusion accuracy and adaptability.

[0215] 3. Knowledge distillation: Transfer knowledge from large cloud models to small edge models, compress model size by 50%, and optimize inference efficiency on edge devices.

[0216] 4. User-guided cold start: Recommend the best time for annotation based on data volatility, quickly generate personalized models, and improve user experience.

[0217] The beneficial effects of this invention are as follows: Based on multimodal data (audio, physiological, environmental, and behavioral), this invention achieves the following advantages through deep learning, GNN, reinforcement learning, knowledge distillation, and user guidance: High accuracy: Feature extraction and GNN modeling improve the accuracy of emotion recognition. Strong adaptability: Incremental learning and knowledge distillation ensure that the model adapts to changes in pet behavior, and edge optimization reduces power consumption. High real-time performance: Reinforcement learning-optimized attention fusion achieves low latency, suitable for real-time applications. Personalized experience: Reduces the initial misidentification rate during cold start with user guidance. Excellent user experience: The app displays emotion probabilities (e.g., "Happy: 85%)", weight visualization (e.g., "Audio contribution: 40%), and labeled recommendations, with simple and transparent interaction.

[0218] The following describes the emotion recognition device, computer equipment, computer-readable storage medium, and computer program product provided in the embodiments of the present invention. The emotion recognition device, computer equipment, computer-readable storage medium, and computer program product described below can be referred to in correspondence with the emotion recognition method described above.

[0219] Please refer to Figure 3 , Figure 3 This is a structural block diagram of an emotion recognition device provided in an embodiment of the present invention. The device may include:

[0220] Module 301 is used to acquire multimodal data of the pet;

[0221] The feature extraction module 302 is used to extract features from multimodal data to obtain feature vectors for each modality;

[0222] The feature aggregation module 303 is used to determine the correlation between each modality using feature vectors, and to perform feature aggregation on each feature vector using a pre-trained graph neural network and correlation to obtain enhanced feature vectors for each modality.

[0223] The emotion recognition module 304 is used to perform attention fusion on each enhanced feature vector to obtain a fused feature vector, and to perform emotion recognition based on the fused feature vector to obtain the target emotion type of the pet.

[0224] Optionally, the feature aggregation module 303 may include:

[0225] The configuration submodule is used to set up feature vector pairs between modes using feature vectors;

[0226] The attention score determination submodule is used to input the feature vectors into a pre-trained attention network to obtain the attention score values ​​between each modality.

[0227] The correlation determination submodule is used to process the attention scores between the target mode and other modes using a normalized exponential function for each target mode, so as to obtain the correlation between the target mode and other modes.

[0228] Optionally, each graph node in the graph neural network corresponds to a different mode, and the graph nodes are interconnected.

[0229] Feature aggregation module 303 may include:

[0230] The adjacency matrix setting submodule is used to set the adjacency matrix according to the relevance; the adjacency matrix contains the weight values ​​of the edges between each graph node, and the weight values ​​are the relevance.

[0231] The graph neural network processing submodule is used to input the feature vectors of each modality into the graph nodes corresponding to each modality, and to use the graph neural network to perform feature aggregation on each graph node based on the adjacency matrix to obtain the enhanced feature vectors of each modality.

[0232] Optionally, the multimodal data includes audio data, physiological data, behavioral data, and environmental data, and the feature vector includes audio features corresponding to the audio data, fluctuation pattern features corresponding to the physiological data, motion features corresponding to the behavioral data, and scene features corresponding to the environmental data.

[0233] Optionally, the emotion recognition module 304 may include:

[0234] The multi-head attention processing submodule is used to process each enhanced feature vector using an attention mechanism based on a pre-trained multi-head attention module to obtain the attention weights for each modality.

[0235] The feature fusion submodule is used to perform weighted fusion of the enhanced feature vectors of each modality using the attention weights of each modality to obtain a fused feature vector;

[0236] The emotion recognition submodule is used to determine the predicted probability of a pet in multiple emotion types by using a pre-trained classifier and fused feature vectors, and to set the emotion type corresponding to the highest predicted probability as the target emotion type.

[0237] Optionally, the device may further include:

[0238] The calculation module is used to determine the first variance using the attention weights of each modality;

[0239] The first training data setting module is used to label the target emotion type of the multimodal data and obtain training data when the maximum prediction probability and the first variance meet the first preset condition; wherein, the first preset condition is that the maximum prediction probability is greater than the first preset threshold and the variance is less than or equal to the second preset threshold.

[0240] The second training data setting module is used to generate confirmation information and output display using prediction probability and attention weight when the maximum prediction probability and the first variance meet the second preset condition, receive the input user-labeled emotion type, and label the multimodal data with user-labeled emotion type to obtain training data; wherein, the second preset condition is that the maximum prediction probability is greater than the third preset threshold and less than or equal to the first preset threshold, or the variance is greater than the second preset threshold and the third preset threshold is less than the first preset threshold.

[0241] The incremental training module is used to perform incremental training of the model using the training data when the amount of training data reaches a preset amount.

[0242] Optionally, the incremental training module may include:

[0243] The training feature extraction submodule is used to extract features from the multimodal data in the training data to obtain the training enhancement feature vectors for each modality.

[0244] The multi-head attention training submodule is used to process each training augmentation feature vector using the multi-head attention module to obtain the initial attention weights for each modality.

[0245] The attention weight adjustment submodule is used to generate contextual states using environmental data, physiological data, and behavioral data. It uses a state-action value function to determine the attention weight adjustment value for each modality based on the contextual state, and uses the attention weight adjustment value to adjust the initial attention weights to obtain the adjusted attention weights for each modality.

[0246] The training feature fusion submodule is used to perform weighted fusion of the training enhancement feature vectors of each modality using the adjusted attention weights to obtain the training fusion feature vector;

[0247] The update submodule is used to determine the predicted emotion type of the pet based on the training fusion feature vector, determine the incremental loss value and reward value based on the predicted emotion type and the emotion type labeled by multimodal data, update the multi-head attention module using the incremental loss value, and update the state-action value function using the reward value.

[0248] Optionally, the incremental training module may include:

[0249] The upload submodule is used to upload training data to the cloud device so that when the amount of training data reaches the preset amount, the cloud device can use the training data to perform incremental training on the teacher model, distill the trained teacher model into a student model, and distribute the student model.

[0250] The update submodule is used to receive the student model and update the local model using the student model.

[0251] Alternatively, cloud devices can be used for:

[0252] The training data is input into the teacher model and the student model respectively to obtain the output data of the teacher model and the output data of the student model.

[0253] The incremental loss is determined based on the student model output data, and the distribution difference between the teacher model output data and the student model output data is determined.

[0254] The distillation loss is determined using incremental loss and distribution difference values, and the student model is updated using the distillation loss.

[0255] Optionally, the feature extraction module 302 may include:

[0256] The preprocessing submodule is used to convert audio data into Mel spectrum and normalize physiological, behavioral, and environmental data.

[0257] The audio feature extraction submodule is used to extract features from the Mel spectrum using a pre-trained audio feature extraction model to obtain audio features;

[0258] The wave pattern feature extraction submodule is used to extract features from physiological data using a pre-trained temporal convolutional model to obtain wave pattern features.

[0259] Run the feature extraction submodule, which uses a pre-trained temporal convolutional model to extract features from behavioral data to obtain motion features;

[0260] The scene feature extraction submodule is used to extract features from environmental data using a pre-trained fully connected layer to obtain scene features.

[0261] Optionally, the device may further include:

[0262] The annotation guidance module is used to determine whether the multimodal data meets the annotation conditions. If the annotation conditions are met, annotation prompts are generated and output using the multimodal data. It accepts user annotation information, adds user annotation information to the multimodal data, and obtains transfer training data.

[0263] The transfer training module is used to perform transfer training on the pre-trained model when the amount of transfer training data reaches a preset amount; based on the pre-trained model that has completed transfer training, it proceeds to the step of extracting features from multimodal data to obtain feature vectors for each modality.

[0264] Optionally, the labeling guidance module may include:

[0265] The variance calculation submodule is used to determine the second variance using physiological data, behavioral data, and environmental data, and to map the second variance to the 0~1 interval to obtain the recommended label score;

[0266] The judgment submodule is used to determine whether the multimodal data meets the labeling conditions when the recommended labeling score is greater than the preset value.

[0267] Please refer to Figure 4 , Figure 4 This is a structural block diagram of a computer device provided in an embodiment of the present invention. The present invention provides a computer device 40, including a processor 41 and a memory 42; wherein, the memory 42 is used to store a computer program; the processor 41 is used to execute the emotion recognition method provided in the foregoing embodiment when executing the computer program.

[0268] For details regarding the specific process of the aforementioned emotion recognition method, please refer to the relevant content provided in the foregoing embodiments, which will not be repeated here.

[0269] Furthermore, the memory 42, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0270] In addition, the computer device 40 also includes a power supply 43, a communication interface 44, an input / output interface 45, and a communication bus 46; wherein, the power supply 43 is used to provide operating voltage for the various hardware devices on the computer device 40; the communication interface 44 can create a data transmission channel between the computer device 40 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this invention, and is not specifically limited here; the input / output interface 45 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0271] In addition, the computer device 40 may also include a display device (not shown) for displaying prompts, such as confirmation information including emotion probability distribution and attention weight values, and annotation information for prompting the user to annotate.

[0272] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the emotion recognition method described in the above embodiments.

[0273] Since the embodiments of the computer program product part correspond to the embodiments of the emotion recognition method part, please refer to the description of the embodiments of the emotion recognition method part for the embodiments of the computer program product part, and will not be repeated here.

[0274] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the emotion recognition method described in the above embodiments.

[0275] Since the embodiments of the computer-readable storage medium portion correspond to the embodiments of the emotion recognition method portion, the embodiments of the storage medium portion are described in the description of the embodiments of the emotion recognition method portion, and will not be repeated here.

[0276] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0277] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0278] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0279] The foregoing has provided a detailed description of the emotion recognition method, apparatus, computer device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. An emotion recognition method, characterized in that, include: Acquire multimodal data about pets; Feature extraction is performed on the multimodal data to obtain feature vectors for each modality; The correlation between each modality is determined using the feature vectors, and the feature vectors are aggregated using a pre-trained graph neural network and the correlation to obtain the enhanced feature vectors of each modality. Attention fusion is performed on each of the enhanced feature vectors to obtain a fused feature vector, and emotion recognition is performed based on the fused feature vector to obtain the target emotion type of the pet.

2. The emotion recognition method according to claim 1, characterized in that, Determining the correlation between modes using the feature vectors includes: The feature vectors are used to set the feature vector pairs between the modes; The feature vectors are input into a pre-trained attention network to obtain attention scores between different modalities. For each target mode, the attention scores between the target mode and other modes are processed using a normalized exponential function to obtain the correlation between the target mode and other modes.

3. The emotion recognition method according to claim 1, characterized in that, Each graph node in the graph neural network corresponds to a mode, and the graph nodes are interconnected. By using a pre-trained graph neural network and the aforementioned relevance, feature aggregation is performed on each feature vector to obtain enhanced feature vectors for each modality, including: An adjacency matrix is ​​set according to the relevance; wherein the adjacency matrix contains the weight values ​​of the edges between each graph node, and the weight values ​​are the relevance. The feature vectors of each modality are input to the graph nodes corresponding to each modality, and the graph neural network is used to aggregate the features of each graph node based on the adjacency matrix to obtain the enhanced feature vectors of each modality.

4. The emotion recognition method according to any one of claims 1 to 3, characterized in that, The multimodal data includes audio data, physiological data, behavioral data, and environmental data. The feature vector includes audio features corresponding to the audio data, fluctuation pattern features corresponding to the physiological data, motion features corresponding to the behavioral data, and scene features corresponding to the environmental data.

5. The emotion recognition method according to claim 4, characterized in that, Attention fusion is performed on each of the enhanced feature vectors to obtain a fused feature vector, and emotion recognition is performed based on the fused feature vector to obtain the target emotion type of the pet, including: The attention weights of each modality are obtained by using a pre-trained multi-head attention module to process each of the enhanced feature vectors through an attention mechanism. The enhanced feature vectors of each modality are weighted and fused using the attention weights of each modality to obtain the fused feature vector; The predicted probabilities of the pet in various emotion types are determined by using a pre-trained classifier and the fused feature vector, and the emotion type corresponding to the highest predicted probability is set as the target emotion type.

6. The emotion recognition method according to claim 5, characterized in that, Also includes: The first variance is determined using the attention weights of each modality; When the maximum predicted probability and the first variance meet the first preset condition, the target emotion type is labeled for the multimodal data to obtain training data; wherein, the first preset condition is that the maximum predicted probability is greater than the first preset threshold and the variance is less than or equal to the second preset threshold. When the maximum prediction probability and the first variance meet the second preset condition, the prediction probability and the attention weight are used to generate confirmation information and output it for display. The input user-labeled emotion type is received, and the user-labeled emotion type is labeled for the multimodal data to obtain training data. The second preset condition is that the maximum prediction probability is greater than the third preset threshold and less than or equal to the first preset threshold, or the variance is greater than the second preset threshold and the third preset threshold is less than the first preset threshold. When the amount of training data reaches a preset amount, incremental training of the model is performed using the training data.

7. The emotion recognition method according to claim 6, characterized in that, Incremental model training using the training data includes: Feature extraction is performed on the multimodal data in the training data to obtain the training enhancement feature vectors for each modality; The multi-head attention module is used to process each of the trained enhanced feature vectors using an attention mechanism to obtain the initial attention weights for each modality; The environmental data, physiological data, and behavioral data are used to generate a context state. The state-action value function is used to determine the attention weight adjustment value of each modality based on the context state. The initial attention weight is then adjusted using the attention weight adjustment value to obtain the adjusted attention weight of each modality. The training enhancement feature vectors of each modality are weighted and fused using the adjusted attention weights to obtain the training fused feature vector; The predicted emotion type of the pet is determined based on the training fusion feature vector. An incremental loss value and a reward value are determined based on the predicted emotion type and the emotion type labeled by the multimodal data. The multi-head attention module is updated using the incremental loss value, and the state-action value function is updated using the reward value.

8. The emotion recognition method according to claim 6, characterized in that, When the amount of training data reaches a preset amount, incremental model training is performed using the training data, including: The training data is uploaded to the cloud device so that when the cloud device determines that the amount of training data has reached a preset amount, it uses the training data to perform incremental training on the teacher model, distills the trained teacher model into a student model, and distributes the student model. Receive the student model and update the local model using the student model.

9. The emotion recognition method according to claim 8, characterized in that, Distilling the trained teacher model into a student model includes: The training data is input into the teacher model and the student model respectively to obtain the teacher model output data and the student model output data. The incremental loss is determined based on the student model output data, and the distribution difference value between the teacher model output data and the student model output data is determined. The distillation loss is determined using the incremental loss and the distribution difference value, and the student model is updated using the distillation loss.

10. The emotion recognition method according to claim 4, characterized in that, Feature extraction is performed on the multimodal data to obtain feature vectors for each modality, including: The audio data is converted into Mel spectrum, and the physiological data, behavioral data, and environmental data are normalized. The audio features are obtained by extracting features from the Mel spectrum using a pre-trained audio feature extraction model. The physiological data are used to extract features using a pre-trained temporal convolutional model to obtain the fluctuation pattern features; The motion features are obtained by extracting features from the behavioral data using a pre-trained temporal convolutional model. The environmental data is used to extract features using a pre-trained fully connected layer to obtain the scene features.

11. The emotion recognition method according to claim 4, characterized in that, Also includes: Determine whether the multimodal data meets the labeling conditions; If the labeling conditions are met, the multimodal data is used to generate and output labeling prompts. User labeling information is accepted as input, and the user labeling information is added to the multimodal data to obtain transfer training data. When it is determined that the amount of transfer training data has reached a preset amount, the pre-trained model is transferred and trained using the transfer training data. Based on the pre-trained model that has completed transfer training, the process proceeds to the step of extracting features from the multimodal data to obtain feature vectors for each modality.

12. The emotion recognition method according to claim 11, characterized in that, Determining whether the multimodal data meets the labeling conditions includes: The second variance is determined using the physiological data, the behavioral data, and the environmental data, and then mapped to the 0~1 interval to obtain the recommendation label score; When the recommended labeling score is determined to be greater than a preset value, the multimodal data is deemed to have met the labeling conditions.

13. An emotion recognition device, characterized in that, include: The acquisition module is used to acquire multimodal data about pets. The feature extraction module is used to extract features from the multimodal data to obtain feature vectors for each modality; The feature aggregation module is used to determine the correlation between each modality using the feature vectors, and to perform feature aggregation on each feature vector using a pre-trained graph neural network and the correlation to obtain the enhanced feature vectors of each modality. The emotion recognition module is used to perform attention fusion on each of the enhanced feature vectors to obtain a fused feature vector, and to perform emotion recognition based on the fused feature vector to obtain the target emotion type of the pet.

14. A computer device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the emotion recognition method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the emotion recognition method as described in any one of claims 1 to 12.

Images