Method and device for recognizing animal emotion based on multi-modal data, and medium

Abstract

The application discloses a method and device for recognizing animal emotion based on multi-modal data and a medium, and relates to the technical field of emotion recognition. The method comprises the following steps: performing multi-modal data fusion on the electroencephalogram feature vector, facial expression feature vector, speech feature vector and motion posture feature vector of a target animal based on a multi-modal fusion module to obtain multi-modal fusion features; and inputting the multi-modal fusion features into an emotion recognition model to determine the emotion type of the target animal. The brain electrical signals, facial images, speech signals and motion acceleration data of the animal are collected, and feature extraction and multi-modal fusion are sequentially performed to recognize the emotion of the animal. The complementarity of each mode in emotion representation is fully utilized, the defects of single mode information deficiency, environmental interference susceptibility and difficulty in distinguishing similar emotions are overcome, and the accuracy and robustness of animal emotion recognition are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and in particular to methods, devices and media for recognizing animal emotions based on multimodal data. Background Technology

[0002] Animal emotion recognition is an important research direction at the intersection of artificial intelligence and animal behavior, with broad application prospects in animal health care, behavioral training, and the development of intelligent animal products. Accurately identifying an animal's emotional state helps to promptly detect abnormalities in its physical and mental health and improve the human-animal interaction experience.

[0003] However, existing technologies exhibit significant limitations when processing animal emotion recognition. Most emotion recognition methods are designed for human physiological characteristics and behavioral habits, while animals and humans differ fundamentally in their ways of representing emotions. Therefore, existing human-oriented emotion recognition methods struggle to effectively process animal emotional information, leading to inaccurate animal emotion recognition results. Summary of the Invention

[0004] Therefore, it is necessary to propose methods, devices, and media for identifying animal emotions based on multimodal data to address the above problems, thereby achieving accurate identification of animal emotions.

[0005] To achieve the above objectives, the first aspect of this application provides a method for recognizing animal emotions based on multimodal data, the method comprising: Collect raw electroencephalogram (EEG) signals, raw facial images, raw speech signals, and raw motion acceleration data of the target animal; The original EEG signals, original facial images, original speech signals, and original motion acceleration signals are preprocessed and feature extracted to obtain the EEG feature vector, facial expression feature vector, speech feature vector, and motion posture feature vector of the target animal. The multimodal fusion module is used to fuse the EEG feature vector, facial expression feature vector, speech feature vector and motion posture feature vector of the target animal to obtain multimodal fusion features. The multimodal fusion features are input into a preset emotion recognition model to determine the emotion type of the target animal.

[0006] Furthermore, the multimodal fusion module includes a modality adaptation layer, a feature sequence splicing layer, a relative position encoding layer, a SwingTransformer, a modality attention weighting layer, and a feature fusion and batch normalization layer connected in sequence. The modality adaptation layer is used to map the EEG feature vector, facial expression feature vector, speech feature vector and motion posture feature vector to the same dimensional space to obtain standard EEG feature vector, standard facial expression feature vector, standard speech feature vector and standard motion posture feature vector with the same dimension. The feature sequence splicing layer is used to splice the standard EEG feature vector, standard facial expression feature vector, standard speech feature vector, and standard motion posture feature vector to obtain the spliced ​​first feature sequence. The relative position encoding layer is used to add position information to the feature vectors corresponding to each modality in the first feature sequence to obtain a second feature sequence with position information. The SwinTransformer is used to perform spatial context encoding on the second feature sequence to obtain the encoded third feature sequence. The modal attention weighting layer is used to determine the weight of each modality based on the third feature sequence, and to perform weighting according to the weight of each modality and the feature vector in the third feature sequence to obtain the weighted feature vector corresponding to each modality. The feature fusion and batch normalization layer is used to fuse, reduce the dimensionality, and batch normalize the feature vectors corresponding to each of the weighted modalities to obtain multimodal fusion features.

[0007] Furthermore, the relative position encoding layer comprises a position bias code and distance calculation layer, a bias matrix generation layer, and the position information fusion layer, which are connected in sequence. The position offset code and distance calculation layer is used to perform position encoding on the feature vectors corresponding to each mode in the first feature sequence, and calculate the relative distance between all mode position pairs based on the position encoding of the feature vectors, wherein the mode position pair is a data pair composed of any two modes in the first feature sequence; The bias matrix generation layer is used to obtain a relative position bias matrix based on the relative distance between all modal position pairs and a preset bias parameter table, wherein the bias parameter table contains a mapping relationship between relative distance and bias parameter values. The location information fusion layer is used to fuse the relative position bias matrix with the first feature sequence to obtain a second feature sequence with relative position information.

[0008] Furthermore, the SwinTransformer includes four sequentially connected coding layers, each of which includes: The first normalization sublayer is used to perform layer normalization on the input feature sequence to obtain the first normalized feature sequence; A multi-head self-attention sub-layer is used to perform window self-attention calculation on the first normalized feature sequence, and the relative position bias matrix is ​​added to the attention score to obtain the attention-weighted feature sequence. The first residual connection sublayer is used to add the attention-weighted feature sequence to the input feature sequence to obtain the first residual feature sequence; The second normalization sublayer is used to perform layer normalization on the first residual feature sequence to obtain the second normalized feature sequence. The feedforward network sublayer is used to perform a nonlinear transformation on the second normalized feature sequence to obtain the feedforward output feature sequence. The second residual connection sub-layer is used to add the feedforward output feature sequence to the first residual feature sequence and output the encoding feature sequence of the current encoding layer. The SwinTransformer uses the second feature sequence as the input to the first coding layer, the coding feature sequence output by each coding layer is the input to the next coding layer, and the coding feature sequence output by the last coding layer is the third feature sequence.

[0009] Furthermore, the modal attention weighting layer includes an attention score calculation layer, a weight normalization layer, and a feature weighting layer connected in sequence: The attention score calculation layer is used to obtain the original attention score of each modality based on the feature vector corresponding to each modality in the third feature sequence and the preset attention score calculation formula. The weight normalization layer is used to normalize the original attention scores of each modality to obtain the attention weights of each modality. The feature weighting layer is used to multiply the attention weight of each modality with the feature vector of the corresponding modality in the third feature sequence to obtain the weighted feature vector of each modality.

[0010] Furthermore, the preprocessing and feature extraction of the original EEG signals, original facial images, original speech signals, and original motion acceleration signals to obtain the target animal's EEG feature vector, facial expression feature vector, speech feature vector, and motion posture feature vector specifically includes: The raw EEG signals, raw facial images, raw speech signals, and raw motion acceleration signals are preprocessed to obtain target EEG data, facial expression data, effective speech data, and target motion data. The dynamic temporal features and static spectral features in different frequency bands of the target EEG data are extracted to obtain the EEG feature vector; Extract the shape and texture features of the facial expression data to obtain a facial expression feature vector; Extract the phonetic features from the effective speech data to obtain a speech feature vector; The motion amplitude features of the target motion data are extracted to obtain the motion posture feature vector.

[0011] Furthermore, the preprocessing of the original EEG signals, original facial images, original speech signals, and original motion acceleration signals to obtain target EEG data, facial expression data, effective speech data, and target motion data specifically includes: The original EEG signal is filtered and spherical interpolation is performed to obtain the first EEG signal; Independent component analysis was performed on the first EEG signal to obtain the second EEG signal after artifact removal. The second EEG signal is smoothed to obtain the target EEG data; Animal face detection and feature point localization are performed on the original facial image to obtain feature point data in the original facial image; Image correction is performed based on the original facial image and the feature point data in the original facial image to determine the facial expression image, wherein the facial expression data includes the facial expression image and the feature point data; The original speech signal is denoised and its feature frequency bands are extracted to obtain effective speech signal segments; The effective speech signal segments are framed and windowed to obtain effective speech data; The original motion acceleration signal is denoised and normalized to obtain the target motion data.

[0012] Furthermore, the step of inputting the multimodal fusion features into a preset emotion recognition model to determine the emotion type of the target animal specifically includes: The multimodal fusion features are input into a preset emotion recognition model, and the probability distribution of the target animal belonging to different emotion types is output. The target animal's emotion type is determined based on the probability distribution of each emotion type, wherein the emotion categories include pleasure, anxiety, fear, calmness, and pain.

[0013] To achieve the above objectives, a second aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method described in the first aspect.

[0014] To achieve the above objectives, a third aspect of this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described in the first aspect.

[0015] The present invention has the following beneficial effects: This invention proposes a method for animal emotion recognition based on multimodal data. The method includes: acquiring raw EEG signals, raw facial images, raw speech signals, and raw motion acceleration data of a target animal; preprocessing and extracting features from the raw EEG signals, raw facial images, raw speech signals, and raw motion acceleration signals to obtain EEG feature vectors, facial expression feature vectors, speech feature vectors, and motion posture feature vectors of the target animal; fusing the EEG feature vectors, facial expression feature vectors, speech feature vectors, and motion posture feature vectors of the target animal using a multimodal fusion module to obtain multimodal fused features; and inputting the multimodal fused features into a preset emotion recognition model to determine the emotion type of the target animal. By acquiring animal EEG signals, facial images, speech signals, and motion acceleration data, and sequentially performing feature extraction and multimodal fusion, animal emotion recognition is achieved. This fully utilizes the complementarity of each modality in emotion representation, overcoming the shortcomings of insufficient information from a single modality, susceptibility to environmental interference, and difficulty in distinguishing similar emotions, significantly improving the accuracy and robustness of animal emotion recognition. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] in: Figure 1 This is a flowchart illustrating the animal emotion recognition method based on multimodal data in an embodiment of the present invention. Figure 2 This is an internal structural diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The animal emotion recognition method of this invention can be widely applied to various animal-related scenarios. In animal health care, when animals are sick or in pain, their emotions will manifest as distress or anxiety. This invention can monitor and identify these abnormal emotions in real time, promptly alerting pet owners or medical personnel to intervene and prevent the condition from worsening. In animal behavior training, trainers can adjust training methods and intensity based on identified animal emotions, such as fear or pleasure, avoiding excessive psychological stress on the animal and improving training effectiveness and the quality of human-animal interaction. In smart animal products, the animal emotion recognition method can be integrated into smart animal shelters, smart collars, automatic feeders, and other devices, enabling the devices to automatically adjust their working status according to the animal's real-time emotions. For example, when fear is detected, soothing music or pheromones can be played, achieving effective intelligent interaction between humans and animals. Furthermore, this invention can also be applied to commercial scenarios such as animal boarding, animal grooming, and animal medical care, providing service personnel with objective animal emotion references and reducing safety risks in their work.

[0020] One embodiment of the present invention proposes a method for identifying animal emotions based on multimodal data, in order to accurately identify animal emotions. For details, please refer to [reference needed]. Figure 1 , Figure 1 This is a flowchart illustrating the animal emotion recognition method based on multimodal data according to an embodiment of the present invention. The method includes: Step 100: Collect raw EEG signals, raw facial images, raw speech signals, and raw motion acceleration data of the target animal.

[0021] In this embodiment, the target animal can be a cat, a dog, or a bird, among other different types of animals. Four different types of raw data are simultaneously collected from the target animal: electroencephalogram (EEG) signals, facial images, speech signals, and motion acceleration data. These four types of data can reflect the animal's emotional state from four dimensions: neurophysiological activity, facial expression, vocal expression, and body movement. The data from different dimensions can complement and verify each other, effectively overcoming the problem of insufficient information from a single modality.

[0022] In this embodiment, the raw EEG signal is the weak voltage fluctuation generated by the electrical activity of neurons in the animal's cerebral cortex, recorded by an EEG device. Optionally, a 4-channel EEG acquisition system is used to acquire the raw EEG signal. Four channels are chosen because most animals have small heads and dense fur; too many electrodes would lead to wearing difficulties, animal resistance, and increased artifact sources. The sampling frequency can be set to 500Hz, which meets the Nyquist sampling requirements of the 0.5Hz to 50Hz EEG band; for example, Delta, Theta, Alpha, Beta, and Gamma waves can be acquired. During acquisition, the animal must wear a dedicated flexible EEG cap designed according to the target animal's head size. The electrodes are made of medical-grade flexible material to reduce skin discomfort and motion artifacts caused by the animal shaking its head or scratching. The electrode impedance is controlled below 20kΩ to ensure the stability of the EEG signal acquisition.

[0023] In this embodiment, the original facial images are captured by a high-definition camera to analyze changes in the animal's facial expressions. Optionally, a 1080P resolution, 30 frames per second high-definition camera is used, equipped with an anti-interference lens, which can maintain clear imaging even when the animal turns its head, lowers its head, or the lighting changes. The camera is usually installed in an area where the animal frequently moves. In order to capture facial features related to the animal's emotions, the camera needs to focus on shooting the animal's eyes, ears, and mouth and nose to accurately obtain the animal's emotions through these facial features. For example, when a cat is afraid, its pupils will dilate and its ears will be lowered to the sides; when a dog is happy, its mouth corners will be slightly upturned and its eyes will squint.

[0024] In this embodiment, the raw voice signal is acquired via a microphone array to analyze the animal's vocal characteristics. Animals express emotions through sounds such as barking, whimpering, purring, and howling. Optionally, a four-microphone array with beamforming and noise reduction capabilities is used to directionally enhance the animal's vocalizations in noisy environments and suppress background noise. The acquisition frequency range can be set from 100Hz to 8kHz, covering the fundamental frequency range and harmonic components of different species. The microphone array is typically installed synchronously with the camera to ensure that the direction of the sound source and the image are consistent.

[0025] In this embodiment, raw motion acceleration data is acquired using a high-precision accelerometer sensor to analyze the animal's body movement posture. The accelerometer sensor is typically fixed to a naturally worn item such as the animal's collar to effectively collect motion acceleration data. It can acquire acceleration values ​​in three axes (X, Y, Z), corresponding to the animal's forward / backward, left / right, and up / down directions, respectively.

[0026] This invention provides a rich information foundation for subsequent fusion recognition by simultaneously collecting data from four complementary modalities. The collection of multimodal data overcomes the limitations of single modality in animal emotion recognition.

[0027] Step 200: Preprocess and extract features from the original EEG signals, original facial images, original speech signals, and original motion acceleration signals to obtain the EEG feature vector, facial expression feature vector, speech feature vector, and motion posture feature vector of the target animal.

[0028] Since the original signal contains noise, artifacts, and redundant information, and the data of different modalities differ greatly in size and distribution, it is necessary to preprocess and extract features for each modality separately to ensure that the extracted feature vectors can truly reflect the emotional state of the animal.

[0029] This invention transforms raw signals into feature vectors through preprocessing and feature extraction tailored to animal characteristics. EEG features highlight the energy and dynamic changes in emotion-related frequency bands; facial expression features quantify animal-specific changes in ear, eye, and mouth morphology; speech features capture the acoustic properties of animal vocalization; and motion features characterize body posture and activity intensity. These feature vectors not only eliminate individual differences, such as between different breeds, ages, and body sizes, but also suppress environmental noise and physiological artifacts, providing more accurate data for subsequent multimodal fusion.

[0030] Step 300: Based on the multimodal fusion module, multimodal data fusion is performed on the target animal's EEG feature vector, facial expression feature vector, speech feature vector and motion posture feature vector to obtain multimodal fusion features.

[0031] Because the features of the four modalities differ significantly in dimensionality, numerical range, and semantic level—that is, multimodal data exhibits heterogeneity—simple concatenation or weighted averaging cannot fully utilize their complementary relationships. Therefore, this embodiment utilizes a multimodal fusion module to enable features from different modalities to interact within the same feature space, ultimately outputting fused features that are rich in information and have low redundancy.

[0032] Step 400: Input the multimodal fusion features into the preset emotion recognition model to determine the emotion type of the target animal.

[0033] In this embodiment, the emotion recognition model is a classifier whose input is multimodal fusion features. After performing a nonlinear transformation on the multimodal fusion features, it outputs the probability distribution of different emotion types, so as to determine the current emotion type of the animal based on the probability distribution of different emotion types.

[0034] In one feasible embodiment, the emotion recognition model also outputs the confidence score for each emotion category. To facilitate use by pet owners, the recognition results can be presented in various ways: for example, displaying "Current Animal Emotion: Fear" on a mobile app, accompanied by a flashing red light or a voice announcement stating "Your animal seems to be afraid." Furthermore, the system can be configured with an anomaly warning function: when the confidence score for a distressing emotion exceeds a preset threshold, an emergency notification is automatically sent to the pet owner's mobile phone, reminding them to pay close attention to the animal's health.

[0035] This embodiment achieves automatic and accurate classification of animal emotions by inputting multimodal fusion features into a specially trained emotion recognition model. Furthermore, by outputting confidence scores and providing anomaly warnings, it can help veterinarians or owners analyze potential causes of illness or environmental stressors.

[0036] This invention, through the simultaneous acquisition of raw data from four modalities—EEG, facial images, speech, and motion acceleration—comprehensively captures animal emotional information from four dimensions: neural electrical activity, facial expressions, vocal expression, and body posture. This overcomes the limitations of single-modal information, which is susceptible to environmental interference or behavioral constraints. A multimodal fusion module deeply fuses the four heterogeneous feature vectors, enabling interaction between different modalities. Even when some modal data quality deteriorates or is missing, robust fusion performance is maintained, significantly improving the accuracy, robustness, and practicality of the recognition.

[0037] In one embodiment of the present invention, the multimodal fusion module includes a modality adaptation layer, a feature sequence splicing layer, a relative position encoding layer, a SwingTransformer, a modality attention weighting layer, and a feature fusion and batch normalization layer connected in sequence.

[0038] In this embodiment, the multimodal fusion module is used to deeply fuse the EEG feature vector, facial expression feature vector, speech feature vector, and motion posture feature vector obtained in step 200. This module consists of six sequentially connected sub-layers: a modality adaptation layer, a feature sequence concatenation layer, a relative position encoding layer, a SwingTransformer, a modality attention weighting layer, and a feature fusion and batch normalization layer. The specific implementation of each sub-layer is described in detail below.

[0039] The modality adaptation layer is used to map EEG feature vectors, facial expression feature vectors, speech feature vectors, and motion posture feature vectors to the same dimensional space, resulting in standard EEG feature vectors, standard facial expression feature vectors, standard speech feature vectors, and standard motion posture feature vectors with the same dimension.

[0040] Because the extracted EEG feature vectors, facial expression feature vectors, speech feature vectors, and posture feature vectors often have different dimensions. For example, an EEG feature vector might be 64-dimensional, a facial expression feature vector might be 128-dimensional, a speech feature vector might be 68-dimensional, and a posture feature vector might be 10-dimensional. The numerical ranges of these feature vectors may also differ; some are between 0 and 1, some are between -1 and 1, and some are dimensionless. Directly feeding these feature vectors with different dimensions and scales into subsequent layers would lead to difficulties in model training, slow convergence, and inability to perform effective attention calculations. Therefore, it is necessary to first map the feature vectors of each modality to a unified dimensional space through a modality adaptation layer, ensuring they have the same feature dimension, which facilitates subsequent concatenation and interaction.

[0041] In this embodiment, the modality adaptation layer independently sets up a fully connected layer for linear projection for each modality. The mathematical form of the fully connected layer is: output vector = input vector × weight matrix + bias, specifically:

[0042]

[0043]

[0044]

[0045] In the formula, , , and These are, respectively, standard EEG feature vectors, standard facial expression feature vectors, standard speech feature vectors, and standard motor posture feature vectors. , , and These are weight matrices for four preset modalities: EEG, facial images, speech, and motion acceleration. , , , These are bias terms for four preset modalities: EEG, facial image, speech, and motion acceleration. The number of rows in the weight matrix equals the target dimension, and the number of columns equals the input dimension; the bias is a learnable vector. In this embodiment, the target dimension for all modalities is uniformly set to 128 dimensions. For example, for a 64-dimensional EEG feature vector, the size of the weight matrix is... After linear transformation, a 128-dimensional standard EEG feature vector is obtained; for the 128-dimensional facial expression feature vector, the size of the weight matrix is... For a 68-dimensional speech feature vector, the weight matrix size is... For a 10-dimensional motion posture feature vector, the weight matrix size is... Bias term These weight matrices and biases are trainable parameters of the model, which are automatically adjusted during training through backpropagation to find the optimal mapping from each original feature space to the unified space.

[0046] To better illustrate this, consider the following example: Suppose a cat's EEG feature vector is [0.2, 0.5, ..., 0.8] (64 numbers in total). After passing through a fully connected EEG layer, a standard 128-dimensional EEG feature vector is obtained, where each element is a weighted sum of the original 64 elements. The weight matrix of the fully connected EEG layer is randomly initialized at the beginning of training. After multiple iterations, it learns to retain important emotional information from the EEG (such as Theta wave energy, Alpha wave stability, etc.) and project it onto a specific dimension of a unified space. Similarly, facial expression feature vectors, speech feature vectors, and motion posture feature vectors are also mapped to the same 128-dimensional space after passing through their respective fully connected layers.

[0047] The modality adaptation layer in this embodiment addresses the issue of dimensional inconsistency in multimodal data heterogeneity. By mapping features from different modalities to the same dimension, it creates conditions for subsequent concatenation and self-attention computation. Simultaneously, training the fully connected layer enables the model to automatically learn which feature dimensions are most important for emotion recognition in each modality, thereby aligning semantic information from different modalities in a unified space and facilitating subsequent layers to capture cross-modal consistency.

[0048] The feature sequence concatenation layer is used to concatenate standard EEG feature vectors, standard facial expression feature vectors, standard speech feature vectors, and standard motion posture feature vectors to obtain the concatenated first feature sequence.

[0049] After the modality adaptation layer, four standard feature vectors with the same dimensions are obtained, representing the four modalities of EEG, facial, speech, and motion. However, these feature vectors do not yet form a structured relationship, and the subsequent SwinTransformer requires a sequence as input to process the relationships between modalities. The feature sequence concatenation layer concatenates these four standard feature vectors into a sequence matrix in a certain order, which serves as the input for subsequent layers.

[0050] In this embodiment, the feature sequence concatenation layer stacks standard feature vectors row-wise to form a matrix of shape [4, 128], denoted as the first feature sequence. Specifically, the concatenation order is agreed to be: the first row is the standard EEG feature vector, the second row is the standard facial feature vector, the third row is the standard speech feature vector, and the fourth row is the standard motion feature vector. This matrix can be regarded as a sequence of length 4. The order within the first feature sequence is fixed and remains consistent during training and inference, enabling the model to perceive the positional identity of each modality through subsequent relative position encoding. The first feature sequence is expressed by the formula:

[0051] The feature sequence concatenation layer in this embodiment transforms four independent modal features into a serialized structure, laying the foundation for subsequent cross-modal interaction using Transformer-like models. This serialized representation preserves the modal identity information, enabling the model to learn the dependencies between different modalities, such as the correlation between EEG and facial expressions, and the synergy between speech and motor posture.

[0052] The relative position encoding layer is used to add positional information to the feature vectors corresponding to each modality in the first feature sequence, so as to obtain a second feature sequence with positional information.

[0053] Because the self-attention mechanism in the Transformer model itself lacks positional awareness, directly inputting the first feature sequence into the SwinTransformer layer will cause the model to treat EEG, facial, speech, and motion as an unordered set, unable to distinguish which vector represents which modality or understand the relative order between modalities. However, in emotion recognition, the order of modalities may implicitly contain certain prior knowledge. For example, EEG, as a physiological signal, is generally considered the most direct indicator of emotion and should be given more attention. Furthermore, the relative distance between different modalities may affect the strength of their interaction. Therefore, positional encoding is needed to inject positional information into the first feature sequence, resulting in a second feature sequence with positional information. Unlike absolute positional encoding, this embodiment uses relative positional encoding, which focuses on the relative offset between any two elements in the sequence, rather than their absolute positions. This is more suitable for scenarios where the number of modalities is fixed and their order has a certain logic. In this embodiment, the relative position encoding layer enables the model to distinguish different modalities and adjust attention weights by using the relative order between modalities. Compared with absolute position encoding, relative position encoding has better generalization to changes in sequence length and better reflects the logical relationship between modalities, thereby optimizing the subsequent fusion effect.

[0054] SwinTransformer is used to perform spatial context encoding on the second feature sequence to obtain the encoded third feature sequence.

[0055] SwinTransformer is a window-based hierarchical Transformer architecture. In this embodiment, since the sequence length is only 4, the entire sequence is considered as a window, meaning the window size equals the sequence length. The role of SwinTransformer is to perform spatial context encoding on the second feature sequence containing positional information. This allows the feature vectors of each modality in the sequence to aggregate information from other modalities, thereby generating a new feature representation that incorporates cross-modal context—the third feature sequence. Through multi-layer stacking, the model can progressively abstract higher-order modal interaction patterns. In this embodiment, SwinTransformer achieves full interaction between modalities, ensuring that the representation of each modality is no longer isolated but includes an understanding of other modalities. Compared to simple feature concatenation, this interactive encoding can more effectively utilize the complementarity and redundancy of multimodal data.

[0056] The modality attention weighting layer is used to determine the weights of each modality based on the third feature sequence, and to weight each modality according to its weight and the feature vector in the third feature sequence to obtain the weighted feature vector corresponding to each modality.

[0057] After being encoded by the SwinTransformer, the third feature sequence shows that the feature vectors of each modality have incorporated information from other modalities. However, different modalities may contribute differently to the current emotion recognition task, and this contribution changes dynamically with the input data. For example, when an animal's face is occluded, the feature quality of the facial modality decreases, and its weight should be reduced; when the environment is noisy, the signal-to-noise ratio of the speech modality is low, and its weight should also be reduced. The purpose of the modality attention weighting layer is to dynamically calculate the importance weight of each modality based on the current input content and to weight the features, enabling the model to automatically focus on reliable and informative modalities while ignoring or suppressing unreliable modalities. The modality attention weighting layer in this embodiment achieves dynamic and adaptive multimodal fusion, significantly improving the robustness of the model in cases of modality quality changes and partial modality loss. It avoids the overall performance collapse caused by the sudden failure of a certain modality.

[0058] The feature fusion and batch normalization layer is used to fuse, reduce the dimensionality, and batch normalize the feature vectors corresponding to each modality after weighting, so as to obtain multimodal fusion features.

[0059] After passing through the modal attention weighting layer, four weighted feature vectors are obtained, each with 128 dimensions. These need to be integrated into a single feature vector as the final multimodal fusion feature, which is then input into the emotion recognition model. Simple concatenation would yield a 512-dimensional vector, but high-dimensional vectors can lead to excessive computation and overfitting risks, and may also introduce redundant information between modalities. Therefore, the feature fusion and batch normalization layer first concatenates the vectors, then reduces the dimensionality through a fully connected layer, and finally standardizes them using batch normalization to obtain a compact and stable fusion feature.

[0060] Specifically, the feature fusion and batch normalization layer consists of three sequentially connected layers: a splicing layer, a dimensionality reduction layer, and a batch normalization layer.

[0061] The concatenation layer concatenates the four weighted 128-dimensional feature vectors sequentially, end to end, to form a 512-dimensional original fused vector:

[0062] In the formula, These are the weighted feature vectors corresponding to EEG, facial features, speech, and movement, respectively. This is the original fusion vector.

[0063] The dimensionality reduction layer maps the original 512-dimensional fused vector to a 128-dimensional fused vector through a fully connected layer:

[0064] In the formula, For dimensionality reduction and fusion vectors, It is a preset dimensionality reduction weight matrix. It is the bias vector.

[0065] It should be noted that the processing flow described in step 300 is for a single sample (i.e., one recognition task). However, in actual training of emotion recognition models, multiple samples are usually organized into a mini-batch and input into the network simultaneously to accelerate and stabilize gradients using matrix operations. Batch normalization utilizes the statistical information in this mini-batch for standardization. During the inference phase, the global statistics accumulated during the training phase are used, without relying on the current batch. The batch normalization layer performs standardization operations on a mini-batch of samples from the batch training phase. Assume that the current training batch contains N samples, and each sample has obtained a 128-dimensional dimensionality-reduced fusion vector after the previous steps. These N dimensionality-reduced fusion vectors are combined into a matrix, where each row is the dimensionality-reduced fusion vector of a single sample.

[0066] Batch normalization calculates statistics independently for each feature dimension. For the th feature... dimension( ), calculate the dimension Mean of a sample and variance Then, for each sample, the first... Standardize the dimension values:

[0067] In the formula, The value of the k-th feature dimension of the n-th sample after standardization. Let be the original value in the k-th dimension of the dimensionality-reduced fusion feature vector of the n-th sample. This is a preset constant to prevent division by zero.

[0068] The mean of the standardized values ​​is variance is To restore the model's expressive power, a learnable scaling factor is introduced. Translation factor Output:

[0069] and These are also model parameters, learned during training. Ultimately, for each sample, a 128-dimensional batch-normalized output vector is obtained. That is, the final multimodal fusion feature.

[0070] In this embodiment, the feature fusion and batch normalization layer integrates the weighted features of the four modalities into a single fused feature. Concatenation preserves all information, dimensionality reduction removes redundancy and lowers the overall dimension, and batch normalization eliminates internal covariate bias by utilizing batch statistics, ensuring a consistent data distribution across different batches and individuals. This provides high-quality input for subsequent emotion recognition models, contributing to improved classification accuracy and generalization ability. Furthermore, the introduction of batch normalization makes model training more stable and faster, and more robust to hyperparameter selection.

[0071] In one embodiment of the present invention, the relative position encoding layer includes a position bias code and distance calculation layer, a bias matrix generation layer and a position information fusion layer connected in sequence.

[0072] In this embodiment, the relative position encoding layer is used to add positional information to the first feature sequence. It comprises three sub-layers connected in sequence: a position encoding and distance calculation layer, a bias matrix generation layer, and a positional information fusion layer. The specific implementation of each sub-layer is described in detail below.

[0073] The positional coding and distance calculation layer is used to encode the feature vectors corresponding to each mode in the first feature sequence, and calculate the relative distance between all mode position pairs based on the positional coding of the feature vectors. Here, a mode position pair is a data pair composed of any two modes in the first feature sequence.

[0074] In this embodiment, the position offset code and distance calculation layer assigns a position identifier to each mode in the sequence and calculates the relative distance between any two modes, providing a basis for the subsequent generation of position offsets.

[0075] The position encoding and distance calculation layer first assigns a unique position index to each modality feature vector in the first feature sequence. For example, the correspondence between position indices and modalities is conventionally: EEG corresponds to position 0, face to position 1, speech to position 2, and motion to position 3. This correspondence is fixed and remains consistent during model training and inference. Then, the position encoding and distance calculation layer calculates the relative distance between all possible modality position pairs in the first feature sequence. For any two position indices i and j, i, j ∈ {0, 1, 2, 3}, the relative distance d... ij =ji. It should be noted that a modal location pair refers to an ordered data pair consisting of any two modalities in the first feature sequence, such as (EEG, facial), (motor, EEG), (EEG, motor), etc. Relative distance d ij The value ranges from -3 to 3, with a total of 7 possible values. By assigning fixed position indices and calculating relative distances, this layer transforms the arrangement of modes into a quantifiable numerical relationship, laying the foundation for the subsequent generation of a learnable bias matrix.

[0076] The bias matrix generation layer is used to obtain the relative position bias matrix based on the relative distance between all modal position pairs and a preset bias parameter table, wherein the bias parameter table contains the mapping relationship between the relative distance and the bias parameter value.

[0077] After obtaining the relative distances of all modality pairs, this embodiment needs to convert these relative distances into bias values ​​that the model can use to adjust the attention weights between different modality pairs in self-attention. First, a bias parameter table needs to be defined, mapping each relative distance to a bias parameter value, and a complete relative position bias matrix is ​​generated accordingly. The bias matrix generation layer transforms the abstract relative distances into bias parameters, enabling the model to learn the positional relationships between modalities. The learnable relative position bias parameter table can automatically adjust the interaction strength between different modality pairs based on the specific task and data characteristics of animal emotion recognition, thereby improving the fusion effect.

[0078] The location information fusion layer is used to fuse the relative position bias matrix with the first feature sequence to obtain a second feature sequence with relative position information.

[0079] In this embodiment, after obtaining the relative position bias matrix, it needs to be fused with the first feature sequence to obtain the second feature sequence. This allows the subsequent SwinTransformer to perceive the relative positional relationship between modalities when calculating self-attention. The goal of fusion is not to modify the feature vectors themselves, but to inject bias information into the calculation of attention scores, thereby affecting the model's attention to different modal pairs without changing the original feature values. The position information fusion layer injects the relative position bias into the subsequent self-attention calculation in a non-intrusive manner, preserving the integrity of the original feature vectors while enabling the model to perceive the sequential relationship between modalities.

[0080] The relative position encoding layer in this invention enables the subsequent SwinTransformer to explicitly perceive the relative order and distance relationships between different modalities during self-attention computation, thereby overcoming the inherent deficiency of the standard Transformer structure in lacking position awareness. Furthermore, relative position encoding better aligns with the practical needs of animal multimodal emotion recognition, where modal order implies prior knowledge and the intensity of intermodal interactions varies with relative distance, thus exhibiting better generalization capabilities.

[0081] In one embodiment of the present invention, the SwinTransformer includes four sequentially connected coding layers, each coding layer having the same structure. The sub-layers contained in each coding layer are described in detail below.

[0082] The first normalization sublayer is used to perform layer normalization on the input feature sequence to obtain the first normalized feature sequence.

[0083] The first normalization sublayer in this embodiment standardizes all feature dimensions of the input feature sequence, giving the feature distribution a stable mean and variance, thereby accelerating model convergence and improving training stability. Unlike batch normalization, layer normalization does not depend on batch size and is suitable for scenarios with short sequence lengths.

[0084] Let the input feature sequence of the current coding layer be X∈ For the first encoding layer, X is the second feature sequence; for subsequent encoding layers, X is the encoded feature sequence output by the previous encoding layer. The first normalization sublayer independently performs layer normalization on the feature vector of each modality of X. For the i-th feature vector, the mean and variance of the feature vector are calculated, and then each element of the feature vector is standardized. Finally, learnable scaling and translation parameters are introduced to obtain the first normalized feature sequence. The first normalization sublayer eliminates the scale differences of the internal dimensions of the feature vectors of each modality, making the subsequent self-attention calculation more stable and preventing certain dimensions from dominating the attention weights due to excessively large values.

[0085] A multi-head self-attention sub-layer is used to perform window self-attention calculation on the first normalized feature sequence and add a relative position bias matrix to the attention score to obtain an attention-weighted feature sequence.

[0086] In this embodiment, since the sequence length is only 4, the window size is set to 4, meaning the window covers the entire sequence, which is equivalent to global self-attention. Furthermore, to inject relative positional information between modalities, the bias matrix generated by the relative position encoding layer needs to be added to the attention score.

[0087] The multi-head self-attention sublayer of the window first passes through three preset linear transformation matrices. Generate query matrices respectively Key matrix Sum matrix :

[0088] in, This is the first normalized feature sequence. The shape of each matrix is... Then, the 128-dimensional feature dimensions are divided into... Each attention head has a dimension. .Will Remodeled into The tensor is then split along the head dimension. For each head... Calculate the attention score matrix:

[0089] in, Let be the attention score matrix of the t-th attention head. Let t be the query matrix of the attention head. The transpose of the key matrix of the t-th attention head , The relative position bias matrix generated by the relative position encoding layer is directly added to the score matrix, thus injecting positional information. Then, each row of the score matrix is ​​Softmax normalized to obtain the attention weight matrix. :

[0090] Finally, multiplying the attention weights by the value matrix yields the output feature vector of the head:

[0091] in, Let be the value matrix of the t-th attention head. Concatenate the outputs of the 8 heads along the feature dimension to obtain... The matrix, then subjected to a linear transformation The feature sequences are fused to obtain the attention-weighted feature sequence:

[0092] The multi-head self-attention sublayer in this embodiment enables the model to capture complex dependencies between modalities from different perspectives through parallel computation of multiple heads. By directly adding the relative position bias matrix to the attention score, the model can perceive the sequential relationship between modalities when performing cross-modal interactions, thereby improving the semantic accuracy of fusion.

[0093] The first residual connection sublayer is used to add the attention-weighted feature sequence to the input feature sequence to obtain the first residual feature sequence.

[0094] In this embodiment, the first residual connection sublayer adds the attention-weighted feature sequence to the original input feature sequence of the current encoding layer element by element to obtain the first residual feature sequence. This addition is performed element-wise at each position; therefore, the shape of the first residual feature sequence remains the same. .

[0095] The second normalization sublayer is used to perform layer normalization on the first residual feature sequence to obtain the second normalized feature sequence.

[0096] In this embodiment, similar to the first normalized sublayer, the second normalized sublayer also employs layer normalization technology, but its application is to the first residual feature sequence. Normalization before the feedforward network stabilizes the input distribution of the feedforward network and improves the efficiency of the nonlinear transformation. The second normalized sublayer provides a stable and uniformly distributed input to the subsequent feedforward network, which helps accelerate the learning of the feedforward network and prevents training instability caused by changes in the input distribution.

[0097] The feedforward sublayer is used to perform a nonlinear transformation on the second normalized feature sequence to obtain the feedforward output feature sequence.

[0098] The feedforward network in this embodiment enhances the model's nonlinear modeling capability by introducing a nonlinear activation function and a higher hidden layer dimension to perform independent nonlinear transformations at each location. The feedforward network sublayers include a first fully connected layer and a second fully connected layer.

[0099] The feedforward network layer processes the feature vectors of the second normalized feature sequence independently. For each element, the following transformation is performed:

[0100]

[0101] in, and These are the weights and biases of the first fully connected layer. and These are the weights and biases of the second fully connected layer, where x is any vector in the second normalized feature sequence. This is the feedforward output feature sequence. GELU (Gaussian Error Linear Unit) is a smooth nonlinear activation function, and its formula is: , It is the cumulative distribution function of the standard normal distribution. After the above transformation, each row of 128-dimensional input is mapped to a 256-dimensional hidden space, and then mapped back to 128 dimensions to obtain the output feature sequence of the feedforward network. Since each row is processed independently, information between different modalities does not interact in the feedforward network, which ensures the locality of the feedforward network.

[0102] The feedforward network layer in this embodiment greatly enhances the model's nonlinear expressive power, enabling the encoder layer to learn high-order feature patterns within each modality. Complementing the cross-modal interaction function of the self-attention layer, the feedforward network focuses on independent in-depth processing of each modality.

[0103] The second residual connection sublayer is used to add the feedforward output feature sequence to the first residual feature sequence to output the encoded feature sequence of the current coding layer.

[0104] In this embodiment, similar to the first residual connection, the second residual connection sublayer is used to add the output of the feedforward network to the first residual feature sequence, further ensuring the smooth flow of gradients and preserving the original information that has not passed through the feedforward network. The second residual connection sublayer further improves the residual path, enabling the output of the feedforward network to fuse with the information of the self-attention path, while maintaining the smooth propagation of gradients. Through two residual connections and two layer normalizations, each coding layer forms a stable and efficient feature extraction module.

[0105] The above six sub-layers constitute one encoding layer. The SwinTransformer in this embodiment contains four such encoding layers, connected sequentially. The first encoding layer takes the second feature sequence output by the relative position encoding layer as input, processes it internally, and outputs the first encoded feature sequence. The second encoding layer takes the output of the first encoding layer as input and outputs the second encoded feature sequence. The third encoding layer outputs the third encoded feature sequence, and the fourth encoding layer outputs the fourth encoded feature sequence, which is the third feature sequence, used as input to the subsequent modality attention weighting layer.

[0106] In this embodiment of the invention, through the stacking of four encoding layers, the SwinTransformer can progressively abstract high-order cross-modal feature patterns. Shallow encoding layers learn simple interactions and local dependencies between modalities, while deep encoding layers capture complex global semantic relationships. This hierarchical structure conforms to the fundamental principles of deep learning, enabling the model to extract useful emotion-related information from the original features step by step, thereby greatly improving the accuracy and generalization ability of emotion recognition.

[0107] In one embodiment of the present invention, the modal attention weighting layer includes an attention score calculation layer, a weight normalization layer and a feature weighting layer connected in sequence.

[0108] In this embodiment, the modal attention weighting layer is used to dynamically weight the third feature sequence output by the SwinTransformer to adaptively adjust the contribution of different modalities to the final fused features. This layer receives the third feature sequence and sequentially passes it through the attention score calculation layer, the weight normalization layer, and the feature weighting layer to output the weighted feature vectors of each modality. The specific implementation of each sublayer is described in detail below.

[0109] The attention score calculation layer is used to obtain the original attention score of each modality based on the feature vector corresponding to each modality in the third feature sequence and the preset attention score calculation formula.

[0110] In this embodiment, the third feature sequence encoded by the SwingTransformer incorporates contextual information from other modalities into the feature vectors of each modality. However, in real-world applications, the data quality of different modalities may dynamically change due to environmental variations, animal behavior, and other factors. For example, when an animal's face is occluded, the features of the facial modality become unreliable; when the environment is noisy, the signal-to-noise ratio of the speech modality decreases; and when an animal is exercising vigorously, the EEG signal may be contaminated with more motion artifacts. Therefore, it is necessary to calculate an importance score for each modality based on the current input content, so that high-quality modalities receive higher scores and low-quality modalities receive lower scores. The role of the attention score calculation layer is to map the feature vector of each modality to a scalar score, which reflects the model's preliminary assessment of the importance of that modality in the current emotion recognition task.

[0111] The attention score calculation layer computes the original attention score for each modality feature vector in the third feature sequence through a learnable attention network. The network structure is as follows: first, a linear transformation is performed, followed by hyperbolic tangent activation, and then a dot product with a context vector. The specific calculation formula is as follows:

[0112] In the formula: It is the score weight matrix, used to map the 128-dimensional input features to a 64-dimensional hidden space; It is the bias vector corresponding to the score weight matrix; It is the hyperbolic tangent activation function; This is a context vector used to aggregate the 64-dimensional hidden representations into a scalar score. The parameters mentioned above ( These are all trainable parameters of the model, which are automatically updated through backpropagation during training.

[0113] After the above calculations, each modality obtains a raw attention score. The scores of the four modalities form a 4-dimensional vector. The attention score calculation layer, through a learnable network, compresses high-dimensional feature vectors into a single scalar score, enabling a quantitative assessment of the importance of each modality. Compared to fixed weights, this data-driven approach dynamically adjusts the score based on the input content, enhancing the model's adaptability to changes in modality quality.

[0114] The weight normalization layer is used to normalize the original attention scores of each modality to obtain the attention weights of each modality.

[0115] In this embodiment, the raw attention scores output by the attention score calculation layer lack comparability between scores of different modalities. To obtain a set of weights with clear probabilistic meaning and comparable values, the raw scores need to be normalized so that the sum of the weights of all modalities is 1, and each weight is between 0 and 1, representing the attention weights of each modality. A commonly used normalization method is the Softmax function, which can transform a vector of arbitrary real values ​​into a probability distribution while preserving the relative magnitudes of the scores.

[0116] The feature weighting layer is used to multiply the attention weight of each modality with the feature vector of the corresponding modality in the third feature sequence to obtain the weighted feature vector of each modality.

[0117] In this embodiment, after obtaining the normalized attention weights for each modality, these weights need to be applied to the corresponding feature vectors. The weights are used as scalar multipliers, multiplied by each element of the feature vector. In this way, high-weight modal feature vectors dominate in subsequent fusion, while low-weight modal feature vectors are attenuated, thus achieving adaptive feature weighting.

[0118] In this embodiment of the invention, the modal attention weighting layer achieves adaptive weighting of multimodal features through an attention score calculation layer, a weight normalization layer, and a feature weighting layer. This layer can automatically evaluate the reliability of each modality based on the content of the input data and adjust the contribution ratio of each modality in subsequent fusion accordingly. When the quality of a modality deteriorates due to occlusion, noise, motion artifacts, etc., its weight is automatically reduced, effectively avoiding the negative impact of that modality on the fusion result; conversely, high-quality modalities receive higher weights and dominate emotion discrimination. This greatly enhances the robustness of the model in complex real-world environments, enabling the animal emotion recognition system to maintain a high recognition accuracy even when some modal data fails. Simultaneously, the attention weights themselves are interpretable and can be used to analyze the basis of model decisions, facilitating subsequent optimization and troubleshooting.

[0119] In one embodiment of the present invention, step 200, preprocessing and feature extraction of the original EEG signal, original facial image, original speech signal, and original motion acceleration signal to obtain the target animal's EEG feature vector, facial expression feature vector, speech feature vector, and motion posture feature vector, specifically includes: S210. Preprocess the raw EEG signal, raw facial image, raw speech signal and raw motion acceleration signal to obtain target EEG data, facial expression data, effective speech data and target motion data.

[0120] In this embodiment, the raw EEG signals, raw facial images, raw speech signals, and raw motion acceleration signals acquired in step 100 are subjected to targeted denoising, correction, normalization, and enhancement processes.

[0121] In a feasible embodiment, S210, the raw EEG signal, raw facial image, raw speech signal, and raw motion acceleration signal are preprocessed to obtain target EEG data, facial expression data, effective speech data, and target motion data, specifically including: S211. The original EEG signal is filtered and spherical interpolation is performed to obtain the first EEG signal.

[0122] Due to the frequent activity of animals, the raw EEG signals often contain power frequency interference, low-frequency drift, and signal loss or abnormalities caused by poor electrode contact. Therefore, the raw EEG signals need to be filtered to preserve the effective EEG frequency bands and remove irrelevant noise. Spherical interpolation is used to repair data from faulty electrode channels and ensure the integrity of multi-channel signals.

[0123] Specifically, a zero-phase-shift FIR filter can be used to filter the raw EEG signal. The filter order is set to 100, with a passband range of 0.5-50Hz. After filtering, baseline drift below 0.5Hz and high-frequency noise above 50Hz are effectively suppressed. After bandpass filtering, an additional notch filter can be added to specifically filter out 50Hz power frequency interference and its harmonics. Due to animal head shaking, scratching, or electrode detachment, the EEG signal amplitude of individual channels may be abnormal. Therefore, it is necessary to first detect abnormal channels: this can be based on signal amplitude, variance, or correlation with adjacent channels. For detected abnormal channels, spherical interpolation is used for estimation, and the first EEG signal is obtained after determining the interpolation value.

[0124] S212. Perform independent component analysis based on the first EEG signal to obtain the second EEG signal after artifact removal.

[0125] Although the first EEG signal has undergone filtering and interpolation, it still contains various physiological and non-physiological artifacts, especially animal-specific electromyography (EMG) artifacts (generated by facial muscle activity, chewing, and head shaking) and motion artifacts (generated by electrode displacement due to head movement). These artifacts overlap with the EEG signal in both the time and frequency domains and cannot be removed by simple filtering. Independent component analysis (ICA) can decompose the multi-channel mixed signal into statistically independent source signal components, each of which may correspond to real EEG activity or a specific artifact source. By identifying and removing artifact components and then reconstructing the signal, a relatively pure second EEG signal can be obtained.

[0126] S213. Smooth the second EEG signal to obtain the target EEG data.

[0127] Although the artifact-free second EEG signal removes obvious artifact components, some high-frequency jitter or irregular spikes may still remain. These high-frequency components are not within the effective frequency band of EEG, but they can interfere with subsequent feature extraction. Smoothing can further filter out these high-frequency noises through sliding window averaging or low-pass filtering, making the signal waveform smoother and more continuous while preserving the main features of EEG.

[0128] S214. Perform animal face detection and feature point localization on the original facial image to obtain feature point data in the original facial image.

[0129] The original facial image contains the animal's entire head and background information. For subsequent facial expression analysis, it's necessary to first locate the animal's facial region from the image, and then precisely locate key feature points within that region, such as ear tips, corners of the eyes, nose tip, and corners of the mouth. Because animal facial structures differ significantly from human ones—for example, dogs have long muzzles and erect or drooping ears, while cats have triangular faces and vertical pupils—generally used human-based detection and localization models cannot be directly applied. Therefore, this embodiment employs a deep learning-based method, fine-tuned using an animal facial dataset to adapt to the facial features of different breeds and postures of animals.

[0130] Specifically, MTCNN (Multi-Task Cascaded Convolutional Network) is used as the base model. MTCNN consists of three cascaded neural networks: P-Net (candidate box generation network), R-Net (refinement network), and O-Net (output network). First, the original facial image is scaled to different sizes to form an image pyramid, which is then input into P-Net to quickly generate a large number of candidate facial windows. Then, R-Net filters the candidate windows and performs bounding box regression. Finally, O-Net outputs the final facial bounding boxes (position and size). To make the model applicable to animals, MTCNN needs to be trained using a dataset with pre-annotated animal facial bounding boxes (e.g., thousands of images containing different breeds and poses of dogs and cats). After training, the model can output the facial regions of animals.

[0131] Within the detected facial bounding box, 36 animal-specific feature points were further located. These feature points are defined differently from those in humans and specifically include: ear tip, midpoint of the inner edge of the auricle, ear root, inner canthus, outer canthus, midpoint of the upper eyelid, midpoint of the lower eyelid, tip of the nose, left ala nasi, right ala nasi, midpoint of the bridge of the nose, left corner of the mouth, right corner of the mouth, midpoint of the upper lip, midpoint of the lower lip, glabella, left brow ridge, right brow ridge, tip of the chin, and other auxiliary points (such as cheek contours). The coordinates of the 36 located feature points were saved as feature point data.

[0132] S215. Perform image correction based on the original facial image and the feature point data in the original facial image to determine the facial expression image, wherein the facial expression data includes the facial expression image and the feature point data.

[0133] In this embodiment, the feature point coordinates are values ​​in the original image coordinate system. Since animal heads may undergo posture changes such as rotation, tilt, and scaling, directly using these coordinates for shape feature extraction will be affected by posture interference. For example, the calculated angle of the corner of the mouth for the same expression may differ under different head postures. Therefore, geometric correction is needed to transform the original facial image to a standard frontal, horizontal, and uniformly sized posture, while simultaneously mapping the feature point coordinates to the corrected image coordinate system. The corrected image is the facial expression image, which, together with the corrected feature point data, constitutes the facial expression data.

[0134] S216. Noise reduction and feature frequency band extraction are performed on the original speech signal to obtain the effective speech signal segment.

[0135] Because raw speech signals are often mixed with environmental noise and non-speech sounds produced by animal activity, such as the sound of claws scratching the ground and the sound of collars clattering, it is necessary to suppress background noise in order to highlight the content of the animal's vocalizations. At the same time, the effective vocal frequency range of animals differs from that of humans, necessitating the extraction of specific characteristic frequency bands and the filtering out of irrelevant extremely low and ultra-high frequency components, thereby improving the signal-to-noise ratio and the accuracy of subsequent feature extraction.

[0136] Specifically, spectral subtraction can be used to reduce noise in the original speech signal. A bandpass filter can also be used to limit the speech signal to the 100Hz-8kHz range. A 100Hz high-pass filter can remove low-frequency environmental boom and DC offset; an 8kHz low-pass filter can remove ultrasonic noise and aliasing. This frequency band retains the main energy of animal vocalizations while significantly reducing out-of-band noise. After noise reduction and bandpass filtering, a continuous and effective speech signal segment is obtained. To save storage and computation, the long signal segment can be further divided into multiple short vocal segments, each corresponding to one animal vocalization event.

[0137] S217. Perform framing and windowing processing on the effective speech signal segments to obtain effective speech data.

[0138] To perform frequency domain analysis, the continuous speech signal needs to be segmented into short-time frames. However, direct framing can lead to spectral leakage at frame boundaries. Therefore, each frame needs to be multiplied by a window function, such as a Hamming window, to smoothly transition to zero at the beginning and end of the frame, reducing discontinuities. Framing and windowing are fundamental steps in all short-time analysis-based speech processing systems, and will not be elaborated upon further here.

[0139] S218. The original motion acceleration signal is denoised and normalized to obtain the target motion data.

[0140] The raw motion acceleration signal was acquired by a triaxial accelerometer. The sensor output included a gravitational acceleration component and the animal's own motion acceleration. Due to changes in the animal's posture, the projection of the gravitational component onto each axis constantly changes, and directly using the raw acceleration would confuse the motion information. Therefore, it is necessary to remove the gravitational component to obtain linear acceleration. Simultaneously, the raw signal contains high-frequency noise, which requires low-pass filtering for noise reduction. Finally, to eliminate amplitude differences between different individuals, normalization processing is performed. The final target motion data is a three-axis linear acceleration time series.

[0141] S220. Extract the dynamic temporal features and static spectral features of the target EEG data in different frequency bands to obtain the EEG feature vector.

[0142] To extract emotion-related features from target EEG data, it is necessary to characterize EEG signals from multiple dimensions. The power or energy of different frequency bands (such as Delta, Theta, Alpha, Beta, and Gamma waves) is closely related to emotional state; at the same time, emotional change is a dynamic process, and the trend of signal change over time also contains important information. Therefore, this embodiment extracts static spectral features and dynamic temporal features to jointly construct an EEG feature vector.

[0143] Specifically, the target EEG data is segmented into multiple time frames with no overlap between frames, enabling the capture of short-term emotional changes. Each channel signal in each frame is multiplied by a Hamming window, and a Fast Fourier Transform (FFT) is performed on the windowed frame signal to obtain the power spectral density. For five classic EEG frequency bands—Delta, Theta, Alpha, Beta, and Gamma waves—the period mean and standard deviation for each channel are calculated. For each band, the period mean is defined as the integral of the power spectral density within that band divided by the band width, reflecting the average energy of that band; the standard deviation is defined as the standard deviation of the power spectral density within that band, reflecting the dispersion of the energy distribution. Based on the means and standard deviations of the five frequency bands, a normal distribution probability density function is constructed, and the integral value of the current frame data on this distribution is calculated to quantify the stability of the EEG signal. The difference between the means of each frequency band between adjacent time frames is calculated. All extracted features are normalized to unify the value range of each dimension. The amplitude differences between different frames and different animals are eliminated to obtain the final EEG feature vector.

[0144] S230. Extract the shape and texture features of the facial expression data to obtain the facial expression feature vector.

[0145] To characterize an animal's emotional state, facial expression features need to be extracted from two aspects: first, geometric shape features based on feature point coordinates, reflecting morphological changes in facial organs, such as drooping ears, dilated pupils, and upturned corners of the mouth; second, texture features based on the image, such as hair direction and wrinkles, reflecting subtle changes in facial skin. Integrating shape and texture features can form a comprehensive description of facial expressions.

[0146] Specifically, based on the coordinates of 36 feature points, the distances between feature points are calculated, such as the distance from the left ear tip to the outer corner of the left eye, the distance from the right ear tip to the outer corner of the right eye, the interocular distance, the distance from the tip of the nose to the left corner of the mouth, the width of the corner of the mouth, and the vertical distance between the midpoint of the upper eyelid and the midpoint of the lower eyelid. These distances can quantify the displacement of facial organs, such as the ear lowering causing a shortening of the distance between the ear tip and the corner of the eye, and pupil dilation causing an increase in the distance between the upper and lower eyelids. The angles between feature points are also calculated, such as the angle between the line connecting the left ear tip and the left ear root and the horizontal line, the angle between the line connecting the left corner of the mouth and the tip of the nose and the horizontal line, and the head deflection angle. The curvature of feature points is also calculated, such as the curvature of the kiss contour and the curvature of the eye socket. Furthermore, the directional gradient histogram algorithm is used to extract texture features from the facial image. The shape feature vector and the texture feature vector are concatenated to obtain the final facial expression feature vector.

[0147] S240. Extract the phonetic features of the effective speech data to obtain the speech feature vector.

[0148] To identify animal emotions, it is necessary to extract acoustic features that characterize vocalization. This embodiment mainly extracts features such as Mel-frequency cepstral coefficients, fundamental frequency, speech rate, and speech energy. Mel-frequency cepstral coefficients can effectively characterize timbre, fundamental frequency reflects pitch, and speech rate and energy reflect rhythm and loudness. These features combined can distinguish different animal emotions.

[0149] S250. Extract the motion amplitude features of the target motion data to obtain the motion posture feature vector.

[0150] Animal body movements, especially tail wagging, body trembling, and pacing, are closely related to their emotional state. For example, a person's tail wags frequently when happy, trembles or curls up when fearful, and paces frequently when anxious. To quantify these movement patterns, it is necessary to extract features of movement amplitude, such as wagging frequency, wagging amplitude, and rate of change of acceleration. By extracting features such as wagging frequency, wagging amplitude, and rate of change, the raw acceleration signal is transformed into quantitative indicators of movement posture that are highly correlated with emotions.

[0151] This invention employs preprocessing and feature extraction tailored to animal characteristics on four raw signals: electroencephalogram (EEG), facial expression, speech, and motion acceleration. This yields EEG feature vectors, facial expression feature vectors, speech feature vectors, and motion posture feature vectors that fully characterize animal emotions. This not only effectively filters out various noises and artifacts caused by animal activity and eliminates scale differences between individual animals, but also extracts multi-dimensional features highly correlated with animal emotions. These feature vectors retain the rich information required for emotion discrimination and possess stability, comparability, and low redundancy.

[0152] In one embodiment of the present invention, step 400, inputting the multimodal fusion features into a preset emotion recognition model to determine the emotion type of the target animal, specifically includes: S410. Input the multimodal fusion features into the preset emotion recognition model and output the probability distribution of the target animal belonging to different emotion types.

[0153] S420. Determine the target animal's emotion type based on the probability distribution of each emotion type, where the emotion categories include pleasure, anxiety, fear, calmness, and pain.

[0154] This embodiment inputs this multimodal fusion feature into a pre-trained emotion recognition model. The model outputs the probability that the animal belongs to different emotion categories, and finally determines the current emotion type of the animal. This embodiment divides animal emotions into five categories: pleasure, anxiety, fear, calmness, and pain. These five categories are common and distinguishable emotional states, which can meet the practical application needs of animal health care, behavior training, etc.

[0155] In this embodiment, the emotion recognition model adopts a Transformer-based encoder-decoder structure. The input is a multimodal fusion feature vector, and the output is a probability vector. The model uses a large amount of labeled data during the training phase. The training dataset is constructed by collecting EEG, facial, speech, and motor data from animals of different breeds, ages, and sexes in various situations, and labeling each time segment with emotion categories based on video and behavioral records. During training, the cross-entropy loss function is used, and focus loss can be combined to alleviate the sample imbalance problem. In the practical application phase, the multimodal fusion feature vector generated in step 300 is input into the trained emotion recognition model. After forward computation, the model outputs a probability distribution, corresponding to the probabilities of five emotion categories: pleasure, anxiety, fear, calmness, and pain, with the sum of all probabilities being 1. Based on the probability distribution, the category with the highest probability value is selected as the current emotion recognition result for the animal. For example, if the output probability distribution is [0.05, 0.10, 0.70, 0.10, 0.05], then the third category has the highest probability, therefore the animal's current emotion is determined to be the third category. In addition, the model can also output the confidence level of the category, which can be used for subsequent decision-making or early warning.

[0156] This invention classifies emotions based on multimodal fusion features, making it easier for users to observe and understand animal emotions, and helping to detect abnormal animal states in a timely manner and take intervention measures.

[0157] Figure 2 An internal structural diagram of a computer device according to one embodiment of the present invention is shown. This computer device can specifically be a terminal or a system. Figure 2 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program causes the processor to perform the steps in the above-described method embodiments. The internal memory may also store a computer program, which, when executed by the processor, causes the processor to perform the steps in the above-described method embodiments. Those skilled in the art will understand that... Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps in the above method embodiments.

[0159] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the steps in the above method embodiments.

[0160] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A method for identifying animal emotions based on multimodal data, characterized in that, The method includes: Collect raw electroencephalogram (EEG) signals, raw facial images, raw speech signals, and raw motion acceleration data of the target animal; The original EEG signals, original facial images, original speech signals, and original motion acceleration signals are preprocessed and feature extracted to obtain the EEG feature vector, facial expression feature vector, speech feature vector, and motion posture feature vector of the target animal. The multimodal fusion module is used to fuse the EEG feature vector, facial expression feature vector, speech feature vector and motion posture feature vector of the target animal to obtain multimodal fusion features. The multimodal fusion features are input into a preset emotion recognition model to determine the emotion type of the target animal.

2. The method as described in claim 1, characterized in that, The multimodal fusion module includes a modality adaptation layer, a feature sequence splicing layer, a relative position encoding layer, a SwingTransformer, a modality attention weighting layer, and a feature fusion and batch normalization layer connected in sequence. The modality adaptation layer is used to map the EEG feature vector, facial expression feature vector, speech feature vector and motion posture feature vector to the same dimensional space to obtain standard EEG feature vector, standard facial expression feature vector, standard speech feature vector and standard motion posture feature vector with the same dimension. The feature sequence splicing layer is used to splice the standard EEG feature vector, standard facial expression feature vector, standard speech feature vector, and standard motion posture feature vector to obtain the spliced ​​first feature sequence. The relative position encoding layer is used to add position information to the feature vectors corresponding to each modality in the first feature sequence to obtain a second feature sequence with position information. The SwinTransformer is used to perform spatial context encoding on the second feature sequence to obtain the encoded third feature sequence. The modal attention weighting layer is used to determine the weight of each modality based on the third feature sequence, and to perform weighting according to the weight of each modality and the feature vector in the third feature sequence to obtain the weighted feature vector corresponding to each modality. The feature fusion and batch normalization layer is used to fuse, reduce the dimensionality, and batch normalize the feature vectors corresponding to each of the weighted modalities to obtain multimodal fusion features.

3. The method as described in claim 2, characterized in that, The relative position encoding layer comprises a position offset code and distance calculation layer, an offset matrix generation layer, and the position information fusion layer, which are connected in sequence. The position offset code and distance calculation layer is used to perform position encoding on the feature vectors corresponding to each mode in the first feature sequence, and calculate the relative distance between all mode position pairs based on the position encoding of the feature vectors, wherein the mode position pair is a data pair composed of any two modes in the first feature sequence; The bias matrix generation layer is used to obtain a relative position bias matrix based on the relative distance between all modal position pairs and a preset bias parameter table, wherein the bias parameter table contains a mapping relationship between relative distance and bias parameter values. The location information fusion layer is used to fuse the relative position bias matrix with the first feature sequence to obtain a second feature sequence with relative position information.

4. The method as described in claim 3, characterized in that, The SwinTransformer comprises four sequentially connected coding layers, each of which includes: The first normalization sublayer is used to perform layer normalization on the input feature sequence to obtain the first normalized feature sequence; A multi-head self-attention sub-layer is used to perform window self-attention calculation on the first normalized feature sequence, and the relative position bias matrix is ​​added to the attention score to obtain the attention-weighted feature sequence. The first residual connection sublayer is used to add the attention-weighted feature sequence to the input feature sequence to obtain the first residual feature sequence; The second normalization sublayer is used to perform layer normalization on the first residual feature sequence to obtain the second normalized feature sequence. The feedforward network sublayer is used to perform a nonlinear transformation on the second normalized feature sequence to obtain the feedforward output feature sequence. The second residual connection sub-layer is used to add the feedforward output feature sequence to the first residual feature sequence and output the encoding feature sequence of the current encoding layer. The SwinTransformer uses the second feature sequence as the input to the first coding layer, the coding feature sequence output by each coding layer is the input to the next coding layer, and the coding feature sequence output by the last coding layer is the third feature sequence.

5. The method as described in claim 2, characterized in that, The modal attention weighting layer comprises, in sequence, an attention score calculation layer, a weight normalization layer, and a feature weighting layer: The attention score calculation layer is used to obtain the original attention score of each modality based on the feature vector corresponding to each modality in the third feature sequence and the preset attention score calculation formula. The weight normalization layer is used to normalize the original attention scores of each modality to obtain the attention weights of each modality. The feature weighting layer is used to multiply the attention weight of each modality with the feature vector of the corresponding modality in the third feature sequence to obtain the weighted feature vector of each modality.

6. The method as described in claim 1, characterized in that, The preprocessing and feature extraction of the original EEG signals, original facial images, original speech signals, and original motion acceleration signals to obtain the target animal's EEG feature vector, facial expression feature vector, speech feature vector, and motion posture feature vector specifically includes: The raw EEG signals, raw facial images, raw speech signals, and raw motion acceleration signals are preprocessed to obtain target EEG data, facial expression data, effective speech data, and target motion data. The dynamic temporal features and static spectral features in different frequency bands of the target EEG data are extracted to obtain the EEG feature vector; Extract the shape and texture features of the facial expression data to obtain a facial expression feature vector; Extract the phonetic features from the effective speech data to obtain a speech feature vector; The motion amplitude features of the target motion data are extracted to obtain the motion posture feature vector.

7. The method as described in claim 6, characterized in that, The preprocessing of the raw EEG signals, raw facial images, raw speech signals, and raw motion acceleration signals to obtain target EEG data, facial expression data, effective speech data, and target motion data specifically includes: The original EEG signal is filtered and spherical interpolation is performed to obtain the first EEG signal; Independent component analysis was performed on the first EEG signal to obtain the second EEG signal after artifact removal. The second EEG signal is smoothed to obtain the target EEG data; Animal face detection and feature point localization are performed on the original facial image to obtain feature point data in the original facial image; Image correction is performed based on the original facial image and the feature point data in the original facial image to determine the facial expression image, wherein the facial expression data includes the facial expression image and the feature point data; The original speech signal is denoised and its feature frequency bands are extracted to obtain effective speech signal segments; The effective speech signal segments are framed and windowed to obtain effective speech data; The original motion acceleration signal is denoised and normalized to obtain the target motion data.

8. The method according to claim 1, characterized in that, The step of inputting the multimodal fusion features into a preset emotion recognition model to determine the emotion type of the target animal specifically includes: The multimodal fusion features are input into a preset emotion recognition model, and the probability distribution of the target animal belonging to different emotion types is output. The target animal's emotion type is determined based on the probability distribution of each emotion type, wherein the emotion categories include pleasure, anxiety, fear, calmness, and pain.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

Images