Old people emotion recognition method based on hierarchical perception enhancement and multi-granularity constraint

Abstract

The invention discloses an elderly emotion recognition method based on hierarchical perception enhancement and multi-granularity constraint, and belongs to the field of emotion recognition. The method comprises the following steps: constructing an audio and video multi-mode spontaneous emotion recognition database for old people, and preprocessing audio data and image data in the database; the method comprises the following steps of: respectively extracting audio and visual multi-scale features by using a double-flow audio-visual encoder, and inputting the audio and visual multi-scale features into a hierarchical perception enhancement module to obtain enhanced single-mode feature representation; in the training stage, deep semantic association learning is realized for the enhanced single-mode feature representation through a bidirectional cyclic consistency collaboration module, and iterative optimization training is completed by using a global coverage cross-mode multi-granularity constraint strategy; in the inference stage, the enhanced single-mode feature representations are cascaded, and sentiment classification is output through a full-connection layer. Compared with the prior art, the accuracy and generalization ability of sentiment recognition tasks of the old people are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition, specifically to a method for emotion recognition in the elderly based on hierarchical perception enhancement and multi-granularity constraints. Background Technology

[0002] Emotion, as a core element of human psychological activity, plays a crucial role in cognitive decision-making and social interaction. Emotional health is not only an important standard of mental health but also a fundamental guarantee for maintaining social function. With the deepening of the concept of active aging, the emotional health of the elderly has received widespread attention, and the emotional recognition mechanisms of the elderly, due to their unique life stage characteristics, exhibit multi-dimensional research value. Therefore, constructing emotional recognition methods that conform to the characteristics of the elderly is a core link in achieving healthy aging and an important breakthrough in alleviating the care pressure on families and society.

[0003] Existing emotion recognition methods can be broadly categorized into two types: those based on non-physiological indicators (such as facial expressions and voice tone) and those relying on physiological signals (such as electroencephalograms and electrocardiograms). Compared to the technical bottlenecks in physiological signal acquisition, such as high equipment invasiveness and strict environmental restrictions, non-physiological signals, with their advantages of non-contact acquisition and significant cost-effectiveness, have become the most promising research direction in the field of affective computing. Previous psychological studies have further confirmed that audio and visual modalities play a significantly dominant role in emotion perception.

[0004] In the development of audiovisual fusion multimodal emotion recognition technology, early methods were mostly based on manually designed features, which greatly limited the performance of emotion recognition. With the breakthrough progress of convolutional neural networks, researchers have achieved automated learning of cross-modal features. Existing research has enabled visual and audio modalities to capture their unique spatiotemporal dynamic features and acoustic prosodic features separately during independent encoding and decoding processes, enhancing the modality-specific representation ability. However, existing methods do not fully simulate the hierarchical processing of human audiovisual cognition, limiting the model's ability to understand modal semantics in more complex scenarios.

[0005] Building upon modality-specific representations, cross-modal semantic learning has become a core challenge hindering the improvement of emotion recognition performance. Due to the inherent differences between audiovisual signals, their latent representations are often distributed across heterogeneous feature spaces, making the establishment of effective cross-modal interaction mechanisms crucial. Early research attempted to construct surface-level associations between modalities but failed to simulate the dynamic collaborative mechanisms between audiovisual modalities in the human nervous system. Existing research has attempted to mine latent similarity relationships between audiovisual signals by imposing multiple constraints. However, most methods are limited to explicit feature spaces and fail to fully explore implicit higher-order semantic associations. Therefore, existing constraint-based audiovisual similarity representation learning does not yet comprehensively cover the multi-granularity requirements.

[0006] Specifically, existing technologies have at least the following shortcomings: Audio-based emotion recognition methods often focus on the direct mapping between acoustic features and semantic labels, neglecting the hierarchical perception mechanism of human hearing. This may limit the model's performance in complex emotional scenarios. Visual-based emotion recognition methods are essentially still at a shallow level of perceptual representation, lacking a hierarchical cognitive guidance mechanism, which may lead to robustness defects in complex emotional scenarios. At the same time, there are still shortcomings in cross-modal fine-grained semantic alignment and multi-granularity learning objectives. In particular, balancing the need for single-modal information integrity with achieving cross-modal semantic depth alignment still faces key challenges.

[0007] To address the aforementioned issues, this invention proposes a method that enhances audiovisual single-modal hierarchical representation and meets the requirements for audiovisual cross-modal multi-granularity alignment, thereby improving the accuracy and generalization ability of emotion recognition tasks for the elderly population. Summary of the Invention

[0008] The purpose of this invention is to propose an elderly emotion recognition method based on hierarchical perception enhancement and multi-granularity constraints to solve the problems of existing technologies in elderly emotion recognition, which lack a single-modal hierarchical processing process that simulates human visual and auditory cognition and are difficult to fully cover cross-modal multi-granularity constraints.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: An elderly emotion recognition method based on hierarchical perception enhancement and multi-granularity constraints includes the following steps: S1. Construct a multimodal spontaneous emotion recognition database for elderly people's audio and video, and preprocess the multimodal data in the database; S2. Input the preprocessed multimodal data into the dual-stream audiovisual encoder in pairs. Extract multi-scale features of audio and vision through the improved feature pyramid network. Then input the obtained features into the hierarchical perception enhancement module. Obtain the enhanced single-modal feature representation through hierarchical feature aggregation in a specific direction. S3. During the training phase, the enhanced single-modal feature representation is passed through the bidirectional cyclic consistency collaborative module, and deep semantic association learning is achieved by utilizing bidirectional cyclic consistency constraints. S4. In the final stage of training, a global cross-modal multi-granularity constraint strategy is used to iteratively optimize the training process by passing the audiovisual single-modal prediction results through the OGM_GE modulation module. S5. In the inference stage, the enhanced single-modal feature representations are cascaded and output as sentiment classifications through a fully connected layer.

[0010] Preferably, the specific workflow for the preprocessing of the multimodal data in S1 is as follows: Each audio segment is segmented based on speech gap detection. The threshold for identifying a gap is set to 30 frames. The audio segments of a single sentence are accurately segmented from continuously repeated sentences and converted into Mel spectrograms, which become the smallest unit of audio modality in the dataset. Based on the audio segmentation, images are extracted from the corresponding video segments at a rate of 5 frames per second. Then, face detection technology is used to accurately identify face regions and segment the images, unify the image resolution, and finally obtain a set of image sequences that focus only on faces, which become the smallest unit of visual modality in the dataset.

[0011] Preferably, the specific workflow of the hierarchical perception enhancement module in S2 is as follows: The image modality employs a top-down semantic guidance mechanism, inputting both deep and shallow features into a cross-scale multi-head self-attention module (CSA). The audio modality adopts a bottom-up high and low frequency feature aggregation strategy, which inputs both shallow and deep features into the cross-scale multi-head self-attention module (CSA). Through the collaborative action of multi-level cross-scale multi-head self-attention modules (CSA), residual connections are used to perform feature fusion between levels, outputting enhanced feature representations after single-modality layer-by-layer perception enhancement.

[0012] Preferably, the specific workflow of the bidirectional circular consistency coordination module in S3 is as follows: The enhanced audio modality features are used to represent the input visual features to generate a branch, and the generator and discriminator are used for adversarial training to generate pseudo visual features. The enhanced visual modality features are used to represent the input audio features to generate a branch, and pseudo-audio features are generated through adversarial training. The pseudo-visual features and pseudo-audio features are learned from their respective enhanced unimodal feature representations through bidirectional cyclic consistency loss, thus establishing a bidirectional feature mapping relationship between modalities.

[0013] Preferably, the global cross-modal multi-granularity constraint strategy described in S4 specifically includes the following: At the pixel level, attention consistency constraints are imposed between the cross-modal representations of the outputs of the cross-scale multi-head self-attention module (CSA) at each level; At the feature level, an enhanced contrast loss constraint is applied before feature fusion; In deep space, apply a bidirectional cyclic consistency constraint; At the decision level, standard cross-entropy loss is used; The overall loss function for the training phase is obtained by weighting the constraints at different levels mentioned above.

[0014] The present invention further protects a computer device, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the instruction, program, code set or instruction set being loaded and executed by the processor to implement the above-mentioned method for elderly emotion recognition based on hierarchical perception enhancement and multi-granularity constraints.

[0015] The present invention further protects a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the instruction, program, code set, or instruction set is loaded and executed by a processor to implement the above-mentioned method for elderly emotion recognition based on hierarchical perception enhancement and multi-granularity constraints.

[0016] Compared with existing technologies, this invention provides a method for emotion recognition in the elderly based on hierarchical perception enhancement and multi-granularity constraints, which has the following beneficial effects: (1) This invention proposes a hierarchical perception enhancement module, which realizes the layer-by-layer enhancement of high-frequency emotional cues in audio in the low-frequency semantic space and the layer-by-layer enhancement of facial expression details in global features. It fully simulates the human hierarchical perception guidance mechanism of audiovisual signals and effectively solves the information loss caused by insufficient single-modal representation in traditional methods. (2) This invention proposes a bidirectional cyclic consistency collaborative module, which utilizes adversarial loss to minimize mapping differences and constructs an implicit bidirectional mapping of the audiovisual latent semantic space to achieve deep cross-modal semantic learning and effectively capture the asymmetric but complementary cross-modal associations in the emotional expression of the elderly. (3) This invention proposes a cross-modal multi-granularity constraint strategy, which realizes the collaborative modeling of micro-changes and macro-semantics, thereby overcoming the limitations in depth and granularity of audiovisual cross-modal interaction. Attached Figure Description

[0017] Figure 1 is a flowchart of the elderly emotion recognition method based on hierarchical perception enhancement and multi-granularity constraints proposed in this invention; Figure 2 is a diagram of the segmentation steps based on speech gaps mentioned in Embodiment 1 of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0019] Example 1: Please refer to Figure 1. This invention proposes a method for emotion recognition in the elderly based on hierarchical perception enhancement and multi-granularity constraints. The method includes the following steps: S101: Data Acquisition and Preprocessing We independently collected spontaneous emotion recognition data on multimodal audio and video from elderly people in East Asia. As shown in Figure 2, each audio segment was segmented based on speech gap detection. The threshold for identifying a gap was set to 30 frames. Audio segments containing a single sentence were accurately segmented from consecutively repeated sentences and converted into Mel spectrograms, becoming the smallest unit of the audio modality in the dataset. Based on the audio segmentation, images of the corresponding video segments were extracted at a rate of 5 frames per second. Then, face detection technology was used to accurately identify face regions and segment the images. The image resolution was standardized to 224×224, resulting in a set of image sequences focusing only on faces, becoming the smallest unit of the visual modality in the dataset. The emotion labels of the audio and visual modality smallest units were aligned one-to-one to construct training samples.

[0020] S102: Single-modal hierarchical feature perception enhancement The preprocessed paired multimodal samples are input into the hierarchical perception enhancement modules. For the image modality, a top-down semantic guidance mechanism is used, inputting both deep and shallow features into the cross-scale multi-head self-attention (CSA) module. For the audio modality, a bottom-up high- and low-frequency feature aggregation strategy is employed, inputting both shallow and deep features into the CSA. Finally, through the collaborative action of multiple CSA modules, residual connections are used for inter-level feature fusion, outputting the enhanced feature representation of a single modality after layer-by-layer hierarchical perception enhancement.

[0021] S103: Bidirectional Cyclic Consistency Collaborative Learning During the training phase, the enhanced unimodal feature representations are processed through a bidirectional cyclic consistency co-processing module to achieve deep semantic association learning using bidirectional cyclic consistency constraints. The enhanced audio modality feature representation is input to the visual feature generation branch, which uses adversarial training between the generator and discriminator to generate pseudo-visual features; similarly, the enhanced visual modality feature representation is input to the audio feature generation branch, which also generates pseudo-audio features through adversarial training. The pseudo-features of the visual and audio modalities are then learned from their respective enhanced unimodal feature representations using bidirectional cyclic consistency loss, establishing a bidirectional feature mapping relationship between modalities.

[0022] S104: Cross-modal multi-granularity constraint learning In the final stage of training, a global, cross-modal, multi-granular constraint strategy is employed, while the audiovisual single-modal prediction results are iteratively optimized through the OGM_GE modulation module. At the pixel level, attention consistency constraints are applied between the cross-modal representations output by each level of the CSA module; at the feature level, enhancement contrast loss constraints are applied before feature fusion; and in the deep space, bidirectional cyclic consistency constraints are applied. The weighted average of these constraints yields the final loss function for the training phase, guiding model optimization.

[0023] S105: Using a trained network for emotion recognition inference The audio and video sentiment data to be evaluated is input into the network after undergoing the same preprocessing. After passing through the hierarchical feature perception enhancement module, the obtained audio-visual single-modal enhanced feature representations are concatenated and passed through a fully connected layer to obtain the predicted sentiment classification.

[0024] S106: Technology Application Emotional recognition technology for the elderly can be deployed on server-side or terminal devices for various business scenarios such as elderly care, health management, and smart services. In home care scenarios, it can connect to home smart monitoring devices or wearable devices to collect and recognize emotional signals such as facial expressions and voice tone in real time, providing emotional state alerts to family members or caregivers. When negative emotions such as loneliness, anxiety, and depression are detected, timely reminders are pushed to assist in the development of personalized care plans. In elderly care institution service scenarios, it can be combined with smart nursing terminals to continuously monitor and analyze the emotional state of the elderly in their daily interactions, guiding caregivers to optimize service methods and adjust... Interactive strategies provide an emotional dimension reference for evaluating the service quality of institutions. In geriatric medical diagnosis and rehabilitation scenarios, it can be integrated with medical diagnostic equipment, combining emotion recognition results with physiological indicators to provide auxiliary references for early screening, disease assessment, and rehabilitation effect tracking of diseases such as geriatric depression and Alzheimer's disease, thus contributing to the formulation of precise treatment plans. In elderly intelligent human-computer interaction scenarios, it can be integrated into devices such as smart speakers and companion robots, adaptively adjusting the interaction mode by recognizing the emotional needs of the elderly. For example, it can use gentle and soothing voice feedback and push positive content to elderly people with low moods, improving the adaptability and comfort of human-computer interaction. Since the elderly emotion recognition technology can complete the emotional state judgment based on non-invasive collection methods without adding extra physiological burden to the elderly, it can improve the accuracy and humanistic care level of various elderly care services and health management systems while ensuring the user experience of the elderly.

[0025] Example 2: Based on Example 1, but with some differences, the scheme in Example 1 will be further described below with specific calculation formulas and example data, as detailed in the following description: S201: Dataset and Label Settings 143 elderly participants (85 women and 58 men) with normal emotional expression abilities were selected for data collection, aged 60-90 years. All participants agreed to the experimental protocol and signed informed consent forms beforehand. Prior to data collection, a professional psychological counselor conducted a psychological assessment of the participants using the Hamilton Depression Rating Scale and the Hamilton Anxiety Rating Scale. To obtain image and audio data in a natural emotional state, a standardized procedure was adopted during the data collection process. Participants repeatedly read a balanced text without words containing obvious emotional biases, while voice and facial expression data were collected simultaneously. Finally, a three-category emotional label was established based on the clinical assessment results: positive, neutral, and negative. After screening, 27 samples were positive, 83 were neutral, and 33 were negative.

[0026] Each audio segment is segmented based on speech gap detection, with a threshold of 30 frames for identifying a gap. This allows for precise segmentation of audio segments containing a single sentence from consecutively repeated statements, forming the smallest unit of the audio modality in the dataset. Based on the audio segmentation, images of the corresponding video segments are extracted at a rate of 5 frames per second. Face detection technology is then used to accurately identify face regions and segment the images, maintaining a uniform resolution of 224×224. This results in a sequence of images focusing solely on faces, forming the smallest unit of the visual modality in the dataset.

[0027] During training, the dataset was divided into training and testing sets in an 8:2 ratio, and five-fold repeated experiments were used to improve the stability of the results.

[0028] S202: Single-modal hierarchical feature perception enhancement Specifically, a visual encoder is first used to extract visual multi-scale features. ,in ∈{1,2,…, }and =4, ∈ , The size of the space of the processed feature map. D For feature dimensions. In multi-scale features, A larger value indicates a deeper level of features. The deeper features and the shallower features are then input together into the cross-scale multi-head self-attention module (CSA). CSA is used to achieve adaptive hierarchical guidance of modal signals. During CSA feature calculation, the deeper-level features are used to generate a guiding query vector. Q Shallower-level features generate key vectors K Sum value vector V Define the input of CSA as follows: and The output after CSA is ,in , ,set up The specific visual modality-level perception enhancement process can be represented by the following formula:

[0029]

[0030] in , For feature dimensions. Through self to The attention mapping is represented, which actually strengthens the shallow features. Medium and deep features Consistent micro-expression detail semantic features. Finally, through the collaborative action of multi-level CSA modules, residual connections are used for inter-level feature fusion, achieving deep feature-guided enhancement of shallow feature micro-expression details. After top-down, layer-by-layer perceptual enhancement, the final output is obtained. for,

[0031] Specifically, the original audio is first converted into a Mel spectrogram as input for the audio modality branch, and then an audio encoder is used to extract multi-scale audio features. ,in ∈{1,2,…, }and =4, ∈ , The size of the space of the processed feature map. D For feature dimensions. Conversely, shallow features and deeper features are input together into CSA, and the input of CSA is defined as... and The output after CSA is ,in , ,set up The specific audio modality-level perceptual enhancement process can be represented by the following formula:

[0032]

[0033] in , For feature dimensions. Through self to The attention mapping is represented, which actually enhances the deep features. Medium and shallow features Consistent high-frequency instantaneous semantic features. Ultimately, through the collaborative action of multi-level CSA modules, the focus on high-frequency semantic information is significantly enhanced. After top-down, layer-by-layer perceptual enhancement, the final output can be obtained. for,

[0034] S203: Bidirectional Cyclic Consistency Collaborative Learning (1) Visual feature generation branch Visual generator By embedding audio features extracted from the audio hierarchy perception enhancement module With a random noise sampled from a standard Gaussian distribution N(0,1) Synthesizing fake visual features To deceive visual feature discriminators To distinguish synthetic features and true characteristics Visual discriminator A fully connected layer is used to perform the binary classification task, while an auxiliary classification head is integrated for image category prediction. To address the instability of traditional GAN ​​training, a more stable training method called WGAN is adopted, with its loss function defined as follows.

[0035]

[0036]

[0037]

[0038] In the above formula, Represents mathematical expectation, and Let represent the parameters of the visual generator and the visual discriminator, respectively. In the last term of equation (7), a gradient penalty term is used to apply the Lipschitz constraint. Specifically, ,in .variable and There are two hyperparameters. Represents the gradient penalty factor. Used to measure the contribution to visual feature classification loss. Here This represents the cross-entropy loss between visual features and their corresponding ground truth labels.

[0039] (2) Audio feature generation branch To address the multimodal cycle consistency loss introduced during cross-modal bidirectional latent feature mapping learning, a similar audio representation generation branch, synthesizing audio features from visual features, is designed. Similar to the visual representation generation branch, the obtained image is first embedded... With random sampled noise vector from a standard Gaussian distribution Connect them to generate synthesized audio signal features in the latent space. Meanwhile, the discriminator For real audio features and synthesized audio features To differentiate them, their objective function can be similarly defined as follows:

[0040]

[0041]

[0042]

[0043] The adversarial loss described by Equation (9) consists of three parts: the first two terms synthesize the distribution difference between the audio feature distribution and the real audio feature distribution through the Wasserstein distance metric, and the last term constrains the discriminator through a gradient penalty term. Equation (10) introduces the audio feature-supervised classification loss. By minimizing the class prediction error between generated features and true features, consistent discriminative performance between the two types of features is ensured.

[0044] (3) Bidirectional cyclic consistency loss The bidirectional circular consistency loss function The definition is as follows:

[0045]

[0046]

[0047] in Indicates the number of samples. These are the coefficients used in the calculation. Based on the principle of minimizing reconstruction error, this invention proposes to use mean squared error to measure the deviation between reconstructed features and true features, and then construct a bidirectional cyclic consistency loss function. This constraint method can effectively quantify the differences in feature distributions between modes, ensure inter-modal consistency, and effectively solve the problem of modal heterogeneity.

[0048] S204: Cross-modal multi-granularity constraint learning Because there are significant distributional differences between audio and visual modalities, the cross-scale attention mechanism may further increase the heterogeneity of distribution between modalities, which is detrimental to multimodal learning. Therefore, an attention consistency loss mechanism is proposed. By quantifying the probability distribution differences between cross-modal attention maps at the same level, the model's dynamic constraints on the cross-modal feature alignment process are strengthened. Specifically, a hierarchical attention consistency loss function is constructed by calculating the weighted sum of Kullback-Leibler (KL) divergences between cross-modal attention maps layer by layer. :

[0049]

[0050] in ∈{1,2,…, }and =3. They represent the first in the network. The audio and visual attention maps output by each CSA module. Then it means the first Attention consistency loss at each level, hyperparameters Used to measure the contribution of each level of loss term to the overall loss.

[0051] An enhanced contrastive learning loss is designed to target the audiovisual modal features output by the hierarchical cognitive enhancement module. By constructing dynamic symmetric regularization constraints, cross-modal feature distributions are driven to converge toward a shared latent semantic space. Specifically, an instance-level cross-modal feature alignment method is designed, which achieves effective alignment across the feature space by explicitly optimizing the cosine similarity maximization objective of matched sample pairs while constraining the cosine similarity minimization of unmatched sample pairs. In batch processing of audio-visual data, using Indicates the first Each audio feature is embedded, using Indicates the first If visual features are embedded, then the enhanced contrastive learning loss... It can be represented as:

[0052]

[0053]

[0054] To accurately decouple emotion classification information from multimodal features, a task-oriented loss calculation is performed on the fused cross-modal representation. This is achieved through standard cross-entropy loss (…). The model quantifies the semantic deviation between predicted labels and true labels, forcing the model to learn sentiment-related discriminative features. Its mathematical expression is:

[0055] in For real labels, For predicting labels, The number of samples; Simultaneously, a bidirectional circular consistency loss is introduced. A cross-modal multi-granularity constraint framework is constructed, ranging from spatiotemporal fine-grained to semantic abstraction. This framework achieves comprehensive and deep fusion of cross-modal features across multiple semantic spaces through global multi-granularity feature alignment and distribution matching. In summary, the overall loss function is as follows, where the hyperparameters... and Used to fine-tune the balance between the three loss terms:

[0056] S105: Using a trained network for emotion recognition inference The audio and video sentiment data to be evaluated is input into the network after undergoing the same preprocessing. After passing through the hierarchical feature perception enhancement module, the obtained audio-visual single-modal enhanced feature representations are concatenated and passed through a fully connected layer to obtain the predicted sentiment classification.

[0057] Example 3 Based on Examples 1-2, but with some differences, the feasibility of the schemes in Examples 1 and 2 is verified below with specific experiments, as detailed in the following description: To verify the performance of the proposed method, several typical audiovisual multimodal emotion recognition methods were selected as baselines and compared under the same dataset and parameter settings. The results are shown in Table 1.

[0058] Table 1. Performance comparison of methods based on self-built databases. Bold text indicates the best result.

[0059] As shown in Table 1, the multimodal method proposed in this invention achieves the best overall performance, demonstrating the effectiveness of combining single-modal hierarchical perceptual enhancement with cross-modal multi-granular constraints. In the audio modality, the method achieves the highest recognition accuracy, thanks to its bottom-up hierarchical perceptual enhancement design, which effectively captures high-frequency emotional cues. In the visual modality, the proposed method remains highly competitive, with a performance gap of less than 1% compared to the best-performing model, HiCMAE.

[0060] To evaluate the generalization ability of the proposed method, cross-database experiments were conducted on the CREMA-D dataset, and the results are shown in Table 2.

[0061] Table 2 Performance Comparison of Methods Based on CREMA-D Database

[0062] As shown in Table 1, the proposed method achieves an accuracy of 55.32%, which, although lower than methods such as OGM_GE, QMF, and MATER, is mainly due to inherent differences between datasets. Specifically, the dataset developed in this invention focuses on assessing the natural psychological state of the elderly, while CREMA-D is a performative emotion recognition dataset with more exaggerated and structured emotional expressions. Despite these domain differences, the proposed method maintains a performance level comparable to methods such as BiGated, EF-GRU, and LF-GRU. These results demonstrate that the proposed method has strong generalization ability across datasets, validating its effectiveness in capturing audiovisual emotional cues.

[0063] In summary, the experimental results of this embodiment show that, compared with various existing audiovisual multimodal baseline methods, the elderly emotion recognition method based on hierarchical perception enhancement and multi-granularity constraints proposed in this invention has significant performance advantages on the self-built East Asian elderly audio-visual multimodal spontaneous emotion recognition database, and also demonstrates its strong generalization ability, thus verifying the effectiveness and feasibility of the technical solutions described in Embodiments 1 and 2.

[0064] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for emotion recognition in the elderly based on hierarchical perception enhancement and multi-granularity constraints, characterized in that, Includes the following steps: S1. Construct a multimodal spontaneous emotion recognition database for elderly people's audio and video, and preprocess the multimodal data in the database; S2. Input the preprocessed multimodal data into the dual-stream audiovisual encoder in pairs. Extract multi-scale features of audio and vision through the improved feature pyramid network. Then input the obtained features into the hierarchical perception enhancement module. Obtain the enhanced single-modal feature representation through hierarchical feature aggregation in a specific direction. S3. During the training phase, the enhanced single-modal feature representation is passed through the bidirectional cyclic consistency collaborative module, and deep semantic association learning is achieved by utilizing bidirectional cyclic consistency constraints. S4. In the final stage of training, a global cross-modal multi-granularity constraint strategy is used to iteratively optimize the training process by passing the audiovisual single-modal prediction results through the OGM_GE modulation module. S5. In the inference stage, the enhanced single-modal feature representations are cascaded and output as sentiment classifications through a fully connected layer.

2. The method for elderly emotion recognition based on hierarchical perception enhancement and multi-granularity constraints according to claim 1, characterized in that, The specific workflow for preprocessing the multimodal data described in S1 is as follows: Each audio segment is segmented based on speech gap detection. The threshold for identifying a gap is set to 30 frames. The audio segments of a single sentence are accurately segmented from continuously repeated sentences and converted into Mel spectrograms, which become the smallest unit of audio modality in the dataset. Based on the audio segmentation, images are extracted from the corresponding video segments at a rate of 5 frames per second. Then, face detection technology is used to accurately identify face regions and segment the images, unify the image resolution, and finally obtain a set of image sequences that focus only on faces, which become the smallest unit of visual modality in the dataset.

3. The method for elderly emotion recognition based on hierarchical perception enhancement and multi-granularity constraints according to claim 1, characterized in that, The specific workflow of the hierarchical perception enhancement module described in S2 is as follows: The image modality adopts a top-down semantic guidance mechanism, which inputs both deep features and shallower features into the cross-scale multi-head self-attention module; The audio modality adopts a bottom-up high and low frequency feature aggregation strategy, which inputs both shallow and deep features into a cross-scale multi-head self-attention module. By leveraging the collaborative efforts of multi-level, cross-scale, multi-head self-attention modules, residual connections are used for inter-level feature fusion, resulting in an enhanced feature representation after single-modal, layer-by-layer perceptual enhancement.

4. The method for elderly emotion recognition based on hierarchical perception enhancement and multi-granularity constraints according to claim 1, characterized in that, The specific workflow of the bidirectional circular consistency coordination module described in S3 is as follows: The enhanced audio modality features are used to represent the input visual features to generate a branch, and the generator and discriminator are used for adversarial training to generate pseudo visual features. The enhanced visual modality features are used to represent the input audio features to generate a branch, and pseudo-audio features are generated through adversarial training. The pseudo-visual features and pseudo-audio features are learned from their respective enhanced unimodal feature representations through bidirectional cyclic consistency loss, thus establishing a bidirectional feature mapping relationship between modalities.

5. The method for elderly emotion recognition based on hierarchical perception enhancement and multi-granularity constraints according to claim 1, characterized in that, The global, cross-modal, multi-granularity constraint strategy described in S4 specifically includes the following: At the pixel level, attention consistency constraints are imposed between the cross-modal representations output by the cross-scale multi-head self-attention modules at each level; At the feature level, an enhanced contrast loss constraint is applied before feature fusion; In deep space, apply a bidirectional cyclic consistency constraint; At the decision level, standard cross-entropy loss is used; The overall loss function for the training phase is obtained by weighting the constraints at different levels mentioned above.

6. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the instruction, program, code set, or instruction set is loaded and executed by the processor to implement the elderly emotion recognition method based on hierarchical perception enhancement and multi-granularity constraints as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by a processor to implement the elderly emotion recognition method based on hierarchical perception enhancement and multi-granularity constraints as described in any one of claims 1-5.

Images