Flying type senile emotion accompanying system based on computer visual identification

Abstract

The invention discloses a flying type senile emotion accompanying system based on computer vision recognition, and relates to the technical field of computer vision. The method comprises the following steps: an emotion recognition module for carrying out related image acquisition and expression recognition on a target old person and extracting a current emotion parameter of the target old person; the emotional resonance module generates a corresponding resonance emotional state according to the current emotional parameter, and controls the robot to perform emotional resonance feedback based on the resonance emotional state to obtain a feedback control result; the flight pacifying module is used for adjusting the flight track mode of the robot in the air according to the feedback control result and executing emotional flight pacifying; and the nursing linkage module is used for acquiring historical interaction information of the target old people and the robot, identifying abnormal behavior signals and automatically triggering a remote nursing linkage mechanism in emotion flight pacifying execution. According to the invention, all-weather, personalized, high-interactivity, safe and controllable intelligent accompanying for the elderly is realized.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a flying-type emotional companionship system for the elderly based on computer vision recognition. Background Technology

[0002] Currently, with the deepening of societal aging, the number of elderly people living alone and those with disabilities in nursing homes is constantly increasing. These elderly individuals are prone to negative emotions such as anxiety, depression, and loneliness due to loneliness, limited social interaction, or physical limitations, which seriously impact their physical and mental health. Traditional human companionship resources are limited and cannot provide round-the-clock, personalized, and emotionally sensitive care. Therefore, developing intelligent emotional companionship systems has become an important demand in elderly care services.

[0003] In related technologies, emotional companionship for the elderly often relies on static robots or fixed terminals, interacting through voice or simple gestures, making it difficult to achieve multi-angle emotion recognition and all-space companionship. Furthermore, their interaction methods are limited, failing to meet the communication needs of elderly people with unclear speech or speaking dialects, thus limiting the effectiveness of emotional companionship. In addition, existing robots generally lack the ability to switch between flight and close-fitting companionship, failing to meet the needs of long-term bedridden elderly for pet-like and human-like interactions, indicating room for improvement. Summary of the Invention

[0004] The purpose of this invention is to provide a flight-type emotional companionship system for the elderly based on computer vision recognition, so as to solve the problems mentioned in the background art.

[0005] This application provides a flight-type emotional companionship system for the elderly based on computer vision recognition, which adopts the following technical solution:

[0006] The emotion recognition module performs relevant image acquisition and facial expression recognition on the target elderly person, and extracts the current emotion parameters of the target elderly person;

[0007] The emotion resonance module generates a corresponding resonance emotion state based on the current emotion parameters, and controls the robot to perform emotion resonance feedback based on the resonance emotion state to obtain feedback control results.

[0008] The flight soothing module adjusts the robot's flight trajectory mode in the air based on the feedback control results to perform emotional flight soothing.

[0009] In the nursing linkage module, during the emotional flight comforting process, the robot acquires historical interaction information between the target elderly person and the robot, identifies abnormal behavioral signals, and automatically triggers the remote nursing linkage mechanism.

[0010] Preferably, the steps of acquiring relevant images and recognizing facial expressions of the target elderly person, and extracting the current emotional parameters of the target elderly person, are as follows:

[0011] The system acquires relevant image sequences of the target elderly person from different angles using a multi-angle camera unit, and dynamically adjusts the exposure value of the camera unit according to the ambient lighting parameters.

[0012] The facial regions of elderly individuals in relevant image sequences were identified using a convolutional neural network model, and multidimensional facial expression feature vectors, including eye corner deformation, mouth corner curvature, eyebrow tension, and facial texture gradient, were extracted.

[0013] Collect relevant speech signals of the target elderly, process the relevant speech signals in frames to obtain multi-frame speech data, and extract multi-dimensional acoustic emotion feature vectors from the multi-frame speech data.

[0014] The multidimensional facial expression feature vector and the multidimensional acoustic emotion feature vector are concatenated and input into the emotion fusion recognition model to calculate the main emotion category and emotion intensity coefficient of the target elderly person.

[0015] An emotion parameter set is constructed based on the main emotion category and emotion intensity coefficient to generate the current emotion parameters of the target elderly person.

[0016] Preferably, the step of concatenating the multidimensional facial expression feature vector with the multidimensional acoustic emotion feature vector, inputting it into the emotion fusion recognition model, and calculating the main emotion category and emotion intensity coefficient of the target elderly person is as follows:

[0017] The multidimensional facial expression feature vector and the multidimensional acoustic emotion feature vector are aligned according to the timestamp, and the time-asynchronous segments are resampled using a sliding time window method;

[0018] The facial expression feature vector and the acoustic emotion feature vector are then input into the multimodal fusion layer for feature concatenation, forming a joint feature matrix;

[0019] The joint feature matrix is ​​input into the emotion fusion recognition model, which performs weighted fusion of facial expression features and acoustic emotion features based on a cross-modal attention mechanism, and outputs an emotion category probability distribution.

[0020] The main emotion category is determined based on the highest probability corresponding item in the emotion category probability distribution, and the emotion intensity coefficient is calculated based on the output feature amplitude.

[0021] Preferably, the steps of determining the main emotion category based on the highest probability corresponding item in the emotion category probability distribution and calculating the emotion intensity coefficient based on the output feature amplitude are as follows:

[0022] Obtain the probability value corresponding to each emotion category from the emotion category probability distribution, and select the item corresponding to the maximum probability value as the main emotion category;

[0023] When the difference between the highest probability value and the second highest probability value is less than a preset threshold, an emotion weighted fusion judgment is performed, and the main emotion category is calculated based on the weighted calculation of multiple emotion weights.

[0024] The amplitude features of the hidden layer feature vector output by the emotion fusion recognition model are extracted and weighted together with the probability distribution entropy value to obtain the emotion intensity coefficient.

[0025] Preferably, the step of generating a corresponding resonant emotional state based on the current emotional parameters, controlling the robot to perform emotional resonance feedback based on the resonant emotional state, and obtaining the feedback control result specifically includes:

[0026] The main emotion category and emotion intensity coefficient are extracted from the current emotion parameters. According to the preset emotion resonance mapping relationship, the main emotion category is mapped to the robot's executable facial expressions, voice and action control parameters to form a resonance state template.

[0027] The amplitude of the resonance state template is adjusted according to the emotional intensity coefficient so that the amplitude of the resonance state template is positively correlated with the emotional intensity coefficient, thereby generating a set of resonance emotional state parameters.

[0028] The set of resonant emotional state parameters is input into the robot control module, and the expression control submodule, voice control submodule and action execution submodule respectively execute the corresponding facial expressions, voice and action feedback;

[0029] By coordinating and controlling the timing, the feedback from the facial expression control submodule, the voice control submodule, and the action execution submodule are linked and executed to obtain the feedback control result.

[0030] Preferably, based on the feedback control result, the robot's flight trajectory mode in the air is adjusted to perform the emotional flight soothing step, specifically as follows:

[0031] Based on the feedback control results, the corresponding flight trajectory mode type of the robot is determined. The flight trajectory mode type includes flexible orbital trajectory, slow descent soothing trajectory, gentle swaying accompanying trajectory, and fixed-point gaze trajectory.

[0032] Based on the flight trajectory pattern type, a flight path control command is generated, and the robot performs emotional flight soothing.

[0033] During the emotional flight comforting process, the flight attitude is adjusted in real time using attitude sensors and spatial positioning units, and synchronized with the lighting control submodule to form a multimodal emotional flight comforting.

[0034] Preferably, during the emotional flight comforting process, the steps of acquiring historical interaction information between the target elderly person and the robot, identifying abnormal behavioral signals, and automatically triggering the remote care linkage mechanism are as follows:

[0035] During the emotional flight comforting operation, historical interaction information between the target elderly and the robot at different time periods is obtained, a personalized emotional memory vector library of the target elderly is constructed, and time series fitting is performed on the emotional memory vector to form a caregiver memory curve;

[0036] Based on the aforementioned caregiver memory curve, abnormal behavioral signals are identified, automatically triggering a remote nursing linkage mechanism and simultaneously triggering emergency assistance confirmation.

[0037] Preferably, in the emotional flight comforting process, the steps of acquiring historical interaction information between the target elderly person and the robot at different time periods, constructing a personalized emotional memory vector library for the target elderly person, and performing time series fitting on the emotional memory vector to form a caregiver memory curve are as follows:

[0038] During the emotional flight comforting operation, historical interaction information between the target elderly person and the robot at different time periods is obtained. The historical interaction information includes facial expression recognition information, voice interaction information, tactile interaction information and environmental context information, and the information is aligned according to timestamps.

[0039] Feature fusion is performed on historical interaction information within each time period to generate an emotional memory vector containing emotion category labeling, emotion intensity coefficient, interaction density index, reaction delay coefficient, and semantic resonance index, thereby constructing an emotional memory vector library for the target elderly.

[0040] Each emotional memory vector in the emotional memory vector library is labeled with a time index, and the emotional memory vector is fitted with a time series based on the time index.

[0041] The emotional change trend is calculated based on the time series fitting results, and a caregiver memory curve is generated to characterize the emotional evolution of the target elderly at different time periods.

[0042] Preferably, based on the caregiver memory curve, abnormal behavioral signals are identified, and a remote nursing linkage mechanism is automatically triggered, simultaneously triggering an emergency assistance confirmation step, specifically as follows:

[0043] During the emotional flight comforting operation, facial expression response data, voice response data, and behavioral response data of the target elderly person are collected in real time. The facial expression response data, voice response data, and phase response data are comprehensively calculated to obtain an emotional state response vector.

[0044] Based on the caregiver memory curve, the normal emotional state range vector of the target elderly is extracted, and the emotional state response vector is compared with the normal emotional state range vector.

[0045] If the emotional state response vector is outside the normal emotional state range vector, then the target elderly person is identified as having abnormal behavior, an abnormal behavior signal is generated, and a remote nursing linkage mechanism is automatically triggered to send an abnormal alarm to the caregiver or family member through the nursing terminal.

[0046] Simultaneously trigger emergency assistance confirmation; if the target elderly person does not respond within the specified time, the status will automatically escalate to emergency linkage.

[0047] In summary, this application includes at least one of the following beneficial technical effects:

[0048] 1. By acquiring relevant images and recognizing facial expressions of the target elderly individuals, the system accurately captures subtle facial dynamics. Combined with intonation, rhythm, and acoustic features from speech signals, a multimodal fusion emotional feature vector is extracted. This transforms the elderly's subjective emotional experience into calculable emotional parameters (including primary emotion category and emotion intensity coefficient), establishing an intelligent perception entry point for elderly emotion recognition. This enables the companion robot to understand and respond to human emotional states in real time, thereby achieving humanized and contextualized companionship. By mapping the identified primary emotion category to executable speech tone, facial expression display, and posture parameters, a resonant emotional state coordinated with the elderly's emotions is generated. The feedback amplitude is dynamically adjusted based on the emotion intensity coefficient, allowing the robot to achieve synchronous resonance with the elderly's emotional state in dimensions such as tone, movement, and lighting effects. This transforms the robot from a passive device performing mechanical actions into an intelligent companion with emotional understanding and expression capabilities, significantly enhancing the elderly's emotional identification and psychological security during interactions. Based on feedback control results, the robot's flight trajectory is dynamically adjusted in the air. Through emotional expression of spatial motion, robot companionship extends from the voice and visual levels to the dynamic behavioral level, creating a multimodal companionship experience with visual soothing, spatial empathy, and emotional guidance functions. During the emotional flight soothing execution, multimodal interaction information between the elderly and the robot is continuously acquired, and a companionship memory curve is generated through emotional memory vector modeling to continuously track the emotional evolution trend of the elderly. When a significant deviation of the elderly's emotional response or behavioral pattern from the historical memory curve is detected, it is automatically identified as an abnormal behavior signal, immediately triggering a remote nursing linkage mechanism to issue an alarm and request confirmation from caregivers or family members. This endows the system with proactive early warning and emergency collaboration capabilities, giving emotional companionship safety protection and risk response characteristics, thereby ensuring the life safety and mental health of the elderly and realizing the intelligent evolution of the companionship process from emotional companionship to safety protection.

[0049] 2. The probability distribution of emotion categories output by the emotion fusion recognition model reflects the system's confidence assessment of different emotional states. By selecting the item corresponding to the highest probability value as the primary emotion category, the complex multi-emotion recognition results can be effectively transformed into unique, actionable emotion labels. This achieves efficient mapping from multi-dimensional recognition results to primary emotion judgment, ensuring the targeted nature and consistent response of caregiving behavior. Elderly individuals often exhibit multiple emotional characteristics simultaneously when experiencing emotional fluctuations or cognitive instability. Simply taking the highest probability may lead to misjudgment. By setting a probability difference threshold and performing weighted fusion judgment, multi-emotion weighted calculations can be performed based on the relative probability distribution of each emotion category, resulting in a more comprehensive emotional outcome that aligns with human psychological patterns. This improves the system's robustness in complex emotional scenarios, enabling the emotional companion robot to have a flexible understanding of ambiguous emotions, enhancing the accuracy of emotion recognition and the naturalness of emotional interaction. The magnitude of the hidden layer feature vector reflects the energy of the feature response within the model, while the entropy value of the probability distribution reflects the uncertainty of the recognition result or the stability of emotions. By weighting and combining the two, an emotion intensity coefficient that better conforms to the laws of psychological intensity change can be obtained. This coefficient not only reflects the direction of emotions but also quantifies the strength of emotions, providing a quantitative basis for subsequent adjustments to flight comfort intensity, voice tone control, and tactile feedback. This endows the system with the ability to perceive emotional energy, enabling the companion robot to evolve from recognizing emotions to understanding the depth of emotions, thereby improving the accuracy and humanization of the companion experience.

[0050] 3. By selecting appropriate flight trajectory modes based on different emotion types and intensities, the robot can visually soothe the emotions of the elderly in various ways. This achieves intelligent mapping between emotion recognition and flight control, making the robot's flight movements more targeted and emotionally adaptable, improving the effectiveness of companionship and the realism of emotional interaction. The trajectory planning results are transformed into specific control commands, enabling the robot to safely and smoothly complete various flight modes, achieving visual soothing and interactive companionship for the elderly. This makes the robot's behavior controllable and safe, while enhancing the elderly's companionship experience and psychological security. Through attitude sensors and spatial positioning units, the robot can perceive its environment and its own state, automatically adjusting its flight attitude to ensure safety and stability. Simultaneously, through linkage with lighting control, visual emotional cues and soothing actions can be presented synchronously, forming a multimodal emotional interaction effect. This enhances the companion robot's emotional expression capabilities, making flight soothing not merely a simple execution of a trajectory, but a comprehensive companionship behavior integrating visual, motor, and emotional signals, thereby enhancing the elderly's psychological comfort and the realism of the companionship experience. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the module connections of an embodiment of a flying-type elderly emotional care system based on computer vision recognition according to the present invention.

[0052] Figure 2 This is a flowchart illustrating the specific steps of an embodiment of a flight-type elderly emotional care system based on computer vision recognition according to the present invention.

[0053] Attached image labels: 1. Emotion Recognition Module. 2. Emotion Resonance Module. 3. Flight Soothing Module. 4. Nursing Coordination Module. Detailed Implementation

[0054] The following examples and... Figures 1-2 The present invention will be described in further detail, but the embodiments of the present invention are not limited thereto.

[0055] This invention discloses a flying-type emotional companionship system for the elderly based on computer vision recognition, specifically including the following steps:

[0056] Emotion recognition module 1 performs relevant image acquisition and facial expression recognition on the target elderly person, and extracts the current emotional parameters of the target elderly person;

[0057] Emotional resonance module 2 generates a corresponding resonance emotional state based on the current emotional parameters, and controls the robot to perform emotional resonance feedback based on the resonance emotional state to obtain feedback control results.

[0058] Flight soothing module 3 adjusts the robot's flight trajectory mode in the air based on the feedback control results to perform emotional flight soothing.

[0059] In the nursing linkage module 4, during the emotional flight comforting process, the robot acquires historical interaction information between the target elderly person and the robot, identifies abnormal behavioral signals, and automatically triggers the remote nursing linkage mechanism.

[0060] In practical applications, image acquisition and facial expression recognition of the target elderly can accurately capture subtle facial dynamic changes. Combined with intonation, rhythm, and acoustic features in the speech signal, a multimodal fusion emotional feature vector is extracted. This transforms the target elderly's subjective emotional experience into calculable emotional parameters (including the primary emotion category and emotion intensity coefficient), establishing an intelligent perception entry point for elderly emotion recognition. This enables the companion robot to understand and respond to human emotional states in real time, thereby achieving humanized and contextualized companionship. By mapping the identified primary emotion category to executable voice intonation, facial expression display, and posture parameters, a resonant emotional state coordinated with the elderly's emotions is generated. The feedback amplitude is dynamically adjusted based on the emotion intensity coefficient, allowing the robot to achieve synchronous resonance with the elderly's emotional state in dimensions such as tone, movement, and lighting effects. This transforms the robot from a passive device performing mechanical actions into an intelligent companion with emotional understanding and expression capabilities, significantly enhancing the elderly's emotional identification and psychological security during interactions. Based on feedback control results, the robot's flight trajectory type in the air (such as circling, slow descent, gentle swaying, fixed gaze, etc.) is dynamically adjusted to create a flexible correspondence between flight actions and the elderly person's current emotional state. Through the emotional expression of spatial movement, robot companionship extends from the voice and visual levels to the dynamic behavioral level, creating a multimodal companionship experience with visual soothing, spatial empathy, and emotional guidance functions. During the execution of emotional flight soothing, multimodal interaction information between the elderly person and the robot (including facial expressions, voice, touch, environmental data, etc.) is continuously acquired, and a companionship memory curve is generated through emotional memory vector modeling to continuously track the elderly person's emotional evolution trend. When a significant deviation of the elderly person's emotional response or behavioral pattern from the historical memory curve is detected, it is automatically identified as an abnormal behavior signal, immediately triggering a remote nursing linkage mechanism to issue an alarm and request confirmation from caregivers or family members. This endows the system with proactive early warning and emergency collaboration capabilities, giving emotional companionship safety protection and risk response characteristics, thereby ensuring the elderly person's life safety and mental health, and realizing the intelligent evolution of the companionship process from emotional companionship to safety protection.

[0061] The steps for acquiring relevant images and recognizing facial expressions of the target elderly person, and extracting their current emotional parameters, are as follows:

[0062] The system acquires relevant image sequences of the target elderly person from different angles using a multi-angle camera unit, and dynamically adjusts the exposure value of the camera unit according to the ambient lighting parameters.

[0063] The facial regions of elderly individuals in relevant image sequences were identified using a convolutional neural network model, and multidimensional facial expression feature vectors, including eye corner deformation, mouth corner curvature, eyebrow tension, and facial texture gradient, were extracted.

[0064] Collect relevant speech signals of the target elderly, process the relevant speech signals in frames to obtain multi-frame speech data, and extract multi-dimensional acoustic emotion feature vectors from the multi-frame speech data.

[0065] The multidimensional facial expression feature vector and the multidimensional acoustic emotion feature vector are concatenated and input into the emotion fusion recognition model to calculate the main emotion category and emotion intensity coefficient of the target elderly person.

[0066] An emotion parameter set is constructed based on the main emotion category and emotion intensity coefficient to generate the current emotion parameters of the target elderly person.

[0067] In practical applications, multi-angle camera units effectively avoid facial feature loss due to head posture changes or occlusion, ensuring complete acquisition of key facial points. Simultaneously, dynamically adjusting exposure values ​​based on ambient lighting parameters maintains stable image brightness and contrast under varying lighting conditions, improving image clarity and feature extraction accuracy. This enhances the system's adaptability and robustness to facial expression changes in natural scenes. Convolutional neural network models efficiently identify facial regions and automatically extract local features, such as changes in eye corners, upturned or downturned corners of the mouth, and tension in the eyebrows. These subtle muscle movements and facial texture gradient changes together constitute a multi-dimensional expression feature vector. Through deep feature learning, complex emotional micro-expressions are captured, improving the system's sensitivity and accuracy in emotion recognition. Speech signals not only contain linguistic content but also rich emotional features, such as speech rate, pitch, intensity, and formant variations. Through frame-based processing, the dynamic changes in speech emotion can be captured on a temporal scale, and multi-dimensional acoustic emotion vectors, such as Mel-frequency cepstral coefficients, fundamental frequency, and energy distribution, can be extracted. This allows the system to consider both visual and auditory channels, thus more comprehensively reflecting the true emotional state of the elderly. By concatenating visual and acoustic features into vectors, a fusion model is used to calculate the primary emotion category (such as pleasure, calmness, anxiety, sadness, etc.) and the corresponding emotion intensity coefficient, achieving a transition from perceiving emotion to quantifying emotion, significantly improving the accuracy and sensitivity of emotion recognition. An emotion parameter set is constructed based on the primary emotion category and emotion intensity coefficient to form an instant emotional profile of the target elderly person, achieving standardized expression of emotion recognition results. This enables the system to continuously track the emotional changes of the elderly person in different times and scenarios, achieving continuous and personalized care responses.

[0068] The steps for concatenating the multidimensional facial expression feature vector with the multidimensional acoustic emotion feature vector, inputting the concatenation into the emotion fusion recognition model, and calculating the main emotion category and emotion intensity coefficient of the target elderly person are as follows:

[0069] The multidimensional facial expression feature vector and the multidimensional acoustic emotion feature vector are aligned according to the timestamp, and the time-asynchronous segments are resampled using a sliding time window method;

[0070] The facial expression feature vector and the acoustic emotion feature vector are then input into the multimodal fusion layer for feature concatenation, forming a joint feature matrix;

[0071] The joint feature matrix is ​​input into the emotion fusion recognition model, which performs weighted fusion of facial expression features and acoustic emotion features based on a cross-modal attention mechanism, and outputs an emotion category probability distribution.

[0072] The main emotion category is determined based on the highest probability corresponding item in the emotion category probability distribution, and the emotion intensity coefficient is calculated based on the output feature amplitude.

[0073] In practical applications, the asynchronous frame rate of image acquisition and the sampling rate of speech signals can lead to emotional feature fusion shifts and matching errors if time alignment is not performed. By using timestamp matching and sliding time window resampling mechanisms, feature segments from different time periods can be dynamically balanced and reconstructed, ensuring the temporal consistency of facial expression and speech features and providing a unified time reference for subsequent fusion models. Through feature concatenation operations in the multimodal fusion layer, complementary information from facial expressions and speech can be integrated at the feature level, enabling the model to simultaneously perceive the correlation between facial micro-expressions and speech emotional cues. The construction of the joint feature matrix helps capture the implicit coupling features between multiple channels, overcoming the limitations of single-modal recognition and enhancing the system's ability to distinguish complex emotional states (such as anxiety and tension, joy and excitement). The emotion fusion recognition model adaptively increases the weight of key modal features and suppresses interference from noisy modal features based on the varying importance of facial expressions and speech in different emotional contexts. Through a cross-modal attention mechanism, it assigns weights and dynamically fuses emotional features from different sources. The model's output probability distribution of emotion categories reflects the likelihood assessment results of different emotional states (such as calmness, joy, sadness, and anxiety), realizing a shift in emotion recognition from feature stacking to semantic fusion. This significantly improves the system's ability to understand and recognize complex emotional cues, making the recognition results closer to human perceptual logic. By determining the primary emotion category using the maximum probability principle, the model can intuitively pinpoint the primary emotional state of the elderly at the current moment. The emotion intensity coefficient, calculated based on the amplitude of the output features, reflects the energy or intensity of emotional fluctuations, achieving a closed loop from probability recognition to behavioral triggering. This enables the robot not only to recognize emotions but also to respond flexibly based on emotional intensity, ultimately realizing humanized and intelligent emotional support.

[0074] The steps of determining the main emotion category based on the highest probability corresponding item in the emotion category probability distribution and calculating the emotion intensity coefficient based on the output feature amplitude are as follows:

[0075] Obtain the probability value corresponding to each emotion category from the emotion category probability distribution, and select the item corresponding to the maximum probability value as the main emotion category;

[0076] When the difference between the highest probability value and the second highest probability value is less than a preset threshold, an emotion weighted fusion judgment is performed, and the main emotion category is calculated based on the weighted calculation of multiple emotion weights.

[0077] The amplitude features of the hidden layer feature vector output by the emotion fusion recognition model are extracted and weighted together with the probability distribution entropy value to obtain the emotion intensity coefficient.

[0078] In practical applications, the probability distribution of emotion categories output by the emotion fusion recognition model reflects the system's confidence assessment of different emotional states (such as happiness, calmness, sadness, anxiety, etc.). By selecting the item corresponding to the highest probability value as the primary emotion category, the complex multi-emotion recognition results can be effectively transformed into unique, actionable emotion labels, achieving efficient mapping from multi-dimensional recognition results to primary emotion judgment, ensuring the targeted nature and consistent response of caregiving behavior. When elderly people experience emotional fluctuations or unstable cognitive states, they often exhibit multiple emotional characteristics simultaneously (such as anxiety mixed with tension, sadness accompanied by anger). In such cases, simply taking the highest probability may lead to misjudgment. By setting a probability difference threshold and performing weighted fusion judgment, multi-emotion weighted calculations can be performed based on the relative probability distribution of each emotion category, thereby obtaining a comprehensive emotion result that is more in line with human psychological laws. This improves the system's robustness in complex emotional scenarios, enabling the emotional companion robot to have a flexible understanding of ambiguous emotions, enhancing the accuracy of emotion recognition and the naturalness of emotional interaction. The magnitude of the hidden layer feature vector reflects the energy of the feature response within the model, while the entropy value of the probability distribution reflects the uncertainty of the recognition result or the emotional stability. By weighting and combining the two, an emotional intensity coefficient that better conforms to the laws of psychological intensity change can be obtained. This coefficient not only reflects the direction of the emotion (such as positive or negative) but also quantifies the strength of the emotion, providing a quantitative basis for subsequent adjustments to the intensity of flight comforting, voice tone control, and tactile feedback. This endows the system with the ability to perceive emotional energy, enabling the companion robot to evolve from recognizing emotions to understanding the depth of emotions, thereby improving the accuracy and humanization of the companion experience.

[0079] The steps of generating a corresponding resonant emotional state based on the current emotional parameters, controlling the robot to perform emotional resonance feedback based on the resonant emotional state, and obtaining the feedback control result are as follows:

[0080] The main emotion category and emotion intensity coefficient are extracted from the current emotion parameters. According to the preset emotion resonance mapping relationship, the main emotion category is mapped to the robot's executable facial expressions, voice and action control parameters to form a resonance state template.

[0081] The amplitude of the resonance state template is adjusted according to the emotional intensity coefficient so that the amplitude of the resonance state template is positively correlated with the emotional intensity coefficient, thereby generating a set of resonance emotional state parameters.

[0082] The set of resonant emotional state parameters is input into the robot control module, and the expression control submodule, voice control submodule and action execution submodule respectively execute the corresponding facial expressions, voice and action feedback;

[0083] By coordinating and controlling the timing, the feedback from the facial expression control submodule, the voice control submodule, and the action execution submodule are linked and executed to obtain the feedback control result.

[0084] In practical applications, through emotional resonance mapping, the primary emotion category is mapped to corresponding facial expressions, vocal tones, and behavioral patterns. This enables the robot to respond in a way that matches the elderly person's emotional state at the visual, auditory, and motor levels. This establishes a bridge between emotion recognition and robot behavior, enabling an executable transformation from perceived emotion to emotional resonance feedback. The robot can proactively engage in emotional interaction, thereby enhancing the anthropomorphism and emotional interaction of the caregiver. Through amplitude adjustment, the robot can display different levels of facial tension, voice volume, and movement amplitude according to the intensity of the elderly person's emotions. This ensures that its response not only matches the emotion type but also reflects emotional energy in intensity, improving the sensitivity and personalization of the caregiver's behavior. The robot can exhibit reasonable resonance responses to both mild and intense emotional fluctuations, enhancing the naturalness and humanization of the interaction. By allocating parameter sets to each sub-module, the robot can simultaneously respond to the emotional changes of the elderly at three levels: visual (facial expressions, light indicators), auditory (voice tone), and motor (gestures, body posture). Through multimodal collaborative execution, the robot's emotional feedback is made more realistic, intuitive, and impactful, effectively promoting emotional resonance and psychological comfort for the elderly, and achieving human-like and interactive companionship. Synchronous control ensures precise timing of facial expressions, voice, and movements, avoiding misalignment or delays between them. This maintains the continuity and natural flow of emotional feedback, improving the interaction quality of the companion robot. It allows the elderly to experience unified and continuous emotional resonance across visual, auditory, and motor perceptions, enhancing the companionship effect and reducing cognitive conflict or discomfort.

[0085] Based on the feedback control results, the robot's flight trajectory mode in the air is adjusted, and the steps of emotional flight soothing are performed, specifically:

[0086] Based on the feedback control results, the corresponding flight trajectory mode type of the robot is determined. The flight trajectory mode type includes flexible orbital trajectory, slow descent soothing trajectory, gentle swaying accompanying trajectory, and fixed-point gaze trajectory.

[0087] Based on the flight trajectory pattern type, a flight path control command is generated, and the robot performs emotional flight soothing.

[0088] During the emotional flight comforting process, the flight attitude is adjusted in real time using attitude sensors and spatial positioning units, and synchronized with the lighting control submodule to form a multimodal emotional flight comforting.

[0089] In practical applications, by selecting appropriate flight trajectory patterns based on different emotion types and intensities, the robot can visually soothe the emotions of the elderly in various ways. For example, a gentle circling trajectory can create a sense of immersion to alleviate loneliness, a slow-descent soothing trajectory is suitable for calming anxiety, a gentle swaying accompanying trajectory enhances the feeling of companionship, and a fixed-point gaze trajectory enables precise attention. This achieves intelligent mapping between emotion recognition and flight control, making the robot's flight movements more targeted and emotionally adaptable, improving the effectiveness of caregiving and the realism of emotional interaction. By translating the trajectory planning results into specific control commands, the robot can safely and smoothly complete various flight modes, achieving visual soothing and interactive companionship for the elderly. This makes the robot's behavior controllable and safe, while enhancing the elderly's experience of companionship and their sense of psychological security. Through attitude sensors and spatial positioning units, the robot can perceive its environment and its own state, automatically adjusting its flight attitude to ensure safety and stability. At the same time, through linkage with lighting control, visual emotional cues and soothing actions can be presented simultaneously, forming a multimodal emotional interaction effect. This enhances the emotional expression ability of the companion robot, making flight soothing not just a simple execution of a movement trajectory, but a comprehensive companionship behavior that integrates visual, motor, and emotional signals, thereby enhancing the psychological comfort of the elderly and the realism of the companionship experience.

[0090] During the emotional flight comforting operation, the steps involved in acquiring historical interaction information between the target elderly person and the robot, identifying abnormal behavioral signals, and automatically triggering the remote care linkage mechanism are as follows:

[0091] During the emotional flight comforting operation, historical interaction information between the target elderly and the robot at different time periods is obtained, a personalized emotional memory vector library of the target elderly is constructed, and time series fitting is performed on the emotional memory vector to form a caregiver memory curve;

[0092] Based on the aforementioned caregiver memory curve, abnormal behavioral signals are identified, automatically triggering a remote nursing linkage mechanism and simultaneously triggering emergency assistance confirmation.

[0093] In practical applications, by collecting historical interaction information such as facial expressions, voice, actions, and environmental context, an emotional memory vector library is constructed. Time series fitting is then performed to form a caregiver memory curve, which reflects the emotional evolution trends of the elderly at different time periods. This allows the system to understand the long-term emotional patterns and fluctuations of the elderly, providing a reference benchmark for abnormal behavior identification and improving the accuracy and scientific nature of caregiving. By comparing the current emotional state with the caregiver memory curve, abnormal behaviors or potential danger signals deviating from normal emotional patterns can be identified. Once an abnormal signal is detected, a remote nursing linkage mechanism is automatically triggered, sending alarm information to caregivers or family members and simultaneously initiating an emergency assistance confirmation process. This enhances the safety assurance capabilities of the caregiver system, enabling the robot not only to provide emotional comfort but also to intervene promptly when the elderly exhibit abnormal emotions or behavioral risks, reducing accidents and enhancing the sense of security and quality of life for the elderly in home or institutional environments.

[0094] During the emotional flight comforting operation, the following steps are taken: First, historical interaction information between the target elderly person and the robot at different time periods is obtained. Then, a personalized emotional memory vector library for the target elderly person is constructed. Finally, time series fitting is performed on the emotional memory vectors to form a caregiver memory curve.

[0095] During the emotional flight comforting operation, historical interaction information between the target elderly person and the robot at different time periods is obtained. The historical interaction information includes facial expression recognition information, voice interaction information, tactile interaction information and environmental context information, and the information is aligned according to timestamps.

[0096] Feature fusion is performed on historical interaction information within each time period to generate an emotional memory vector containing emotion category labeling, emotion intensity coefficient, interaction density index, reaction delay coefficient, and semantic resonance index, thereby constructing an emotional memory vector library for the target elderly.

[0097] Each emotional memory vector in the emotional memory vector library is labeled with a time index, and the emotional memory vector is fitted with a time series based on the time index.

[0098] The emotional change trend is calculated based on the time series fitting results, and a caregiver memory curve is generated to characterize the emotional evolution of the target elderly at different time periods.

[0099] In practical applications, by integrating facial expressions, voice, touch, and environmental context information, the integrity and temporal consistency of emotion analysis data can be ensured. This allows the system to accurately capture the emotional state and interaction habits of the elderly, providing a reliable basis for subsequent analysis of emotional change trends and identification of abnormal behavior. Multimodal historical interaction information is transformed into quantifiable emotion memory vectors. These vectors not only contain emotion categories and intensities but also reflect interaction frequency, response speed, and the degree of resonance with language and context, forming personalized emotion profiles for the elderly. This provides the robot with precise behavioral decision-making support and supports the development of personalized care strategies and the tracking of long-term emotion changes. Time series analysis transforms discrete emotion memory vectors into continuous emotion evolution trajectories. By labeling each vector with a time index and fitting it, the regularity and trends of emotional changes in the elderly over different time periods can be revealed. This allows the system to quantify and predict the dynamic changes in the elderly's emotions over time, providing a scientific basis for abnormal emotion detection and subsequent care intervention, while also improving the real-time response capability and personalized adaptability of robot care. The companionship memory curve visually presents the trajectory of the elderly's emotional evolution. The companionship memory curve not only reflects the ups and downs of emotions, but also reveals the periodicity and long-term trend of the elderly's emotional fluctuations. This enables the robot to formulate personalized flight trajectories and intervention strategies based on historical emotional trends, thereby achieving precise long-term emotional companionship and improving the effectiveness of companionship and the psychological comfort of the elderly.

[0100] Based on the aforementioned caregiver memory curve, abnormal behavioral signals are identified, automatically triggering a remote nursing linkage mechanism and simultaneously initiating an emergency assistance confirmation step, specifically as follows:

[0101] During the emotional flight comforting operation, facial expression response data, voice response data, and behavioral response data of the target elderly person are collected in real time. The facial expression response data, voice response data, and phase response data are comprehensively calculated to obtain an emotional state response vector.

[0102] Based on the caregiver memory curve, the normal emotional state range vector of the target elderly is extracted, and the emotional state response vector is compared with the normal emotional state range vector.

[0103] If the emotional state response vector is outside the normal emotional state range vector, then the target elderly person is identified as having abnormal behavior, an abnormal behavior signal is generated, and a remote nursing linkage mechanism is automatically triggered to send an abnormal alarm to the caregiver or family member through the nursing terminal.

[0104] Simultaneously trigger emergency assistance confirmation; if the target elderly person does not respond within the specified time, the status will automatically escalate to emergency linkage.

[0105] In practical applications, by comprehensively calculating these data to form an emotional state response vector, the current psychological and emotional state of the elderly can be accurately reflected. This allows the robot to instantly perceive the emotional changes and potential risks of the elderly during caregiving, improving the system's safety and intelligence. By comparing the real-time collected emotional states of the elderly with their historical emotional change patterns, and using the normal emotional state range defined by the caregiving memory curve, the system can determine whether the current state is abnormal, achieving personalized anomaly detection standards. This enables the robot to identify potential behavioral abnormalities by combining the elderly's long-term emotional patterns, rather than relying solely on single or general thresholds, improving the accuracy and reliability of anomaly identification. When abnormal behavior or emotions of the elderly are detected, an alarm signal is immediately generated and relevant personnel are notified through a remote care mechanism, enabling proactive intervention and enhancing the safety assurance capabilities of the caregiving system. The robot not only provides emotional companionship but can also intervene promptly when abnormal emotions or behaviors occur, reducing the risk of accidents and providing timely and controllable safety assurance for the elderly. After an abnormal event occurs, the safety status of the elderly is further confirmed, and higher-level emergency response measures are activated when the elderly are unable to respond in a timely manner. This establishes a multi-layered safety protection mechanism, enabling the care system to automatically upgrade its response in potential emergencies, ensuring the safety of the elderly's lives, while reducing the real-time monitoring pressure on caregivers, and realizing an intelligent and safe emotional care system.

[0106] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A flight-type emotional companionship system for the elderly based on computer vision recognition, characterized in that, Includes the following steps: The emotion recognition module (1) performs relevant image acquisition and expression recognition on the target elderly person and extracts the current emotion parameters of the target elderly person; The emotion resonance module (2) generates a corresponding resonance emotion state based on the current emotion parameters, controls the robot to perform emotion resonance feedback based on the resonance emotion state, and obtains feedback control results. The flight comfort module (3) adjusts the robot's flight trajectory mode in the air according to the feedback control results and performs emotional flight comfort. The nursing linkage module (4) acquires historical interaction information between the target elderly person and the robot during the emotional flight comforting process, identifies abnormal behavior signals, and automatically triggers the remote nursing linkage mechanism.

2. The flying-type elderly emotional companionship system based on computer vision recognition according to claim 1, characterized in that, The emotion recognition module (1) performs relevant image acquisition and expression recognition on the target elderly person, and extracts the current emotion parameters of the target elderly person in the following steps: The system acquires relevant image sequences of the target elderly person from different angles using a multi-angle camera unit, and dynamically adjusts the exposure value of the camera unit according to the ambient lighting parameters. The facial regions of elderly individuals in relevant image sequences were identified using a convolutional neural network model, and multidimensional facial expression feature vectors, including eye corner deformation, mouth corner curvature, eyebrow tension, and facial texture gradient, were extracted. Collect relevant speech signals of the target elderly, process the relevant speech signals in frames to obtain multi-frame speech data, and extract multi-dimensional acoustic emotion feature vectors from the multi-frame speech data. The multidimensional facial expression feature vector and the multidimensional acoustic emotion feature vector are concatenated and input into the emotion fusion recognition model to calculate the main emotion category and emotion intensity coefficient of the target elderly person. An emotion parameter set is constructed based on the main emotion category and emotion intensity coefficient to generate the current emotion parameters of the target elderly person.

3. The flying-type elderly emotional companionship system based on computer vision recognition according to claim 2, characterized in that, The step of concatenating the multidimensional facial expression feature vector with the multidimensional acoustic emotion feature vector, inputting it into the emotion fusion recognition model, and calculating the main emotion category and emotion intensity coefficient of the target elderly person is as follows: The multidimensional facial expression feature vector and the multidimensional acoustic emotion feature vector are aligned according to the timestamp, and the time-asynchronous segments are resampled using a sliding time window method; The facial expression feature vector and the acoustic emotion feature vector are then input into the multimodal fusion layer for feature concatenation, forming a joint feature matrix; The joint feature matrix is ​​input into the emotion fusion recognition model, which performs weighted fusion of facial expression features and acoustic emotion features based on a cross-modal attention mechanism, and outputs an emotion category probability distribution. The main emotion category is determined based on the highest probability corresponding item in the emotion category probability distribution, and the emotion intensity coefficient is calculated based on the output feature amplitude.

4. The flying-type elderly emotional companionship system based on computer vision recognition according to claim 3, characterized in that, The steps of determining the main emotion category based on the highest probability corresponding item in the emotion category probability distribution and calculating the emotion intensity coefficient based on the output feature amplitude are as follows: Obtain the probability value corresponding to each emotion category from the emotion category probability distribution, and select the item corresponding to the maximum probability value as the main emotion category; When the difference between the highest probability value and the second highest probability value is less than a preset threshold, an emotion weighted fusion judgment is performed, and the main emotion category is calculated based on the weighted calculation of multiple emotion weights. The amplitude features of the hidden layer feature vector output by the emotion fusion recognition model are extracted and weighted together with the probability distribution entropy value to obtain the emotion intensity coefficient.

5. A flight-type elderly emotional companionship system based on computer vision recognition according to claim 4, characterized in that, The emotional resonance module (2) generates a corresponding resonant emotional state based on the current emotional parameters, controls the robot to perform emotional resonance feedback based on the resonant emotional state, and obtains the feedback control result. The specific steps are as follows: The main emotion category and emotion intensity coefficient are extracted from the current emotion parameters. According to the preset emotion resonance mapping relationship, the main emotion category is mapped to the robot's executable facial expressions, voice and action control parameters to form a resonance state template. The amplitude of the resonance state template is adjusted according to the emotional intensity coefficient so that the amplitude of the resonance state template is positively correlated with the emotional intensity coefficient, thereby generating a set of resonance emotional state parameters. The set of resonant emotional state parameters is input into the robot control module, and the expression control submodule, voice control submodule and action execution submodule respectively execute the corresponding facial expressions, voice and action feedback; By coordinating and controlling the timing, the feedback from the facial expression control submodule, the voice control submodule, and the action execution submodule are linked and executed to obtain the feedback control result.

6. The flying-type elderly emotional companionship system based on computer vision recognition according to claim 1, characterized in that, The flight soothing module (3), based on the feedback control result, adjusts the robot's flight trajectory mode in the air and performs the emotional flight soothing steps, specifically: Based on the feedback control results, the corresponding flight trajectory mode type of the robot is determined. The flight trajectory mode type includes flexible orbital trajectory, slow descent soothing trajectory, gentle swaying accompanying trajectory, and fixed-point gaze trajectory. Based on the flight trajectory pattern type, a flight path control command is generated, and the robot performs emotional flight soothing. During the emotional flight comforting process, the flight attitude is adjusted in real time using attitude sensors and spatial positioning units, and synchronized with the lighting control submodule to form a multimodal emotional flight comforting.

7. A flight-type elderly emotional companionship system based on computer vision recognition according to claim 1, characterized in that, The nursing linkage module (4), in the process of emotional flight comforting, acquires historical interaction information between the target elderly person and the robot, identifies abnormal behavioral signals, and automatically triggers the remote nursing linkage mechanism. The specific steps are as follows: During the emotional flight comforting operation, historical interaction information between the target elderly and the robot at different time periods is obtained, a personalized emotional memory vector library of the target elderly is constructed, and time series fitting is performed on the emotional memory vector to form a caregiver memory curve; Based on the aforementioned caregiver memory curve, abnormal behavioral signals are identified, automatically triggering a remote nursing linkage mechanism and simultaneously triggering emergency assistance confirmation.

8. A flight-type elderly emotional companionship system based on computer vision recognition according to claim 7, characterized in that, The emotional flight comforting process involves acquiring historical interaction information between the target elderly person and the robot at different time periods, constructing a personalized emotional memory vector library for the target elderly person, and performing time series fitting on the emotional memory vector to form a caregiver memory curve. The specific steps are as follows: During the emotional flight comforting operation, historical interaction information between the target elderly person and the robot at different time periods is obtained. The historical interaction information includes facial expression recognition information, voice interaction information, tactile interaction information and environmental context information, and the information is aligned according to timestamps. Feature fusion is performed on historical interaction information within each time period to generate an emotional memory vector containing emotion category labeling, emotion intensity coefficient, interaction density index, reaction delay coefficient, and semantic resonance index, thereby constructing an emotional memory vector library for the target elderly. Each emotional memory vector in the emotional memory vector library is labeled with a time index, and the emotional memory vector is fitted with a time series based on the time index. The emotional change trend is calculated based on the time series fitting results, and a caregiver memory curve is generated to characterize the emotional evolution of the target elderly at different time periods.

9. A flight-type elderly emotional companionship system based on computer vision recognition according to claim 7, characterized in that, The steps of identifying abnormal behavioral signals based on the caregiver memory curve, automatically triggering the remote nursing linkage mechanism, and simultaneously triggering emergency assistance confirmation are as follows: During the emotional flight comforting operation, facial expression response data, voice response data, and behavioral response data of the target elderly person are collected in real time. The facial expression response data, voice response data, and phase response data are comprehensively calculated to obtain an emotional state response vector. Based on the caregiver memory curve, the normal emotional state range vector of the target elderly is extracted, and the emotional state response vector is compared with the normal emotional state range vector. If the emotional state response vector is outside the normal emotional state range vector, then the target elderly person is identified as having abnormal behavior, an abnormal behavior signal is generated, and a remote nursing linkage mechanism is automatically triggered to send an abnormal alarm to the caregiver or family member through the nursing terminal. Simultaneously trigger emergency assistance confirmation; if the target elderly person does not respond within the specified time, the status will automatically escalate to emergency linkage.

Images