Older person interactive digital human management system based on multi-modal semantic understanding

Abstract

The application belongs to the technical field of digital people and data processing, and discloses an old people interactive digital people management system based on multi-modal semantic understanding, which comprises the following steps: collecting multi-modal data in the process of user interaction with digital people in real time; performing emotion recognition and emotion evolution trend determination to determine intervention trigger level; determining current cognitive behavior intervention stage; determining physical intervention interval; generating voice intervention data and digital people interaction control parameters; recording intervention instruction issuing time stamp, extracting interaction effective time stamp, performing time response correction, outputting effective intervention response interval and target emotion feedback data; performing intervention effect evaluation and identifying abnormal emotion fluctuation to determine real intervention response state; updating cognitive behavior intervention stage, voice intervention data and digital people interaction control parameters; and improving the control precision of digital people intervention execution and guaranteeing the stability and safety of human-computer interaction.

Description

Technical Field

[0001] This application relates to the field of digital human and data processing technology, and more specifically, to an interactive digital human management system for the elderly based on multimodal semantic understanding. Background Technology

[0002] Multimodal semantic understanding is a key means and core step in the process of human-computer interaction and intelligent services. The existing smart elderly care field has begun to use interactive digital humans to provide daily companionship and emotional care services for elderly users. Multimodal semantic recognition and effective control of digital human behavior in this process are used to ensure the effectiveness and security of the interaction. However, traditional interactive digital human management systems still face many technical bottlenecks when facing complex scenarios caused by the emotional fluctuations of elderly users and various constraints.

[0003] In real-world interactive scenarios, elderly users' emotional expressions are often characterized by high-frequency fluctuations in recognized emotional features due to factors such as momentary facial twitches, brief pauses in speech, or slow language delivery. Traditional cognitive behavioral intervention systems often employ simple threshold judgment logic and lack effective confirmation of emotional change trends, ignoring the interference of short-term signal fluctuations. This can easily lead to misjudgments of the timing of care responses, resulting in over-intervention when unnecessary or missed opportunities when intervention is needed. Furthermore, after an intervention command is issued, there is a buffer time between the digital human's actual action and the elderly user's emotional response. This buffer time is often longer for elderly users due to their relatively slower cognitive processing speed. However, traditional interactive digital human management systems do not establish a time alignment relationship between digital human action execution and user feedback, leading to a time discrepancy between intervention actions and emotional feedback. Due to misalignment; and during the intervention process, users may exhibit subconscious resistance postures or actions such as avoiding eye contact or leaning back due to emotional fluctuations. However, traditional cognitive behavioral intervention systems rely solely on preset intervention strategy templates, ignoring the changes in elderly users' acceptance of interactive interventions. This could lead to the digital human suddenly approaching when the elderly user is extremely sensitive, causing further psychological stimulation. On the other hand, emotional care for the elderly is a dynamic guidance process accompanied by emotional fluctuations. When receiving care, elderly users may experience psychological triggers due to interactive content. Traditional interactive digital human management systems often use fixed state feedback logic, making it difficult to distinguish between normal transitional emotional fluctuations induced by care. This can easily lead to misjudging normal transitional reactions as care failures, prematurely interrupting the interaction or incorrectly switching care stages, disrupting the continuity of the emotional care process, and ultimately resulting in poor guidance effects.

[0004] In view of this, this application proposes an interactive digital human management system for the elderly based on multimodal semantic understanding to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, this application provides the following technical solution: an interactive digital human management system for the elderly based on multimodal semantic understanding, comprising: The data acquisition module collects multimodal data in real time during the interaction between the user and the digital human, including user emotion data, user posture data, and digital human action execution data. The emotion recognition module performs emotion recognition based on user emotion data, determines the emotion evolution trend based on the emotion recognition results, and determines the intervention trigger level. The control parameter generation module, when the intervention trigger level meets the preset trigger conditions, matches the target cognitive intervention mode based on the emotion evolution trend and determines the current cognitive behavior intervention stage; combines the user's real-time multimodal data and digital human action execution data in the current cognitive behavior intervention stage to determine the physical intervention range of the digital human on the user; and generates voice intervention data and digital human interaction control parameters corresponding to the current cognitive behavior intervention stage based on the physical intervention range. The feedback recognition module controls the digital human to perform intervention interactions based on digital human interaction control parameters and voice intervention data; records the timestamp of intervention command issuance, extracts the interaction effective timestamp from the digital human action execution data, performs time response correction on the user's emotional data before and after intervention, and outputs the effective intervention response range and the target emotional feedback data within the effective intervention response range. The parameter update module determines the changes in the user's emotional state after intervention based on the target emotional feedback data and performs an intervention effect evaluation. In the intervention effect evaluation, it identifies abnormal emotional fluctuations caused by the current cognitive-behavioral intervention stage to determine the true intervention response state. Based on this true intervention response state, it updates the cognitive-behavioral intervention stage, as well as the voice intervention data and digital human interaction control parameters. The modules are connected to each other via wired and / or wireless means.

[0006] Furthermore, methods for performing emotion recognition based on user emotion data include: Extract user voice data and user facial image data from user emotion data; Voice emotion features are constructed by measuring the fundamental frequency of speech, speech rate and speech pause duration of user speech data; facial expression features are constructed by measuring the range of eyebrow changes, eyelid opening and closing and mouth corner deflection of user facial image data. The difference between each item in the speech emotion feature and the preset speech baseline value is calculated, and the sum is obtained by weighting the values ​​based on the preset speech element ratios. The difference between each item in the facial expression feature and the preset facial expression baseline value is calculated, and the sum is obtained by weighting the values ​​based on the preset facial expression element ratios. Identify the visible proportion of the face, the body orientation deflection angle, and the limb movement speed in user posture data within the same sampling period; if the visible proportion of the face is less than a preset proportion threshold or the body orientation deflection angle is greater than a preset deflection threshold, set the facial expression weight to be less than the voice weight; match the limb movement speed with a preset speed range to obtain the posture emotion compensation amount. The intensity of facial expression emotion and the intensity of voice emotion are weighted and summed using facial expression weight and voice weight, and then superimposed with the postural emotion compensation amount to obtain the comprehensive emotion intensity. The system matches voice emotion features and facial expression features with a preset emotion parameter table, outputs emotion category labels, and combines the comprehensive emotion intensity to output the emotion recognition result.

[0007] Furthermore, methods for determining the trend of emotion evolution include: The combined emotional intensity and emotional category labels of consecutive sampling periods are used to form an emotional sequence; Calculate the intensity difference between adjacent comprehensive emotion intensities in the emotion sequence. If there is an intensity difference greater than a preset fluctuation threshold, the corresponding comprehensive emotion intensity is judged as an outlier. Replace the outlier with the mean of the comprehensive emotion intensities before and after it, and output the adjusted emotion sequence. Identify the first and last comprehensive emotional intensity of the adjusted emotional sequence and calculate the difference to obtain the first-to-last intensity difference; compare the first-to-last intensity difference with the trend fluctuation threshold range and output the trend determination result; By matching the trend judgment results and emotion category labels with a preset intervention trigger mapping table, the corresponding intervention trigger level is determined.

[0008] Furthermore, when the intervention trigger level meets the preset trigger conditions, the methods for matching the target cognitive intervention model based on the emotional evolution trend and determining the current cognitive behavioral intervention stage include: When the intervention trigger level is not lower than the preset trigger threshold, the difference between the first and last strengths of the corresponding trend judgment result is extracted; if several pre-trigger levels are higher than the high-risk level and the difference between the first and last strengths is greater than the upper limit of the trend fluctuation threshold range, the target cognitive intervention mode is set to emergency intervention mode; if several pre-trigger levels are not higher than the high-risk level and the difference between the first and last strengths is in the stable range within the trend fluctuation threshold range, the target cognitive intervention mode is set to stable intervention mode. Obtain the set of preset intervention stages corresponding to the target cognitive intervention model, including the emotional soothing stage and the cognitive guidance stage; compare the overall emotional intensity of the current sampling period with the preset stage switching threshold. If the overall emotional intensity is greater than the preset stage switching threshold, or the intensity difference between the beginning and end is greater than zero, then confirm that the current cognitive behavioral intervention stage is the emotional soothing stage. If the overall emotional intensity is not greater than the preset stage switching threshold, and the intensity difference between the beginning and end is not greater than zero, then the current cognitive behavioral intervention stage is confirmed as the cognitive guidance stage.

[0009] Furthermore, methods for determining the range of physical intervention by digital humans on users include: Identify real-time body orientation deflection angle, real-time limb movement speed, and real-time comprehensive emotion intensity in real-time multimodal data; obtain user facial orientation vector and digital human gaze vector and calculate cosine similarity; The difference between the real-time body orientation deflection angle and the preset deflection threshold is calculated, and the degree of posture avoidance is calculated by combining cosine similarity; the degree of limb movement is calculated based on real-time limb movement speed and real-time comprehensive emotional intensity. The degree of user resistance is obtained by weighted summation of the degree of posture avoidance and the degree of limb movement; the degree of user resistance is converted into a change in safe distance, and summed with a preset baseline safe distance to output the dynamic safe boundary distance; Obtain the real-time spatial distance from the digital human's motion execution data, and calculate the distance difference between the real-time spatial distance and the dynamic safety boundary distance; Set positive and negative boundary tolerances. If the distance difference is less than the negative boundary tolerance, set the far distance range as the physical intervention range and control the digital human to execute the priority retreat command. If the distance difference is greater than the positive boundary tolerance, and the current cognitive behavioral intervention stage is the emotional soothing stage, then the close distance range is set as the physical intervention range, and the digital human is controlled to execute the slow approach command. If the distance difference is between the positive boundary tolerance and the negative boundary tolerance, then the physical intervention range of the previous sampling period is maintained or the medium distance range is set as the physical intervention range.

[0010] Furthermore, the methods for generating voice intervention data and digital human interaction control parameters corresponding to the current cognitive behavioral intervention stage based on this physical intervention interval include: Based on the current cognitive behavioral intervention stage, the corresponding target speech content template is matched, and the basic interactive speech rate and basic volume are obtained. The basic interactive speech rate and basic volume are proportionally adjusted based on the physical intervention interval and distance difference, and the speech intervention data is output. Based on the physical intervention range, the target speed range and movement amplitude range of the digital human are matched, and the current cognitive and behavioral intervention stage is combined to determine whether to perform touch interaction and generate touch interaction labels. The target speed range, motion range, and touch interaction labels are integrated to form the digital human interaction control parameters.

[0011] Furthermore, methods for time-response correction of user sentiment data before and after intervention include: When an intervention command containing voice intervention data and digital human interaction control parameters is issued, the timestamp of the intervention command issuance is recorded; Real-time monitoring of acceleration or joint angle change rate in digital human motion execution data; if acceleration or joint angle change rate is greater than the preset corresponding motion response threshold, record the interaction activation timestamp; calculate the sum of the interaction activation timestamp and the user's cognitive buffer time to obtain the theoretical feedback timestamp. User emotional data between the intervention instruction issuance timestamp and the theoretical feedback timestamp is treated as transitional emotional data and discarded; the theoretical feedback timestamp is used as the starting point, and the effective intervention response interval is divided according to the preset evaluation duration, and user emotional data within the effective intervention response interval is used as the target emotional feedback data.

[0012] Furthermore, the calculation methods for user cognitive buffer time include: Obtain the historical comprehensive emotional intensity of the previous sampling period before the intervention interaction, and obtain the stage delay ratio corresponding to the current cognitive behavioral intervention stage; calculate the ratio of the historical comprehensive emotional intensity to the preset stable intensity to obtain the emotional buffer ratio; calculate the product of the preset basic cognitive response time, the stage delay ratio, and the emotional buffer ratio to obtain the user's cognitive buffer time.

[0013] Furthermore, based on the target emotional feedback data, the changes in the user's emotional state after the intervention are determined, and the intervention effectiveness is evaluated. In the process of evaluating the intervention effectiveness, methods for identifying abnormal emotional fluctuations triggered by the current cognitive-behavioral intervention stage to determine the true intervention response state include: The time period corresponding to the target emotional feedback data is divided into the preceding response period and the following response period; the comprehensive emotional intensity peak of the preceding response period and the following response period are extracted respectively as the preceding intensity peak and the following intensity peak, and the terminal emotional intensity of the following response period is obtained at the same time. If the preceding intensity peak is greater than the comprehensive emotional intensity of the sampling period before the intervention interaction, and the subsequent intensity peak and the terminal emotional intensity are both less than the preceding intensity peak, then the current real intervention response state is determined to be an intervention transition state. If both the preceding and subsequent intensity peaks are greater than the overall emotional intensity of the sampling period before the intervention interaction, and the final emotional intensity is not less than the subsequent intensity peak, then the current real intervention response state is determined to be an intervention failure state. If neither the preceding nor the following intensity peak is greater than the overall emotional intensity of the sampling period prior to the intervention interaction, then the current actual intervention response state is determined to be in a stable intervention state.

[0014] Furthermore, the methods for updating the cognitive behavioral intervention phase, as well as the voice intervention data and digital human interaction control parameters, include: If the actual intervention response state is in the intervention transition state, then the current cognitive behavioral intervention stage remains unchanged, and the corresponding execution cycle is extended; If the actual intervention response status is an intervention failure status, the current cognitive behavioral intervention stage will be transformed into the emotional soothing stage, and the corresponding voice intervention data and digital human interaction control parameters will be regenerated. If the actual intervention response status is in a stable state, the current cognitive behavioral intervention stage will be advanced to the cognitive guidance stage, and the voice intervention data and digital human interaction control parameters will be updated simultaneously.

[0015] The technical effects and advantages of the multimodal semantic understanding-based interactive digital human management system for the elderly in this application are as follows: By collecting and processing initial multimodal data during the interaction between elderly users and the digital human, and obtaining data related to the elderly users' emotions and the digital human's performance, a digital human management system for elderly interaction based on multimodal semantic understanding was finally realized. Compared with existing experience, by combining voice, facial expression, and posture information, the system avoids the distortion of emotion recognition in complex scenarios caused by single-modal data. By judging the trend of emotion changes in continuous sampling periods, the system eliminates signal jitter caused by instantaneous emotional fluctuations. By analyzing the degree of user resistance and determining the physical intervention range, the system avoids further intervention stimulation to elderly users caused by the digital human approaching too closely or repeatedly moving back and forth. By correcting the execution time response, the system corrects the time attribution misalignment between intervention actions and real emotional feedback. By identifying abnormal emotional fluctuations in the intervention effect, the system determines the real intervention response state and avoids erroneous switching of intervention strategies. The system improves the control precision of the digital human's intervention performance and ensures the stability and safety of human-computer interaction. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the interactive digital human management system for the elderly based on multimodal semantic understanding, as described in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] Example 1 Please see Figure 1 As shown in this embodiment, the interactive digital human management system for the elderly based on multimodal semantic understanding includes: The data acquisition module collects multimodal data in real time during the interaction between the user and the digital human, including user emotion data, user posture data, and digital human action execution data. The emotion recognition module performs emotion recognition based on user emotion data, determines the emotion evolution trend based on the emotion recognition results, and determines the intervention trigger level. The control parameter generation module, when the intervention trigger level meets the preset trigger conditions, matches the target cognitive intervention mode based on the emotion evolution trend and determines the current cognitive behavior intervention stage; combines the user's real-time multimodal data and digital human action execution data in the current cognitive behavior intervention stage to determine the physical intervention range of the digital human on the user; and generates voice intervention data and digital human interaction control parameters corresponding to the current cognitive behavior intervention stage based on the physical intervention range. The feedback recognition module controls the digital human to perform intervention interactions based on digital human interaction control parameters and voice intervention data; records the timestamp of intervention command issuance, extracts the interaction effective timestamp from the digital human action execution data, performs time response correction on the user's emotional data before and after intervention, and outputs the effective intervention response range and the target emotional feedback data within the effective intervention response range. The parameter update module determines the changes in the user's emotional state after intervention based on the target emotional feedback data and performs an intervention effect evaluation. In the intervention effect evaluation, it identifies abnormal emotional fluctuations caused by the current cognitive-behavioral intervention stage to determine the true intervention response state. Based on this true intervention response state, it updates the cognitive-behavioral intervention stage, as well as the voice intervention data and digital human interaction control parameters. The modules are connected to each other via wired and / or wireless means.

[0019] In this embodiment, when a user interacts with a digital human, the user is often in a state of emotional sensitivity or intense fluctuation. In such cases, the digital human is needed as an interaction medium to interact with the elderly user through methods such as dialogue and physical movements. The user's emotional data includes audio stream data collected by the digital human's sound recording module and facial image data of the elderly user collected by the digital human's camera. User posture data refers to the posture exhibited by the elderly user during interaction when they exhibit resistance or subconscious defensiveness due to emotions; this data is collected by the digital human's camera and skeletal recognition sensor, including, for example, body orientation angle, facial visibility ratio, and limb movement speed. Digital human motion execution data refers to parameters of the digital human during interaction, such as distance, speed, and joint rotation angle. Here, "digital human" refers to a robot or interactive device that achieves human-computer interaction through recognizing commands and related digital data.

[0020] Methods for performing sentiment recognition based on user sentiment data include: Extract user voice data and user facial image data from user emotion data.

[0021] The user voice data includes audio content such as the elderly user's speech, sighs, crying, or pauses and silences; the user facial images include frontal facial images, side facial images, and image frames from different angles generated during the interaction.

[0022] Voice emotion features are constructed by measuring the fundamental frequency of speech, speech rate, and duration of pauses in user speech data.

[0023] The fundamental frequency of speech is obtained by extracting the frequency of user speech data. Speech rate refers to the number of effective syllables or words uttered by elderly users per unit time. Speech pause duration refers to the duration of silent intervals that occur between consecutive sentences by elderly users. Since human emotions easily affect language expression, the above three acoustic parameters are combined into speech emotion features.

[0024] Facial expression features are formed by measuring the range of eyebrow changes, eyelid opening and closing, and mouth corner deflection in user facial image data.

[0025] Among them, eyebrow change range refers to the vertical displacement of key feature points in the eyebrow area of ​​elderly users relative to the normal eyebrow position; eyelid opening and closing range refers to the proportion of the vertical distance between the upper and lower eyelids of elderly users to the overall height of the eye socket; mouth corner deflection range refers to the combined horizontal and vertical displacement of key feature points on the left and right corners of the mouth relative to the normal mouth position; since the muscles in the eyebrow, eyelid, and mouth areas are the most obvious and emotionally recognizable areas in human facial expressions, the above three facial features are combined into facial expression features.

[0026] The difference between each of the speech emotion features and the preset speech baseline value is calculated, and the sum is obtained by weighting the values ​​based on the preset speech element ratios.

[0027] In reality, the basic frequency of speech of elderly users is usually low, with male elderly users mostly concentrated in the range of 100Hz to 170Hz and female elderly users mostly concentrated in the range of 160Hz to 230Hz. The speech rate of elderly users is generally slower, and the pauses between sentences are significantly longer. Some elderly users may have a long period of silence or repeated pauses when expressing complex emotions. Moreover, the speech of elderly users may contain dialect words or unclear pronunciation, resulting in large fluctuations in the duration of speech pauses and speech rate. Therefore, the preset speech reference value in this embodiment refers to the reference value set based on the distribution of three acoustic parameters obtained by statistically analyzing a large number of elderly users in a normal calm state: basic frequency of speech, speech rate, and duration of speech pauses.

[0028] The preset speech element ratio refers to the proportional coefficient set for the difference in basic speech frequency, speech rate, and speech pause duration. It is set based on the information contribution of each acoustic parameter to the expression of emotion obtained from a large number of psychoacoustic experimental records, and higher weights are given to acoustic parameters that are more sensitive to emotions.

[0029] The intensity of voice emotion was obtained by weighted summation of the differences between each acoustic parameter and its corresponding baseline value, thus quantifying the emotional fluctuations of each corresponding elderly user in their speech.

[0030] The difference between each facial expression feature and the preset expression baseline value is calculated, and the emotional intensity of the expression is obtained by weighted summation based on the preset expression element ratios.

[0031] It should be noted that there are significant differences in facial expression characteristics between elderly users and younger users. Due to the loose skin and numerous wrinkles on the faces of elderly users, the detection accuracy of key facial feature points is easily affected. For example, deep forehead wrinkles and nasolabial folds may affect the accurate measurement of the range of eyebrow changes and the range of mouth corner deflection. At the same time, when expressing the same emotion, the range of facial muscle activity of elderly users is usually smaller than that of younger users, and the changes in expression are more subtle. For example, when expressing sadness, the range of mouth corner deflection of elderly users may only be 50% to 70% of that of younger users. Therefore, in this embodiment, the preset expression benchmark value refers to a set of unified standard expression features obtained by statistically analyzing the expression features of a large number of users in a calm and relaxed state. Even if there are differences in the facial structure of different elderly users, facial features can be compared with the preset expression benchmark value by making certain proportional adjustments.

[0032] The preset facial expression element ratio refers to the ratio coefficient of the range of eyebrow changes, eyelid opening and closing, and mouth corner deflection. It is a statistical setting of the information contribution of each facial region to the expression of emotion, obtained based on experimental data of the facial expression recognition region.

[0033] By calculating the intensity of facial expressions, the geometrical features of facial changes of different dimensions are transformed into an independent parameter that reflects emotional fluctuations.

[0034] Identify the proportion of visible faces, body orientation angle, and limb movement speed in user posture data from the same sampling period.

[0035] Data extraction is performed on user posture data within the same sampling period for the aforementioned user voice data and user facial image data. The visible facial ratio refers to the average proportion of the number of successfully identified facial key feature points in all image frames within the corresponding sampling period to the total number of complete facial key feature points of the user. The smaller this value, the greater the degree to which the elderly user's face is obscured. The body orientation deflection angle refers to the angle between the normal direction of the elderly user's torso and the direction in which the elderly user faces the digital human. In this embodiment, it is extracted using a skeletal key point detection algorithm. The larger this value, the greater the degree of the elderly user's turn. The limb movement speed refers to the average displacement of the key points at the extremities of the elderly user per unit time. The larger this value, the more intense the elderly user's limb activity.

[0036] It should be noted that the overall range of limb movement speed of elderly users is significantly lower, and the speed of body orientation deflection is slower. It usually takes more transition time to turn from facing the interactive digital human to turning to the side. At the same time, some elderly users have limited range of body orientation deflection angle due to cervical or lumbar spine diseases, and may not be able to complete large-scale turning movements.

[0037] If the visible proportion of the face is less than the preset proportion threshold or the body orientation deflection angle is greater than the preset deflection threshold, the facial expression weight is set to be less than the voice weight.

[0038] The preset ratio threshold refers to the lower limit of the visible proportion of the face, which is set based on experience in detecting a large number of samples using facial key point detection algorithms. When the proportion of visible key feature points on the face is lower than this threshold, it means that the accuracy of facial expression features cannot effectively reflect the corresponding emotional expression.

[0039] The preset deflection threshold refers to the upper limit of the body's facing deflection angle. Based on the parameter configuration settings of the digital human camera, when the body's facing deflection angle exceeds this deflection threshold, it means that the corresponding camera cannot effectively capture the facial features of elderly users.

[0040] In this embodiment, the initial facial expression weight and voice weight are the same, both being 0.5. When the visible proportion of the face is less than the corresponding threshold or the body orientation deflection angle is greater than the corresponding threshold, it indicates that the facial features may be distorted due to physical occlusion. Therefore, the facial features cannot be effectively used to determine emotions. Reducing the facial expression weight reduces the information contribution of the facial features, thereby indirectly improving the accuracy of emotion judgment.

[0041] The posture and emotion compensation amount is obtained by matching the limb movement speed with a preset speed range.

[0042] In this embodiment, the preset speed range refers to the range set based on common sense of human kinematics and a large amount of experimental data generated by stress behavior in psychological experiments, which is used to divide the speed of multiple limb movements. For example, the low speed range corresponds to the state of elderly users sitting still or moving slowly, and the high speed range corresponds to the stress state of elderly users pacing violently or trembling.

[0043] Postural emotion compensation refers to the compensation value associated with each speed range, and specific values ​​are set based on the relationship between the intensity of different limb activities and the degree of emotional expression in a large number of psychological experiments.

[0044] This compensation value is used to address the emotion recognition problem when elderly users are silent and their faces are obscured. It also takes into account the influence of the elderly user's posture on emotion judgment. For example, if an elderly user is extremely anxious and is silent while looking down, the corresponding compensation value can be obtained by obtaining the high-frequency limb shaking movements of the elderly user. For example, when an elderly user is sitting in a wheelchair or armchair, the overall limb movement speed is relatively low, but the hands may exhibit subtle high-frequency movements such as repeatedly rubbing the hands or gripping the armrests due to anxiety. Although the overall limb movement speed is not high, emotion recognition can still be carried out by obtaining the compensation amount of this high-frequency movement.

[0045] The intensity of facial expression and the intensity of voice emotion are weighted and summed using facial expression weight and voice weight, and then superimposed with the postural emotion compensation amount to obtain the comprehensive emotion intensity.

[0046] The adjusted facial expression weights and voice weights are used to weight and sum the facial expression intensity and voice emotion intensity to obtain a basic emotion intensity. Furthermore, a posture emotion compensation is introduced to supplement the hidden emotional information reflected by the posture of elderly users, thereby forming the overall true emotional strength of elderly users within the corresponding sampling period, i.e., the comprehensive emotion intensity.

[0047] The system matches voice emotion features and facial expression features with a preset emotion parameter table, outputs emotion category labels, and combines the comprehensive emotion intensity to output the emotion recognition result.

[0048] The preset emotion parameter table refers to a data table that maps features to emotion categories, constructed based on psychological emotion classification knowledge and a large number of emotion judgment samples. The table records the emotion categories corresponding to different combinations of voice features and facial expression features, such as anger, sadness, anxiety, joy, and calmness. The emotion category label refers to the name of the emotion category output after searching and matching in the above emotion parameter table.

[0049] Methods for determining the trend of emotion evolution include: The comprehensive emotion intensity and emotion category label of continuous sampling periods are used to form an emotion sequence, in which the comprehensive emotion intensity and corresponding emotion category label of continuous sampling periods are arranged in chronological order to form the emotion sequence.

[0050] Calculate the intensity difference between adjacent comprehensive emotion intensities in the emotion sequence. If there is an intensity difference greater than a preset fluctuation threshold, the corresponding comprehensive emotion intensity is judged as an outlier.

[0051] In this embodiment, the intensity difference refers to the absolute value of the difference between adjacent comprehensive emotional intensities in the emotional sequence; the preset fluctuation threshold refers to the maximum reasonable change in emotional intensity between adjacent sampling periods; and the upper limit of the rate of emotional change under normal circumstances is determined based on psychological experimental data, that is, under normal circumstances, elderly users will not experience drastic changes higher than the preset fluctuation threshold between two adjacent sampling periods.

[0052] If the intensity difference is greater than the preset fluctuation threshold, it indicates that it is likely not caused by real emotional changes, but by environmental noise or false signals caused by elderly users such as clearing their throats or sneezing. Therefore, the comprehensive emotional intensity of the corresponding sampling period is judged as an outlier.

[0053] The outlier is replaced by the average of the combined emotional intensities of the adjacent values ​​before and after it, and the adjusted emotional sequence is output.

[0054] The process involves calculating the mean of the combined emotional intensity of two adjacent sampling periods before and after an outlier. To avoid directly removing outliers and causing a break in the timeline, the calculated mean is used to replace the outlier, ensuring the continuity of the emotional sequence in the time dimension and thus obtaining an adjusted emotional sequence.

[0055] Identify the first and last comprehensive emotional intensities of the adjusted emotional sequence and calculate the difference to obtain the first-to-last intensity difference.

[0056] The first comprehensive emotional intensity refers to the earliest comprehensive emotional intensity in the emotional adjustment sequence, and the last comprehensive emotional intensity refers to the latest comprehensive emotional intensity in the emotional adjustment sequence. The difference between the two is obtained by calculating the first and last intensity difference. If the value is positive, it means that the overall emotion is developing in the direction of intensification in the corresponding time period. If it is negative, it means that the overall emotion is developing in the direction of calming down. If it is close to 0, it means that the overall emotion remains basically stable.

[0057] The difference between the strength of the beginning and end of the trend is compared with the trend fluctuation threshold range, and the trend determination result is output.

[0058] The trend fluctuation threshold range refers to a set of numerical ranges used to define the direction of emotional changes, including, for example, an upward judgment range, a stable judgment range, and a downward judgment range. The boundary values ​​of each numerical range are set based on the distribution of different trends obtained from a large amount of historical emotional fluctuation data. The trend judgment result is output by determining which range of the trend fluctuation threshold range the difference between the beginning and the end falls into.

[0059] By matching the trend judgment results and emotion category labels with a preset intervention trigger mapping table, the corresponding intervention trigger level is determined.

[0060] The preset intervention trigger mapping table is a data table set up based on clinical experience of cognitive behavioral intervention and the distribution of a large number of historical samples that have undergone emotion recognition. It is used to store the intervention trigger level corresponding to each emotion category and the trend of change of the emotion. The higher the level, the more unstable the current elderly user's emotions are, and the more obvious the need to perform cognitive intervention, reflecting the priority of performing cognitive intervention.

[0061] When the intervention trigger level meets the preset trigger conditions, the methods for matching the target cognitive intervention model based on the emotional evolution trend and determining the current cognitive behavioral intervention stage include: When the intervention trigger level is not lower than the preset trigger threshold, the difference between the first and last intensity of the corresponding trend judgment result is extracted.

[0062] The preset trigger threshold refers to the minimum risk level set by the digital human's internal configuration to initiate cognitive behavioral intervention, which is set based on the distribution of scenarios requiring no intervention and requiring intervention in historical samples. Several pre-trigger levels are not lower than the preset trigger threshold, indicating that the elderly user's emotional state has reached a level that requires the digital human to actively intervene. At this time, the corresponding beginning and end intensity difference is extracted to further determine the direction and magnitude of emotional changes within the corresponding time period.

[0063] If the intervention trigger level is higher than the high-risk level, and the difference between the first and last strengths is greater than the upper limit of the trend fluctuation threshold range, then the target cognitive intervention mode will be set to emergency intervention mode.

[0064] The high-risk level is the higher-level interval in the intervention trigger level, used to indicate that elderly users are in a state of emotional deterioration or need to be comforted. It is set based on the distribution of comprehensive emotional intensity in historical high-risk emotional samples.

[0065] If the intervention trigger level is higher than the high-risk level and the difference between the first and last intensity is greater than the upper limit of the trend fluctuation threshold range, it indicates that the increase in the emotional intensity of elderly users in the corresponding time period has exceeded the stable range. Therefore, the required target cognitive intervention mode is set as the emergency intervention mode. This mode refers to the priority intervention mode set by the digital human for high-risk states, including intervention strategies such as rapid soothing, reducing stimulation, and timely guidance.

[0066] If the intervention trigger level is not higher than the high-risk level, and the difference between the first and last strengths is within a stable range within the trend fluctuation threshold range, then the target cognition intervention mode will be set to the stable intervention mode.

[0067] The stable interval refers to the interval within the trend fluctuation threshold range used to represent the stable change state of emotions. Based on the statistical results of the intensity changes of elderly users' emotions in historical samples when there were no significant changes and only normal fluctuations; several pre-trigger levels are not higher than the high-risk level, and the intensity difference between the beginning and the end is within the stable interval, indicating that although elderly users have some emotional fluctuations in the corresponding time period, the overall change is small. Therefore, the target cognitive intervention mode is set as the stable intervention mode. This mode refers to the digital human intervening in the elderly user's life using a slower pace of guidance without overstimulating the elderly user.

[0068] Obtain a set of pre-set intervention stages corresponding to the target cognitive intervention model, including the emotional soothing stage and the cognitive guidance stage.

[0069] The pre-set intervention stage set is a sequence of stages pre-divided for the target cognitive intervention mode in different time periods. Each intervention stage can be divided into two types: the emotional soothing stage and the cognitive guidance stage. The emotional soothing stage is the initial intervention stage used to reduce the level of negative emotions of elderly users. The cognitive guidance stage is the process by which the digital human guides elderly users to eliminate negative emotions through voice dialogue or behavior, which belongs to the deep intervention stage.

[0070] The overall emotional intensity of the current sampling period is compared with the preset stage switching threshold. If the overall emotional intensity is greater than the preset stage switching threshold, or the intensity difference between the beginning and end is greater than zero, then the current cognitive behavioral intervention stage is confirmed as the emotional soothing stage.

[0071] The preset stage switching threshold refers to the pre-set boundary value of emotional intensity used to distinguish between soothing and guidance. It is set based on the statistical results of elderly users successfully accepting guidance under different comprehensive emotional intensities in historical samples. If the comprehensive emotional intensity is greater than the preset stage switching threshold, it means that the elderly user's emotions are in a high state. If the corresponding intensity difference between the beginning and end is greater than 0, it means that the elderly user's emotions are still rising in the corresponding time period. If either condition is met, it can be considered that soothing needs to be performed. Therefore, the current cognitive behavioral intervention stage is determined to be the emotional soothing stage.

[0072] If the overall emotional intensity is not greater than the preset stage switching threshold, and the intensity difference between the beginning and end is not greater than zero, then the current cognitive behavioral intervention stage is confirmed as the cognitive guidance stage.

[0073] If the overall emotional intensity is not greater than the preset stage switching threshold, it means that the emotional intensity level of the elderly user is within an acceptable range. At the same time, if the difference between the intensity at the beginning and the end is not greater than 0, it means that the elderly user's emotions have not continued to deteriorate during the corresponding time period, are relatively stable, or may even decline. At this time, the current cognitive behavioral intervention stage is determined to be the cognitive guidance stage.

[0074] Methods for determining the range of physical intervention by digital humans on users include: Identify real-time body orientation deflection angle, real-time limb movement speed, and real-time comprehensive emotion intensity in real-time multimodal data.

[0075] Real-time multimodal data refers to multimodal data collected in real time during the current cognitive behavioral intervention phase. Real-time body orientation angle, real-time limb movement speed, and real-time comprehensive emotional intensity are extracted from this data as the data basis for subsequent operations.

[0076] Obtain the user's facial orientation vector and the digital human's gaze vector, and calculate the cosine similarity.

[0077] The formula for calculating cosine similarity is as follows: ;in The cosine similarity between the user's facial orientation vector and the digital human's gaze vector represents the similarity between the two vectors. This represents the user's facial orientation vector, which is a vector estimated based on the elderly user's facial key points or head posture to represent the front of the elderly user's face. This represents the digital human's gaze vector, which is the vector pointing from the digital human's camera or torso in the direction the digital human's gaze is directed.

[0078] In this embodiment, to facilitate judgment, the direction of the digital human's gaze vector is reversed. If the cosine similarity is closer to 1, it means that the elderly user and the digital human are almost face to face. If it is closer to 0, it means that the elderly user has obvious posture deviation and avoidance. If it is closer to -1, it means that the elderly user is facing away from the digital human, which may indicate a state of resistance.

[0079] The difference between the real-time body orientation deflection angle and the preset deflection threshold is calculated, and the degree of posture avoidance is calculated by combining cosine similarity.

[0080] The formula for calculating the degree of posture avoidance is as follows: ;in Indicates the degree of avoidance in posture; Indicates cosine similarity; Indicates the real-time body orientation deflection angle; Indicates the preset deflection threshold; This indicates that the maximum value is being taken, used to filter out negative values; when an elderly user is facing the digital human figure, The value tends to 0, meaning that even if the body is turned, it will not affect the judgment of the elderly user's face orientation; this posture avoidance level is used to represent the overall avoidance level of the elderly user through posture in the corresponding sampling period.

[0081] The degree of limb movement is calculated based on real-time limb movement speed and real-time comprehensive emotional intensity.

[0082] The formula for calculating the degree of limb movement is as follows: ;in This value indicates the degree of limb movement and quantifies the overall intensity of limb activity caused by the influence of emotions on limb movement. This represents the result after normalizing the difference between the real-time emotion intensity and the preset stable intensity. Indicates the real-time overall emotional intensity. This indicates the preset stable intensity, which is a threshold value for the intensity of a normal emotional state, set based on a large amount of comprehensive emotional intensity calculation experience. This indicates that the maximum value is taken, and is used to filter out negative values. This indicates the real-time speed of limb movement.

[0083] The process involves first calculating the difference between the real-time comprehensive emotional intensity and the preset stable intensity to reflect the degree to which the elderly user's current emotions exceed the normal state. This value is then normalized to eliminate dimensional differences. The normalized value is used as a weight to weight the real-time limb movement speed, thus adding the emotional influence to the limb activity and ultimately obtaining the degree of limb movement.

[0084] The degree of user resistance is obtained by weighting and summing the degree of posture avoidance and the degree of body movement.

[0085] The study weights and unifies the quantities of posture avoidance and body movement into a single parameter, namely, the degree of user resistance, which represents the overall level of rejection that elderly users have towards interacting with the digital human during the current sampling period. The degree of posture avoidance and body movement are first standardized to the same dimension before being weighted. It should be noted that the weights of posture avoidance and body movement are based on the frequency of instances in a large amount of historical interaction data where posture avoidance and body movement led to clear resistance to interaction from elderly users. A lower degree of user resistance indicates a higher level of acceptance of the digital human interaction among elderly users, while a higher degree indicates significant resistance.

[0086] The user's level of resistance is converted into a change in the safe distance, and the result is summed with the preset baseline safe distance to output the dynamic safe boundary distance.

[0087] The change in safe distance refers to the distance increment obtained by multiplying the user's resistance level by the preset distance conversion coefficient. The preset distance conversion coefficient is a proportional coefficient that maps the resistance level to the back distance, which is determined based on the physical movement range of the digital human, the size of the interactive space, and a large amount of experimental data on human-computer interaction safe distance. For example, if the distance conversion coefficient is set to back 10cm for each user's resistance level, then when the user's resistance level is 3, the change in safe distance is 30cm.

[0088] The preset baseline safe distance refers to the distance set based on a large amount of experimental data on human-computer interaction safe distance. It is used to represent the basic interaction distance that elderly users maintain with the digital human when they are in a non-resistant state. The dynamic safe boundary distance is obtained by summing the preset baseline safe distance and the change in safe distance. It is used to represent the interaction distance adjusted after taking into account the resistance state of elderly users.

[0089] Obtain the real-time spatial distance from the digital human's motion execution data, and calculate the distance difference between the real-time spatial distance and the dynamic safety boundary distance.

[0090] The distance difference is obtained by calculating the difference between the real-time spatial distance of the digital human and the dynamic safety boundary distance, which is used to represent the deviation of the real-time spatial distance from the dynamic safety boundary distance.

[0091] Set positive and negative boundary tolerances. If the distance difference is less than the negative boundary tolerance, set the far distance range as the physical intervention range, and control the digital human to execute the priority retreat command.

[0092] The positive boundary tolerance and negative boundary tolerance refer to two buffer amounts for the positive and negative directions, which are used to avoid the digital human from frequently switching between different physical intervention ranges due to small fluctuations in measurement when the distance difference is close to 0.

[0093] Negative boundary tolerance refers to the maximum amount of entry that the digital human is allowed to enter within the dynamic safety boundary distance, while positive boundary tolerance refers to the minimum amount of distance that the digital human is required to move away from the safety boundary. Both are set based on the digital human's sensor measurement configuration and the minimum displacement of its single movement, to ensure that interval switching is triggered only when the offset of the distance difference exceeds the sensor's allowable noise range and the bottom mechanical structure's response dead zone.

[0094] When the distance difference is less than the negative boundary tolerance, it means that the digital human has entered the safety boundary of the elderly user. At this time, a long distance range is set. This long distance range refers to the physical intervention range in which the digital human should prioritize maintaining a large distance and actively retreat. At the same time, a retreat command is issued to ensure that the digital human will retreat to the outside of the safety boundary after receiving the command, so as to avoid further stimulation if the elderly user shows obvious resistance.

[0095] If the distance difference is greater than the positive boundary tolerance, and the current cognitive behavioral intervention stage is the emotional soothing stage, then the close distance range is set as the physical intervention range, and the digital human is controlled to execute the slow approach command.

[0096] If the distance difference is greater than the positive boundary tolerance, it means that the real-time distance of the digital human is significantly greater than the dynamic safety boundary distance. At the same time, if the corresponding current cognitive behavior intervention stage is the emotional soothing stage, a close distance range can be set. This close distance range refers to the physical intervention range in which the digital human is allowed to maintain a relatively close interaction distance under certain conditions, and a slow approach command is issued to guide the digital human to slowly approach the elderly user at a low speed.

[0097] If the distance difference is between the positive boundary tolerance and the negative boundary tolerance, then the physical intervention range of the previous sampling period is maintained or the medium distance range is set as the physical intervention range.

[0098] When the distance difference is between the positive and negative boundary tolerances, it indicates that the digital human is close to the dynamic safety boundary distance, but has not yet reached the point where forced retreat or slow approach is allowed. In this case, due to fluctuations in sensor measurements, slight body shaking or breathing fluctuations of elderly users, the distance difference may fluctuate slightly. Therefore, to avoid frequent and repeated movements of the digital human, the physical intervention range used in the previous sampling period is maintained first in this state. If no effective physical intervention range is generated in the previous sampling period, the medium distance range is set as the physical intervention range. The medium distance range refers to the buffered interaction range in which the digital human can maintain an appropriate distance. It is an intermediate state that was finally determined by performing a large number of human-computer interaction experiments.

[0099] The methods for generating voice intervention data and digital human interaction control parameters corresponding to the current cognitive behavioral intervention stage based on this physical intervention interval include: Based on the current cognitive behavioral intervention stage, the corresponding target speech content template is matched, and the basic interactive speech rate and basic volume are obtained.

[0100] The digital human database stores pre-set speech templates for different cognitive-behavioral intervention stages. Users can select the corresponding target speech content template according to the specific type of cognitive-behavioral intervention stage. For example, the speech template for the emotional soothing stage is mainly composed of soothing and low-stimulation statements, while the speech template for the cognitive guidance stage is mainly composed of open-ended questions and behavioral guidance statements.

[0101] The basic interactive speech rate and basic volume are the default speech output speed and loudness set for the cognitive behavioral intervention stage. For example, the emotional soothing stage usually uses a slower speech rate and a softer volume to avoid creating a sense of pressure due to excessively fast speech. The cognitive guidance stage usually uses a slightly faster and clearer speech rate and a moderate volume to ensure the accuracy of information transmission.

[0102] Based on the physical intervention range and distance difference, the basic interactive speech rate and basic volume are proportionally adjusted to output speech intervention data.

[0103] Since the distance between the digital human and the elderly user in actual interaction also affects the effect of cognitive behavioral intervention on the elderly user, it is necessary to make corresponding proportional adjustments to the basic interactive speech rate and basic volume based on the specific physical intervention range and distance difference.

[0104] When the physical intervention range is a long-distance range, the voice received by the elderly user will be weaker due to distance attenuation because the digital human is far away. Therefore, the basic volume is appropriately amplified according to a certain ratio, while the basic interactive speech speed is appropriately slowed down to ensure that the elderly user can still hear the sentences clearly at a greater distance. When the physical intervention range is a short-distance range, the volume of the digital human is close to the elderly user, and excessive volume may cause resistance from the elderly user. Therefore, the basic volume is appropriately reduced according to a certain ratio, while maintaining or slightly increasing the basic interactive speech speed to maintain normal interaction. The corresponding ratio is set based on acoustic theory and the correspondence between distance difference and sound propagation in real time.

[0105] The voice intervention data includes the voice content that needs to be output interactively, as well as the corresponding adjusted interactive speech rate and volume parameters.

[0106] Based on the target speed range and movement amplitude range of the digital human matched with the physical intervention range, and combined with the current cognitive behavioral intervention stage, it is determined whether to perform touch interaction and generate touch interaction labels.

[0107] The target speed range refers to the upper and lower limits of movement speed within the allowable range of the corresponding physical intervention range, set based on the physical intervention range. For example, when the digital human is in the far-distance range, it needs to quickly move backward, so the upper limit of the allowable movement speed is higher at this time. When it is in the close-distance range, in order to avoid causing resistance from elderly users, the upper limit of the allowable movement speed is lower.

[0108] The range of motion refers to the upper and lower limits of the range of motion that a robotic arm or head can perform within the allowable range of the corresponding physical intervention range. For example, when a digital human is in a close range, in order to avoid excessive movement that may create a sense of oppression in the field of vision of elderly users, the upper limit of the allowable range of motion is relatively small.

[0109] Touch interaction tags are control markers constructed to determine whether the digital human is allowed to perform contact-type actions such as lightly touching the arm or patting the shoulder during the current cognitive behavioral intervention stage. They are determined based on the physical intervention range and the current cognitive behavioral intervention stage. In this embodiment, touch interaction is only allowed when the physical intervention range is the close range and the current cognitive behavioral intervention stage is the emotional soothing stage. In other cases, touch interaction is set to be disallowed to avoid further impacting the emotions of elderly users at inappropriate times.

[0110] The target speed range, the range of motion, and the touch interaction label are integrated to form the digital human interaction control parameters. The target speed range, the range of motion, and the touch interaction label corresponding to the cognitive behavior intervention stage are matched to form the digital human interaction control parameters, which are used to control the digital human to perform interactive operations.

[0111] Methods for time-response correction of user sentiment data before and after intervention include: When an intervention command containing voice intervention data and digital human interaction control parameters is issued, the timestamp of the intervention command issuance is recorded.

[0112] In this embodiment, by merging voice intervention data and digital human interaction control parameters into a complete intervention command and issuing it, the precise timestamp of the command issuance is identified.

[0113] Real-time monitoring of acceleration or joint angle change rate in digital human motion execution data; if acceleration or joint angle change rate is greater than the preset corresponding motion response threshold, record the interaction activation timestamp.

[0114] Acceleration refers to the linear acceleration value of the lower limbs or bottom motion module of the digital human during operation, and the joint angle change rate refers to the speed of angle change of the motion joint structure such as the arm joint or head rotation joint of the digital human in real-time operation. Based on human-computer interaction experience obtained from a large number of human-computer interaction experiments, preset corresponding thresholds for acceleration and joint angle change rate are set for the corresponding actions. If the value of either variable is higher than the preset corresponding threshold, it means that the digital human's action has actually produced physical movement, so the timestamp is recorded as the interaction activation timestamp.

[0115] The theoretical feedback timestamp is obtained by summing the timestamp of the interaction taking effect with the user's cognitive buffer time.

[0116] Since elderly users need a period of cognitive process after perceiving external stimuli before they can produce obvious emotional changes, this period is called the user's cognitive buffer time. Therefore, the sum of the interaction take-off timestamp and the user's cognitive buffer time is calculated to obtain the earliest time when the elderly user's emotions begin to show effective response changes, which is the theoretical feedback timestamp.

[0117] User sentiment data between the intervention command issuance timestamp and the theoretical feedback timestamp is treated as transitional sentiment data and discarded.

[0118] In the time interval between the intervention command issuance time stamp and the theoretical feedback time stamp, the digital human may not have started actual movement yet, or the digital human may have just started moving but the elderly user has not produced an effective response. Therefore, the user's emotional data in this time interval cannot reflect the real response to the intervention. This data is treated as transitional emotional data and is removed to prevent contamination of other data.

[0119] Starting from the theoretical feedback timestamp, the effective intervention response interval is divided according to the preset evaluation duration, and the user emotional data within the effective intervention response interval is used as the target emotional feedback data.

[0120] The preset assessment duration refers to the observable time for collecting and assessing changes in the emotional response of elderly users to a single cognitive behavioral intervention. It is set based on the time range of emotional changes that elderly users experienced after fully receiving a cognitive behavioral intervention stimulus in historical records. It needs to be long enough to cover the complete process of the elderly user's emotions from response, fluctuation, and then to stabilization. The effective intervention response interval refers to the time window formed after the preset assessment duration, starting from the theoretical feedback timestamp. The emotional data of elderly users within this window reflects the complete emotional change process of the elderly users and is therefore used as the target emotional feedback data.

[0121] The calculation methods for user cognitive buffer time include: Obtain the historical comprehensive emotional intensity of the previous sampling period before the intervention interaction, and obtain the phase delay ratio corresponding to the current cognitive behavioral intervention stage.

[0122] The stage delay ratio refers to the time delay coefficient set for different cognitive behavioral intervention stages. The cognitive speed of elderly users to external stimuli varies in different stages. For example, in the emotional soothing stage, elderly users are usually in a state of emotional tension or high pressure, and their attention is more scattered. They need a longer processing time for external cognitive behavioral interventions. Therefore, the stage delay ratio corresponding to the emotional soothing stage is set to a value greater than 1 to extend the user's cognitive buffer time. For the cognitive guidance stage, elderly users are relatively calm and react quickly. Therefore, the stage delay ratio for this stage is set to a value close to and less than 1.

[0123] The emotional buffer ratio is obtained by calculating the ratio of the historical comprehensive emotional intensity to the preset stable intensity.

[0124] The preset stable intensity refers to the intensity threshold of a normal emotional state set based on a large amount of experience in calculating comprehensive emotional intensity. The emotional buffer ratio is obtained by calculating the ratio of the historical comprehensive emotional intensity of the previous sampling period to the preset stable intensity.

[0125] If the emotional level in the previous sampling period is high, then the emotional buffer ratio at this time will be greater than 1, indicating that the elderly user's current sampling period will be affected by the emotions of the previous sampling period; if the emotional level in the current sampling period is close to the preset stable intensity, then the emotional buffer ratio at this time will be close to 1, and it will not have an effective impact on the emotions of the current sampling period.

[0126] In this embodiment, a lower limit constraint is imposed on the value of the emotion buffer ratio, for example, it is set to 0.6; this is to prevent the emotion buffer ratio from being much less than 1 due to the low emotion level in the previous sampling period. Since this situation would cause the emotion buffer ratio to be close to 0, the user's cognitive buffer time would eventually be close to 0, which is obviously contrary to common sense.

[0127] The user's cognitive buffer time is obtained by multiplying the preset basic cognitive response time by the ratio of stage delay and the emotional buffer ratio.

[0128] The preset basic cognitive response time refers to the response time of elderly users in a standard calm state, which is set based on cognitive psychology theory. The impact of the corresponding cognitive behavior intervention stage is reflected by multiplying the calculated stage delay ratio with the preset basic cognitive response time. Then, the impact of the emotion buffer ratio is introduced by multiplying it with the emotion buffer ratio. Finally, the user cognitive buffer time is obtained by combining the historical impact and the current stage impact.

[0129] Based on the target emotional feedback data, determine the changes in the user's emotional state after the intervention and perform an intervention effectiveness evaluation. In the intervention effectiveness evaluation, identify abnormal emotional fluctuations triggered by the current cognitive-behavioral intervention stage to determine the true intervention response state. Methods for determining this include: The time period corresponding to the target emotional feedback data is divided into the preceding response period and the following response period.

[0130] Since the response of elderly users to intervention actions is generally not linear, but rather a process of fluctuation followed by stabilization, the time period corresponding to the target emotional feedback data can be divided into two parts: the pre-response period refers to the part in which the elderly user's emotions are generated and the intensity of the emotions increases, and the post-response period refers to the part in which the later soothing and guidance have an effect on the elderly user and the emotions gradually calm down.

[0131] It should be noted that the emotional venting process of elderly users is usually slower, the slope of the emotional rise in the pre-response period is relatively gentle, the emotional peak occurs relatively late, and the emotional decline process of elderly users is also longer, with a relatively gentle emotional decline curve in the post-response period. Some very elderly users may need more time to recover from emotional fluctuations to baseline levels.

[0132] The combined emotional intensity peaks of the preceding and following response periods are extracted separately as the preceding intensity peak and the following intensity peak, respectively. At the same time, the terminal emotional intensity of the following response period is obtained.

[0133] The maximum value of the overall emotional intensity is extracted from the preceding and subsequent response periods, namely the preceding intensity peak and the subsequent intensity peak; the terminal emotional intensity refers to the latest overall emotional intensity in the subsequent response period.

[0134] If the preceding intensity peak is greater than the overall emotional intensity of the sampling period before the intervention interaction, and the subsequent intensity peak and the final emotional intensity are both less than the preceding intensity peak, then the current real intervention response state is determined to be an intervention transition state.

[0135] If the peak intensity of the preceding period is greater than the overall emotional intensity of the previous sampling period, it indicates that the elderly user experienced an emotional rise in the preceding period. Meanwhile, if both the peak intensity of the subsequent period and the emotional intensity at the end are less than the peak intensity of the preceding period, it indicates that the elderly user experienced an emotional rise in the preceding period, but then experienced an emotional decline in the subsequent period. This is a normal emotional cathartic transition response to the intervention, and therefore the corresponding real intervention response state is determined to be the intervention transition state.

[0136] If both the preceding and subsequent intensity peaks are greater than the overall emotional intensity of the sampling period before the intervention interaction, and the final emotional intensity is not less than the subsequent intensity peak, then the current actual intervention response state is determined to be an intervention failure state.

[0137] If both the preceding and subsequent intensity peaks are greater than the overall emotional intensity of the previous sampling period, it indicates that the intervention caused the elderly user's emotions to rise and not fall back. Furthermore, if the final emotional intensity is not less than the subsequent intensity peak, it indicates that the elderly user still maintains a high emotional intensity at the final time. This suggests that the intervention was ineffective or even had a negative effect on the elderly user. Therefore, the actual intervention response state is defined as the intervention failure state.

[0138] If neither the preceding nor the following intensity peak is greater than the overall emotional intensity of the sampling period prior to the intervention interaction, then the current actual intervention response state is determined to be in a stable intervention state.

[0139] The fact that the peak intensity of the preceding period was not greater than the overall emotional intensity of the previous sampling period indicates that the elderly user did not experience an increase in emotion after receiving the intervention stimulus. Furthermore, the peak intensity of the subsequent period was also not greater than the overall emotional intensity of the previous sampling period, indicating that the elderly user maintained their emotions within a controllable range after receiving the intervention stimulus. Therefore, this actual intervention response state was determined as the intervention stable state.

[0140] The methods for updating cognitive behavioral intervention phases, as well as voice intervention data and digital human interaction control parameters, include: If the actual intervention response state is in the intervention transition state, then the current cognitive behavioral intervention stage remains unchanged, and the corresponding execution cycle is extended.

[0141] If the actual intervention response state is in the intervention transition state, it means that although the elderly user experienced an emotional rise, it eventually showed a downward trend, thus belonging to a normal transition state. In this embodiment, we choose to continue to the corresponding cognitive behavioral intervention stage and extend the execution cycle of this stage to give the elderly user more time to complete emotional catharsis and self-regulation, avoiding forced switching of the stage. Since the elderly user's emotional regulation ability is weak, the duration of emotional catharsis is often longer, and some elderly users may cry or repeatedly talk during the emotional catharsis process. If the interaction management stage is forcibly switched at this time, it is easy to interrupt the elderly user's emotional release process, leading to stronger negative emotional reactions.

[0142] If the actual intervention response status is an intervention failure status, the current cognitive behavioral intervention stage will be transformed into an emotional soothing stage, and the corresponding voice intervention data and digital human interaction control parameters will be regenerated.

[0143] If the actual intervention response status is an intervention failure status, it means that the interaction strategy used at this time cannot have an effective effect on the elderly user. Therefore, an emotional assessment is performed again based on the current status of the elderly user, and corresponding new voice data and new interaction parameters are generated. The system is then forced to switch back to the emotional soothing stage to avoid further stimulating the elderly user.

[0144] If the actual intervention response status is in a stable state, the current cognitive behavioral intervention stage will be advanced to the cognitive guidance stage, and the voice intervention data and digital human interaction control parameters will be updated simultaneously.

[0145] If the actual intervention response state is in a stable state, it means that the intervention interaction in the current cognitive behavioral intervention stage has successfully stabilized the elderly user's emotions and the elderly user has a high degree of acceptance of the interaction. Therefore, if it is still in the emotional soothing stage, it can be promoted to the cognitive guidance stage. If it is already in the cognitive guidance stage, it will remain unchanged. At the same time, based on the elderly user's real-time emotional state and physical intervention range, new voice intervention data and digital human interaction control parameters are regenerated to update the original data, ensuring a reasonable stage progression.

[0146] This embodiment collects and processes initial multimodal data during the interaction between elderly users and the digital human to obtain data related to the elderly user's emotions and the digital human's performance, ultimately realizing an interactive digital human management system for the elderly based on multimodal semantic understanding. Compared with existing experience, by combining voice, facial expression, and posture information, it avoids the distortion of emotion recognition in complex scenarios caused by single-modal data; by judging the trend of emotion changes in continuous sampling periods, it eliminates signal jitter caused by instantaneous emotional fluctuations; by analyzing the degree of user resistance and determining the physical intervention range, it avoids further intervention stimulation to the elderly user caused by the digital human approaching too closely or repeatedly moving back and forth; by correcting the execution time response, it corrects the time attribution misalignment between intervention actions and real emotional feedback; by identifying abnormal emotional fluctuations in the intervention effect, it determines the real intervention response state and avoids erroneous switching of intervention strategies; it improves the control accuracy of the digital human's execution intervention and ensures the stability and safety of human-computer interaction.

[0147] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0148] All formulas in this manual are dimensionless and calculated numerically. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0149] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. An interactive digital human management system for the elderly based on multimodal semantic understanding, characterized in that, include: The data acquisition module collects multimodal data in real time during the interaction between the user and the digital human, including user emotion data, user posture data, and digital human action execution data. The emotion recognition module performs emotion recognition based on user emotion data, determines the emotion evolution trend based on the emotion recognition results, and determines the intervention trigger level. The control parameter generation module, when the intervention trigger level meets the preset trigger conditions, matches the target cognitive intervention mode based on the emotion evolution trend and determines the current cognitive behavioral intervention stage; combined with the user's real-time multimodal data and digital human action execution data in the current cognitive behavioral intervention stage, it determines the physical intervention range of the digital human on the user; Based on this physical intervention interval, voice intervention data and digital human interaction control parameters corresponding to the current cognitive behavioral intervention stage are generated; The feedback recognition module controls the digital human to perform intervention interactions based on digital human interaction control parameters and voice intervention data; records the timestamp of intervention command issuance, extracts the interaction effective timestamp from the digital human action execution data, performs time response correction on the user's emotional data before and after intervention, and outputs the effective intervention response range and the target emotional feedback data within the effective intervention response range. The parameter update module determines the changes in the user's emotional state after the intervention based on the target emotional feedback data and performs an intervention effect evaluation. In evaluating the effectiveness of the intervention, abnormal emotional fluctuations triggered by the current cognitive-behavioral intervention stage are identified to determine the true intervention response status; the cognitive-behavioral intervention stage, as well as the voice intervention data and digital human interaction control parameters, are updated based on this true intervention response status; the various modules are connected via wired and / or wireless means.

2. The elderly interactive digital human management system based on multimodal semantic understanding according to claim 1, characterized in that, The methods for performing emotion recognition based on user emotion data include: Extract user voice data and user facial image data from user emotion data; Voice emotion features are constructed by measuring the fundamental frequency of speech, speech rate and speech pause duration of user speech data; facial expression features are constructed by measuring the range of eyebrow changes, eyelid opening and closing and mouth corner deflection of user facial image data. The difference between each item in the speech emotion feature and the preset speech baseline value is calculated, and the sum is obtained by weighting the values ​​based on the preset speech element ratios. The difference between each item in the facial expression feature and the preset facial expression baseline value is calculated, and the sum is obtained by weighting the values ​​based on the preset facial expression element ratios. Identify the visible proportion of the face, the body orientation deflection angle, and the limb movement speed in user posture data within the same sampling period; if the visible proportion of the face is less than a preset proportion threshold or the body orientation deflection angle is greater than a preset deflection threshold, set the facial expression weight to be less than the voice weight; match the limb movement speed with a preset speed range to obtain the posture emotion compensation amount. The intensity of facial expression emotion and the intensity of voice emotion are weighted and summed using facial expression weight and voice weight, and then superimposed with the postural emotion compensation amount to obtain the comprehensive emotion intensity. The system matches voice emotion features and facial expression features with a preset emotion parameter table, outputs emotion category labels, and combines the comprehensive emotion intensity to output the emotion recognition result.

3. The elderly interactive digital human management system based on multimodal semantic understanding according to claim 2, characterized in that, The methods for determining the trend of emotion evolution include: The combined emotion intensity and emotion category labels from consecutive sampling periods are used to form an emotion sequence; Calculate the intensity difference between adjacent comprehensive emotion intensities in the emotion sequence. If there is an intensity difference greater than a preset fluctuation threshold, the corresponding comprehensive emotion intensity is judged as an outlier. Replace the outlier with the mean of the comprehensive emotion intensities before and after it, and output the adjusted emotion sequence. Identify the first and last comprehensive emotional intensity of the adjusted emotional sequence and calculate the difference to obtain the first-to-last intensity difference; compare the first-to-last intensity difference with the trend fluctuation threshold range and output the trend determination result; By matching the trend judgment results and emotion category labels with a preset intervention trigger mapping table, the corresponding intervention trigger level is determined.

4. The elderly interactive digital human management system based on multimodal semantic understanding according to claim 3, characterized in that, The method for matching the target cognitive intervention pattern and determining the current cognitive behavioral intervention stage based on the emotional evolution trend when the intervention trigger level meets the preset trigger conditions includes: When the intervention trigger level is not lower than the preset trigger threshold, the difference between the first and last strengths of the corresponding trend judgment result is extracted; if several pre-trigger levels are higher than the high-risk level and the difference between the first and last strengths is greater than the upper limit of the trend fluctuation threshold range, the target cognitive intervention mode is set to emergency intervention mode; if several pre-trigger levels are not higher than the high-risk level and the difference between the first and last strengths is in the stable range within the trend fluctuation threshold range, the target cognitive intervention mode is set to stable intervention mode. Obtain the set of preset intervention stages corresponding to the target cognitive intervention model, including the emotional soothing stage and the cognitive guidance stage; compare the overall emotional intensity of the current sampling period with the preset stage switching threshold. If the overall emotional intensity is greater than the preset stage switching threshold, or the intensity difference between the beginning and end is greater than zero, then confirm that the current cognitive behavioral intervention stage is the emotional soothing stage. If the overall emotional intensity is not greater than the preset stage switching threshold, and the intensity difference between the beginning and end is not greater than zero, then the current cognitive behavioral intervention stage is confirmed as the cognitive guidance stage.

5. The elderly interactive digital human management system based on multimodal semantic understanding according to claim 4, characterized in that, The methods for determining the physical intervention range of the digital human on the user include: Identify real-time body orientation deflection angle, real-time limb movement speed, and real-time comprehensive emotion intensity in real-time multimodal data; obtain user facial orientation vector and digital human gaze vector and calculate cosine similarity; The difference between the real-time body orientation deflection angle and the preset deflection threshold is calculated, and the degree of posture avoidance is calculated by combining cosine similarity; the degree of limb movement is calculated based on real-time limb movement speed and real-time comprehensive emotional intensity. The degree of user resistance is obtained by weighted summation of the degree of posture avoidance and the degree of limb movement; the degree of user resistance is converted into a change in safe distance, and summed with a preset baseline safe distance to output the dynamic safe boundary distance; Obtain the real-time spatial distance from the digital human's motion execution data, and calculate the distance difference between the real-time spatial distance and the dynamic safety boundary distance; Set positive and negative boundary tolerances. If the distance difference is less than the negative boundary tolerance, set the far distance range as the physical intervention range and control the digital human to execute the priority retreat command. If the distance difference is greater than the positive boundary tolerance, and the current cognitive behavioral intervention stage is the emotional soothing stage, then the close distance range is set as the physical intervention range, and the digital human is controlled to execute the slow approach command. If the distance difference is between the positive boundary tolerance and the negative boundary tolerance, then the physical intervention range of the previous sampling period is maintained or the medium distance range is set as the physical intervention range.

6. The elderly interactive digital human management system based on multimodal semantic understanding according to claim 5, characterized in that, The method for generating voice intervention data and digital human interaction control parameters corresponding to the current cognitive behavioral intervention stage based on the physical intervention interval includes: Based on the current cognitive behavioral intervention stage, the corresponding target speech content template is matched, and the basic interactive speech rate and basic volume are obtained. The basic interactive speech rate and basic volume are proportionally adjusted based on the physical intervention interval and distance difference, and the speech intervention data is output. Based on the physical intervention range, the target speed range and movement amplitude range of the digital human are matched, and the current cognitive and behavioral intervention stage is combined to determine whether to perform touch interaction and generate touch interaction labels. The target speed range, motion range, and touch interaction labels are integrated to form the digital human interaction control parameters.

7. The elderly interactive digital human management system based on multimodal semantic understanding according to claim 6, characterized in that, The methods for time-response correction of user emotional data before and after intervention include: When an intervention command containing voice intervention data and digital human interaction control parameters is issued, the timestamp of the intervention command issuance is recorded; Real-time monitoring of acceleration or joint angle change rate in digital human motion execution data; if acceleration or joint angle change rate is greater than the preset corresponding motion response threshold, record the interaction activation timestamp; calculate the sum of the interaction activation timestamp and the user's cognitive buffer time to obtain the theoretical feedback timestamp. User emotional data between the intervention instruction issuance timestamp and the theoretical feedback timestamp is treated as transitional emotional data and discarded; the theoretical feedback timestamp is used as the starting point, and the effective intervention response interval is divided according to the preset evaluation duration, and user emotional data within the effective intervention response interval is used as the target emotional feedback data.

8. The elderly interactive digital human management system based on multimodal semantic understanding according to claim 7, characterized in that, The calculation method for the user cognitive buffer duration includes: Obtain the historical comprehensive emotional intensity of the previous sampling period before the intervention interaction, and obtain the stage delay ratio corresponding to the current cognitive behavioral intervention stage; calculate the ratio of the historical comprehensive emotional intensity to the preset stable intensity to obtain the emotional buffer ratio; calculate the product of the preset basic cognitive response time, the stage delay ratio, and the emotional buffer ratio to obtain the user's cognitive buffer time.

9. The elderly interactive digital human management system based on multimodal semantic understanding according to claim 8, characterized in that, The process involves determining changes in the user's emotional state after intervention based on target emotional feedback data, and then evaluating the effectiveness of the intervention. Methods for identifying aberrant emotional fluctuations triggered by the current cognitive-behavioral intervention phase in the evaluation of intervention effectiveness, in order to determine the true state of intervention response, include: The time period corresponding to the target emotional feedback data is divided into the preceding response period and the following response period; the comprehensive emotional intensity peak of the preceding response period and the following response period are extracted respectively as the preceding intensity peak and the following intensity peak, and the terminal emotional intensity of the following response period is obtained at the same time. If the preceding intensity peak is greater than the comprehensive emotional intensity of the sampling period before the intervention interaction, and the subsequent intensity peak and the terminal emotional intensity are both less than the preceding intensity peak, then the current real intervention response state is determined to be an intervention transition state. If both the preceding and subsequent intensity peaks are greater than the overall emotional intensity of the sampling period before the intervention interaction, and the final emotional intensity is not less than the subsequent intensity peak, then the current real intervention response state is determined to be an intervention failure state. If neither the preceding nor the following intensity peak is greater than the overall emotional intensity of the sampling period prior to the intervention interaction, then the current actual intervention response state is determined to be in a stable intervention state.

10. The elderly interactive digital human management system based on multimodal semantic understanding according to claim 9, characterized in that, The methods for updating the cognitive behavioral intervention phase, as well as the voice intervention data and digital human interaction control parameters, include: If the actual intervention response state is in the intervention transition state, then the current cognitive behavioral intervention stage remains unchanged, and the corresponding execution cycle is extended; If the actual intervention response status is an intervention failure status, the current cognitive behavioral intervention stage will be transformed into the emotional soothing stage, and the corresponding voice intervention data and digital human interaction control parameters will be regenerated. If the actual intervention response status is in a stable state, the current cognitive behavioral intervention stage will be advanced to the cognitive guidance stage, and the voice intervention data and digital human interaction control parameters will be updated simultaneously.

Images