Self-adaptive interaction method and system of health-care robot for home-based care for the elderly

Abstract

The application is suitable for the field of human-computer interaction, and provides a self-adaptive interaction method system of a health-care robot for home-based care, which comprises the following steps: constructing an archive of multi-modal interaction channels for the old users, recording the historical success rate, average response delay and user preference weight of each channel; collecting the input signals of each channel in the interaction process in real time, and dynamically counting the instant success rate of each channel in the interaction attempt by using a sliding window; when the instant success rate of the main channel is continuously lower than a first threshold value, switching to the channel with the highest instant success rate; when the instant success rate of all channels is lower than a second threshold value, executing a multi-channel fusion strategy; when the channel is switched, outputting voice and screen prompts for informing the user of the switching result, and storing the switching record in the archive. The application significantly improves the success rate, continuity and naturalness of the interaction between the old people and the health-care robot, and has good individualized adaptive ability.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, specifically to an adaptive interaction method and system for elderly care robots designed for home-based elderly care. Background Technology

[0002] With the increasing aging of the population, home-based elderly care has become the mainstream model, and elderly care robots, as an important tool to assist the elderly in their lives, have received widespread attention. However, the elderly exhibit significant individual differences and age-related degenerative characteristics in perception, movement, and cognition. For example, hearing loss leads to difficulties in speech recognition, hand tremors affect gesture interaction, vision decline makes it difficult to see screens clearly, and arthritis leads to inaccurate touch operations. Most existing elderly care robots adopt a single interaction method (such as supporting only voice or only touch screen), which cannot adapt to the different changes in the elderly, resulting in low interaction success rates and poor user experience.

[0003] Current adaptive interaction methods mostly focus on optimizing a single modality, such as speech enhancement algorithms to improve speech recognition rates for the elderly, gesture magnification algorithms to compensate for hand tremors, and font magnification to improve visual readability. While these methods improve the performance of a single channel to some extent, they lack a comprehensive evaluation and dynamic switching mechanism between different channels. When a channel fails due to environmental interference or fluctuations in user status, it cannot actively switch to other more reliable channels, forcing the elderly to repeatedly try the failed channels.

[0004] Existing technologies have attempted to allow users to manually select interaction methods (such as setting voice or gesture modes), but elderly people often find it difficult to determine which channel is most suitable for them, and manually switching between modes is an additional burden for elderly people with limited mobility or cognitive impairment. Therefore, an adaptive interaction method and system for home-based elderly care robots is proposed to address the aforementioned problems. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of this invention is to provide an adaptive interaction method and system for elderly care robots for home-based elderly care, so as to solve the problems existing in the above-mentioned background technology.

[0006] This invention is implemented as follows: an adaptive interaction method for elderly care robots for home-based elderly care, the method comprising the following steps:

[0007] A profile of multimodal interaction channels for elderly users is constructed, which records the historical success rate, average response latency, and user preference weight of each channel. The multimodal interaction channels include voice channels, gesture channels, touch channels, and visual channels.

[0008] The system collects input signals from each channel in real time during the interaction process and uses a sliding window to dynamically calculate the instantaneous success rate of each channel during the interaction attempts.

[0009] When the real-time success rate of the primary channel is continuously lower than the first threshold, switch to the channel with the highest real-time success rate. When the real-time success rate of all channels is lower than the second threshold, execute the multi-channel fusion strategy.

[0010] When switching channels, voice and screen prompts are output to inform the user of the switching result, and the parameters of the interactive interface are adjusted. At the same time, the switching record is stored in the file.

[0011] As a further aspect of the present invention: the step of dynamically calculating the instantaneous success rate of each channel during interactive attempts using a sliding window specifically includes:

[0012] A sliding window with a set number of iterations is used to record the success or failure status of each channel in each interaction, and the continuity flag used to determine the interruption of the interaction is continuously monitored. The continuity flag includes voice detection status, user head orientation data, gaze dwell time, and environmental sound events.

[0013] When the interaction is uninterrupted, the sliding window is maintained, and the instantaneous success rate of each channel is calculated based on the ratio of the number of successful interactions to the sliding window.

[0014] When the interaction is interrupted, the sliding window data is frozen, and the window data, dialogue history and unfinished task data are saved as a temporary state packet. At the same time, the interruption timer is started, and environmental noise and user state parameters are collected.

[0015] Once the interaction resumes, the system determines whether to restore the original sliding window or clear it by comparing the duration of the interrupt timer with a preset threshold.

[0016] When the time is less than the preset threshold, the original sliding window is restored, and the estimated success rate of each channel after restoration is calculated using environmental noise and user status parameters. Based on the estimated success rate, the primary channel after restoration is selected, and the real-time success rate is continuously calculated and output.

[0017] As a further aspect of the present invention: the step of calculating the estimated success rate of each channel after recovery using environmental noise and user status parameters specifically includes:

[0018] The environmental noise is decomposed into frequency bands, and the degree of interference of each frequency band noise on the voice channel is evaluated to obtain the noise confidence correction factor. The noise confidence correction factor for the other channels is the default value.

[0019] The user status parameters were decomposed into visual stability, head posture flexibility, and hand tremor amplitude, and the capability confidence correction factor of each channel was evaluated.

[0020] For each channel, the instantaneous success rate before the interruption is multiplied by the corresponding noise confidence correction factor and capability confidence correction factor to obtain the estimated success rate.

[0021] The historical success rates of each channel in the archives are incorporated and used as long-term weights to adjust the estimated success rates.

[0022] As a further aspect of the present invention: the step of executing the multi-channel fusion strategy specifically includes:

[0023] If the second threshold is lower than the first threshold, all channels will fail when the instant success rate of all channels is lower than the second threshold.

[0024] Simultaneously, it receives audio signals from the voice channel, motion trajectory signals from the gesture channel, touch position signals from the touch channel, and image signals from the vision channel, and performs independent intent recognition on the signals from each channel to obtain the intent recognition results and corresponding confidence scores for each channel.

[0025] Based on the historical success rate of each channel in the archives, the confidence score is weighted and adjusted, with channels that have a higher historical success rate being assigned a higher weighting coefficient.

[0026] The weighted and corrected intent recognition results from each channel are merged, and the intent recognition result with the highest confidence score is output.

[0027] As a further aspect of the present invention: the step of independently identifying the intent of the signals in each channel specifically includes:

[0028] Based on the voice channel, the ASR recognition result is input into the intent classifier, which outputs the initial intent label and the initial confidence score. When incomplete sentence features are detected, the initial intent label is completed according to the interaction history, and the initial confidence score is processed using a preset discount coefficient. When repetitive sentence features are detected, the initial intent label is kept unchanged, and the current initial confidence score is weighted and averaged with the previous confidence score.

[0029] Based on the gesture channel, the gesture recognition results of multiple consecutive frames are temporally smoothed, and the smoothed intent label and its confidence score are output.

[0030] Based on the touch channel, the screen position of the touch point is mapped to the intent label. The probability of intentional operation is calculated based on the trajectory continuity and jitter amplitude of the touch point, and this probability is used as the confidence score.

[0031] Based on the visual channel, the gaze detection model outputs the category of the gaze target object and the gaze duration. When the gaze duration exceeds the preset duration and there is no voice or gesture input, the intention label is obtained by mapping the gaze object category. The confidence score is determined based on the ratio of the gaze duration to the preset maximum duration.

[0032] As a further aspect of the present invention, the method further includes:

[0033] A fixed-length temporal intent queue is established for each channel, storing the intent tags and confidence scores identified by each channel in chronological order;

[0034] After each new intent tag is identified, the intent stability index of the temporal intent queue of that channel is calculated. The stability index is determined based on the consistency of adjacent intent tags within the queue.

[0035] When the new intent label is inconsistent with the previous intent label in the queue and the stability index is lower than the preset stability threshold, the current recognition result is determined to be a temporal jump.

[0036] Replace the intent label with the historical intent label with the highest confidence score in the queue, and adjust the confidence score using a preset jump discount factor.

[0037] Another object of the present invention is to provide an adaptive interactive system for elderly care robots for home-based elderly care, the system comprising:

[0038] The profile recording module is used to build profiles for multimodal interaction channels for elderly users, which records the historical success rate, average response latency, and user preference weights of each channel. The multimodal interaction channels include voice channels, gesture channels, touch channels, and visual channels.

[0039] The data acquisition and statistics module is used to collect the input signals of each channel in real time during the interaction process, and to dynamically calculate the instantaneous success rate of each channel during the interaction attempt using a sliding window.

[0040] The channel switching module is used to switch to the channel with the highest real-time success rate when the real-time success rate of the primary channel is continuously lower than the first threshold. When the real-time success rate of all channels is lower than the second threshold, a multi-channel fusion strategy is executed.

[0041] The switching feedback module is used to output voice and screen prompts to inform the user of the switching result when switching channels, adjust the interactive interface parameters, and store the switching record in the file.

[0042] As a further aspect of the present invention: the data collection and statistics module includes:

[0043] The window monitoring unit is used to record the success or failure status of each channel in each interaction using a set number of sliding windows, and continuously monitors the continuity flag used to determine the interruption of the interaction. The continuity flag includes voice detection status, user head orientation data, gaze dwell time and environmental sound events.

[0044] The success rate statistics unit is used to maintain the sliding window when the interaction is uninterrupted, and to calculate the real-time success rate of each channel based on the ratio of the number of successful interactions to the sliding window.

[0045] The interruption freeze unit is used to freeze the sliding window data when the interaction is interrupted, and save the window data, dialogue history and unfinished task data as a temporary state packet. At the same time, it starts the interruption timer and collects environmental noise and user state parameters.

[0046] The recovery determination unit is used to determine whether to restore the original sliding window or clear the sliding window by comparing the duration of the interrupt timer with a preset threshold after the interaction is restored.

[0047] The prediction selection unit is used to restore the original sliding window when the time is less than a preset threshold. It calculates the predicted success rate of each channel after restoration using environmental noise and user status parameters, selects the main channel after restoration based on the predicted success rate, and continuously calculates and outputs the real-time success rate.

[0048] As a further aspect of the present invention: the channel switching module includes:

[0049] The threshold triggering unit is used to preset a second threshold that is lower than the first threshold. When the instantaneous success rate of all channels is lower than the second threshold, all channels will fail.

[0050] An independent recognition unit is used to simultaneously receive audio signals from the voice channel, motion trajectory signals from the gesture channel, touch position signals from the touch channel, and image signals from the visual channel, and to perform independent intent recognition on the signals from each channel to obtain the intent recognition results and corresponding confidence scores for each channel.

[0051] The weighted correction unit is used to weight and correct the confidence score based on the historical success rate of each channel in the archive, where the channel with the higher historical success rate is assigned a higher weighting coefficient.

[0052] The fusion output unit is used to fuse the weighted and corrected intent recognition results of each channel and output the intent recognition result with the highest confidence score.

[0053] Compared with the prior art, the beneficial effects of the present invention are:

[0054] This invention, by constructing a profile, can permanently memorize the interaction ability characteristics of each elderly person, overcoming the shortcomings of existing technologies that rely solely on instantaneous or offline assessments. It provides data support for adaptive decision-making, and the real-time collection and statistical analysis of instantaneous success rates can sensitively capture transient changes in channel capabilities caused by fatigue, medication, emotional fluctuations, or environmental interference. Compared to traditional fixed thresholds or manual switching methods, it offers faster response speeds and lower misjudgment rates. Furthermore, it can automatically switch to the backup channel with the highest success rate, achieving dynamic optimization among multiple channels. This allows the elderly to obtain the most reliable interaction method without manual intervention, significantly reducing the frustration and cognitive burden caused by channel failure. In addition, a multi-channel fusion strategy is used as a fallback to identify user intent, ensuring that interaction continues even in extreme situations such as when the elderly person is in a very poor condition or in an extremely noisy environment. In summary, this invention significantly improves the success rate, continuity, and naturalness of interaction between the elderly and the elderly care robot, and possesses excellent personalized adaptive capabilities. Attached Figure Description

[0055] Figure 1 A flowchart illustrating an adaptive interaction method for elderly care robots designed for home-based senior living.

[0056] Figure 2 This is a flowchart illustrating the real-time success rate of each channel during interaction attempts in an adaptive interaction method for elderly care robots designed for home-based senior living, utilizing a sliding window.

[0057] Figure 3 This is a flowchart illustrating the process of calculating the estimated success rate of each channel after recovery using environmental noise and user state parameters in an adaptive interaction method for elderly care robots designed for home-based senior living.

[0058] Figure 4 This is a flowchart illustrating the execution of a multi-channel fusion strategy in an adaptive interaction method for elderly care robots designed for home-based senior living.

[0059] Figure 5 This is a flowchart illustrating the independent intent recognition of signals from each channel in an adaptive interaction method for elderly care robots designed for home-based senior living.

[0060] Figure 6 This is a schematic diagram of the structure of an adaptive interactive system for elderly care robots designed for home-based senior living.

[0061] Figure 7 This is a schematic diagram of the data collection and statistics module in an adaptive interactive system for elderly care robots designed for home-based senior living.

[0062] Figure 8 This is a schematic diagram of the channel switching module in an adaptive interactive system for elderly care robots designed for home-based senior living. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0064] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0065] like Figure 1 As shown in the figure, this invention provides an adaptive interaction method for elderly care robots for home-based elderly care, the method comprising the following steps:

[0066] S100, construct a profile of multimodal interaction channels for elderly users, which records the historical success rate, average response latency and user preference weight of each channel. The multimodal interaction channels include voice channel, gesture channel, touch channel and visual channel.

[0067] The S200 collects the input signals of each channel in real time during the interaction process and uses a sliding window to dynamically calculate the instantaneous success rate of each channel during the interaction attempt.

[0068] S300: When the real-time success rate of the primary channel is continuously lower than the first threshold, switch to the channel with the highest real-time success rate. When the real-time success rate of all channels is lower than the second threshold, execute the multi-channel fusion strategy to ensure the recognition of the user's intent.

[0069] S400: When switching channels, output voice and screen prompts to inform the user of the switching result, adjust the interactive interface parameters, and store the switching record in the file.

[0070] It should be noted that when switching channels, voice and screen prompts are output simultaneously, and the interactive interface parameters (such as font size, button layout, voice volume, etc.) are automatically adjusted, so that elderly users can intuitively understand the current interaction mode and adapt quickly, reducing learning costs and operational errors. As for why the switching records are stored in the aforementioned file, it is mainly for online self-optimization, so that the system can continuously learn the user's long-term ability decline trend (such as hearing loss year by year, and hand tremors gradually worsening) as the number of uses increases, thereby dynamically updating the preference weights and historical success rate data in the file, making the robot increasingly able to understand elderly users.

[0071] In this embodiment of the invention, by constructing a personalized profile that includes historical success rates, response latency, and user preference weights, the interactive ability characteristics of each elderly person can be memorized in the long term. This overcomes the shortcomings of existing technologies that rely solely on instantaneous or offline assessments, providing stable and accurate data support for adaptive decision-making. Real-time acquisition of signals from each channel and dynamic statistical analysis of instantaneous success rates using a sliding window can keenly capture instantaneous changes in channel capabilities caused by fatigue, medication, emotional fluctuations, or environmental interference. Compared to traditional fixed thresholds or manual switching methods, this approach offers faster response times and lower misjudgment rates. Furthermore, a two-tiered decision-making mechanism using a first and second threshold is employed. When the success rate of the primary channel continuously falls below the first threshold, the system automatically switches to the backup channel with the highest current success rate. This achieves dynamic optimization among multiple channels, allowing the elderly to obtain the most reliable interaction method without manual intervention, significantly reducing the frustration and cognitive burden caused by channel failure. When the success rates of all channels fall below the second threshold, a multi-channel fusion strategy is activated to identify user intent as a fallback, ensuring that interaction can continue even in extreme situations such as when the elderly person is in a very poor condition or in an extremely noisy environment. This avoids service interruption and significantly improves the robustness and security of the system.

[0072] like Figure 2 As shown, in a preferred embodiment of the present invention, the step of dynamically calculating the real-time success rate of each channel in the interactive attempt using a sliding window specifically includes:

[0073] S201, a sliding window with a set number of times is used to record the success or failure status of each channel in each interaction, and the continuity flag used to determine the interruption of the interaction is continuously monitored. The continuity flag includes voice detection status, user head orientation data, gaze dwell time and environmental sound events.

[0074] S202, when the interaction is not interrupted, maintain the sliding window and calculate the instantaneous success rate of each channel based on the ratio of the number of successful interactions to the sliding window.

[0075] S203: When the interaction is interrupted, freeze the sliding window data and save the window data, dialogue history and unfinished task data as a temporary state packet. At the same time, start the interrupt timer and collect environmental noise and user state parameters.

[0076] S204, When the interaction is restored, the original sliding window is restored or the sliding window is cleared by comparing the duration of the interrupt timer with a preset threshold.

[0077] S205: When the time is less than the preset threshold, restore the original sliding window, calculate the estimated success rate of each channel after recovery using environmental noise and user status parameters, select the main channel after recovery based on the estimated success rate, and continuously calculate and output the real-time success rate.

[0078] In this embodiment of the invention, the sliding window is a commonly used data structure in signal processing and statistics. Here, a window with a fixed number of interactions (e.g., the most recent 20 interactions) is used to record the success or failure result of each attempt for each interaction channel. Specifically, for the continuity indicator, the system does not solely rely on voice activity detection to determine whether an interaction is in progress. Instead, it integrates the user's head orientation (whether facing the robot), gaze persistence (whether looking at the interactive interface), and environmental sound events (such as doorbells, telephone rings, etc.). For example, if the user suddenly turns their head to look out the window, or if there is continuous high-decibel noise in the environment, even without a voice timeout, it will be considered a potential interaction interruption. This multi-dimensional indicator monitoring provides a basis for subsequent interaction interruption determination, which is highly consistent with the characteristic that elderly people are prone to involuntary interruptions due to external interference or distraction during interaction. The calculation of the instant success rate is relatively conventional and will not be described in detail. Under normal interaction conditions, the instant success rate dynamically reflects the user's recent performance on that channel, rather than a lifetime average. This makes the system highly sensitive to fluctuations in ability caused by short-term fatigue, emotions, or medication effects in the elderly. Because the sliding window size is fixed and the computational load is stable, it is suitable for real-time operation in robot embedded systems.

[0079] When interaction is interrupted, the system doesn't simply discard data from the interruption period or clear the sliding window. Instead, it freezes the current sliding window data (stopping window updates) and preserves the success or failure status of each channel at the moment before the interruption. Simultaneously, it packages and saves the current dialogue history (e.g., a sentence the user just finished speaking) and unfinished interactive tasks (e.g., unconfirmed reminders), while an interruption timer records the duration of the interruption. Furthermore, it actively collects environmental noise (e.g., background TV or air conditioner sounds) and user status parameters (e.g., fatigue level and heart rate detected by wearable devices or vision) during the interruption. Freezing the window prevents invalid or erroneous interaction attempts from contaminating success rate statistics, ensuring the accuracy of the immediate success rate. Saving dialogue history and unfinished tasks allows for continuation from the breakpoint after the interruption, eliminating the need for the elderly to repeat themselves and reducing the memory burden on cognitively impaired seniors. Moreover, actively collecting environmental noise and user status parameters during the interruption provides valuable preliminary information for predicting the success rate of each channel after recovery. For example, if a significant increase in environmental noise is detected during the interruption, the success rate of the voice channel may be lower after recovery, allowing for advance preparation to switch to the gesture or touch channel. By comparing the interruption duration with a preset threshold (which can be determined based on the statistical distribution of the elderly user's historical interruption behavior), a quantitative assessment of the interruption severity is achieved. Restoring the original sliding window after a short interruption (such as answering a phone call or coughing) maintains data continuity and avoids the loss of long-term statistical information due to brief interference. Clearing the sliding window and re-accumulating data after a long interruption (such as the elderly person going to the bathroom or being called away) promptly discards outdated data, allowing the immediate success rate to reflect the current true state more quickly. However, for short interruptions, further explanation is needed. First, the original sliding window is restored, and then the estimated success rate of each channel after restoration is calculated using the environmental noise and user status parameters collected during the interruption. Based on this, the primary channel is selected after restoration.

[0080] like Figure 3 As shown in the preferred embodiment of the present invention, the step of calculating the estimated success rate of each channel after recovery using environmental noise and user status parameters specifically includes:

[0081] S2051, decompose the environmental noise by frequency band and evaluate the degree of interference of each frequency band noise to the voice channel to obtain the noise confidence correction factor, while the noise confidence correction factor of the other channels is the default value;

[0082] S2052 decomposes user state parameters into visual stability, head posture flexibility, and hand tremor amplitude, and evaluates the capability confidence correction factor for each channel.

[0083] S2053, For each channel, multiply its instantaneous success rate before interruption by the corresponding noise confidence correction factor and capability confidence correction factor to obtain the estimated success rate;

[0084] S2054 introduces the historical success rate of each channel in the archive and uses it as a long-term weight to correct the estimated success rate.

[0085] In this embodiment of the invention, environmental noise is first decomposed into different frequency bands using frequency domain analysis methods in signal processing (such as Fast Fourier Transform). The unique aspect is that instead of using total noise energy as a single indicator, the actual interference level of each frequency band on speech recognition is evaluated separately, and a frequency band-interference weight mapping table is pre-established. For example, low-frequency noise (air conditioner hum) mainly masks the fundamental frequency information of speech, while high-frequency noise (television voice) interferes with consonant recognition; the two have different influence weights. By multiplying the energy of each frequency band by its corresponding weight, summing the results, and normalizing, a noise confidence correction factor for the speech channel is obtained (with a value range of 0-1, the larger the noise, the smaller the factor). For gesture, touch, and visual channels, since they are not affected by environmental noise, the noise confidence correction factor is directly set to 1. The above processing accurately distinguishes the differentiated impact of various noise sources on speech recognition, avoiding the bias of a single total energy assessment. This makes the noise correction in the estimated success rate more consistent with the real physical environment. Then, the user state parameters are decoupled into three independent dimensions strongly correlated with specific interaction channels: gaze stability directly affects the interaction capability of the visual channel, head posture flexibility affects the speech channel, and hand tremor amplitude affects the gesture and touch channels. This achieves decoupled assessment of multiple concurrent physiological degenerative characteristics in the elderly, independently calculating correction factors for each channel that reflect its true ability state, avoiding misjudgments caused by treating the overall user state as a single indicator. The calculation of the estimated success rate uses a conventional weighted processing method. The immediate success rate before interruption reflects the user's recent historical performance on that channel, while the noise and user state parameters collected during the interruption reflect real-time changes in the environment and physiology. Combining these two through multiplication yields a comprehensive and predictive estimated success rate. Furthermore, the long-term weight is obtained by recording the success or failure status of each user interaction attempt and accumulating and updating the historical success rate of each channel using an exponentially weighted moving average method, which is stored in the user's profile. The historical success rate in the archives reflects the inherent reliability of the channel and is not easily affected by short-term fluctuations. A weighted average can be used for correction.

[0086] Final estimated success rate = Estimated success rate × (1-α) + Historical success rate × α

[0087] Here, α is a preset long-term weight. This correction method effectively prevents the estimated success rate from deviating significantly from the user's actual ability due to single or short-term extreme situations, while also enabling the estimated success rate to smoothly track the long-term decline trend of the elderly's abilities.

[0088] like Figure 4 As shown, in a preferred embodiment of the present invention, the step of executing the multi-channel fusion strategy specifically includes:

[0089] S301, the preset second threshold is lower than the first threshold. When the instant success rate of all channels is lower than the second threshold, all channels fail.

[0090] S302 simultaneously receives audio signals from the voice channel, motion trajectory signals from the gesture channel, touch position signals from the touch channel, and image signals from the vision channel, and performs independent intent recognition on the signals from each channel to obtain the intent recognition results and corresponding confidence scores for each channel.

[0091] S303, based on the historical success rate of each channel in the archive, the confidence score is weighted and adjusted, with channels with higher historical success rates being assigned higher weighting coefficients;

[0092] S304 merges the weighted and corrected intent recognition results of each channel and outputs the intent recognition result with the highest confidence score.

[0093] In this embodiment of the invention, when all channels are determined to be faulty, switching channels is no longer attempted. Instead, a fallback multi-channel fusion decision is initiated. Simultaneously, audio signals from the voice channel, motion trajectory signals from the gesture channel, touch position signals from the touch channel, and image signals from the visual channel are received. Intent recognition is performed independently on each channel's signal, yielding the intent recognition result and its corresponding confidence score for each channel. A unique aspect of this embodiment lies in the subsequent weighted correction and fusion. Based on the historical success rate of each channel in the archive, the confidence scores are weighted and corrected, with channels having higher historical success rates assigned higher weighting coefficients. For example, a user who has consistently used the voice channel and has a high historical success rate will have a higher voice weight in the fusion process. Finally, the weighted and corrected intent recognition results from each channel are fused, and the intent with the highest confidence score is output as the final user intent. Even in the extreme case where all channels are unreliable, it can still select the most reliable one from multiple potentially imperfect recognition results, ensuring that the interaction is not completely interrupted. Because it incorporates long-term historical success rates for weighting, the fusion result better aligns with users' long-term usage habits and inherent channel capabilities, exhibiting higher robustness and user satisfaction compared to simply relying on current instantaneous confidence. It should be noted that the multi-channel fusion decision is triggered only when all channels fail and will not interfere with efficient channel switching under normal circumstances.

[0094] like Figure 5 As shown in the preferred embodiment of the present invention, the step of independently identifying the intent of the signal in each channel specifically includes:

[0095] S3021, based on the voice channel, inputs the ASR recognition result into the intent classifier, outputs the initial intent label and initial confidence score. When incomplete sentence features are detected, the initial intent label is completed according to the interaction history, and the initial confidence score is processed using a preset discount coefficient. When repetitive sentence features are detected, the initial intent label is kept unchanged, and the current initial confidence score is weighted and averaged with the previous confidence score.

[0096] S3022, based on the gesture channel, performs temporal smoothing on the gesture recognition results of multiple consecutive frames, and outputs the smoothed intent label and its confidence score;

[0097] S3023, based on the touch channel, maps the screen position of the touch point to an intent label, calculates the probability of intentional operation based on the trajectory continuity and jitter amplitude of the touch point, and uses the probability as a confidence score;

[0098] S3024, Based on the visual channel, the gaze detection model is used to output the gaze target object category and gaze duration. When the gaze duration exceeds the preset duration and there is no voice or gesture input, the intention label is obtained according to the gaze object category. The confidence score is determined according to the ratio of the gaze duration to the preset maximum duration.

[0099] S3025 establishes a fixed-length temporal intent queue for each channel, storing the intent tags and confidence scores identified by each channel in chronological order;

[0100] S3026, After each new intent tag is identified, calculate the intent stability index of the temporal intent queue of the channel. The stability index is determined based on the consistency of adjacent intent tags in the queue.

[0101] S3027, when the new intent label is inconsistent with the previous intent label in the queue and the stability index is lower than the preset stability threshold, the current recognition result is determined to be a temporal jump.

[0102] S3028, replace the intent label with the historical intent label with the highest confidence score in the queue, and correct the confidence score using a preset jump discount coefficient.

[0103] In this embodiment of the invention, regarding the voice channel, when incomplete sentence features are detected, the initial intent label is completed based on the interaction history, and the confidence score is reduced using a preset discount coefficient to reflect the uncertainty of the completion result; when repetitive sentence features are detected, it is determined that the user is repeatedly expressing the same intent due to cognitive impairment. At this time, the intent label of the previous output remains unchanged, and the current initial confidence score and the previous confidence score are weighted and averaged. This stabilizes the output and reflects the confidence improvement brought about by repeated confirmation through weighted averaging. In the gesture channel, the gesture recognition results of multiple consecutive frames are temporally smoothed (e.g., using majority voting or Kalman filtering) to filter out instantaneous misidentifications caused by hand tremors, and the smoothed intent label and its confidence score are output. In the touch channel, the screen position of the touch point is directly mapped to a preset intent label, and the probability of intentional operation is calculated based on the continuity of the touch point's trajectory and the amplitude of tremors (smooth trajectory and small tremors result in a high probability, and vice versa), and this probability is used as the confidence score. In the visual channel, the gaze detection model is used to output the category of the gaze target object and the gaze duration. When the gaze duration exceeds a preset duration (set based on the statistical distribution of the historical gaze behavior of elderly users, or using a fixed empirical value recommended by cognitive experiments) and there is no accompanying voice or gesture input, the intent label is obtained by mapping the gaze object category, and the confidence score is determined based on the ratio of the gaze duration to the preset maximum duration. The independent recognition methods of the above four channels are each targeted, solving unique problems such as incomplete / repetitive speech expression by the elderly, interference from hand tremors, high touch error rate, and unclear gaze intent expression.

[0104] Building upon this, this embodiment also provides a temporal stability correction mechanism to prevent sudden changes in intent labels due to occasional errors within a single channel. A fixed-length temporal intent queue (e.g., storing the most recent 10 identified intent labels and confidence scores) is established for each channel, stored chronologically. After each new intent label is identified, the intent stability index of the temporal intent queue for that channel is calculated. This index is determined based on the consistency of adjacent intent labels within the queue, such as by counting the maximum length of consecutive identical intents in the queue or calculating the proportion of identical adjacent labels. When a new intent label is inconsistent with the previous intent label in the queue, and the stability index is lower than a preset stability threshold (e.g., more than two of the most recent three intents in the queue are different, indicating that the user intent is inherently unstable or has high recognition noise), the current identification result is determined to be a temporal jump, likely an identification error rather than a genuine change in user intent. In this case, the new intent label is replaced with the historical intent label with the highest confidence score in the queue, and a preset jump discount coefficient is used to correct the current confidence score, thereby suppressing sudden changes in intent caused by a single identification error. The jump discount coefficient is a preset positive number less than 1, used to penalize recognition results judged as time-series jumps. Its value can be fixed based on experimental experience or dynamically adjusted based on the user's historical recognition accuracy. For example, if the recent intent recognition labels on the voice channel of an elderly person with cognitive impairment are "drink water," "drink water," "take medicine," and "drink water," the stability index is low (because it changes back and forth). If the new intent recognition label is "eat," which is inconsistent with the previous "drink water" and has low stability, then correction is triggered, replacing it with the historical intent with the highest confidence in the queue (which may be "drink water"), and lowering the confidence. In summary, this mechanism effectively avoids frequent intent jumps caused by user cognitive fluctuations or recognition noise, making the output smoother and more reliable.

[0105] like Figure 6 As shown, this embodiment of the invention also provides an adaptive interactive system for elderly care robots for home-based elderly care, the system comprising:

[0106] The profile recording module 100 is used to construct profiles for multimodal interaction channels for elderly users, which records the historical success rate, average response latency, and user preference weights of each channel. The multimodal interaction channels include voice channels, gesture channels, touch channels, and visual channels.

[0107] The data acquisition and statistics module 200 is used to acquire the input signals of each channel in real time during the interaction process, and to dynamically calculate the instantaneous success rate of each channel during the interaction attempt using a sliding window.

[0108] The channel switching module 300 is used to switch to the channel with the highest current instant success rate when the instant success rate of the primary channel is continuously lower than the first threshold. When the instant success rate of all channels is lower than the second threshold, a multi-channel fusion strategy is executed to ensure the recognition of the user's intent.

[0109] The switching feedback module 400 is used to output voice and screen prompts to inform the user of the switching result when switching channels, adjust the interactive interface parameters, and store the switching record in the file.

[0110] like Figure 7 As shown, in a preferred embodiment of the present invention, the data collection and statistics module 200 includes:

[0111] The window monitoring unit 201 is used to record the success or failure status of each channel of each interaction using a set number of sliding windows, and continuously monitor the continuity flag used to determine the interruption of the interaction. The continuity flag includes voice detection status, user head orientation data, gaze dwell time and environmental sound events.

[0112] The success rate statistics unit 202 is used to maintain the sliding window when the interaction is not interrupted, and to calculate the real-time success rate of each channel based on the ratio of the number of successful interactions to the sliding window.

[0113] The interruption freeze unit 203 is used to freeze the sliding window data when the interaction is interrupted, and save the window data, dialogue history and unfinished task data as a temporary state packet. At the same time, it starts the interruption timer and collects environmental noise and user state parameters.

[0114] The recovery determination unit 204 is used to determine whether to restore the original sliding window or clear the sliding window by comparing the duration of the interrupt timer with a preset threshold after the interaction is restored.

[0115] The prediction selection unit 205 is used to restore the original sliding window when the time is less than a preset threshold, calculate the predicted success rate of each channel after restoration using environmental noise and user status parameters, select the main channel after restoration based on the predicted success rate, and continuously calculate and output the real-time success rate.

[0116] like Figure 8 As shown, in a preferred embodiment of the present invention, the channel switching module 300 includes:

[0117] The threshold triggering unit 301 is used to preset a second threshold that is lower than the first threshold. When the instantaneous success rate of all channels is lower than the second threshold, all channels fail.

[0118] The independent recognition unit 302 is used to simultaneously receive audio signals from the voice channel, motion trajectory signals from the gesture channel, touch position signals from the touch channel, and image signals from the visual channel, and to perform independent intent recognition on the signals of each channel to obtain the intent recognition results and corresponding confidence scores for each channel.

[0119] The weighted correction unit 303 is used to weight and correct the confidence score based on the historical success rate of each channel in the archive, wherein the channel with the higher historical success rate is assigned a higher weighting coefficient.

[0120] The fusion output unit 304 is used to fuse the weighted and corrected intent recognition results of each channel and output the intent recognition result with the highest confidence score.

[0121] The above description only details the preferred embodiments of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0122] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the disclosure in the specification and embodiments. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. An adaptive interaction method for elderly care robots designed for home-based elderly care, characterized in that, The method includes the following steps: A profile of multimodal interaction channels for elderly users is constructed, which records the historical success rate, average response latency, and user preference weight of each channel. The multimodal interaction channels include voice channels, gesture channels, touch channels, and visual channels. The system collects input signals from each channel in real time during the interaction process and uses a sliding window to dynamically calculate the instantaneous success rate of each channel during the interaction attempts. When the real-time success rate of the primary channel is continuously lower than the first threshold, switch to the channel with the highest real-time success rate. When the real-time success rate of all channels is lower than the second threshold, execute the multi-channel fusion strategy. When switching channels, voice and screen prompts are output to inform the user of the switching result, and the parameters of the interactive interface are adjusted. At the same time, the switching record is stored in the file.

2. The adaptive interaction method for elderly care robots for home-based elderly care according to claim 1, characterized in that, The step of dynamically calculating the real-time success rate of each channel during interactive attempts using a sliding window specifically includes: A sliding window is used to record the success or failure status of each channel in each interaction a set number of times, and the continuity flag used to determine the interruption of the interaction is continuously monitored. The continuity flag includes voice detection status, user head orientation data, gaze dwell time and environmental sound events. When the interaction is uninterrupted, the sliding window is maintained, and the instantaneous success rate of each channel is calculated based on the ratio of the number of successful interactions to the sliding window. When the interaction is interrupted, the sliding window data is frozen, and the window data, dialogue history and unfinished task data are saved as a temporary state packet. At the same time, the interruption timer is started, and environmental noise and user state parameters are collected. Once the interaction resumes, the system determines whether to restore the original sliding window or clear it by comparing the duration of the interrupt timer with a preset threshold. When the time is less than the preset threshold, the original sliding window is restored, and the estimated success rate of each channel after restoration is calculated using environmental noise and user status parameters. Based on the estimated success rate, the primary channel after restoration is selected, and the real-time success rate is continuously calculated and output.

3. The adaptive interaction method for elderly care robots for home-based elderly care according to claim 2, characterized in that, The step of calculating the estimated success rate of each channel after recovery using environmental noise and user status parameters specifically includes: The environmental noise is decomposed into frequency bands, and the degree of interference of each frequency band noise on the voice channel is evaluated to obtain the noise confidence correction factor. The noise confidence correction factor for the other channels is the default value. The user status parameters were decomposed into visual stability, head posture flexibility, and hand tremor amplitude, and the capability confidence correction factor of each channel was evaluated. For each channel, the instantaneous success rate before the interruption is multiplied by the corresponding noise confidence correction factor and capability confidence correction factor to obtain the estimated success rate. The historical success rates of each channel in the archives are incorporated and used as long-term weights to adjust the estimated success rates.

4. The adaptive interaction method for elderly care robots for home-based elderly care according to claim 1, characterized in that, The steps for implementing the multi-channel fusion strategy specifically include: If the second threshold is lower than the first threshold, all channels will fail when the instant success rate of all channels is lower than the second threshold. Simultaneously, it receives audio signals from the voice channel, motion trajectory signals from the gesture channel, touch position signals from the touch channel, and image signals from the vision channel, and performs independent intent recognition on the signals from each channel to obtain the intent recognition results and corresponding confidence scores for each channel. Based on the historical success rate of each channel in the archives, the confidence score is weighted and adjusted, with channels that have a higher historical success rate being assigned a higher weighting coefficient. The weighted and corrected intent recognition results from each channel are merged, and the intent recognition result with the highest confidence score is output.

5. The adaptive interaction method for elderly care robots for home-based elderly care according to claim 4, characterized in that, The step of independently identifying the intent of the signals from each channel specifically includes: Based on the voice channel, the ASR recognition result is input into the intent classifier, which outputs the initial intent label and the initial confidence score. When incomplete sentence features are detected, the initial intent label is completed according to the interaction history, and the initial confidence score is processed using a preset discount coefficient. When repetitive sentence features are detected, the initial intent label is kept unchanged, and the current initial confidence score is weighted and averaged with the previous confidence score. Based on the gesture channel, the gesture recognition results of multiple consecutive frames are temporally smoothed, and the smoothed intent label and its confidence score are output. Based on the touch channel, the screen position of the touch point is mapped to the intent label. The probability of intentional operation is calculated based on the trajectory continuity and jitter amplitude of the touch point, and this probability is used as the confidence score. Based on the visual channel, the gaze detection model outputs the category of the gaze target object and the gaze duration. When the gaze duration exceeds the preset duration and there is no voice or gesture input, the intention label is obtained by mapping the gaze object category. The confidence score is determined based on the ratio of the gaze duration to the preset maximum duration.

6. The adaptive interaction method for elderly care robots for home-based elderly care according to claim 5, characterized in that, The method further includes: A fixed-length temporal intent queue is established for each channel, storing the intent tags and confidence scores identified by each channel in chronological order; After each new intent tag is identified, the intent stability index of the temporal intent queue of that channel is calculated. The stability index is determined based on the consistency of adjacent intent tags within the queue. When the new intent label is inconsistent with the previous intent label in the queue and the stability index is lower than the preset stability threshold, the current recognition result is determined to be a temporal jump. Replace the intent label with the historical intent label with the highest confidence score in the queue, and adjust the confidence score using a preset jump discount factor.

7. An adaptive interactive system for elderly care robots designed for home-based senior living, characterized in that: The system includes: The profile recording module is used to build profiles for multimodal interaction channels for elderly users, which records the historical success rate, average response latency, and user preference weights of each channel. The multimodal interaction channels include voice channels, gesture channels, touch channels, and visual channels. The data acquisition and statistics module is used to collect the input signals of each channel in real time during the interaction process, and to dynamically calculate the instantaneous success rate of each channel during the interaction attempt using a sliding window. The channel switching module is used to switch to the channel with the highest real-time success rate when the real-time success rate of the primary channel is continuously lower than the first threshold. When the real-time success rate of all channels is lower than the second threshold, a multi-channel fusion strategy is executed. The switching feedback module is used to output voice and screen prompts to inform the user of the switching result when switching channels, adjust the interactive interface parameters, and store the switching record in the file.

8. The adaptive interactive system for home-based elderly care robots according to claim 7, characterized in that, The data collection and statistics module includes: The window monitoring unit is used to record the success or failure status of each channel in each interaction using a set number of sliding windows, and continuously monitors the continuity flag used to determine the interruption of the interaction. The continuity flag includes voice detection status, user head orientation data, gaze dwell time and environmental sound events. The success rate statistics unit is used to maintain the sliding window when the interaction is uninterrupted, and to calculate the real-time success rate of each channel based on the ratio of the number of successful interactions to the sliding window. The interruption freeze unit is used to freeze the sliding window data when the interaction is interrupted, and save the window data, dialogue history and unfinished task data as a temporary state packet. At the same time, it starts the interruption timer and collects environmental noise and user state parameters. The recovery determination unit is used to determine whether to restore the original sliding window or clear the sliding window by comparing the duration of the interrupt timer with a preset threshold after the interaction is restored. The prediction selection unit is used to restore the original sliding window when the time is less than a preset threshold. It calculates the predicted success rate of each channel after restoration using environmental noise and user status parameters, selects the main channel after restoration based on the predicted success rate, and continuously calculates and outputs the real-time success rate.

9. The adaptive interactive system for home-based elderly care robots according to claim 7, characterized in that, The channel switching module includes: The threshold triggering unit is used to preset a second threshold that is lower than the first threshold. When the instantaneous success rate of all channels is lower than the second threshold, all channels will fail. An independent recognition unit is used to simultaneously receive audio signals from the voice channel, motion trajectory signals from the gesture channel, touch position signals from the touch channel, and image signals from the visual channel, and to perform independent intent recognition on the signals from each channel to obtain the intent recognition results and corresponding confidence scores for each channel. The weighted correction unit is used to weight and correct the confidence score based on the historical success rate of each channel in the archive, where the channel with the higher historical success rate is assigned a higher weighting coefficient. The fusion output unit is used to fuse the weighted and corrected intent recognition results of each channel and output the intent recognition result with the highest confidence score.

Images