Senior proactive care system using autonomous ai agent orchestration based on adaptive heartbeat engine and multimodal sensor fusion
The adaptive heartbeat engine and multimodal sensor fusion in senior care systems dynamically adjust intervention cycles and improve sensor accuracy, ensuring timely emergency responses and personalized care through integrated IoT device orchestration.
Patent Information
- Application Number
- KR1020260063454
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2026-04-08
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2046-04-08
AI Technical Summary
Existing senior care systems are limited by passive responses, lack of long-term conversational context, inaccurate sensor data fusion, and inability to dynamically adjust intervention cycles, leading to delayed emergency responses and resource wastage.
An adaptive heartbeat engine dynamically adjusts intervention cycles based on user activity and risk levels, combined with multimodal sensor fusion for precise emotional and physical state assessment, and an autonomous AI agent orchestrates multiple IoT devices for integrated care services.
Prevents delays in emergency responses, minimizes resource waste, and provides personalized, stable care interventions across the residential space, enhancing user engagement and accuracy.
Smart Images

Figure 112026042776738-PAT00002_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to an active senior care system utilizing an adaptive heartbeat engine and autonomous AI agent orchestration based on multimodal sensor fusion. Background Technology
[0002] The rapid aging of the population in modern society is emerging as a serious social challenge worldwide.
[0003] In particular, South Korea is entering a super-aging society, where the population of people aged 65 or older is predicted to increase from 14.9% in 2019 to 46.5% in 2067.
[0004] The increase in the elderly population is creating a vast demand for care services, including chronic disease management, alleviation of loneliness among the elderly living alone, and assistance with daily living; however, the working-age population supporting this is rapidly declining, leading to a dual challenge of a shortage of care personnel and rising costs.
[0005] Among these, mild cognitive impairment (MCI) and dementia are diseases that impose a tremendous economic and psychological burden not only on the patients themselves but also on their families and society as a whole; therefore, the consensus in the academic community is that continuous cognitive stimulation and systematic lifestyle management are essential alongside medication.
[0006] To meet these care demands, various assistive devices for senior care have been developed.
[0007] However, existing senior care assistive devices are largely limited to responding to passive alarms or simple voice commands.
[0008] These systems adopt a reactive architecture where functions only work when the user first recognizes the device, operates it, or speaks to it; consequently, they have limitations as they are not effective tools for seniors whose cognitive abilities have declined or who have become isolated from the outside world due to depression.
[0009] In addition, existing smart speakers or early-stage chatbot services lack the ability to maintain conversational context over the long term and have limitations in remembering users' individual lifestyle habits or emotional states to perform customized interventions.
[0010] Meanwhile, existing senior care robot systems provide a function to monitor the patient's condition, but they have a problem in that they cannot flexibly respond to changes in risk levels over time or changes in the user's activity patterns because the cycle for status checks is fixed.
[0011] For example, if the status is checked at the same fixed cycle as during the day, even when the risk of falls increases during late-night hours, the response to emergencies may be delayed.
[0012] Conversely, checking the status at unnecessarily short intervals, even during periods when users are active stably, results in a waste of computational resources and power.
[0013] In addition, existing care robot systems often rely on data from a single sensor when determining whether to intervene.
[0014] It is difficult to accurately determine the user's status with a single sensor alone, and false positives caused by sensor malfunctions or environmental factors can occur frequently.
[0015] For example, if the user's condition is determined solely by motion detection sensors, it is difficult to distinguish between a situation where the user is simply reading quietly and an actual emergency.
[0016] Furthermore, existing care robots operate based on a single device, which limits their ability to detect or respond to dangerous situations occurring in areas outside the space where the robot is physically located.
[0017] Various IoT devices, such as wearable devices and smart home devices, operate individually, but a system in which a single agent integrates and orchestrates them to collaborate has not yet been fully implemented.
[0018] Furthermore, existing care robot systems lack specific mechanisms to prevent conflicts between tasks when multiple operations—such as physical motion control, voice output, and app operation—occur simultaneously, leading to issues such as reduced operational stability or inconsistent responses to the user.
[0019] Therefore, there is an increasing technical demand for active senior care systems that dynamically adjust status check cycles based on user activity patterns and time-based risk levels, accurately determine the need for intervention by fusing multimodal sensor data such as voice, facial, and environmental data, and expand the care domain to the entire residential space by having a single autonomous AI agent integrally orchestrate multiple IoT devices.
[0020] Meanwhile, Korean Published Patent No. 10-2022-0109694 describes a "care service system for elderly people living alone using a companion robot and the method thereof." Prior art literature
[0021] Korean Patent Publication No. 10-2022-0109694 (Published August 5, 2022) The problem to be solved
[0022] Accordingly, the present invention has been devised against the aforementioned background and aims to provide an active senior care system capable of preventing delays in responding to emergency situations, such as the risk of falls during late-night hours, while minimizing unnecessary computational resources and power waste during stable activity hours, through an adaptive heartbeat engine that repeatedly generates a wake-up signal according to a preset cycle and dynamically adjusts the cycle by reflecting the user's activity index, risk level by time of day, and health status indicators.
[0023] Furthermore, the present invention aims to provide an active senior care system capable of precisely identifying the user's emotional and physical condition by resolving the false detection and judgment inaccuracies of existing methods relying on a single sensor, through a multimodal sensor fusion judgment unit that collects and fuses at least two of the user's voice data, facial data, environmental data, and activity data to calculate an intervention index.
[0024] Furthermore, the present invention aims to provide a senior active care system capable of realizing active care intervention in an emotionally acceptable manner while minimizing user resistance by establishing intervention strategies through reasoning based on a Large Language Model (LLM) and generating personalized conversation scripts utilizing a long-term memory storage that stores the user's past conversation history and preferences, and a persona setting unit that defines conversational tone and behavioral principles.
[0025] In addition, the present invention aims to provide a senior active care system that can prevent conflicts between voice output, motion control, and content playback of a care device and ensure stable robot operation by having a lane queue-based task serialization engine sequentially process execution commands authorized from an agent orchestration unit.
[0026] In addition, the present invention aims to provide an active senior care system that can expand the entire residential space into an intelligent care area beyond the physical range of a single care device through a multi-device collaboration orchestration unit that links with external IoT devices such as wearable devices, intelligent pill dispensers, and smart home devices.
[0027] The objectives of the present invention are not limited thereto, and other unmentioned objectives will be clearly understood by a person skilled in the art from the description below. means of solving the problem
[0028] To achieve this objective, one embodiment of the present invention relates to a senior active care system that monitors a user's condition and actively provides care services, comprising: an adaptive heartbeat engine unit that repeatedly generates a wake-up signal according to a preset cycle, and dynamically adjusts the cycle by reflecting at least one of the user's activity index, risk level by time period, and health status indicator; a multimodal sensor fusion judgment unit that, in response to the wake-up signal of the adaptive heartbeat engine unit, collects at least two of the user's voice data, facial data, environmental data, and activity data, and calculates an intervention index by fusion analysis of the collected data; and an agent orchestration unit that, when the intervention index calculated by the multimodal sensor fusion judgment unit exceeds a preset threshold, establishes an intervention strategy through inference based on a Large Language Model (LLM) and generates execution commands to control a care device according to the established intervention strategy. The present invention provides a senior active care system utilizing an adaptive heartbeat engine and multimodal sensor fusion-based autonomous AI agent orchestration, characterized by including: a lane queue-based task serialization engine unit that sequentially processes the execution commands authorized from the agent orchestration unit to serially execute at least one of voice output, motion control, and content playback of the care device.
[0029] Additionally, the adaptive heartbeat engine unit dynamically switches the cycle based on a state machine comprising: a dormant state in which the cycle is set to the first cycle when no movement is detected during the user's sleep time; a daily activity state in which the cycle is set to the second cycle when normal activity is detected during the activity time; an attention state in which the cycle is shortened to the third cycle when no movement is detected for more than a preset time during the activity time; and an emergency state in which the cycle is shortened to the fourth cycle when abnormal signs are detected. This provides a senior active care system utilizing an adaptive heartbeat engine and multimodal sensor fusion-based autonomous AI agent orchestration.
[0030] In addition, the multimodal sensor fusion judgment unit extracts at least one acoustic feature among jitter, shimmer, and Mel frequency cepstrum coefficient (MFCC) from the voice data, extracts at least one visual feature among valence and arousal from the facial data, and calculates the intervention index in a weighted sum manner by assigning a preset weight to each of the extracted acoustic feature and the visual feature. This provides an active senior care system utilizing an adaptive heartbeat engine and a multimodal sensor fusion-based autonomous AI agent orchestration.
[0031] Additionally, the agent orchestration unit comprises a long-term memory storage unit that stores past conversation history with the user, cognitive training results, and preference information, and a persona setting unit that stores persona information including the conversational tone, personality traits, and behavioral principles of the care device; and when establishing the intervention strategy, the system generates a personalized conversation script by reflecting the context information of the user stored in the long-term memory storage unit and the persona information stored in the persona setting unit, thereby providing an active senior care system utilizing an adaptive heartbeat engine and multimodal sensor fusion-based autonomous AI agent orchestration.
[0032] Additionally, the system further includes a multi-device collaboration orchestration unit that receives the user's biometric data or environmental data from an external IoT device and transmits the received data to the multimodal sensor fusion judgment unit or the adaptive heartbeat engine unit; wherein the external IoT device includes at least one of a wearable device, an intelligent pill box, and a smart home device, thereby providing a senior active care system utilizing an adaptive heartbeat engine and multimodal sensor fusion-based autonomous AI agent orchestration. Effects of the invention
[0033] According to one embodiment of the present invention, by dynamically adjusting the cycle of the weather signal by reflecting the user's activity index, risk level by time of day, and health status indicators, it is possible to prevent delays in responding to emergency situations, such as the risk of falls during late-night hours, while minimizing unnecessary computational resources and power waste during stable activity hours.
[0034] In addition, the multimodal sensor fusion judgment unit fuses and analyzes at least two of the user's voice data, facial data, environmental data, and activity data to calculate an intervention index, thereby resolving the false detection and judgment inaccuracies of existing methods that rely on a single sensor and enabling precise identification of changes in the user's emotional and physical state.
[0035] Furthermore, the agent orchestration unit establishes intervention strategies through reasoning based on a Large Language Model (LLM) and generates personalized conversation scripts by reflecting the user's contextual information stored in long-term memory and the conversational tone and behavioral principles defined in the persona setting unit, thereby minimizing user resistance and enabling active care intervention in an emotionally acceptable manner.
[0036] In addition, the lane queue-based task serialization engine sequentially processes execution commands authorized from the agent orchestration unit, thereby preventing conflicts between the voice output, motion control, and content playback of the care device and ensuring stable and consistent robot operation.
[0037] In addition, the multi-device collaboration orchestration unit receives user biometric data or environmental data from external IoT devices such as wearable devices, intelligent pill dispensers, and smart home devices, and transmits it to the multimodal sensor fusion judgment unit or the adaptive heartbeat engine unit, thereby having the effect of expanding the entire residential space into an intelligent care area beyond the physical range of a single care device. Brief explanation of the drawing
[0038] FIG. 1 is a block diagram showing a senior active care system utilizing an adaptive heartbeat engine and multimodal sensor fusion-based autonomous AI agent orchestration according to an embodiment of the present invention. FIG. 2 is a state machine flowchart of an adaptive heartbeat engine unit according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a multimodal sensor fusion-based intervention determination process according to an embodiment of the present invention. FIG. 4 is a lane queue-based multi-task priority processing flowchart according to an embodiment of the present invention. FIG. 5 is a multi-device collaboration orchestration data flow diagram according to an embodiment of the present invention. FIG. 6 is an overall flowchart of an active intervention process according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating an adaptive difficulty adjustment process for a cognitive enhancement quiz according to an embodiment of the present invention. FIG. 8 is a security structure diagram of a guardian app linkage and local gateway according to an embodiment of the present invention. Specific details for implementing the invention
[0039] Hereinafter, some embodiments of the present invention will be described in detail with reference to the exemplary drawings. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the present invention, if it is determined that a detailed description of related known components or functions could obscure the essence of the invention, such detailed description is omitted.
[0040] In addition, terms such as first, second, A, B, (a), (b), etc., may be used when describing the components of the present invention. These terms are intended only to distinguish the components from other components, and the essence, order, or sequence of the components is not limited by the terms. Where it is stated that a component is "connected," "combined," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that another component may also be "connected," "combined," or "connected" between each component.
[0041] FIG. 1 is a block diagram illustrating a senior active care system utilizing an adaptive heartbeat engine and autonomous AI agent orchestration based on multimodal sensor fusion according to an embodiment of the present invention. FIG. 2 is a state machine flowchart of the adaptive heartbeat engine unit according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating an intervention judgment process based on multimodal sensor fusion according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating a lane queue-based multi-task priority processing flowchart according to an embodiment of the present invention. FIG. 5 is a data flowchart of multi-device collaborative orchestration according to an embodiment of the present invention. FIG. 6 is an overall flowchart of the active intervention process according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating an adaptive difficulty adjustment process for a cognitive enhancement quiz according to an embodiment of the present invention. FIG. 8 is a security structure diagram of guardian app integration and local gateway according to an embodiment of the present invention.
[0042] As illustrated in these drawings, a senior active care system (100) utilizing an adaptive heartbeat engine and a multimodal sensor fusion-based autonomous AI agent orchestration according to an embodiment of the present invention comprises: an adaptive heartbeat engine unit (110) that repeatedly generates a wake-up signal according to a preset cycle, and dynamically adjusts the cycle by reflecting at least one of the user's activity index, risk level by time period, and health status indicator; and a multimodal sensor fusion judgment unit (120) that, in response to the wake-up signal of the adaptive heartbeat engine unit (110), collects at least two of the user's voice data, facial data, environmental data, and activity data, and calculates an intervention index by fusion analysis of the collected data. The system may be characterized by including: an agent orchestration unit (130) that, when the intervention index calculated by the multimodal sensor fusion judgment unit (120) exceeds a preset threshold, establishes an intervention strategy through inference based on a large language model (LLM) and generates an execution command to control the care device (200) according to the established intervention strategy; and a lane queue-based task serialization engine unit (140) that sequentially processes the execution command authorized from the agent orchestration unit (130) to serially execute at least one of voice output, motion control, and content playback of the care device (200).
[0043] Below, each component is described in detail.
[0044] Adaptive heartbeat engine unit (110)
[0045] The adaptive heartbeat engine unit (110) generates a wake-up signal repeatedly according to a preset period, and can dynamically adjust the period by reflecting at least one of the user's activity index, risk level by time period, and health status indicator.
[0046] The adaptive heartbeat engine unit (110) can variably set the wake-up cycle according to the user's real-time situation, for example, within a range of at least 5 minutes to a maximum of 120 minutes, in order to resolve the inefficiency of the existing fixed cycle method.
[0047] The period calculation of the adaptive heartbeat engine unit (110) can be performed according to the following [Equation 1].
[0048] [Mathematical Formula 1]
[0049] T_hb = T_base × ϕ(A) × γ(R) × ω(S)
[0050] Here, T_hb is the adaptive heartbeat period, T_base is the base period (e.g., 30 minutes), φ(A) is the activity index-based weighting factor, γ(R) is the risk weighting factor, and ω(S) is the schedule importance factor.
[0051] For example, the above φ(A) can be set to 0.5 to shorten the cycle if the user's activity level decreases by more than 70% compared to the average of the last hour, and set to 2.0 to extend the cycle if regular activity (such as performing gymnastics) is confirmed.
[0052] For example, the above γ(R) can be set to 0.1 when abnormal noise is detected or a fall is suspected, so that the cycle can be immediately shortened to the minimum level.
[0053] For example, the above ω(S) can be set to 0.5 to shorten the cycle when the time for taking medication is imminent, and maintained at 1.0 when there is no separate schedule.
[0054] Here, T_hb can be clipped to a range between a preset minimum period (e.g., 5 minutes) and a maximum period (e.g., 120 minutes).
[0055] That is, if the T_hb value calculated by [Equation 1] is smaller than the minimum period, it can be set to the minimum period, and if it is larger than the maximum period, it can be set to the maximum period.
[0056] Meanwhile, the adaptive heartbeat engine unit (110) can dynamically switch cycles based on a state machine that cycles through the following four states depending on the user's state.
[0057] First, the Idle-Deep state is entered when no movement is detected during the sleep time period (e.g., 21:00 to 07:00), and the cycle can be set to the first cycle (e.g., 120 minutes) to check only the environmental sensors (temperature, humidity, etc.).
[0058] Second, the Active-Daily state is entered when normal activity is detected during the activity time period (e.g., 07:00 to 21:00), and the cycle is set to the second cycle (e.g., 30 minutes) to perform care tasks such as greeting, medication guidance, and suggesting cognitive quizzes.
[0059] Third, the Caution-Inactive state is entered when no movement is detected for more than a preset time (e.g., 1 hour) during the activity period, and the cycle can be shortened to a third cycle (e.g., 15 minutes) to perform status checks via voice call.
[0060] Fourth, the Emergency-Alert is entered when abnormal signs (e.g., fall detection by a wearable device, sudden change in heart rate, etc.) are detected, and the cycle can be shortened to the fourth cycle (e.g., 5 minutes) to perform real-time camera streaming and emergency contact with a guardian.
[0061] Here, the fourth period is shorter than the third period, the third period is shorter than the second period, and the second period may be shorter than the first period.
[0062] For example, if the user has not moved for one hour at 2:00 PM (activity time), the adaptive heartbeat engine unit (110) switches from a daily activity state to an attention state and sets φ(A) to 0.5 to shorten the heartbeat cycle to 15 minutes.
[0063] As another example, if 30 minutes have passed since the sensor in the direction of the bathroom was activated at 2 a.m. but no returning movement was detected, the adaptive heartbeat engine unit (110) can switch from a dormant state to an emergency state and shorten the cycle to 5 minutes.
[0064] As another example, if the user performs the gymnastics program normally at 10:00 AM, the adaptive heartbeat engine unit (110) can increase energy efficiency by setting φ(A) to 2.0 to extend the heartbeat cycle to 60 minutes.
[0065] Multimodal sensor fusion judgment unit (120)
[0066] The multimodal sensor fusion judgment unit (120) can collect at least two of the user's voice data, facial data, environment data, and activity data in response to a weather signal from the adaptive heartbeat engine unit (110), and calculate an intervention index by fusion analyzing the collected data.
[0067] The multimodal sensor fusion judgment unit (120) can comprehensively determine the user's state by integrating multiple data modalities to prevent malfunction of a single sensor or false detection due to environmental factors.
[0068] The multimodal sensor fusion judgment unit (120) may include a voice analysis module (121) that extracts acoustic features from voice data, a face analysis module (122) that extracts visual features from face data, an environment sensor module (123) that collects environment data, and an activity detection module (124) that collects activity data.
[0069] The voice analysis module (121) can collect the user's voice through a microphone array mounted on the care device (200) and extract at least one acoustic feature among jitter, shimmer, and Mel-Frequency Cepstral Coefficients (MFCC) from the collected voice.
[0070] For example, jitter is an indicator that measures the irregularity of the vocal cord vibration cycle, and in the case of users with worsening depression, jitter values may tend to increase by more than 1.04% compared to normal.
[0071] As another example, MFCC can detect patterns of cognitive decline, such as a decrease in speech rate or ambiguity in pronunciation, by analyzing changes in the frequency spectrum of speech.
[0072] As another example, if the range of the fundamental frequency (F0) narrows and changes to a monotone tone, the voice analysis module (121) can determine the user's emotional apathy state.
[0073] The facial analysis module (122) can collect a user's facial image through a camera mounted on the care device (200) and extract at least one visual feature among emotional polarity (Valence) and arousal (Arousal) from the collected facial image.
[0074] For example, the facial analysis module (122) can perform facial expression analysis based on a Valence-Arousal model and determine that emotional care is needed when the Valence value is 0.3 or less (negative emotion).
[0075] As another example, the facial analysis module (122) can perform an analysis of Action Units (AU) based on the Facial Action Coding System (FACS) and determine that if there is a lack of coordination between AU6 (squinting of the eyes) and AU12 (pulling of the corners of the mouth), it is a fake smile (Social Smile) and initiate emotional care.
[0076] As another example, the facial analysis module (122) can track the user's facial expression changes over the long term and determine that if a pattern of decreasing diversity of facial expressions is detected, it is an early sign of cognitive decline.
[0077] The environment sensor module (123) can collect at least one of indoor light intensity, temperature, and humidity through a sensor mounted on the care device (200) or an external IoT device.
[0078] For example, if the indoor illuminance is measured to be 10 Lux or less during the activity time, the environment sensor module (123) can determine that the user is in an abnormal state of being active in a dark environment.
[0079] The activity detection module (124) can collect user movement data through a human body detection sensor (PIR) mounted on the care device (200) or an accelerometer of an external wearable device.
[0080] The calculation of the intervention index of the multimodal sensor fusion judgment unit (120) can be performed according to the following [Equation 2].
[0081] [Mathematical Formula 2]
[0082] I_int = Σ(w_i × s_i) - C_context
[0083] Here, I_int is an intervention index, s_i is an abnormal score for each sensor modality (voice, face, environment, activity), w_i is a confidence weight for each sensor modality, and C_context is a context correction coefficient that reflects the user's usual behavioral pattern stored in long-term memory.
[0084] The above context correction coefficient (C_context) can serve to prevent unnecessary intervention by downwardly correcting the intervention index when the user has a habit of living a quiet life.
[0085] The multimodal sensor fusion judgment unit (120) can determine the intervention level differentially according to the value of the calculated intervention index (I_int) as follows.
[0086] For example, if I_int is 0.8 or higher, high-intensity intervention (emergency situation dissemination) can be initiated.
[0087] As another example, if I_int is 0.5 or higher and less than 0.8, a moderate-intensity intervention (attempting conversation with a companion) can be initiated.
[0088] As another example, if I_int is less than 0.5, intervention can be withheld and wait until the next heartbeat cycle.
[0089] Agent Orchestration Department (130)
[0090] The agent orchestration unit (130) can establish an intervention strategy through inference based on a large language model (LLM) when the intervention index calculated by the multimodal sensor fusion judgment unit (120) exceeds a preset threshold, and can generate an execution command to control the care device (200) according to the established intervention strategy.
[0091] The agent orchestration unit (130) may include a long-term memory storage unit (131) that stores past conversation history with the user, cognitive training results, and preference information, and a persona setting unit (132) that stores persona information including the conversation tone, personality traits, and behavioral principles of the care device (200).
[0092] The long-term memory storage (131) can store detailed log records and summary memories in JSONL format in parallel, thereby enabling the agent to understand the user over the long term and continue personalized conversations.
[0093] For example, the long-term memory storage (131) can store the user's cognitive quiz correct answer rate over the past week, preferred music genre, family relationship information, memories mentioned in past conversations, etc.
[0094] As another example, the long-term memory storage (131) can store health management data such as the user's medication history, exercise performance history, and sleep patterns.
[0095] As another example, the long-term memory storage (131) stores the daily emotional state change trends of the user, which can be used by the agent orchestration unit (130) to dynamically adjust the intensity of emotional care.
[0096] The persona setting unit (132) can define the conversational tone, personality traits, and behavioral principles of the care device (200) to help minimize user resistance and form an emotional bond.
[0097] For example, in the persona setting section (132), specific personality traits such as “an affectionate 8-year-old grandson,” specific conversational tones such as “did you do that?” and behavioral principles such as “prioritizing health but encouraging without sounding like nagging” can be defined.
[0098] As another example, the persona setting section (132) may define personality traits such as "strict health coach" and conversational tones such as "you must...".
[0099] As another example, the persona information of the persona setting section (132) can be set or changed remotely by the guardian through the guardian terminal device (300).
[0100] The agent orchestration unit (130) can perform the following step-by-step reasoning process when establishing an intervention strategy.
[0101] First, in the context collection stage, the agent orchestration unit (130) can comprehensively collect the current time, mission list, the time of the user's last conversation, and sensor data.
[0102] Second, in the intervention necessity assessment stage, the agent orchestration unit (130) can determine whether the current situation requires intervention through a large language model (LLM) based on the collected data.
[0103] For example, reasoning such as "Current time 10:30 AM, mission list has a mission to suggest gymnastics at 10:00 AM, current user is in front of care device (200) (PIR detection), since there is a 30-minute delay, gymnastics should be suggested immediately" can be performed.
[0104] Third, in the intervention strategy formulation stage, the agent orchestration unit (130) can generate a personalized conversation script that minimizes user resistance by reflecting the persona information stored in the persona setting unit (132).
[0105] Fourth, in the tool call and execution step, the agent orchestration unit (130) can generate execution commands for the operation control, voice output, content playback, etc. of the care device (200) to perform the determined action and transmit them to the lane queue-based task serialization engine unit (140).
[0106] Lane queue-based work serialization engine unit (140)
[0107] The lane queue-based task serialization engine (140) can sequentially process execution commands authorized from the agent orchestration unit (130) to serially execute at least one of voice output, motion control, and content playback of the care device (200).
[0108] The lane queue-based task serialization engine (140) can enter all execution commands into a queue and check the completion status of the currently executing task to sequentially deliver the next task to the care device (200) in order to prevent hardware resource conflicts of the care device (200) when multiple execution commands occur simultaneously.
[0109] The lane queue-based task serialization engine (140) can classify execution commands into the following three classes and process them according to priority.
[0110] First, Level 1 (Interrupt-Emergency) is an execution command that responds to emergency situations such as fall detection or heart rate abnormalities, and can be executed as a top priority by stopping all ongoing tasks.
[0111] For example, if a fall is detected from a wearable device, the lane queue-based task serialization engine (140) can immediately stop the cognitive quiz content currently being performed and prioritize the execution of user status check voice output and guardian emergency notification transmission.
[0112] Second, Level 2 (Interactive-Realtime) is an execution command corresponding to a user's direct call and answer to a question, which can be executed as soon as the current operation is completed.
[0113] For example, when a user calls out "Dasom-ah," the lane queue-based task serialization engine (140) can execute a voice response to the user's question after the background music currently being played is completed.
[0114] Third, Level 3 (Background-Scheduled) is an execution command corresponding to regular well-being checks by heartbeat, diary writing suggestions, etc., and can be executed only when the user is resting or the care device (200) is idle.
[0115] For example, if the heartbeat engine recognizes the time for the diary writing mission at 7:00 PM, the lane queue-based task serialization engine (140) can execute a diary writing suggestion voice output after confirming that the user is not currently performing other activities (such as eating).
[0116] The processing algorithm of the lane queue-based task serialization engine (140) can determine the next execution task according to the following [Equation 3].
[0117] [Mathematical Formula 3]
[0118]
[0119] Here, T_next is the next task to be executed, T_new is a newly received execution command, Dequeue(Q) is an operation to extract the next task from the queue (Q) according to the first-in, first-out (FIFO) method, and Robot_Idle may be the idle state of the care device (200).
[0120] Tasks of level 1 (Interrupt-Emergency) are executed immediately without entering the queue and regardless of the Robot_Idle condition, while tasks of levels 2 and 3 can be sequentially extracted and executed when the care device (200) is idle after entering the queue.
[0121] Multi-device collaboration orchestration unit (150)
[0122] The multi-device collaboration orchestration unit (150) can receive user biometric data or environmental data from an external IoT device and transmit the received data to the multimodal sensor fusion judgment unit (120) or the adaptive heartbeat engine unit (110).
[0123] The multi-device collaboration orchestration unit (150) can abstract various communication protocols such as MQTT, Zigbee, and REST API to enable the agent orchestration unit (130) to control external IoT devices in an integrated manner.
[0124] The above external IoT device may include at least one of a wearable device, an intelligent pill dispenser, and a smart home device.
[0125] The wearable device can collect the user's real-time heart rate (HR) and heart rate variability (HRV) and transmit them to a multi-device collaborative orchestration unit (150).
[0126] For example, if the user's SDNN (standard deviation of heart rate variability) value drops by more than 2 standard deviations compared to the normal range (e.g., 109.1 ms), the multi-device collaborative orchestration unit (150) can transmit the data to the adaptive heartbeat engine unit (110) to trigger a transition to an emergency state.
[0127] As another example, when a wearable device detects an impact of 70G or more (estimated fall), the multi-device collaborative orchestration unit (150) can simultaneously transmit the event to the adaptive heartbeat engine unit (110) and the multimodal sensor fusion judgment unit (120).
[0128] As another example, the wearable device collects the user's sleep patterns (sleep time, number of times tossing and turning during sleep, etc.) and stores them in a long-term memory storage (131), and the agent orchestration unit (130) can use them to provide comments on sleep quality during morning greeting conversations.
[0129] The intelligent medicine container can detect whether medicine has been taken and transmit it to the multi-device collaborative orchestration unit (150).
[0130] For example, if the opening / closing signal of the intelligent medicine bottle is not detected even though 30 minutes have passed since the medication schedule, the multi-device collaborative orchestration unit (150) can transmit the data to the agent orchestration unit (130) to perform medication recommendation through the care device (200).
[0131] As another example, the intelligent medicine dispenser can collect information on the remaining amount of medicine and send a medicine re-prescription notification to the guardian terminal device (300).
[0132] As another example, the intelligent pill dispenser can accumulate data on the user's medication adherence by recording the information in a long-term memory storage (131) when medication is completed.
[0133] Smart home devices may include smart plugs, smart lights, gas valve controllers, etc.
[0134] For example, if it is detected that TV usage time is more than 5 hours longer than usual through a smart plug, the multi-device collaboration orchestration unit (150) can transmit the data to the agent orchestration unit (130) to activate the conversational companion skill.
[0135] As another example, in the event of an emergency, the multi-device collaborative orchestration unit (150) can simultaneously send commands to switch the smart lights to 100% brightness and shut off the gas valve.
[0136] As another example, when it is time to sleep, the multi-device collaborative orchestration unit (150) can gradually lower the brightness of the smart lights and adjust the indoor temperature to an appropriate sleep temperature (e.g., 22°C).
[0137] Care device (200)
[0138] The care device (200) may be a physical device that interacts directly with the user according to the execution command of the agent orchestration unit (130).
[0139] The care device (200) may include a recognition module (210) that converts the user's voice into text and recognizes faces and objects, an expression module (220) that outputs the response of the agent orchestration unit (130) as voice and facial expression, a motion control module (230) that controls the joint movements of the care device (200), and a content playback module (240) that runs a cognitive training app, video, music, etc.
[0140] For example, the care device (200) may be a robot dedicated to senior care (for example, a Kebbi robot).
[0141] As another example, the care device (200) may be an interaction device of various forms, such as a smart display or a tablet terminal.
[0142] As another example, the care device (200) may be an articulated robot equipped with 12 servo motors capable of controlling the movement of the arms, neck, and body.
[0143] Guardian terminal device (300)
[0144] The guardian terminal device (300) can provide an interface for the guardian to manage the operation settings of the agent orchestration unit (130) and monitor the status of the user.
[0145] The guardian terminal device (300) may include a persona setting interface that can remotely set or change persona information of the persona setting unit (132), an activity dashboard that visualizes and provides a log of the user's daily activities, and a mission assignment interface that allows the guardian to input a specific care mission and transmit it to the agent orchestration unit (130).
[0146] For example, the guardian terminal device (300) may be a guardian-only application installed on a smartphone.
[0147] As another example, if a guardian inputs "Please tell them to make sure to take their medicine this afternoon" through the guardian terminal device (300), the agent orchestration unit (130) can determine the appropriate timing and perform the corresponding mission through the care device (200).
[0148] As another example, when an emergency occurs, the guardian terminal device (300) can receive an emergency notification containing real-time video and location information from the agent orchestration unit (130).
[0149] Gateway section (400)
[0150] The gateway section (400) may be a central control plane that manages all data flow between the care device (200) and the guardian terminal device (300).
[0151] The gateway unit (400) can ensure real-time performance by using a WebSocket-based real-time communication protocol and maintain data integrity by allocating an independent session to each user.
[0152] The gateway unit (400) can perform all inference and data processing in a local environment, taking into account the sensitivity of personal health data and home video information.
[0153] For example, the guardian terminal device (300) receives only the care report summarized by the agent orchestration unit (130), and may adopt a dual security structure in which a session for real-time video verification is formed only when there is an explicit request from the guardian and voice approval from the user.
[0154] As another example, the gateway unit (400) can isolate and manage sessions so that when multiple users use the system in different locations, each user's personalized data and persona information are not confused.
[0155] As another example, the gateway unit (400) can support an offline mode so that basic care functions can continue in the local environment even when a network failure occurs.
[0156] Meanwhile, the senior active care system (100) according to one embodiment of the present invention can provide integrated care services to the user through the following specific service scenarios.
[0157] Service Scenario 1: Intelligent Medication Management
[0158] The agent orchestration unit (130) can perform medication guidance missions based on the medication time set by the guardian.
[0159] For example, if the user is not detected near the care device (200) via the activity detection module (124) at the time of medication, the agent orchestration unit (130) may delay the medication guidance until the user is detected, and as soon as the user is detected, generate a personalized conversation script such as "Senior, where have you been? It is time to take your medicine now" and output it through the care device (200).
[0160] As another example, when the user completes taking the medication, the agent orchestration unit (130) can record the medication completion information in the long-term memory storage (131) and send a completion notification to the guardian terminal device (300).
[0161] Service Scenario 2: Adaptive Cognitive Enhancement Quiz
[0162] The agent orchestration unit (130) can perform a cognitive enhancement quiz service that dynamically adjusts the difficulty level according to the user's cognitive state.
[0163] The difficulty level of the agent orchestration unit (130) can be adjusted according to the following [Equation 4].
[0164] [Mathematical Formula 4]
[0165] D_next = D_current + α × (R_correct - R_target)
[0166] Here, D_next is the difficulty level of the next quiz, D_current is the current difficulty level, α is the difficulty adjustment coefficient, R_correct is the correct answer rate over the past week, and R_target is the target correct answer rate (e.g., 70%).
[0167] Here, α can be set to 0.5, for example, in which case D_next is adjusted upward by 0.05 compared to D_current when R_correct is 80%, which is 10% higher than the target accuracy rate (70%), and D_next is adjusted downward by 0.1 compared to D_current when R_correct is 50%, which is 20% lower than the target accuracy rate.
[0168] For example, if R_correct is 85% or higher, D_next is adjusted upward so that difficult problems such as 'words backward' can be selected.
[0169] As another example, if R_correct is 50% or less, D_next is lowered so that low-difficulty problems such as 'Guess the name from the picture' can be selected.
[0170] As another example, if a decrease in the user's concentration is detected through the facial analysis module (122) during the quiz, the agent orchestration unit (130) can stop the quiz and switch to playing music or having a conversation with a companion.
[0171] The results of the quiz performance are stored in a long-term memory storage (131), and if signs of cognitive decline are detected, they can be immediately visualized on the activity dashboard of the guardian terminal device (300).
[0172] Service Scenario 3: Robot-Assisted Gymnastics Guide
[0173] The agent orchestration unit (130) can directly guide the user's physical activity by utilizing the motion control module (230) of the care device (200).
[0174] For example, the motion control module (230) can adjust the joint angle (e.g., left_shoulder_y: 110) of the care device (200) in accordance with the flow of the gymnastics video to demonstrate the movement in a way that makes it easy for the user to follow along.
[0175] As another example, if the activity detection module (124) detects a cessation of the user's movement during gymnastics, the agent orchestration unit (130) can generate an encouraging message such as "Keep going a little longer! You're almost there!" and output it through the care device (200).
[0176] As another example, after the gymnastics are completed, the care device (200) can perform a celebratory dance to provide a positive reward to the user and record the results of the gymnastics performance in a long-term memory storage (131).
[0177] Service Scenario 4: Active Emotional Care
[0178] The agent orchestration unit (130) can determine the user's emotional state based on the analysis results of the multimodal sensor fusion judgment unit (120) and actively perform emotional care.
[0179] For example, if a keyword related to loneliness or sadness is detected in the user's speech through the voice analysis module (121), the agent orchestration unit (130) can retrieve a list of music that the user enjoyed listening to from the long-term memory storage (131) and play music through the content playback module (240) or perform a dance through the motion control module (230) to draw attention.
[0180] As another example, if the Valence value is consistently measured to be 0.3 or less through the facial analysis module (122), the agent orchestration unit (130) can send a notification requesting a video call connection to the guardian terminal device (300).
[0181] As another example, the agent orchestration unit (130) can induce the user to write a diary after dinner by asking a question such as, "What was the happiest moment of your day?" and convert the voice-recognized answer into text through the recognition module (210) and store it in the long-term memory storage (131).
[0182] Service Scenario 5: Writing an Autobiography and Diary
[0183] The agent orchestration unit (130) can perform an emotional care service that transcribes the user's speech into text and semantically summarizes it to leave a record of life.
[0184] For example, the agent orchestration unit (130) can classify diary data accumulated for one year in the long-term memory storage (131) by topic (family, hometown, dreams, etc.) and produce it in the form of a digital autobiography, and share it with family members through the guardian terminal device (300).
[0185] Service Scenario 6: Nostalgia Evocation and Personalized Video / Music Provision
[0186] The agent orchestration unit (130) can maximize cognitive stimulation by combining family photo data uploaded by the guardian with user preference information.
[0187] For example, the agent orchestration unit (130) can first say, "Grandma, do you remember this photo? It is from when you went on a trip to Jeju Island 20 years ago!" and display the photo on the display of the content playback module (240).
[0188] As another example, if depression is detected as a result of the emotional analysis of the multimodal sensor fusion judgment unit (120), the agent orchestration unit (130) can select the brightest song from the list of trot songs that the user usually enjoys listening to and play it through the content playback module (240), and control the care device (200) to shake its body to the rhythm through the motion control module (230).
[0189] Service Scenario 7: Companion Conversation
[0190] The agent orchestration unit (130) can actively initiate a conversational companion by the wake-up signal of the adaptive heartbeat engine unit (110) even if the user does not explicitly call it.
[0191] For example, if it is detected that the user is alone for a long time during the activity period, the agent orchestration unit (130) can retrieve the previous day's conversation content from the long-term memory storage (131) and initiate a conversation that continues the context, such as "What was the name of the dog you met when you went for a walk yesterday?"
[0192] Service Scenario 8: Multi-Detection Based Emergency Response
[0193] A cross-device emergency response scenario through a multi-device collaboration orchestration unit (150) can be performed as follows.
[0194] First, in the detection phase, if the wearable device detects an impact of 70G or more (estimated fall) and the accelerometer remains stationary, the multi-device collaborative orchestration unit (150) can transmit the event to the adaptive heartbeat engine unit (110) to switch to an emergency state.
[0195] Second, in the care device dispatch phase, the agent orchestration unit (130) can instruct the care device (200) to move to the last location where the user was detected.
[0196] Third, in the situation verification and environment optimization stage, the care device (200) can check the user's condition through a camera, switch the smart light to 100% brightness through the multi-device collaboration orchestration unit (150), and shut off the gas valve.
[0197] Fourth, in the final reporting stage, if the user's response is not recognized, the agent orchestration unit (130) can transmit real-time video and location information to the guardian terminal device (300) through the gateway unit (400) and transmit a rescue request to 119.
[0198] As described above, according to one embodiment of the present invention, the adaptive heartbeat engine unit dynamically adjusts the cycle of the weather signal by reflecting the user's activity index, risk level by time of day, and health status indicators, thereby preventing delays in response to emergency situations such as the risk of falls during late-night hours, while minimizing unnecessary computational resources and power waste during stable activity hours.
[0199] In addition, the multimodal sensor fusion judgment unit fuses and analyzes at least two of the user's voice data, facial data, environmental data, and activity data to calculate an intervention index, thereby resolving the false detection and judgment inaccuracies of existing methods that rely on a single sensor and enabling precise identification of changes in the user's emotional and physical state.
[0200] Furthermore, the agent orchestration unit establishes intervention strategies through reasoning based on a Large Language Model (LLM) and generates personalized conversation scripts by reflecting the user's contextual information stored in long-term memory and the conversational tone and behavioral principles defined in the persona setting unit, thereby minimizing user resistance and enabling active care intervention in an emotionally acceptable manner.
[0201] In addition, the lane queue-based task serialization engine sequentially processes execution commands authorized from the agent orchestration unit, thereby preventing conflicts between the voice output, motion control, and content playback of the care device and ensuring stable and consistent robot operation.
[0202] In addition, the multi-device collaboration orchestration unit receives user biometric data or environmental data from external IoT devices such as wearable devices, intelligent pill dispensers, and smart home devices, and transmits it to the multimodal sensor fusion judgment unit or the adaptive heartbeat engine unit, thereby having the effect of expanding the entire residential space into an intelligent care area beyond the physical range of a single care device.
[0203] Although it has been described above that all components constituting an embodiment of the present invention are combined or operate as a single unit, the present invention is not necessarily limited to such an embodiment. That is, within the scope of the purpose of the present invention, all components may be selectively combined in one or more ways to operate.
[0204] The foregoing description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols
[0205] 100: Senior Active Care System Utilizing Autonomous AI Agent Orchestration Based on Adaptive Heartbeat Engine and Multimodal Sensor Fusion 110: Adaptive Heartbeat Engine Unit 120: Multimodal sensor fusion judgment unit 121 : Voice Analysis Module 122 : Facial Analysis Module 123 : Environmental sensor module 124: Activity detection module 130 : Agent Orchestration Department 131: Long-term memory storage 132 : Persona Settings 140: Lane Queue-based Task Serialization Engine 150: Multi-device Collaboration Orchestration Department 200 : Care device 210: Recognition Module 220 : Expression Module 230: Operation control module 240 : Content Playback Module 300 : Guardian terminal device 400: Gateway section
Claims
Claim 1 A senior active care system that monitors the user's condition and actively provides care services comprises: an adaptive heartbeat engine unit that repeatedly generates a wake-up signal according to a preset cycle, and dynamically adjusts the cycle by reflecting at least one of the user's activity index, risk level by time of day, and health status indicator; a multimodal sensor fusion judgment unit that, in response to the wake-up signal from the adaptive heartbeat engine unit, collects at least two of the user's voice data, facial data, environmental data, and activity data, and calculates an intervention index by fusion analysis of the collected data; an agent orchestration unit that, when the intervention index calculated by the multimodal sensor fusion judgment unit exceeds a preset threshold, establishes an intervention strategy through inference based on a Large Language Model (LLM) and generates execution commands to control a care device according to the established intervention strategy; and a lane queue-based task serialization engine unit that sequentially processes the execution commands authorized by the agent orchestration unit to serially execute at least one of voice output, motion control, and content playback of the care device.A senior active care system utilizing an adaptive heartbeat engine and multimodal sensor fusion-based autonomous AI agent orchestration, comprising: a lane queue-based task serialization engine; wherein the execution command is classified into one of the following: Grade 1, which is an execution command responding to an emergency situation that stops all ongoing tasks and is executed with the highest priority; Grade 2, which is an execution command corresponding to a direct call or answer to a question from the user that is executed immediately upon completion of the current operation; and Grade 3, which is an execution command corresponding to a regular care task based on a weather signal that is executed only when the care device is idle; wherein the execution command of Grade 1 is not entered into a queue and is executed immediately regardless of whether the care device is idle, and the execution commands of Grade 2 and Grade 3 are entered into the queue, and then sequentially extracted and executed in a first-in, first-out manner when the care device is idle after checking the completion status of the currently executing task. Claim 2 In claim 1, the adaptive heartbeat engine unit is characterized by dynamically switching the cycle based on a state machine comprising: a dormant state in which the cycle is set to a first cycle when no movement is detected during the user's sleep time; a daily activity state in which the cycle is set to a second cycle when normal activity is detected during the activity time; an attention state in which the cycle is shortened to a third cycle when no movement is detected for more than a preset time during the activity time; and an emergency state in which the cycle is shortened to a fourth cycle when an abnormal sign is detected. Claim 3 A senior active care system utilizing an adaptive heartbeat engine and multimodal sensor fusion-based autonomous AI agent orchestration, wherein, in claim 1, the multimodal sensor fusion judgment unit extracts at least one acoustic feature among jitter, shimmer, and Mel frequency cepstrum coefficient (MFCC) from the voice data, extracts at least one visual feature among valence and arousal from the facial data, and calculates the intervention index in a weighted sum manner by assigning a preset weight to each of the extracted acoustic feature and the visual feature. Claim 4 A senior active care system utilizing an adaptive heartbeat engine and multimodal sensor fusion-based autonomous AI agent orchestration, wherein, in claim 1, the agent orchestration unit comprises a long-term memory storage unit that stores past conversation history with the user, cognitive training results, and preference information, and a persona setting unit that stores persona information including the conversational tone, personality traits, and behavioral principles of the care device, and when establishing the intervention strategy, generates a personalized conversation script by reflecting the context information of the user stored in the long-term memory storage unit and the persona information stored in the persona setting unit. Claim 5 A senior active care system utilizing an adaptive heartbeat engine and multimodal sensor fusion-based autonomous AI agent orchestration, wherein, in claim 1, it further comprises a multi-device collaboration orchestration unit that receives the user's biometric data or environmental data from an external IoT device and transmits the received data to the multimodal sensor fusion judgment unit or the adaptive heartbeat engine unit, and wherein the external IoT device includes at least one of a wearable device, an intelligent pill dispenser, and a smart home device.
Citation Information
Patent Citations
Intelligent health-care patient wrist strap based on multi-modal sensing and AI
CN120549457A
Intelligent AI personal healthcare safety control system that analyzes risk data and predicts and manages emergency situations through network communication
KR102747926B1
Wearable device healthcare system
US20240148340A1
Systems and methods for determining and performing interventions using objective-based measurements and specifying the interventions using a medical description language that is capable of serving as input to a large language model
US20260057989A1
