A hierarchical empathic meal-assisting robot control method and system based on multimodal intention recognition
By using multimodal intention recognition and hierarchical empathic guidance closed loop, voice and facial movement features are collected, and a large language model is used to determine dining intention and switch guidance strategies. This solves the problem of lack of emotional interaction in meal assistance robots, achieves efficient dining intention recognition and adaptive eating guidance, and improves the dining experience and safety.
Patent Information
- Application Number
- CN202611399238.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-10
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-09-10
AI Technical Summary
Existing meal assistance robots lack emotional interaction, are unable to effectively identify and respond to the elderly's dining desires, resulting in a deterioration of the eating experience. Furthermore, they cannot provide tiered empathetic guidance based on differences in users' interaction abilities, leading to interaction interruptions and safety hazards.
By using a multimodal willingness recognition and hierarchical empathy guidance closed loop, voice signals, human body key points and facial movement features are collected to generate natural language semantic descriptions. A large language model is used for cross-modal reasoning to determine dining willingness, and the eating guidance strategy is switched according to the interaction ability, including conversational persuasion and loop playback guidance.
It improves the accuracy and robustness of meal intention recognition, adapts to elderly groups with different communication abilities, realizes the intelligent and humanized meal assistance process, avoids the safety hazards of forced feeding, and enhances the eating experience.
Smart Images

Figure CN122876412B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human-computer interaction technology, and in particular to a hierarchical empathic meal-assisting robot control method and system based on multimodal intention recognition. Background Technology
[0002] For vulnerable groups such as the elderly with disabilities, physical impairments, and cognitive impairments, the decline in physical function and mobility often prevents them from eating independently, requiring long-term reliance on family members or professional caregivers for meal assistance. Traditional manual feeding not only consumes significant human and time resources but also lacks emotional connection, easily leading to resistance and loss of appetite, thus worsening the eating experience. Against this backdrop, Meal-Assistance Robots (MARs) can provide self-feeding support for elderly and disabled individuals with impaired motor function, significantly improving their quality of life and independence. These robots help reduce reliance on caregivers, alleviate the persistent shortage of elderly care workers, and reduce the enormous social costs associated with care, demonstrating significant economic and social benefits.
[0003] However, most existing meal-assistance robots focus on the physical aspects of food acquisition and delivery, generally neglecting the emotional interaction and guidance needed during the eating process for the elderly. Multiple studies show that approximately 15%-30% of elderly people living at home in the community experience decreased appetite / anorexia due to cognitive impairment, social isolation, emotional problems, and inappropriate care interactions. The prevalence is even higher among elderly people in hospitals and nursing homes, reaching 38% and 42%, respectively. Long-term anorexia significantly increases the risk of malnutrition, frailty, and even death in the elderly. Although refusal to eat is common in elderly care, it has not received sufficient attention. An international survey of 1,545 healthcare workers showed that 38.7% of respondents still considered anorexia a normal aging phenomenon, making it easily overlooked by doctors and family members as simply a lack of appetite in old age. In real-world elderly care scenarios, a large number of refusal-to-eat behaviors do not stem from physical inability to eat, but rather from psychological and emotional unwillingness to eat. Service robots are rapidly developing towards intelligence and human-machine emotional interaction, with specialized robots for niche scenarios such as elderly care becoming a new industry trend.
[0004] Invention application CN121973192A proposes a robot control method, device, equipment, and medium based on emotion enhancement, involving the field of artificial intelligence in finance, healthcare, insurance, and banking. The method includes: acquiring the user's visual information and natural language commands to generate an emotion embedding vector; inputting the visual information, natural language commands, and emotion embedding vector into a preset large language model, which generates a comprehensive command that simultaneously includes task semantics and emotional strategies; generating an action token based on the task semantics and emotional strategies, the action token including action parameters for controlling robot movement and language response content; driving the robot to perform physical actions based on the action parameters and simultaneously outputting language response content. However, this method only performs general emotion adjustment and passive action execution based on commands, lacking multimodal temporal semantic reasoning for dining intentions, and failing to achieve a closed-loop dining assistance control system that integrates layered empathic persuasion and intention review based on differences in user interaction capabilities. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a hierarchical empathy-based meal assistance robot control method and system that achieves accurate perception and adaptive guidance through multimodal intention recognition and hierarchical empathy guidance closed loop, thereby improving the intelligence and experience of meal assistance.
[0006] The objective of this invention can be achieved through the following technical solutions: A hierarchical empathic meal-assisting robot control method based on multimodal willingness recognition includes the following steps: Collect the target user's voice signal, human body key point coordinates, and facial motion unit features; Voice events, action events, and facial expression events are generated based on the voice signal, the coordinates of the human body key points, and the facial motion unit features, respectively. The continuous numerical features of the voice events, action events, and facial expression events are converted into natural language semantic descriptions, and the natural language semantic descriptions are temporally aggregated within a sliding window to obtain a scene semantic summary. The semantic summary of the scene is input into a large language model for cross-modal reasoning to determine the target user's current dining intention category; When the willingness to eat is determined to be "acceptable", the robotic arm is controlled to perform the food picking and delivery actions. When the dining intention category is determined to be rejection, the interaction ability level of the target user is read; When the interaction capability level is high, conversational persuasion content is generated by combining a large language model with a user database, and the digital human is controlled to guide the eating conversationally. When the interaction capability level is low, the digital human is controlled to play a loop of guiding messages to guide eating. After the feeding guidance is completed, the target user's willingness to eat is reassessed, and when the willingness to eat changes to accept, the robotic arm is controlled to perform the food picking and delivery actions.
[0007] Furthermore, generating the voice event specifically includes: The speech signal is transcribed into text through real-time speech recognition, and speech rate, volume, fundamental frequency and pitch are extracted simultaneously. Speech events with timestamps are generated through speech activity detection.
[0008] Furthermore, generating the action event specifically includes: Based on the coordinates of the human body's key points, the system identifies eating-related behaviors such as nodding, shaking, tilting the head, opening the mouth, and waving the hands through temporal motion analysis, and outputs action events with timestamps.
[0009] Furthermore, generating the facial expression event specifically includes: The facial motion unit features are extracted. The facial motion unit includes inner eyebrow raising AU1, outer eyebrow raising AU2, eyebrow frowning AU4, cheek lifting AU6, nose root wrinkling AU9, and mouth corner pulling AU12. Facial expression events with timestamps are output.
[0010] Furthermore, the step of converting the continuous numerical features of the voice event, the action event, and the facial expression event into a natural language semantic description specifically includes: The speech rate in the speech event is mapped to semantic labels of slow speech rate, normal speech rate, or fast speech rate through a semantic mapping function. The volume in the voice event is mapped to semantic labels of low volume, normal volume, and high volume using a semantic mapping function. The intonation in the speech event is mapped into semantic tags of rising intonation, falling intonation, and steady intonation through a semantic mapping function; The average intensity of each facial action unit in the facial expression event within the sliding window is compared with a preset activation threshold. When the average intensity is greater than the preset activation threshold, it is determined as the activation intensity of the facial action unit. The activation intensity of the facial action unit is then mapped to a semantic expression containing the name and intensity of the facial action unit through a semantic mapping function. The temporal changes in the coordinates of key human body points in the action event are mapped to preset action categories through a semantic mapping function, and the duration of the preset action category and its coverage within the sliding window are mapped to a structured description of the action.
[0011] Furthermore, the step of performing temporal aggregation of the natural language semantic description within a sliding window to obtain a scene semantic summary specifically includes: Within a continuously advancing sliding window of fixed width, events falling within the sliding window are acquired; the semantic content and prosodic features corresponding to the speech events within the sliding window are summarized. The duration of each behavior within the sliding window is recorded, and the sliding window coverage is calculated to generate an action behavior summary. Calculate the average activation value of facial motion units within the sliding window and the trend of changes across the window to generate an expression summary; The aggregated results of the voice events, action events, and facial expression events within the sliding window are concatenated into a scene semantic summary using a structured concatenation operator.
[0012] Furthermore, the step of inputting the scene semantic summary into a large language model for cross-modal reasoning to determine the target user's current dining intention category specifically includes: The percentage of time during which voice, motion, and facial expressions are active within the sliding window is calculated as a temporal continuity factor. The weighted confidence is calculated based on the modal confidence, corresponding decision weight, and corresponding temporal continuity factor of the independent outputs of speech, action, and facial expression. The decision weight of speech is higher than that of action, and the decision weight of action is greater than that of facial expression. The sum of the decision weights of the three is 1. The scene semantic summary, the weighted confidence score, and the preset prompt template are input into the large language model to infer and determine the target user's current dining intention category.
[0013] Furthermore, after determining that the dining intention category is rejection, the step of reading the target user's interaction ability level further includes: When the feeding guidance reaches the maximum number of persuasion rounds and the dining intention category remains in refusal, it is determined that the feeding is continuously refused, and the robotic arm is controlled to return to the initial position and trigger a prompt.
[0014] Furthermore, the control of the robotic arm to perform the food-grabbing and delivery actions specifically includes: Identify the dishes in the plate area based on an object detection model; The system uses a large language model to determine the feeding strategy based on the dish name; it then controls the end effector of the robotic arm to execute the feeding strategy and deliver the food to the target user's mouth. During delivery, force control sensors installed at the end of the robotic arm monitor changes in force. When an abnormal change in force occurs, the robotic arm is controlled to stop the delivery action and retract to a safe position, while an alarm is issued.
[0015] A hierarchical empathic meal-assisting robot control system based on multimodal willingness recognition includes: The sensing device, including a microphone and a depth camera, is used to collect the target user's voice signals, human body key point coordinates, and facial motion unit features; Interactive devices, including touch screens and speakers, are used to present digital human figures and output dialogue or guiding voices; The mechanical actuator includes a six-degree-of-freedom robotic arm, an end effector, and a force control sensor located at the end of the robotic arm; A controller is connected to the sensing device, the interaction device, and the mechanical actuator. The controller includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, it implements a hierarchical empathic meal-assisting robot control method based on multimodal intention recognition.
[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. Unifying heterogeneous perceptual features and eliminating transient noise significantly improves the accuracy and robustness of dining intention recognition: By using a semantic mapping function, the continuous numerical features of three types of heterogeneous perceptual signals—speech, action, and facial expression—are uniformly converted into natural language semantic descriptions. This eliminates the dimensional differences in sampling frequency, feature scale, and semantic expression among different modalities, enabling large language models to directly understand underlying perceptual information and perform cross-modal semantic reasoning. Simultaneously, combining temporal speech aggregation with a sliding window effectively suppresses the impact of occasional actions, environmental noise, and transient disturbances on the discrimination results. This not only characterizes the continuity and dynamic evolution of user behavior but also compresses the input scale and reduces the reasoning complexity of large language models.
[0017] 2. Adapts to elderly groups with different communication abilities, effectively solving the problem of interaction interruption caused by the single interaction strategy of traditional meal assistance: The system automatically switches the eating guidance strategy based on the interaction ability level of the target user. For highly interactive users with communication ability, a large language model combined with the user database is used to generate targeted multi-round dialogue-style persuasion. For low-interaction users with limited communication, the system switches to a multi-dimensional loop playback mode of guidance without language feedback. This avoids the interaction stagnation caused by the inability of low-interaction elderly to respond. The same system can cover the entire spectrum of elderly users from normal communication to cognitive impairment, which greatly improves the success rate of empathic persuasion and the universality of the system.
[0018] 3. Construct a closed-loop control system of "perception-guidance-reassessment-execution" to realize the transformation of meal assistance robots from passive mechanical feeding to proactive emotional care: It breaks through the limitations of existing meal assistance robots that assume users are willing to eat, deeply embedding multimodal meal willingness recognition and emotional guidance into the robot's control logic. After recognizing refusal, it proactively uses a digital human to provide layered empathetic guidance to alleviate the elderly's resistance. After a single round of guidance, it reassesses the willingness in real time, and only triggers the robotic arm to perform the food picking and delivery action when the user's willingness turns to acceptance. This avoids the safety hazards and bad experience of forcibly feeding when the user is resistant, and significantly improves the humanization and intelligence of the meal assistance process. Attached Figure Description
[0019] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the temporal speech aggregation method of the present invention; Figure 3 This is a flowchart of the hierarchical empathic feeding guidance strategy of the present invention; Figure 4 This is a schematic diagram of the layered empathic eating guidance method of the present invention; Figure 5 This is a system module diagram of the present invention; Figure 6 This is a distribution diagram of failure types in this invention; Figure 7 NASA-TLX scale scores for four HI subjects; Figure 8 The graph shows the evaluation results of technology acceptance and user experience. Figure 9 This is a comparative evaluation chart of robots and human caregivers across four dimensions. Detailed Implementation
[0020] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0021] Example 1 This embodiment provides a hierarchical empathic meal-assisting robot control method based on multimodal intention recognition, such as... Figure 1 As shown, it includes the following steps: The underlying behavioral information of elderly target users is collected in parallel using a microphone array and an Intel RealSense D405 depth camera. Based on the collected voice signals, human key point coordinates and facial motion unit features, voice events, motion events and facial expression events are generated respectively. Voice event generation: After the microphone captures the user's voice, it uses Baidu AI Cloud's real-time speech recognition to transcribe the speech into text, and simultaneously extracts speech rate, volume (RMS), and base frequency (BF). Prosodic features such as intonation are used to filter out environmental noise using speech activity detection (VAD) to generate speech events with independent timestamps.
[0022] Action event generation: The depth camera acquires human images at a sampling rate of 30fps. The MediaPipe Holistic human keypoint detection model is used to extract the coordinates of key points of the whole skeleton. Through temporal motion analysis, the behavior closely related to eating is identified, including nodding, shaking, tilting the head, opening the mouth and waving the hand, and the action events with independent timestamps are output.
[0023] Facial expression event generation: The OpenFace 3.0 tool is used to extract facial action unit (AU) features associated with dining intentions, covering AU1 (inner eyebrow raising), AU2 (outer eyebrow raising), AU4 (frowning / eyebrow lowering), AU6 (cheek lifting), AU9 (nose root wrinkling) and AU12 (pulling corners of mouth), to generate facial expression events with independent timestamps.
[0024] The generated heterogeneous event stream is written to an independent buffer channel, retaining the original timestamp attribute, thus achieving asynchronous event management and thread-safe data transmission.
[0025] To address the inconsistencies in sampling rate, feature scale, and dimensions among speech, actions, and facial expressions, semantic mapping and temporal aggregation are performed to unify the mapping of continuous numerical features generated by different modalities into natural language semantic descriptions. For example, speech rate is mapped to semantic labels such as "slow speech rate" and "normal speech rate"; the duration and window coverage of actions are mapped to structured descriptions such as "waving hands (lasting 4.0s, occupying 80% of the window)"; and AU activation intensity is mapped to semantic expressions such as "increased eyebrow downward pressure" and "moderate wrinkling of the nasal root". After unified semantic mapping, information from different modalities is transformed into a consistent natural language space, effectively eliminating differences in feature scale and dimensions, and providing a unified input representation for subsequent semantic inference. The specific mapping formula is as follows: in Indicates the first semantic mapping function of class modality, Indicates the first The original feature space of the modality, This represents the natural language semantic space; the mapping transforms the continuous numerical features of each modality into a unified natural language semantic description. , , These correspond to three perceptual modalities: speech, action, and facial expression.
[0026] Table 1. Rules for determining the conversion of multimodal features of speech, facial expressions, and actions into natural language labels. To characterize the persistence and dynamic evolution of target user behavior, a fixed-width sliding window (window length) is used. Step length Proceed continuously along the timeline to obtain the current window. The system contains three asynchronous semantic event streams: voice events, action events, and facial expression events. The sliding window and its internally accumulated multimodal event set are as follows: in, , , These correspond to three perception modalities: speech, motion, and facial expression. Indicates the current moment; Indicates The right endpoint is, and the length is The sliding window interval; It means falling in the window Belongs to the first A collection of events for a modal.
[0027] When a multimodal persistent event (such as an action or speech) crosses the left or right boundary of the current sliding window, the processing rule of "intra-window truncation for coverage, and full event duration for persistence description" is adopted. Let the event time interval be... The current window is ,in If the event interval intersects with the current window, that is... If the event is not specified, it is included in the event representation of the current window, but only the portion intersecting with the window is retained for time statistics within the window. Specifically, the valid duration within the window is: Time coverage is The portion outside the window is not included in the coverage calculation of the current window. Meanwhile, to avoid artificially shortening the duration of the same persistent event in adjacent windows due to sliding window boundaries, the event duration is not truncated by the window boundaries: for ended events, their full duration is reported. For ongoing events, report the cumulative duration up to the current moment. Therefore, duration and window coverage are preserved as two independent temporal features.
[0028] like Figure 2 As shown, in obtaining the event sets of each modality After completing the semantic mapping of continuous numerical features, the overall features of each channel within the window are statistically analyzed through temporal speech aggregation: the speech channel summarizes the semantic content and prosodic features within the window; the action channel calculates the duration of each behavior and calculates the window coverage to generate a behavior summary; the facial expression channel calculates the mean activation value of action units and the cross-window change trend. The specific formula for the above temporal aggregation process is as follows: in This represents a structured splicing operator. For channel-specific aggregation functions of mode k, This represents the semantic summary of the scene obtained by aggregation within the window.
[0029] The semantic summary of the scenario is input into a large language model for cross-modal reasoning. Considering the characteristics of the meal assistance scenario, this paper designs a domain-knowledge-enhanced prompt template, explicitly encoding decision rules such as modal priority, temporal continuity, multimodal consistency, trend changes, and conflict resolution into the reasoning process. Specific rules include: linguistic semantics, as the most direct expression of intent, has the highest decision weight; action information serves as an auxiliary basis for judgment; facial expressions serve as reference information; behaviors that occur continuously or show an increasing trend have higher discriminative value; when different modalities conflict, comprehensive reasoning is performed by combining modal priority and temporal context. The model ultimately outputs the elderly person's current meal intention category (Accept or Reject) and the corresponding decision basis, providing reliable decision support for subsequent hierarchical empathic eating guidance strategy selection and robot-assisted meal control. Based on the above rules, multimodal intention judgment can be formalized into a weighted confidence model, as shown below: in, express Time category The weighted confidence level; Indicates the first Modal decision weights satisfy > > And the sum of the three is 1; Represents the k-th modality pair category The modal confidence is output independently by each sensing module; Indicates the first The temporal continuity factor of a modality is defined as the time continuity of that modality within a window. The percentage of time the user is in an active state. In equation (4), This indicates the predicted category of dining intention; c is the candidate category, with values of either accept (Acc) or reject (Rej). Represents the inference function of a large language model; The scene semantic summary obtained by equation (3); The weighted confidence score obtained from equation (5) is used as the reference input for reasoning in the large language model; Enhance the prompt template for domain knowledge.
[0030] The specific process of the weighted fusion decision-making mentioned above is as follows: Input parameters and initialization Get the current sliding window In-generated scene semantic summarization Modal confidence of independent outputs of each sensing mode (in These represent the speech modality, action modality, and facial expression modality, respectively. (representing the two candidate categories of acceptance and rejection, respectively), and the preset modal decision weights. (satisfy and ), and preset domain knowledge enhancement prompt templates. .
[0031] Calculate the time continuity factor for each mode. Time-cumulative statistics were performed separately for the three modalities of speech, motion, and facial expression: Initialization mode Cumulative active time within the current sliding window ; Traverse the current sliding window Each sampling time point within If the mode is determined exist If you are always active, your cumulative active time will increase. ,in The sampling time interval; Calculate modes Time continuity factor This is used to characterize the proportion of active time of this mode within the sliding window.
[0032] Calculate the weighted confidence score for each candidate category. For the two candidate categories, Accept (Acc) and Reject (Rej) Calculate their comprehensive weighted confidence scores respectively. : Initialize category Weighted confidence level ; The calculation formula, which integrates the decision weights of each modality, modal independence confidence scores, and time continuity factors, is as follows: In the formula, For the decision weights of the corresponding modes, For modality Category confidence level For modality The time continuity factor.
[0033] Large language model reasoning and decision output combining domain knowledge rules Scene semantic summarization Weighted confidence level and domain knowledge enhancement prompt templates The data is passed as input to a large language model, and inference scores for the two candidate categories are calculated separately: In the formula, Represents the category of the output of a large language model The reasoning confidence score; Compare the scores of the two candidate categories: if the accept score is greater than or equal to the rejection score ( If the target user's current dining intention prediction result is "accept", then the prediction result is determined to be "accept". If the acceptance score is less than the rejection score ( If the target user's current dining intention prediction result is "rejection", then the prediction result is determined to be "rejection". ); The final output predicts the category of dining intentions. And its corresponding natural language decision-making basis.
[0034] When the willingness to eat is determined to be "acceptable", the robotic arm is controlled to perform the food picking and delivery actions. When the dining intention category is determined to be rejection, the interaction ability level of the target user is read; The target user's interaction ability level was obtained based on the limit values of the World Health Organization (WHO) International Classification of Functioning, Disability and Health (ICF) items d310 (receiving verbal information) and d330 (speaking) related to verbal communication, as well as through clinical assessment using the Mini-Mental State Examination (MMSE). High-Interaction (HI) users: Communication function is absent or slightly limited (ICF threshold 0-1) and cognitive function is normal (MMSE). point); Low-Interaction (LI) users: Exhibit moderate to severe communication difficulties (ICF threshold). Level 1) or cognitive decline (MMSE) point).
[0035] The evaluation results are pre-entered into the user profile database as interactive capability tags, and the system directly reads the corresponding interactive capability tags after identifying the target user.
[0036] After determining user interaction capabilities, the system automatically switches to the corresponding empathic eating guidance strategy based on user type. The tiered empathic eating guidance process is as follows: Figure 3 As shown. When a target user is determined to be a high-interaction user, the system calls the large language model and combines it with the user profile database to analyze the reasons for food refusal (physical factors, emotional factors, or objective factors) based on health records (such as medical history and symptoms of physical discomfort), social relationships (such as information on children and relatives), and personal preference information (such as dietary preferences and entertainment preferences such as opera). For physical refusal to eat, we will tailor health rehabilitation and maintenance communication strategies. For emotionally driven refusal to eat, we can use fun post-meal incentives to stimulate the desire to eat, taking into account entertainment preferences. For objective reasons of food refusal, we combine the image and voice cloning of relatives to provide emotional support and counseling. The large language model generates targeted, multi-turn conversational persuasion content, presents a digital human image through an interactive interface, and drives voice output to achieve dynamic and personalized conversational eating guidance.
[0037] When the target user is identified as a low-interaction user, the system switches to a loop-guided mode to address their inability to provide effective verbal question-and-answer feedback. Without relying on the user's verbal response, the system generates and plays short, friendly, and infectious one-way guiding messages in a loop, based on multiple preset dimensions such as food incentives, interest rewards, food conservation, health care, and emotional encouragement.
[0038] After a single round of eating guidance (multiple rounds of dialogue for HI users or a single loop of guidance for LI users) ends, the signal is collected again and the willingness to eat is reassessed. Both types of feeding guidance strategies have a maximum limit on the number of persuasion rounds. wheel; If the willingness to eat changes to acceptance in any of the 5 rounds, the feeding guidance process is immediately terminated, and the robotic arm is controlled to perform the food picking and delivery actions; If the willingness to eat is still judged as rejected after reaching the upper limit for 5 consecutive rounds, it is recorded as a failure of feeding guidance, judged as a persistent refusal to eat, and the robotic arm is controlled to reset to the initial safe position and a prompt signal is issued to notify the caregiver to intervene.
[0039] The entire layered, empathic eating guidance mechanism is based on offline survey data, combined with a large language model to achieve personalized and precise guidance, while embedding humane termination rules to balance practicality and humanistic care. Authentic empathic eating guidance dialogues include... Figure 4 As shown, the specific design of the dialogue includes the following three levels: (1) Research on the causes of food refusal. The three core reasons for elderly people's resistance to eating are: physical reasons - which are physiological food refusal, mainly including various physical discomforts such as no appetite, abdominal bloating and discomfort, toothache, physical weakness and inability to eat, and physical discomfort after eating; emotional reasons - which are psychological food refusal, mainly including emotional and psychological problems such as subjective resistance to eating, bland taste of food, loneliness and boredom when eating alone, and lack of motivation to eat; objective reasons - which are situational food refusal, mainly including objective realities such as eating snacks before meals, not feeling hungry at the moment, and planning to postpone meal time. The specific tiered empathic eating guidance prompt template is shown in Table 2.
[0040] Table 2. Layered Empathic Eating Guidance Template (2) User-Specific Database. To achieve personalized and empathetic guidance and persuasion for each individual, this embodiment establishes a dedicated database for elderly users, comprehensively recording and retaining core user information to provide data support for intelligent identification and precise persuasion. The core stored content of the database includes: user health status (past medical history, postoperative recovery, symptoms of physical discomfort, dietary health taboos, etc.); user social relationships (children's residence, companionship of relatives and friends, family interaction patterns, etc.); user's daily preferences and taboos (dietary preferences, entertainment hobbies, daily leisure habits, dietary taboos, behavioral taboos, etc.).
[0041] (3) Intelligent identification and tiered persuasion. This embodiment integrates the Qwen large language model to identify and analyze the semantics of elderly users' dialogues in real time, accurately determine the category of the user's refusal to eat, and match the user information in the database to output soft, warm, and personalized persuasion content: For physical refusal to eat, persuasion scripts are customized based on the user's health records, focusing on core points such as health rehabilitation and physical care to provide caring persuasion; For emotional refusal to eat, the user's daily interests and preferences are combined to guide the user to eat immediately through fun incentives after meals, mobilizing their enthusiasm for eating; For objective refusal to eat, the user's social relationship information is combined to provide emotional guidance by focusing on family companionship, alleviating the elderly's loneliness and resistance.
[0042] Example 2 This embodiment provides a hierarchical empathic meal-assisting robot control system based on multimodal intention recognition. It mainly comprises four parts: a perception module, an interaction module, a mechanical execution module, and a core main control module. Each module establishes communication and data transmission connections with the controller via hardware cables. Figure 5 As shown: Perception Module Depth camera: Mounted at the end of the robotic arm and connected to the main control board via a USB cable. Used for food detection (identifying the location and type of dishes on the plate area) and dining intention recognition (collecting the user's facial expressions, head posture, and key movement points).
[0043] Microphone and audio acquisition unit: Connected to the main control board via USB data cable, used to acquire the target user's voice signal in real time for voice interaction and intention analysis.
[0044] Force control sensor: Installed between the end effector of the robotic arm and the actuator, and connected to the main control board via a USB data cable. It is used to detect force in real time during food delivery and feeding, and trigger safety retraction and shutdown protection when abnormal force changes occur.
[0045] Interactive module Display Interactive Terminal (Tablet / Screen): Connected to the main control board via HDMI cable, it runs the digital human interactive interface. The system supports switching between the virtual digital human avatar and the family member clone avatar, and presents a multi-round dialogue interaction panel to achieve layered, empathetic feeding guidance.
[0046] Speaker unit: In conjunction with the audio module, it connects to the main controller via a USB data cable and is used to output digital human dialogue voice, looped guidance messages, and safety alarm prompts.
[0047] Mechanical Actuation Module Multi-degree-of-freedom robotic arm: Connects to the main control board via a USB data cable to receive motion planning and control commands.
[0048] End effector: The robotic arm is equipped with a replaceable spoon or fork at the end. The system adaptively switches the actuator according to the shape of the food it recognizes and performs the corresponding scooping or forking action to deliver the food smoothly to the target user's mouth to complete the feeding.
[0049] Core main control module Edge computing main control board (controller): It centrally connects the depth camera, robotic arm, force control sensor, display screen and microphone speaker unit, and serves as the central data processing and control hub. It is responsible for multimodal intention reasoning, digital human dialogue generation, food detection and robotic arm motion planning and safety control.
[0050] Example 3 To comprehensively evaluate the overall performance of the hierarchical empathic eating guidance robot system proposed in this embodiment, all experiments in this embodiment were conducted in real-world scenarios. The following experiments were designed from four dimensions: physical execution, perception and understanding, interactive guidance, and user experience: (1) Feeding success rate experiment to evaluate the reliability of the robotic arm's physical food picking and delivery under different categories of Chinese dishes; (2) Dining intention recognition experiment to verify the accuracy of the multimodal semantic reasoning framework in recognizing the dining intention of the elderly, and to quantify the independent contribution of each modality through an ablation experiment; (3) Hierarchical empathic eating guidance experiment to test the effectiveness of the hierarchical empathic eating guidance mechanism in guiding the elderly from a state of refusal to eat to a state of acceptance; (4) Human-computer interaction comprehensive evaluation experiment to evaluate the user's acceptance of the robot, cognitive load, and user experience based on a standardized scale.
[0051] Based on a hierarchical framework of interaction ability, eight elderly subjects (aged ≥60 years and ≤90 years) were recruited from a nursing home. The inclusion criteria for subjects included: (1) varying degrees of difficulty in daily eating (requiring assistance from others or completely dependent on others for feeding); (2) no severe hearing or visual impairment affecting basic communication; and (3) no severe swallowing dysfunction requiring nasogastric feeding or gastrostomy. All subjects signed informed consent forms, and the eight subjects were divided into a high-interaction (HI) group (n=4) and a low-interaction (LI) group (n=4).
[0052] The experiment was conducted in a dedicated test room in an elderly care facility, with stable indoor lighting conditions. The experimental hardware included: (1) the complete hardware platform of the layered empathic eating guidance and meal assistance robot developed in this paper (including a digital human display terminal, a six-degree-of-freedom robotic arm, an RGB-D camera, a microphone array, and a speaker); (2) standard experimental plates and four representative Chinese dishes; (3) video and audio recording equipment for offline behavior analysis; and (4) a tablet device for data recording and questionnaire completion. The system software was deployed on an NVIDIA Jetson Orin NX development board, the digital human interactive interface ran locally, all calculations and data processing were completed on the device, and user data was not transmitted to the outside, thus ensuring user privacy and data security.
[0053] Before the experiment, participants signed informed consent forms, completed basic information collection and pre-testing of scales, and were introduced to the robot's basic functions and safety precautions by the experimenter. Following this, participants entered a system familiarization phase, engaging in simple voice and visual interactions with the robot to become acquainted with its basic operating modes. The formal experiment consisted of three tasks: a feeding success rate test, a meal intention recognition test, and a tiered empathic feeding guidance test (the order of the tasks was randomized among participants, with appropriate breaks in between). After the experiment, participants completed a human-computer interaction evaluation scale and a semi-structured open-ended interview.
[0054] Eight subjects underwent feeding tests in a randomized order, with each food item subjected to 20 independent feeding-delivery cycles (a total of 200 cycles). Each cycle was defined as follows: the robotic arm starts from the starting position → visually locates the target food → performs the corresponding scooping or forking action → delivers the food to a preset position in front of the elderly person's mouth → the elderly person completes biting and eating → the robotic arm returns to the starting position. During the feeding process, a force control sensor at the end of the robotic arm monitors changes in force in real time. When the elderly person strongly resists, coughs, or suddenly experiences physical discomfort, an abnormal change in the force at the end will occur. Based on this, the system determines that there is a safety risk, immediately stops the current delivery action, and retracts the robotic arm to a safe position away from the elderly person's face. At the same time as retraction, an alarm is sounded by a speaker to alert the caregivers present to intervene in time. The operator selects the target food category through the system interface, and the robot automatically identifies the food location and selects the corresponding feeding strategy. Evaluation indicators include: (1) feeding success rate (number of successful cycles / total number of cycles × 100%); (2) average cycle time (complete cycle time from the start of the robotic arm to the return to the starting position); (3) failure type distribution (scooping / forking failure, food identification error, strategy selection error, delivery process drop, other system abnormalities).
[0055] The feeding performance varies significantly under different food forms, as shown in Table 3.
[0056] Table 3 Feeding success rate and average cycle time The feeding success rate for fried chicken nuggets and apple chunks was 90%, while that for watermelon chunks was 85%, with the shortest average cycle time (45.7-46.9s). Chunk-shaped ingredients are intact and have stable outlines, making them less prone to slipping during fork handling, which helps ensure stable delivery. The feeding success rate for granular rice was 85%, with an average cycle time of 48.7±2.0s. The feeding success rate for sliced dishes (shredded green peppers and pork, stir-fried greens, scrambled eggs with tomatoes, and shredded celery and pork) was concentrated between 65% and 85%, with average cycle times mostly exceeding 50s. Sliced ingredients are loose in texture, easily broken, and stick together, making them prone to falling during scooping and transport, reducing the success rate of complete feeding and increasing operational time. The performance of strip-shaped ingredients showed the most significant differentiation: noodles had a feeding success rate of 85%, with an average cycle time of 48.5±2.1s; while rice noodles, due to their soft and slippery texture, were extremely prone to tangling and slipping, resulting in a feeding success rate of only 60%, with an average cycle time of 51.4±3.4s.
[0057] Based on the aforementioned differences in feeding performance, further statistical analysis was conducted on all 200 feeding cycles of failure cases to analyze the main factors affecting the system's feeding success rate, such as... Figure 6 As shown in the figure, the results indicate that the highest percentage of failures occurred when scooping or forking (35.0%), mainly due to differences in texture, shape, and structural stability of different foods (e.g., rice noodles easily slip, watermelon chunks easily break, and rice grains easily spill), suggesting that the end effector's adaptability to the complex physical characteristics of Chinese food still has room for improvement. Dropping during delivery accounted for 27.5%, mainly related to changes in acceleration during the robotic arm's movement and the contact stability between food and utensils. System anomalies (robotic arm communication anomalies, joint protection triggering, and brief interruptions in visual communication) accounted for 20.0%. In contrast, food recognition errors accounted for only 10.0%, and strategy selection errors accounted for 7.5%, indicating that food recognition and LLM-driven feeding strategy decisions have good reliability. Overall, physical execution layer errors (scooping / forking failures + delivery dropping) accounted for 62.5% of all failure cases, representing the main bottleneck affecting feeding success rates; perception and decision-making layer errors accounted for only 17.5% combined.
[0058] This experiment evaluated the performance of a multimodal semantic reasoning framework in recognizing elderly participants' willingness to eat (Accept / Reject) based on their actual interaction with a meal-assistance robot. Eight participants were divided into two groups, HI (n=4) and LI (n=4), according to their ICF (Information Functional Scale) and MMSE (Morphological Examination) cognitive assessment results. Each session followed a unified closed-loop process of "willingness assessment - feeding or tiered empathic feeding guidance": the system first assessed the participant's willingness to eat using a complete multimodal framework (voice + action + facial expression fusion), then inquired about their true willingness as the gold standard (Ground Truth). If the participant's true willingness was Accept, the robot directly delivered food; if it was Reject, the system triggered tiered empathic feeding guidance, re-collecting multimodal signals and reassessing willingness after each round of guidance (up to 5 rounds).
[0059] To fairly compare the discriminative capabilities of each modality, this study employed a synchronous ablation protocol: at each intention judgment point, the system simultaneously recorded the inference outputs of five model configurations: (1) speech only, (2) action only, (3) facial expression only, (4) speech + action, and (5) complete multimodal (speech + action + facial expression). The inference output of the complete multimodal configuration served as the system's decision-making tool to drive the robot's actual behavior, while the other four configurations ran silently in the sidebar, without interfering with the interaction process. All five configurations were evaluated on the exact same set of judgment points, ensuring the consistency of the data source. Evaluation metrics included overall accuracy, precision, recall, and F1 score. The ablation experiment results are shown in Tables 4 and 5.
[0060] Table 4 Ablation Experiment Results of HI Group Table 5 Ablation Experiment Results of Group LI During the actual interaction, the HI group generated 42 willingness assessment points (28 initial assessments + 14 reassessments during the layered empathic feeding guidance process), of which 28 were genuine Accept and 14 were genuine Reject. The LI group, due to the higher frequency of triggering layered empathic feeding guidance, generated 99 willingness assessment points (28 initial assessments + 71 reassessments during the layered empathic feeding guidance process), of which 26 were genuine Accept and 73 were genuine Reject. The two groups combined had 141 assessment points.
[0061] The complete multimodal method achieved 90.5% and 89.9% accuracy in the HI and LI groups, respectively, with F1 scores of 92.9% and 82.1%, respectively. This indicates that under the controlled experimental conditions, multimodal fusion can improve the performance of recognizing the dining intentions of users with different interaction capabilities.
[0062] The ablation results show that different modalities function differently for different users. The action modality exhibited the most stable performance in both groups (HI: 83.3%, LI: 78.8%), demonstrating good cross-user robustness. The performance of the speech modality significantly decreased in the LI group (accuracy 56.6%, F1 score 33.8%), indicating that low-interaction users have limited language expression abilities, and speech information contributes insufficiently to intention judgment. The facial expression modality had the lowest overall discriminative ability (HI 69.0%, LI 54.5%), validating its role as auxiliary reference information. Notably, the speech + action dual-modality combination already approaches the performance level of the complete multimodal system (HI F1: 90.9%, LI F1: 72.7%), indicating that speech and action are the core information channels for dining intention recognition, and the facial expression modality can provide further improvement. This finding has practical deployment implications: in application scenarios with limited computing resources or sensor configuration, the quality of both speech and action signals can be prioritized to achieve discriminative performance close to that of the complete system.
[0063] When the multimodal semantic reasoning framework detects that the elderly person is in a Reject state, the system directly selects the corresponding empathic eating guidance strategy based on pre-grouped labels: For HI users, the system analyzes the reasons for refusal to eat (physical factors, emotional factors, or objective factors) through a large language model and generates targeted, personalized empathic eating guidance content; for LI users, the system outputs short guidance messages in a loop around multiple dimensions such as food inducement, interest rewards, health care, and emotional encouragement. After each round of persuasion, the system calls the multimodal semantic reasoning framework again: if the elderly person's willingness changes to Accept and an actual feeding is completed, it is recorded as a successful tiered empathic eating guidance; if the elderly person still refuses after reaching the maximum number of persuasion rounds (N_max=5), it is recorded as a failed tiered empathic eating guidance. Evaluation indicators include: (1) success rate of stratified empathic eating guidance (number of successful cases divided by the total number of empathic eating guidance scenarios × 100%); (2) average number of persuasion rounds (average number of persuasion rounds in successful scenarios); (3) average duration of stratified empathic eating guidance (average persuasion time in all triggering scenarios, including failed scenarios, unit: seconds).
[0064] The statistical results of the tiered empathic eating guidance behavior of the two groups of subjects are shown in the table below. In the HI group, tiered empathic eating guidance was triggered 9 times in 28 sessions (trigger rate 32.1%), with a total of 14 rounds of persuasion and an average time of 45.7 seconds per session. In the LI group, tiered empathic eating guidance was triggered 20 times in 28 sessions (trigger rate 71.4%), with a total of 71 rounds of persuasion and an average time of 54.6 seconds per session. In the 29 guidance scenarios in this experiment, the state transition was successfully completed 27 times within the upper limit of 5 rounds of persuasion (success rate 93.1%). This result indicates that the tiered empathic eating guidance method proposed in this paper has the potential to promote the elderly from a state of food refusal to a state of acceptance. The only two failures of stratified empathic feeding guidance in this experiment occurred in the LI group. In both failures, the system correctly identified the patient as a persistent refusal state in all rounds of persuasion and did not misjudge the situation. Based on the subjects' medical history, the two failures corresponded to two types of triggers: digestive dysfunction and severe cognitive impairment, respectively.
[0065] Table 6. Statistical Results of Layered Empathic Eating Guidance Behavior From the interaction process, the two groups of users exhibited different interaction characteristics. The HI group relied on language interaction to complete the change of intention, requiring an average of only 1.56 rounds of persuasion, but each round took a relatively long time (including multiple rounds of personalized dialogue generation and user response). The LI group, due to less language feedback, mainly relied on a cyclical guidance strategy, requiring an average of 3.55 rounds of persuasion, but each round of interaction time was shorter (only the guiding voice was played). The average time per session for the final tiered empathic eating guidance was not significantly different between the two groups (45.7s vs 54.6s), indicating that adopting differentiated strategies for users with different interaction abilities can complete the tiered empathic eating guidance task within a reasonable time frame.
[0066] The complete multimodal response resulted in two instances of misjudging acceptance as rejection in the HI group and three instances in the LI group. The two instances in the HI group and two in the LI group occurred during the initial judgment phase, while the additional instance in the LI group occurred during the re-judgment phase of the empathic feeding guidance process (a boundary state after multiple rounds of persuasion). This phenomenon suggests that highly interactive users exhibit more pronounced behavioral changes after accepting persuasion (richer verbal and behavioral feedback), making it easier for the system to capture their shift in intent. Conversely, low-interaction users, due to weaker and more gradual behavioral feedback, still have room for improvement in the accuracy of state perception during the hierarchical empathic feeding guidance process.
[0067] To further verify the effectiveness and human-computer interaction quality of the meal-assisting robot method of this invention from the perspective of subjective experience, the system conducted subjective scales and comparative evaluations from three levels: cognitive load, technology acceptance, and human-computer comparison. Assessment tools and scale design NASA-TLX Task Load Index Assessment: Used to assess the cognitive load level of target users during system interaction, including six sub-dimensions: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration, using a 7-point scale (1 point indicates very low, 7 points indicate very high). The Technology Acceptance Model (TAM) assessment is used to evaluate users' acceptance of and willingness to continue using meal-assisting robots. It covers eight dimensions: Feeding Effectiveness, Intention Understanding, Personalized Feeding, Interaction Naturalness, Emotional Comfort, Trust & Safety, Robot Acceptance, and Continuance Intention. A 5-point scale is used for scoring (1 point indicates strong disagreement, and 5 points indicate strong agreement). Comparative evaluation of care experience: The subjective experience of the meal assistance robot and human caregivers in the actual feeding service process was compared from four dimensions: feeding assistance, understanding of intentions, feeding persuasion and emotional affinity. A 5-point scale was used for scoring (1 point is very low and 5 points is very high).
[0068] Evaluation Results and Data Analysis Cognitive Load (NASA-TLX) analysis: such as Figure 7 As shown in the evaluation results, the NASA-TLX scores of the test subjects were mainly distributed between 1 and 4 points, and the overall subjective load was at a low level, indicating that the multimodal perception and hierarchical interaction design of the present invention did not impose an excessive cognitive and operational burden on the users. The scores on negative emotion dimensions such as frustration were extremely low (1-2 points), reflecting the good performance of digital human empathic interaction in alleviating anxiety and resistance in the elderly. Technology Acceptance (TAM) analysis: such as Figure 8As shown, user ratings for core functionalities such as feeding effectiveness, personalized feeding, and trust and safety were all concentrated in the range of 4-5 points (high satisfaction), indicating that the system's physical food delivery and personalized digital human care functions have been fully recognized by users. High scores were also achieved in the dimensions of interaction naturalness and willingness understanding, validating the practical value of natural language understanding and hierarchical persuasion strategies based on a large language model. Comparative analysis of robots and human caregivers: such as Figure 9 As shown, in terms of functional care, the meal-assisting robot exhibited performance close to that of human caregivers: its score in "Intention Understanding" remained stable at 4 points (compared to 4-5 points for human caregivers); its score in "Feeding Guidance" reached 4-5 points, with some users scoring on par with human caregivers (5 points); and its score in "Feeding Assistance" reached 4-5 points, even surpassing human care in some cases due to its precise and stable movements. In the "Emotional Affinity" dimension, the robot scored 3-4 points. While there is still room for improvement compared to skilled human caregivers, it has effectively established a warm and gentle emotional connection, fully demonstrating the significant feasibility and technological advantages of a tiered, empathetic feeding guidance mechanism in replacing or assisting human care.
[0069] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A layered empathetic assistive robot control method based on multi-modal intention recognition, characterized by, Includes the following steps: Collect the target user's voice signal, human body key point coordinates, and facial motion unit features; Voice events, action events, and facial expression events are generated based on the voice signal, the coordinates of the human body key points, and the facial motion unit features, respectively. The continuous numerical features of the voice events, action events, and facial expression events are converted into natural language semantic descriptions, and the natural language semantic descriptions are temporally aggregated within a sliding window to obtain a scene semantic summary. The semantic summary of the scene is input into a large language model for cross-modal reasoning to determine the target user's current dining intention category; When the willingness to eat is determined to be "acceptable", the robotic arm is controlled to perform the food picking and delivery actions. When the dining intention category is determined to be rejection, the interaction ability level of the target user is read; wherein, the interaction ability level includes high interaction and low interaction, the high interaction is that the communication function is non-existent or slightly limited and the cognitive function is normal, and the low interaction is that there is moderate or above communication difficulty or cognitive decline. When the interaction capability level is high, the reasons for the target user's food refusal are analyzed by combining a large language model with health records, social relationships, and personal preference information from the user database, wherein: For refusal to eat caused by physical factors, we will customize health rehabilitation and maintenance communication scripts. For food refusal caused by emotional factors, we can stimulate the willingness to eat by combining entertainment preferences with fun incentives after meals. For food refusal caused by objective factors, provide emotional support and guidance by combining the image and voice cloning of relatives; The large language model generates targeted, multi-turn conversational persuasion content, presents a digital human image through an interactive interface, and drives voice output to provide conversational eating guidance. When the interaction capability level is low, in response to the inability to establish effective language question-and-answer feedback, the control system switches to the loop guidance mode and controls the digital human to play the guidance text in a loop to guide eating. After guiding the user to eat, the system re-determines the target user's willingness to eat, and when the willingness to eat changes to accept, it controls the robotic arm to perform the eating and delivery actions. When the feeding guidance reaches the maximum number of persuasion rounds and the dining intention category remains in refusal, it is determined that the feeding is continuously refused, and the robotic arm is controlled to return to the initial position and a prompt is triggered. 2.The hierarchical co-empathic assistive robot control method based on multi-modal intention recognition according to claim 1, wherein, Generating the voice event specifically includes: The speech signal is transcribed into text through real-time speech recognition, and speech rate, volume, fundamental frequency and pitch are extracted simultaneously. Speech events with timestamps are generated through speech activity detection. 3.The hierarchical co-empathic assistive robot control method based on multi-modal intention recognition according to claim 1, wherein, Generating the action event specifically includes: Based on the coordinates of the human body's key points, the system identifies eating-related behaviors such as nodding, shaking, tilting the head, opening the mouth, and waving the hands through temporal motion analysis, and outputs action events with timestamps.
4. The hierarchical empathic meal-assisting robot control method based on multimodal intention recognition according to claim 1, characterized in that, Generating the facial expression event specifically includes: The facial motion unit features are extracted. The facial motion unit includes inner eyebrow raising AU1, outer eyebrow raising AU2, eyebrow frowning AU4, cheek lifting AU6, nose root wrinkling AU9, and mouth corner pulling AU12. Facial expression events with timestamps are output.
5. The hierarchical empathic meal-assisting robot control method based on multimodal intention recognition according to claim 1, characterized in that, The step of converting the continuous numerical features of the voice event, the action event, and the facial expression event into a natural language semantic description specifically includes: The speech rate in the speech event is mapped to semantic labels of slow speech rate, normal speech rate, or fast speech rate through a semantic mapping function. The volume in the voice event is mapped to semantic labels of low volume, normal volume, and high volume using a semantic mapping function. The intonation in the speech event is mapped into semantic tags of rising intonation, falling intonation, and steady intonation through a semantic mapping function; The average intensity of each facial action unit in the facial expression event within the sliding window is compared with a preset activation threshold. When the average intensity is greater than the preset activation threshold, it is determined as the activation intensity of the facial action unit. The activation intensity of the facial action unit is then mapped to a semantic expression containing the name and intensity of the facial action unit through a semantic mapping function. The temporal changes in the coordinates of key human body points in the action event are mapped to preset action categories through a semantic mapping function, and the duration of the preset action category and its coverage within the sliding window are mapped to a structured description of the action.
6. The hierarchical empathic meal-assisting robot control method based on multimodal intention recognition according to claim 1, characterized in that, The step of performing temporal aggregation of the natural language semantic description within a sliding window to obtain a scene semantic summary specifically includes: Within a continuously advancing sliding window of fixed width, events falling within the sliding window are acquired; the semantic content and prosodic features corresponding to the speech events within the sliding window are summarized. The duration of each behavior within the sliding window is recorded, and the sliding window coverage is calculated to generate an action behavior summary. Calculate the average activation value of facial motion units within the sliding window and the trend of changes across the window to generate an expression summary; The aggregated results of the voice events, action events, and facial expression events within the sliding window are concatenated into a scene semantic summary using a structured concatenation operator.
7. The hierarchical empathic meal-assisting robot control method based on multimodal intention recognition according to claim 1, characterized in that, The step of inputting the scene semantic summary into a large language model for cross-modal reasoning to determine the target user's current dining intention category specifically includes: The percentage of time during which voice, motion, and facial expressions are active within the sliding window is calculated as a temporal continuity factor. The weighted confidence is calculated based on the modal confidence, corresponding decision weight, and corresponding temporal continuity factor of the independent outputs of speech, action, and facial expression. The decision weight of speech is higher than that of action, and the decision weight of action is greater than that of facial expression. The sum of the decision weights of the three is 1. The scene semantic summary, the weighted confidence score, and the preset prompt template are input into the large language model to infer and determine the target user's current dining intention category.
8. The hierarchical empathic meal-assisting robot control method based on multimodal intention recognition according to claim 1, characterized in that, The control robotic arm performs food-grabbing and delivery actions, specifically including: Identify the dishes in the plate area based on an object detection model; The system uses a large language model to determine the feeding strategy based on the dish name; it then controls the end effector of the robotic arm to execute the feeding strategy and deliver the food to the target user's mouth. During delivery, force control sensors installed at the end of the robotic arm monitor changes in force. When an abnormal change in force occurs, the robotic arm is controlled to stop the delivery action and retract to a safe position, while an alarm is issued.
9. A hierarchical empathic meal-assisting robot control system based on multimodal willingness recognition, characterized in that, include: The sensing device, including a microphone and a depth camera, is used to collect the target user's voice signals, human body key point coordinates, and facial motion unit features; Interactive devices, including touch screens and speakers, are used to present digital human figures and output dialogue or guiding voices; The mechanical actuator includes a six-degree-of-freedom robotic arm, an end effector, and a force control sensor located at the end of the robotic arm; A controller is connected to the sensing device, the interaction device, and the mechanical actuator. The controller includes a memory and a processor. The memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Robot control method, device and equipment based on emotion enhancement and medium
CN121973192A
Control method of feeding robot based on multi-modal large model
CN121179437A
Method of group cognitive-behavioural therapy for obesity in women
RU2838785C1
