A multi-modal oral cavity perception and personalized feeding pose estimation method for bedridden hemiplegic subjects and a system thereof

Abstract

The application discloses a mouth multi-modal perception and personalized feeding pose estimation method for bedridden hemiplegic objects and a system thereof, and belongs to the technical field of nursing robots and robot-assisted feeding control. The affected side, the healthy side and the lateral lying direction of the cared object are acquired; images are collected through double RGB-D units, a first camera is arranged on the front side of the bedside to acquire the overall facial region, and a second camera is arranged on the bed side to compensate for the healthy side corner occlusion; key points and head posture are recognized, a mapping from the image to the human body anatomical coordinate is established according to the camera external parameter and the head posture, and the healthy side corner is determined as a candidate delivery target; an object-specific mouth-face asymmetry model is called; candidate points and quality indicators of each camera are calculated, and a target point is obtained through weighted fusion; the target point, a recommended entering direction and an accessible window are calculated, and a standardized pose package containing deliverable confidence and risk labels is output. If the confidence is insufficient or the target point drifts, the corner is occluded, the mouth is not opened or the risk of leakage increases, a pause feeding or manual review flag is output. The application can stably output the healthy side corner pose under lateral lying, head and facial rotation and local occlusion, and trigger pause or manual review when the risk increases.

Description

Technical Field

[0001] This invention belongs to the fields of nursing robots, computer vision, artificial intelligence perception, RGB-D depth perception, multimodal signal processing, bedside edge computing, and robot-assisted feeding control technology. Specifically, it relates to a method and system for oral multimodal perception and personalized feeding pose estimation for bedridden hemiplegic patients. Background Technology

[0002] Existing meal-assisting robots, nursing robots, or mouth-interaction devices typically identify the mouth region of a person using a monocular camera, RGB-D camera, or facial landmark detection algorithms, and use the center of the mouth, the center of the lips, or the geometric center of the face as a reference target for food delivery, object handing, or interaction. For individuals eating in a normal seated position, this type of method can complete mouth localization under a relatively stable frontal facial posture; some solutions also combine a depth camera to obtain the three-dimensional coordinates of the mouth and convert the visual localization results into target points in the robotic arm's coordinate system.

[0003] For bedridden hemiplegic patients, real-world care scenarios are more complex. The patient may be in a position with the head of the bed elevated and lying on their side, resulting in rotation, tilting, and partial occlusion of the head and face relative to the bed and robot's coordinate system. Hemiplegia or hemiplegic-like motor disorders can cause inconsistent mouth closure, facial asymmetry, drooling, food leakage, and differences in mouth opening coordination. Bedding, pillows, caregiver's hands, tableware, and the robotic arm's end effector can also periodically obscure the mouth area, making single-camera mouth centering prone to target point jumps or incorrect delivery in such scenarios.

[0004] The standard mouth center positioning method does not distinguish between the affected and healthy sides, and it is easy to set the target on the side with weaker closure ability or higher risk of leakage, which does not meet the nursing needs of hemiplegic patients to be fed small amounts slowly from the corner of the mouth on the healthy side.

[0005] Single-camera positioning is not stable enough under conditions of side-lying, head tilting, and partial occlusion; when one corner of the mouth is obscured by a pillow, blanket, tableware, or robotic arm, it is prone to missing depth data, misidentification of key points, and target drift.

[0006] Existing mouth recognition results mostly remain at the level of two-dimensional key points or three-dimensional coordinates, lacking recommended entry direction, spoon surface orientation, reachable window, and delivery confidence that can be directly used by the robot control terminal.

[0007] Existing methods typically lack object-specific oral-facial models, making it difficult to incorporate information such as long-term facial asymmetry, mouth corner closure ability, oral opening and closing range, drooling, and food leakage risk into a unified estimate.

[0008] If the sensing module and the robot control module are too tightly coupled, it will be difficult to upgrade subsequent functions such as accidental aspiration warning, slow feeding in human-in-loop environment and safety pause independently, and the boundaries of patent protection will also be easily confused. Summary of the Invention

[0009] This invention provides a method and system for oral multimodal perception and personalized feeding pose estimation for bedridden hemiplegic patients. When the bedridden hemiplegic patient is in a side-lying feeding position with head and face rotated and obstructed, how to stably identify the state of the healthy side of the mouth corner and oral surface, and output a personalized feeding pose with confidence, entry direction and reachability window, so that the robot control end can complete safe meal delivery without repeatedly understanding the complex oral surface state.

[0010] This invention is achieved through the following technical solution: A method for oral multimodal perception and personalized feeding pose estimation for bedridden hemiplegic patients, the method comprising the following steps: Step 1: Obtain the affected side, healthy side, and the patient's side-lying position in bed; Step 2: Simultaneously acquire color images, depth maps, timestamps, intrinsic and extrinsic parameters of the face in a side-lying position using at least two RGB-D image acquisition units with complementary perspectives. The first RGB-D camera is positioned in front of the bed or diagonally in front of the headboard to obtain the overall visible area of ​​the subject's face. The second RGB-D camera is positioned beside the bed or near the robotic arm to compensate for insufficient observation of the corner of the mouth, the mouth fissure area, or partially occluded areas on the healthy side in the side-lying position. The data acquired by the two cameras are synchronized in time and transformed in coordinates within the same edge computing unit. Step 3: Based on the facial color image, identify facial key points, mouth center, left and right corners of the mouth, head posture, mouth opening and closing status, and occluded areas; Step 4: Based on the camera extrinsic parameters, head pose rotation matrix, and lateral lying direction, establish a mapping relationship from image coordinates to human anatomical coordinates, normalize the detected left / right corner of the mouth key points to the left and right corner of the mouth, and use the healthy side corner of the mouth as a candidate for delivery of the mouth corner; Step 5: Create or call the SubjectOralProfile, a model of oral-facial asymmetry specific to the object. The model includes the affected side, the healthy side, the baseline head posture, the relative positions of the left and right corners of the mouth, the center of the oral fissure, the mouth corner closure ability, the mouth opening range, the drooling risk level, the food leakage risk level, and the historical manual correction offset. Step 6: Calculate the candidate 3D points and multiple quality indicators of the healthy side corner of the mouth from each RGB-D image acquisition unit, and perform weighted fusion to obtain the fused target points of the healthy side corner of the mouth. P fuse ; Step 7: Calculate the final delivery target point by combining the mouth-to-face model, food characteristics, and the robot's reachable space. P target Recommended entry direction, spoon face orientation, permissible entry depth, and lateral safety boundaries; Step 8: Calculate the deliverability confidence score C deliver With risk labels, a standardized FeedingPosePacket pose package is output to the robot control terminal; Step 9: If the confidence level is insufficient or the target point drifts continuously, the corner of the mouth is obscured, the mouth is not open, or the risk of leakage increases, output a pause delivery or manual review flag.

[0011] Furthermore, step 1 specifically involves determining the affected side and the healthy side based on nursing records, limb activity assessments, manual input, or visual posture recognition. The lateral decubitus direction includes left lateral decubitus, right lateral decubitus, and semi-lateral decubitus, and is provided by nursing bed position data, bed surface pressure distribution, or visual posture recognition results.

[0012] Furthermore, step 5 specifically involves representing the asymmetric vector of the aperture as follows: D asym = [ d left , d right , h open , c close , r leak ] in d left and d right This indicates the spatial offset of the left and right corners of the mouth relative to the center of the mouth opening. h open Indicates the range of mouth opening and closing. c close This indicates a score for the ability to close the corners of the mouth. r leak This indicates a risk score for drooling or leakage.

[0013] Furthermore, step 6 specifically involves, for the k-th RGB-D image acquisition unit, calculating the overall confidence level: q k = normalize( w 1· s k + w 2. d k + w 3· o k + w 4. a k + w 5.t k ) in s k Indicates the reliability of key point detection. d k Indicates the availability of depth data. o k Indicates the occlusion of the reverse fraction. a k This indicates the degree of fit between the camera's viewing angle and the mouth surface normal, and is determined by the angle between the camera's optical axis and the local normal of the healthy side of the mouth corner. t k This indicates the time synchronization quality of the dual-channel RGB-D frames and is obtained by mapping the difference in the dual-channel timestamps. w 1 to w 5 represents the weighting parameter; When both cameras are working P fuse = Σ( q k · P k ) / Σ q k If only one camera's confidence level meets the threshold, the result of that camera is used and the deliverability confidence level is reduced; if both cameras' confidence levels are below the threshold, the deliverability status is output.

[0014] Furthermore, step 7 specifically involves calculating the final delivery target point by combining the mouth-surface model, food characteristics, and the robot's reachable space. P target Recommended entry direction d in Spoon side facing n spoon Allowed to enter depth d epthrange and lateral safety boundaries w indowxy , in P target by P fuse Based on this, a slight offset is made along the local normal and the recommended entry direction on the outer side of the corner of the mouth. The offset amount is determined by the mouth corner closure ability, mouth opening range, food viscosity, and single spoonful size. The recommended entry direction is to slowly enter from the outer side of the healthy side of the mouth towards the center of the oral fissure or the entrance of the oral cavity, avoiding the corner of the mouth, cheek, gums, nose, and identified obstructed areas on the affected side.

[0015] Furthermore, step 8 specifically involves calculating the deliverability confidence level. C deliver Risk labels: C deliver = σ ( a 1· Q p + a 2. Q s + a 3· Q o + a 4. Q m + a 5. Q r - a 6· E ) Where σ is the normalization function. Q p Indicates the quality of target point positioning. Q s Indicates the confidence level of the assessment of the affected and healthy sides. Q o Indicates the degree of obstruction visibility. Q m This represents the evaluation value of the mouth opening and closing ability, and is obtained by mapping the mouth corner closure ability score (cclose) and mouth opening amplitude (hopen) in SubjectOralProfile. Q r This indicates the accessibility rating of the implementing agency. E This indicates risk items such as drooling, food leakage, abnormal head movements, or changes in the target point. a 1 to a 6 represents the weight parameter.

[0016] Furthermore, step 8 specifically involves outputting a standardized FeedingPosePacket pose package, including targetPoint, orientation, approachVector, spoonNormal, depthRange, reachableWindow, deliverConfidence, riskLabels, and validFlag, for use by the robot control terminal. If the confidence level is insufficient, or the target point continuously drifts, the corner of the mouth is obscured, the mouth is not open, or the risk of leakage increases, a flag indicating that food delivery should be paused or manual review should be output. A per-mouth log (PerBiteLog) is recorded, with fields including biteIndex, timestamp, posePacket, confidence, riskLabels, manualCorrection, and robotAcceptFlag, supporting model updates and post-event traceability.

[0017] Furthermore, the priority for acquiring the affected side and the healthy side is as follows: first, read the nursing record or the information of the affected side entered by the caregiver; when the nursing record is missing or there is a conflict, collect the upper limb range of motion, facial symmetry, mouth corner closure ability and the caregiver's interactive confirmation results to determine the final affected and healthy side; The unified coordinate system is the bed coordinate system, the robot base coordinate system, or a visual coordinate system with a fixed transformation relationship with the robot base coordinate system. The three-dimensional points in each camera coordinate system are transformed to the unified coordinate system using camera calibration parameters. The dual RGB-D image acquisition units are both used to observe the face and mouth area of ​​the same object. The data acquired by the two cameras are fused with confidence weights for the same healthy side corner of the mouth in a unified coordinate system, rather than being used separately to observe the face and the object.

[0018] Furthermore, the healthy side and affected side information are derived from nursing records, medical record annotations, limb activity assessments, or manual input by caregivers, and serve as a prerequisite constraint for subsequent dual RGB-D confidence fusion and FeedingPosePacket output; The SubjectOralProfile, a model of object-specific mouth-to-face asymmetry, is either manually calibrated and initialized or automatically updated by manual correction labels and mouth-to-mouth logs (PerBiteLog) during multiple feeding processes.

[0019] A multimodal oral cavity perception and personalized feeding pose estimation system for bedridden hemiplegic patients is characterized in that the system uses the aforementioned multimodal oral cavity perception and personalized feeding pose estimation method for bedridden hemiplegic patients, and the system includes... The lateral information acquisition module is used to read nursing records, limb assessments, or caregiver input to determine the affected side, healthy side, and lateral decubitus position. The multi-view RGB-D acquisition module includes at least two RGB-D image acquisition units with complementary viewpoints, used to acquire facial color images, depth maps, timestamps, intrinsic parameters, and extrinsic parameters; The mouth and face state recognition module is used to detect key facial points, the center of the mouth opening, the left and right corners of the mouth, head posture, the opening and closing state of the mouth corners, and the occluded area; The Individualized Modeling module is used to create or call object-specific face-asymmetric models, such as SubjectOralProfile. The confidence fusion module is used to transform the three-dimensional candidate points of the corner of the mouth obtained by each image acquisition unit into a unified coordinate system, and to perform weighted fusion by combining key point confidence, depth validity, occlusion ratio, viewpoint quality and historical stability to obtain the target point of the corner of the mouth on the healthy side. The pose estimation module is used to output feeding pose estimation results based on the target point of the healthy side mouth corner, mouth surface state, food characteristics and robot reachability constraints; The interface output module is used to output a standardized FeedingPosePacket to the robot control terminal.

[0020] The beneficial effects of this invention are: This invention improves the traditional mouth center positioning to mouth corner pose estimation on the healthy side, which is more suitable for slow feeding scenarios in hemiplegic subjects where there is a significant difference in the ability to close the mouth corner on one side.

[0021] This invention uses complementary observations from dual RGB-D cameras and occlusion confidence fusion to maintain relatively stable mouth corner target estimation even when there is head deflection, partial occlusion, or single-path depth failure.

[0022] This invention records the affected and healthy sides of the subject, facial asymmetry, mouth opening range, mouth corner closure ability, and historical correction amount through SubjectOralProfile, so that the estimation of feeding posture for each feeding has an individualized basis.

[0023] This invention outputs pose-level data packets instead of just key point coordinates, enabling clear decoupling of the sensing end, robot trajectory planning end, force-controlled feeding end, accidental aspiration warning end, and human-in-the-loop verification end.

[0024] When the confidence of delivery is insufficient, the target point drifts continuously, the corner of the mouth is covered, the mouth is not open, or the risk of leakage increases, the system can output a pause delivery or manual review flag, thereby reducing the risk of misdelivery from the status perception level. Attached Figure Description

[0025] Figure 1 This is a system hardware architecture diagram of the present invention.

[0026] Figure 2 This is a deployment scenario diagram of the bedside side-lying feeding and multimodal sensing system of the present invention.

[0027] Figure 3 This is a deployment scenario diagram of the forward RGB-D port surface monitoring of the present invention.

[0028] Figure 4 This is a scene diagram of lateral mouth corner pose estimation and robot delivery according to the present invention.

[0029] Figure 5 This is a geometric relationship diagram between the dual RGB-D camera of the present invention and the corner of the mouth on the affected and healthy sides.

[0030] Figure 6 This is a schematic diagram of the individualized oral-surface model and dual-camera confidence fusion of the present invention.

[0031] Figure 7This is a schematic diagram of the feeding pose output and reachable window of the present invention.

[0032] Figure 8 This is a flowchart of the overall method of the present invention.

[0033] Figure 9 This is a flowchart of the interface data packet and port-by-port closed-loop process of the present invention. Detailed Implementation

[0034] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0035] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0036] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0037] The following is in conjunction with the appendix to this application specification. Figures 1-4 The technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0038] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0039] This invention can be deployed in the state perception subsystem of a bedside safe feeding system. The subsystem includes a multi-view RGB-D image acquisition unit, a camera mounting bracket or bedside mounting base, an edge computing unit, a care recipient information input terminal, an oral-facial state recognition module, a patient / healthy side and lateral decubitus orientation determination module, an individualized oral-facial modeling module, a dual-camera confidence fusion module, a feeding pose estimation module, and a data interface connected to a robot control terminal.

[0040] Step 1: Obtain the affected side, healthy side, and the patient's side-lying position in bed; Step 2: Simultaneously acquire color images, depth maps, timestamps, intrinsic and extrinsic parameters of the face in a side-lying position using at least two RGB-D image acquisition units with complementary perspectives. The first RGB-D camera is positioned in front of the bed or diagonally in front of the headboard to obtain the overall visible area of ​​the subject's face. The second RGB-D camera is positioned beside the bed or near the robotic arm to compensate for insufficient observation of the corner of the mouth, the mouth fissure area, or partially occluded areas on the healthy side in the side-lying position. The data acquired by the two cameras are synchronized in time and transformed in coordinates within the same edge computing unit. Step 3: Based on the facial color image, identify facial key points, mouth center, left and right corners of the mouth, head posture, mouth opening and closing status, and occluded areas; Step 4: Based on the camera extrinsic parameters, head pose rotation matrix, and lateral lying direction, establish a mapping relationship from image coordinates to human anatomical coordinates, normalize the detected left / right corner of the mouth key points to the left and right corner of the mouth, and use the healthy side corner of the mouth as a candidate for delivery of the mouth corner; Step 5: Create or call the SubjectOralProfile, a model of oral-facial asymmetry specific to the object. The model includes the affected side, the healthy side, the baseline head posture, the relative positions of the left and right corners of the mouth, the center of the oral fissure, the mouth corner closure ability, the mouth opening range, the drooling risk level, the food leakage risk level, and the historical manual correction offset. Step 6: Calculate the candidate 3D points and multiple quality indicators of the healthy side corner of the mouth from each RGB-D image acquisition unit, and perform weighted fusion to obtain the fused target points of the healthy side corner of the mouth. P fuse ; Step 7: Calculate the final delivery target point by combining the mouth-to-face model, food characteristics, and the robot's reachable space. P target Recommended entry direction, spoon face orientation, permissible entry depth, and lateral safety boundaries; Step 8: Calculate the deliverability confidence score C deliver With risk labels, a standardized FeedingPosePacket pose package is output to the robot control terminal; Step 9: If the confidence level is insufficient or the target point drifts continuously, the corner of the mouth is obscured, the mouth is not open, or the risk of leakage increases, output a pause delivery or manual review flag.

[0041] In a preferred deployment, a first RGB-D camera is positioned in front of the bed or diagonally in front of the headboard to capture the overall visible area of ​​the subject's face; a second RGB-D camera is positioned beside the bed or near the robotic arm to compensate for insufficient observation of the corner of the mouth, the mouth fissure area, or partially obscured areas on the healthy side when the subject is lying on their side. Both cameras acquire RGB images, depth maps, timestamps, intrinsic parameters, and extrinsic parameters, and time synchronization and coordinate transformation are performed within the same edge computing unit.

[0042] The above scene diagram corresponds to a multi-view perception deployment method in the bedside side-lying feeding state. The forward RGB-D camera is used to acquire overall facial information, and the side or near-mouth camera is used to compensate for the occlusion of the healthy side corner of the mouth area. The feeding execution end performs safe delivery based on the output healthy side corner of the mouth pose packet. System hierarchy division

[0043] Data processing and input: The original inputs of this invention include at least five categories: First, the patient's healthy side prior knowledge, which can be determined by nursing records, medical record annotations, limb activity assessments, or manual input by the caregiver; second, the lateral lying position in bed, including left lateral, right lateral, and semi-lateral positions, which can be provided by bed position data, bed surface pressure distribution, or visual posture recognition results; third, dual RGB-D camera data, including RGB images, depth maps, timestamps, camera intrinsic and extrinsic parameters; fourth, oral and facial state data, including facial key points, oral fissure center, left and right corners of the mouth, oral opening and closing amplitude, mouth corner closure status, drooling or food leakage risk characteristics, head posture angle, and occlusion confidence; and fifth, optional contextual inputs, including food characteristics, spoon quantity range, manual correction by the caregiver to the previous target point, and the patient's degree of mouth opening cooperation.

[0044] The preprocessing process includes time synchronization of the dual RGB-D cameras, conversion of 3D points in each camera coordinate system to the bedside robot base coordinate system or unified world coordinate system using camera calibration parameters; face region detection and facial key point extraction in the RGB images; hole filling, outlier removal and local smoothing in the depth map; and anatomical direction mapping of the left and right corners of the mouth in the image based on the lateral lying direction and head posture to avoid misjudgment of the affected and healthy sides due to screen flipping, head rotation or mirror input.

[0045] Identification of affected and healthy sides and mapping of anatomical orientation: The affected side of the patient is labeled "AffectedSide," and the healthy side is labeled "HealthySide," with values ​​of "Left" or "Right." The system prioritizes reading the affected side information from nursing records or input by caregivers. When nursing records are unavailable or conflicting, the system can collect data on upper limb range of motion, facial symmetry, mouth closure ability, and caregiver interaction confirmation to determine the final affected / healthy side.

[0046] In visual inspection, the left and right sides of an image are not necessarily equivalent to the left and right sides of an anatomical human body. The system establishes a mapping relationship from image coordinates to anatomical coordinates based on camera extrinsic parameters, the head pose rotation matrix Rhead, and the side-lying direction SideLyingDirection. This normalizes the detected left / right mouth corner keypoints to the left and right corners of the human body. If HealthySide corresponds to the right side of the human body, the right corner of the mouth is used as the candidate for delivery; if HealthySide corresponds to the left side of the human body, the left corner of the mouth is used as the candidate for delivery.

[0047] Object-specific face-asymmetric model: The system collects baseline oral-surface data of the subject in its current lateral recumbent position during the initial record-keeping or preparation phase before each formal feeding, and establishes a SubjectOralProfile. This model includes the affected side, the healthy side, the baseline head posture, the relative positions of the left and right corners of the mouth, the center of the oral fissure, the ability to close the corners of the mouth, the range of mouth opening, the risk level of drooling, the risk level of food leakage, and the historical manual correction offset.

[0048] In one embodiment, the aperture asymmetry vector Dasym can be represented as: D asym = [ d left , d right , h open , c close , r leak ] in, d left and d right This indicates the spatial offset of the left and right corners of the mouth relative to the center of the mouth opening. h open Indicates the range of mouth opening and closing. c close This indicates a score for the ability to close the corners of the mouth. r leakThis represents a risk score for drooling or leakage. This vector is not a purely mathematical model for protection outside the specific scene, but rather a data structure used by the system during cornering, target correction, and delivery confidence calculation.

[0049] Dual RGB-D camera confidence fusion: To address the localized occlusion caused by the lateral decubitus position, this invention employs a dual-camera complementary perspective for mouth corner estimation. For the first... k The system outputs candidate 3D points for the healthy side of the mouth corner from the road camera. P k Head posture R k Mouth corner test score s k Deep efficiency d k Occlusion of reverse fractions o k Viewpoint Angle Fit Score a k Time-synchronized scores t k The camera's overall confidence level q k It can be obtained by weighting multiple quality indicators.

[0050] When both cameras are active, the system will P k After being converted to a unified coordinate system, weighted fusion is performed. When the confidence of only one camera meets the threshold, the system adopts the result of that camera and reduces the deliverability confidence. When both cameras are below the threshold, the system outputs a "not deliverable" status, prompting the robot control terminal to pause food delivery and wait for the caregiver to adjust the body position or remove the obstruction.

[0051] Estimation of target point at the corner of the mouth on the healthy side and feeding posture: The system uses the fused healthy side corner of the mouth point P fuse Based on this, the final delivery target point is generated according to the object-specific mouth face model and food characteristics. P target To avoid utensils directly pressing against the corners of the mouth or touching the gums, P target A slight offset can be made along the local normal and the recommended entry direction on the outer side of the corner of the mouth. The amount of offset is determined by the mouth corner closure ability, mouth opening range, food viscosity, and the amount of food scooped per mouth.

[0052] The feeding pose includes the position vector, pose quaternion or rotation matrix, and recommended approach direction. d in Spoon side facing n spoon Allowed to enter depthd epthrange and lateral safety boundaries w indowxy and deliverability confidence C deliver The recommended approach is to slowly enter from the outer corner of the healthy side of the mouth towards the center of the oral fissure or the entrance of the mouth, avoiding the corner of the mouth, cheek, gums, nose, and identified obstructed areas on the affected side. Step S101: Read nursing records, limb assessments, or caregiver input to determine the affected side, healthy side, and lateral decubitus position; Step S102: Simultaneously acquire dual RGB-D images, depth maps, camera parameters, and current head pose; Step S103: Detect key facial points, center of the mouth opening, left and right corners of the mouth, mouth opening and closing range, and occluded areas; Step S104: Based on the head pose of the camera external participants, map the left and right sides of the image to the left and right sides of the human anatomy to determine the candidate targets of the healthy side corner of the mouth; Step S105: Establish or call the object-specific mouth-face asymmetry model, and read the mouth corner closure capability, mouth opening range, and historical correction amount; Step S106: Calculate the overall confidence of the candidate results of each camera and complete the weighted fusion of the three-dimensional target points of the two cameras; Step S107: Combining the mouth-face model, food characteristics, and robot reachable space, calculate... P target , d in , n spoon , d epthrange and w indowxy ; Step S108: Calculate the deliverability confidence score C deliver And risk labels; if the conditions are not met, output a pause or manual review flag; Step S109: Output a standardized FeedingPosePacket to the robot control terminal and record a log message.

[0053] Data Structure and Interface Design: To decouple the sensing end from the robot control end, this invention defines a standardized data structure and interface. The robot control end only needs to read the target point, posture, recommended entry direction, reachable window, confidence level, and risk label from the FeedingPosePacket to determine whether to proceed with food delivery trajectory planning or pause for manual review.

[0054]

[0055] Model training, inference, and deployment optimization: If a deep learning model is used to detect facial landmarks, corners of the mouth, or estimate occlusion, it can be trained using bedside oral-facial image data that includes different lateral lying positions, head turning angles, degrees of occlusion, lighting conditions, and degrees of facial asymmetry. Training labels may include the left / right corners of the mouth (anatomical), the center of the oral fissure, the state of the mouth opening and closing, the occluded area, the head posture angle, and the corner of the mouth delivered from the healthy side (verified manually).

[0056] To meet the real-time requirements of the bedside feeding system, latency can be reduced during the inference phase by employing methods such as ROI pruning, lightweight keypoint modeling, local depth map sampling, temporal smoothing of candidate points, adaptive camera frame rate, and co-deployment of edge computing. Display images, debugging waveforms, and visualization diagrams are not included in the final decision-making process at the control end; pose output is based on timestamped data packets to ensure the reproducibility of the fusion judgment and the integrability of the engineering.

[0057] Confidence fusion formula: For the k RGB-D image acquisition unit, overall confidence level q k It can be calculated using the following formula: q k = normalize( w 1· s k + w 2. d k + w 3· o k + w 4. a k + w 5. t k ) in s k Indicates the reliability of key point detection. d k Indicates the availability of depth data. o k Indicates the occlusion of the reverse fraction. a k This indicates the degree of fit between the camera's viewing angle and the mouth surface normal, and is determined by the angle between the camera's optical axis and the local normal of the healthy side of the mouth corner. t k This indicates the time synchronization quality of the dual-channel RGB-D frames and is obtained by mapping the difference in the dual-channel timestamps. w 1 to w 5 represents the weight parameter.

[0058] When all multiple cameras are active, the fused target point at the corner of the mouth on the healthy side can be represented as: P fuse = Σ( q k · P k ) / Σ q k The above expression can be generalized to two or more RGB-D cameras. For occluded, depth-invalid, or excessively tilted head views, the corresponding... q k Reduced; for viewpoints with high visibility of the corner of the mouth on the healthy side, continuous depth, and good time synchronization quality, the corresponding q k Increase.

[0059] Deliveryability confidence formula: In a preferred embodiment, confidence can be delivered. C deliver The quality of mouth corner localization, the quality of determining the affected and healthy sides, the visibility of obstruction, the ability of the mouth corner to open and close, the robot's accessibility, and risk factors are jointly determined: C deliver = σ ( a 1· Q p + a 2. Q s + a 3· Q o + a 4. Q m + a 5. Q r - a 6· E ) Where σ is the normalization function. Q p Indicates the quality of target point positioning. Q s Indicates the confidence level of the assessment of the affected and healthy sides. Q o Indicates the degree of obstruction visibility. Q m This represents the evaluation value of the mouth opening and closing ability, and is obtained by mapping the mouth corner closure ability score (cclose) and mouth opening amplitude (hopen) in SubjectOralProfile. Q r This indicates the accessibility rating of the implementing agency.E This indicates risk items such as drooling, food leakage, abnormal head movements, or changes in the target point. a 1 to a 6 represents the weight parameter.

[0060] Pose output representation: The feeding pose estimation result of the present invention can be expressed as: T feed = { p , R , W , d , C} in, p Represents the three-dimensional coordinates of the target point on the healthy side of the mouth in a unified coordinate system; R Indicates the recommended orientation of the utensils or the end of the feeding device; W This indicates the accessible window that the executing agency is allowed to enter; d Indicates the recommended direction; C This indicates the confidence level of delivery capability. T feed It is not a single key point coordinate, but a comprehensive state output for the robot control end.

[0061] Elevate the head of the bed and feed while the baby is lying on their left side: The patient was in a head-elevated, left-side lying position, but the nursing record showed right-sided hemiplegia. The system identified the left corner of the mouth as a candidate delivery target for the healthy side. A forward-facing RGB-D camera acquired overall facial key points and head posture, while a lateral RGB-D camera compensated for depth information in the mouth corner area. After converting the two mouth corner candidate points to the robot's base coordinate system, the system performed confidence fusion and output the left healthy side mouth corner. P target And the recommended direction is to enter from the outside and diagonally in front of the corner of the mouth. If C deliver If the value exceeds the threshold, the robot control system will perform slow, small-volume delivery; if a sudden change in the target point is detected or the corner of the mouth is obscured by a blanket, the process will pause and prompt the caregiver to make adjustments.

[0062] Single-view perspective temporarily disabled: When one of the cameras is briefly obstructed by a caregiver's hand or tableware, the obstruction score (ok) for that camera's viewpoint decreases, and the overall confidence score (qk) decreases. If the other camera can still stably observe the corner of the mouth on the healthy side, the system uses the valid viewpoint result and reduces the overall Cdeliver score; if both cameras are below the threshold, validFlag is set to false, and the control terminal does not issue new delivery trajectories.

[0063] The number of RGB-D acquisition units can be two, or it can be expanded to three or more, as long as complementary perspectives and unified coordinate fusion can be formed.

[0064] A unified coordinate system can be the bed coordinate system, the robot base coordinate system, or a vision coordinate system that has a fixed transformation relationship with the robot base coordinate system.

[0065] Individualized mouth-face models can be initialized manually or automatically updated by manual correction labels and mouth-to-mouth logs during multiple feeding processes.

[0066] The deliverability confidence threshold can be adaptively adjusted based on the object's status, food characteristics, and care risk level.

[0067] The interface can take the form of ROS2 topics, HTTP / REST, shared memory, industrial bus, or internal message structure of the robot controller, without affecting the core protection concept of this invention.

Claims

1. A method for oral multimodal perception and personalized feeding pose estimation for bedridden hemiplegic patients, characterized in that, The method includes the following steps: Step 1: Obtain the affected side, healthy side, and the patient's side-lying position in bed; Step 2: Simultaneously acquire color images, depth maps, timestamps, intrinsic and extrinsic parameters of the face in a side-lying position using at least two RGB-D image acquisition units with complementary perspectives. The first RGB-D camera is positioned in front of the bed or diagonally in front of the headboard to obtain the overall visible area of ​​the subject's face. The second RGB-D camera is positioned beside the bed or near the robotic arm to compensate for insufficient observation of the corner of the mouth, the mouth fissure area, or partially occluded areas on the healthy side in the side-lying position. The data acquired by the two cameras are synchronized in time and transformed in coordinates within the same edge computing unit. Step 3: Based on the facial color image, identify facial key points, mouth center, left and right corners of the mouth, head posture, mouth opening and closing status, and occluded areas; Step 4: Based on the camera extrinsic parameters, head pose rotation matrix, and lateral lying direction, establish a mapping relationship from image coordinates to human anatomical coordinates, normalize the detected left / right corner of the mouth key points to the left and right corner of the mouth, and use the healthy side corner of the mouth as a candidate for delivery of the mouth corner; Step 5: Create or call the SubjectOralProfile, a model of oral-facial asymmetry specific to the object. The model includes the affected side, the healthy side, the baseline head posture, the relative positions of the left and right corners of the mouth, the center of the oral fissure, the mouth corner closure ability, the mouth opening range, the drooling risk level, the food leakage risk level, and the historical manual correction offset. Step 6: Calculate the candidate 3D points and multiple quality indicators of the healthy side corner of the mouth from each RGB-D image acquisition unit, and perform weighted fusion to obtain the fused target points of the healthy side corner of the mouth. P fuse ; Step 7: Calculate the final delivery target point by combining the mouth-to-face model, food characteristics, and the robot's reachable space. P target Recommended entry direction, spoon face orientation, permissible entry depth, and lateral safety boundaries; Step 8: Calculate the deliverability confidence score C deliver With risk labels, a standardized FeedingPosePacket pose package is output to the robot control terminal; Step 9: If the confidence level is insufficient or the target point drifts continuously, the corner of the mouth is obscured, the mouth is not open, or the risk of leakage increases, output a pause delivery or manual review flag.

2. The method according to claim 1, characterized in that, Specifically, step 1 involves determining the affected and healthy sides based on nursing records, limb activity assessments, manual input, or visual posture recognition. The lateral decubitus direction includes left lateral decubitus, right lateral decubitus, and semi-lateral decubitus, and is provided by nursing bed position data, bed surface pressure distribution, or visual posture recognition results.

3. The method according to claim 1, characterized in that, Specifically, step 5 involves representing the asymmetric vector of the aperture as follows: D asym = [ d left , d right , h open , c close , r leak ] in d left and d right This indicates the spatial offset of the left and right corners of the mouth relative to the center of the mouth opening. h open Indicates the range of mouth opening and closing. c close This indicates a score for the ability to close the corners of the mouth. r leak This indicates a risk score for drooling or leakage.

4. The method according to claim 1, characterized in that, Step 6 specifically involves, for the k-th RGB-D image acquisition unit, calculating the overall confidence level: q k = normalize( w 1· s k + w 2· d k + w 3· o k + w 4· a k + w 5· t k ) in s k Indicates the reliability of key point detection. d k Indicates the availability of depth data. o k Indicates the occlusion of the reverse fraction. a k This indicates the degree of fit between the camera's viewing angle and the mouth surface normal, and is determined by the angle between the camera's optical axis and the local normal of the healthy side of the mouth corner. t k This indicates the time synchronization quality of the dual-channel RGB-D frames and is obtained by mapping the difference in the dual-channel timestamps. w 1 to w 5 represents the weighting parameter; When both cameras are working P fuse = Σ( q k · P k ) / S q k If only one camera's confidence level meets the threshold, the result of that camera is used and the deliverability confidence level is reduced; if both cameras' confidence levels are below the threshold, the deliverability status is output.

5. The method according to claim 1, characterized in that, Step 7 specifically involves calculating the final delivery target point by combining the mouth-face model, food characteristics, and the robot's reachable space. P target Recommended entry direction d in Spoon side facing n spoon Allowed to enter depth d epthrange and lateral safety boundaries w indowxy , in P target by P fuse Based on this, a slight offset is made along the local normal and the recommended entry direction on the outer side of the corner of the mouth. The offset amount is determined by the mouth corner closure ability, mouth opening range, food viscosity, and single spoonful size. The recommended entry direction is to slowly enter from the outer side of the healthy side of the mouth towards the center of the oral fissure or the entrance of the oral cavity, avoiding the corner of the mouth, cheek, gums, nose, and identified obstructed areas on the affected side.

6. The method according to claim 1, characterized in that, Specifically, step 8 involves calculating the deliverability confidence level. C deliver Risk labels: C deliver = σ ( a 1· Q p + a 2· Q s + a 3· Q o + a 4· Q m + a 5· Q r - a 6· E ) Where σ is the normalization function. Q p Indicates the quality of target point positioning. Q s Indicates the confidence level of the assessment of the affected and healthy sides. Q o Indicates the degree of obstruction visibility. Q m This represents the evaluation value of the mouth opening and closing ability, and is obtained by mapping the mouth corner closure ability score (cclose) and mouth opening amplitude (hopen) in SubjectOralProfile. Q r This indicates the accessibility rating of the implementing agency. E This indicates risk items such as drooling, food leakage, abnormal head movements, or changes in the target point. a 1 to a 6 represents the weight parameter.

7. The method according to claim 1, characterized in that, Step 8 specifically involves outputting a standardized FeedingPosePacket, including targetPoint, orientation, approachVector, spoonNormal, depthRange, reachableWindow, deliverConfidence, riskLabels, and validFlag, for use by the robot control terminal. If the confidence level is insufficient, the target point drifts continuously, the corner of the mouth is obscured, the mouth is not open, or the risk of leakage increases, a flag indicating that food delivery should be paused or manual review should be output. A per-mouth log (PerBiteLog) is recorded, with fields including biteIndex, timestamp, posePacket, confidence, riskLabels, manualCorrection, and robotAcceptFlag, supporting model updates and post-event traceability.

8. The method according to claim 1, characterized in that, The priority for obtaining the affected and healthy sides is as follows: first, read the nursing records or the information of the affected side entered by the caregiver; when the nursing records are missing or conflicting, collect the range of upper limb movement, facial symmetry, mouth corner closure ability and the results of the caregiver's interactive confirmation to determine the final affected and healthy sides. The unified coordinate system is the bed coordinate system, the robot base coordinate system, or a visual coordinate system with a fixed transformation relationship with the robot base coordinate system. The three-dimensional points in each camera coordinate system are transformed to the unified coordinate system using camera calibration parameters. The dual RGB-D image acquisition units are both used to observe the face and mouth area of ​​the same object. The data acquired by the two cameras are fused with confidence weights for the same healthy side corner of the mouth in a unified coordinate system, rather than being used separately to observe the face and the object.

9. The method according to claim 1, characterized in that, The healthy and affected side information comes from nursing records, medical record annotations, limb activity assessments, or manual input by caregivers, and serves as a prerequisite constraint for subsequent dual RGB-D confidence fusion and FeedingPosePacket output; The SubjectOralProfile, a model of object-specific mouth-to-face asymmetry, is either manually calibrated and initialized or automatically updated by manual correction labels and mouth-to-mouth logs (PerBiteLog) during multiple feeding processes.

10. A multimodal oral cavity perception and personalized feeding pose estimation system for bedridden hemiplegic patients, characterized in that, The system uses the oral multimodal perception and personalized feeding pose estimation method for bedridden hemiplegic patients as described in any one of claims 1-9, and the system includes, The lateral information acquisition module is used to read nursing records, limb assessments, or caregiver input to determine the affected side, healthy side, and lateral decubitus position. The multi-view RGB-D acquisition module includes at least two RGB-D image acquisition units with complementary viewpoints, used to acquire facial color images, depth maps, timestamps, intrinsic parameters, and extrinsic parameters; The mouth and face state recognition module is used to detect key facial points, the center of the mouth opening, the left and right corners of the mouth, head posture, the opening and closing state of the mouth corners, and the occluded area; The Individualized Modeling module is used to create or call object-specific face-asymmetric models, such as SubjectOralProfile. The confidence fusion module is used to transform the three-dimensional candidate points of the corner of the mouth obtained by each image acquisition unit into a unified coordinate system, and to perform weighted fusion by combining key point confidence, depth validity, occlusion ratio, viewpoint quality and historical stability to obtain the target point of the corner of the mouth on the healthy side. The pose estimation module is used to output feeding pose estimation results based on the target point of the healthy side mouth corner, mouth surface state, food characteristics and robot reachability constraints; The interface output module is used to output a standardized FeedingPosePacket to the robot control terminal.

Images