A multi-modal physiological signal real-time warning and emergency feeding interruption system for bedridden eating and its working method

Abstract

This invention discloses a multimodal physiological signal aspiration / choking real-time early warning and emergency feeding interruption system and method for bedridden feeding. The system simultaneously acquires RGB images, depth images, and timestamps. It determines the laryngeal ROI based on facial and human posture key points, and filters based on depth validity, location rationality, and temporal continuity. Swallowing motion features are extracted through depth difference, relative displacement, velocity, and local optical flow. Simultaneously, rPPG respiratory rhythm and signal quality features are extracted based on facial RGB images. Dynamic weighted fusion is performed combining head and face posture and the quality of each channel to generate a swallowing risk score and level. Based on this, a feeding safety gating signal is output to control the robot to perform allow, pause, or stop feeding. This achieves non-contact swallowing and breathing monitoring without thermal imaging or contact sensors, improving the safety and reliability of automated bedridden feeding.

Description

Technical Field

[0001] This invention belongs to the field of medical assisted feeding robot technology, specifically relating to a multimodal physiological signal aspiration / choking real-time early warning and emergency feeding interruption system for bedridden feeding and its working method. Background Technology

[0002] Bedridden patients, patients with impaired swallowing function, and individuals requiring assisted feeding are prone to risks such as incomplete swallowing, impaired swallowing and breathing coordination, aspiration of food into the airway, and delayed choking response during feeding. In scenarios equipped with automated feeding arms or bedside assisted feeding devices, the feeding process typically includes stages such as food retrieval, delivery, waiting at the mouth, withdrawal, and preparation for the next bite. If the risk of aspiration or choking is not promptly identified during the waiting or delivery stage, the feeding mechanism may continue to advance the food, potentially increasing the risk. Therefore, feeding monitoring results need to be linked in real-time with the feeding mechanism, rather than simply providing post-event displays or alarms.

[0003] In existing technologies, monitoring of swallowing or choking mainly includes visual observation by caregivers, monocular camera observation, contact laryngeal motion sensors, neck accelerometers, laryngeal flexion sensors, surface electromyography, acoustic cough detection, breathing belts, or thermal imaging. Some published studies have demonstrated that swallowing movements can be quantified through laryngeal motion, oral motion, or depth sensing information.

[0004] In the field of non-contact physiological signal detection, remote photoplethysmography (rPPG) technology can extract blood flow-related signals from subtle changes in facial skin color over time, and further estimate heart rate, respiratory waveforms, or respiratory rate. Existing rPPG methods include CHROM, POS, ICA, PCA, and deep learning models such as PhysNet. For example, the paper "Robust pulse rate from chrominance-based rPPG, IEEE Transactions on Biomedical Engineering, 2013" discloses the CHROM method, the paper "Algorithmic Principles of Remote PPG, IEEE Transactions on Biomedical Engineering, 2017" discloses the POS method, and the paper "Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks, BMVC, 2019" discloses the PhysNet model. The above method demonstrates the feasibility of estimating physiological rhythms from RGB video, but when it is directly applied to the safety control of automatic feeding in bed, the coupling problem between swallowing events, robotic arm status and real-time control interface still needs to be solved.

[0005] In terms of robot system integration, ROS2 provides standardized communication interfaces for perception, decision-making, and execution modules through node, topic, message timestamp, tf2 coordinate system, and message_filters time synchronization mechanisms. Existing feeding robots or bedside robot solutions typically develop visual detection, robotic arm motion control, and interface display as independent functional modules, but there is insufficient disclosure regarding the safety gate linkages between swallowing confirmation, respiratory phase, choking risk, and feeding interruption during the feeding process.

[0006] In summary, existing technologies have at least the following shortcomings: First, detection based on absolute depth changes in the larynx is easily affected by the patient's overall body movement, head rotation, and drift of the region of interest in the larynx; a single absolute depth change is insufficient to effectively distinguish between local larynx movements and the patient's overall body movements. Second, a single depth feature has limited ability to perceive small-amplitude local larynx movements. Third, when existing multimodal fusion uses fixed weights or simple rules, signal quality deteriorates when a sensing channel is obstructed, experiences head movement, loses depth, or changes in illumination; this low-quality modality may still significantly impact the final risk assessment. Fourth, contact breathing bands, pharyngeal sensors, neck accelerometers, or surface electromyography electrodes can affect patient comfort, nursing procedures, and robotic arm movements. Fifth, mixing display and computational data can transmit refresh delays, scaling issues, and rendering frame drops to the control link. Sixth, many solutions only output identification tags or alarm information, lacking a hierarchical safety gating mechanism for real-time linkage with the feeding actuator.

[0007] Therefore, the relevant technical field requires a non-contact, multimodal feeding safety monitoring solution that can stably locate the larynx region based on RGB-D data, extract local motion features of the larynx through depth difference between the larynx region and the reference region and local image optical flow, and combine respiratory rhythm, respiratory recovery features and signal quality obtained based on facial RGB images to dynamically weight and fuse different sensing channels for quality perception, and generate a comprehensive risk score, risk level and feeding safety gating signal. Summary of the Invention

[0008] This invention provides a multimodal physiological signal aspiration / choking real-time early warning and emergency feeding interruption system and its working method for bedridden feeding. It addresses the problems in automated bedridden feeding scenarios where the laryngeal region of interest is easily affected by head movement, body movement, and posture changes; single depth changes are difficult to distinguish between local laryngeal movements and overall body movements; and dynamic changes in the signal quality of different sensing modalities lead to insufficient reliability of fixed fusion strategies. Through adaptive generation and scoring of multiple candidate laryngeal regions of interest, reference region depth difference, relative displacement, relative velocity, local image optical flow, rPPG respiratory sensing, signal quality assessment, dynamic weight fusion, comprehensive risk scoring and risk level, and three-state feeding safety gating, a closed-loop safety control system in conjunction with a feeding robot is achieved.

[0009] This invention is achieved through the following technical solution: A multimodal physiological signal aspiration / choking real-time early warning and emergency feeding interruption system for bedridden feeding, the system includes an RGB-D camera 1, a camera bracket 6, an industrial control computer 2, a feeding robotic arm 3, an alarm unit 4, and an independent emergency stop unit 5; The RGB-D camera 1 is mounted on a camera bracket 6 next to the bed, with the optical axis of the RGB-D camera 1 facing the face, jaw and throat area of ​​the bedridden patient 7. The industrial control computer 2 is equipped with an RGB-D acquisition node 211, a laryngeal movement detection node 212, an rPPG breathing node 213, a risk fusion and safety gating node 214, a visualization node 215, and a feeding control node 216. The RGB-D acquisition node 211 acquires RGB frames and depth frames, and encapsulates the camera intrinsic parameters, extrinsic parameters, frame number and acquisition timestamp into a ROS2 message; The laryngeal motion detection node 212 acquires facial key points and human posture key points based on RGB-D data, generates multiple candidate laryngeal regions of interest, scores them based on depth validity, skin color prior, position rationality, and temporal continuity, and determines the target laryngeal region of interest; sets a reference region, calculates relative depth, relative laryngeal displacement, relative laryngeal velocity, and local vertical optical flow, obtains local laryngeal motion characteristics, motion risk, and motion signal quality, and publishes / swallow / state; The rPPG respiratory node 213 extracts rPPG signals based on the temporal color changes of the facial skin region, obtains respiratory waveform, respiratory rate, respiratory rhythm characteristics, respiratory recovery characteristics, respiratory signal quality and timestamp, and publishes / swallow / respiration; The risk fusion and safety gate node 214 subscribes to / swallow / state, / swallow / respiration and / feeding / arm_status, completes the time alignment before multimodal fusion through the time synchronization module 8, dynamically determines the fusion weight according to the signal quality of each sensing channel, performs dynamic weighted fusion of motion risk and respiratory risk, obtains comprehensive risk score and risk level, and publishes / swallow / safety_gate. The visualization node 215 is used to display RGB images, depth images, target throat region of interest, reference region, relative displacement of the throat, relative velocity of the throat, local optical flow characteristics, rPPG waveform, respiratory characteristics, comprehensive risk score, risk level, and safety gating status; The feeding control node 216 controls the feeding robotic arm 3 to perform actions such as allowing feeding, pausing feeding, or stopping feeding according to the safety door output interface 9; when feeding is allowed, the normal feeding action continues; when feeding is paused, the tableware is stopped from being pushed toward the patient's mouth and the risk is reassessed; when feeding is stopped, the current process is stopped and the end retraction, prohibition of the next food delivery, and alarm are performed according to the preset safety strategy. The independent emergency stop unit 5 acts directly on the feeding robot arm 3 through a hard-wired circuit, while simultaneously driving the alarm unit 4, and is set in parallel with the ROS2 software link.

[0010] A method for real-time early warning and emergency feeding interruption based on multimodal physiological signals for bedridden feeding, the method using the real-time early warning and emergency feeding interruption system for multimodal physiological signals for bedridden feeding as described in claim 1, the method comprising the following steps: Step S1: Collect color images, depth images, and corresponding timestamp data of the object being fed, and perform time-series synchronization processing.

[0011] Step S2: Generate multiple candidate regions of interest for the throat based on facial key points and human pose key points. Score the candidate regions based on depth validity, skin color prior, position rationality and temporal continuity, and determine the target throat region of interest.

[0012] Step S3: Calculate the relative depth, relative displacement, and relative velocity of the throat based on the depth difference between the region of interest and the reference region of the target throat, and obtain the vertical motion features of the throat by combining the optical flow of the local image.

[0013] Step S4: Extract rPPG signals based on the temporal color changes of the facial skin region, and obtain respiratory rhythm features, respiratory recovery features, and corresponding respiratory signal quality.

[0014] Step S5: Acquire laryngeal motion features, respiratory features, head and face posture, and signal quality of each sensing channel, and dynamically determine the corresponding fusion weights based on the signal quality of each sensing channel.

[0015] Step S6: Perform weighted fusion of motion risk and respiratory risk according to dynamic fusion weights to obtain a comprehensive risk score, and determine the risk level based on the comprehensive risk score.

[0016] Step S7: Generate a feeding safety gate signal based on the comprehensive risk score or risk level, and control the feeding robot to perform operations such as allowing feeding, pausing feeding, or stopping feeding.

[0017] Furthermore, step S1 specifically involves acquiring color image frames and depth image frames captured by the RGB-D camera, encapsulating each frame into a ROS2 message with a header.stamp timestamp, and performing precise or approximate time synchronization; the RGB-D camera is set on a bedside bracket so that its field of view covers the patient's face, jaw region, and throat region.

[0018] Furthermore, step S2 specifically involves performing camera intrinsic parameter calibration, depth-color image alignment, and invalid depth removal; generating multiple candidate regions of interest for the throat based on facial key points and human pose key points, and scoring them according to depth information, effective depth ratio, skin color prior, location rationality, and temporal continuity, selecting the region with the highest comprehensive score or that meets the preset scoring conditions as the target throat region of interest.

[0019] Further, step S3 specifically involves setting a reference region with a predetermined spatial relationship to the target throat region of interest, obtaining the representative depth D_throat(t) of the target throat region and the representative depth D_ref(t) of the reference region, calculating the relative depth D_rel(t), and obtaining the relative displacement disp(t) and relative velocity vel(t) of the throat based on the resting baseline and the time interval between adjacent frames; performing local image optical flow estimation within the target throat region to obtain the local vertical optical flow flow(t), thereby obtaining the local motion characteristics or motion risk of the throat.

[0020] Furthermore, step S4 specifically involves calculating the RGB color mean trajectory within the region of interest on the facial skin, performing color normalization, illumination change suppression, and motion artifact suppression, extracting the rPPG signal based on the temporal color changes in the facial skin region, and obtaining the respiratory waveform, respiratory rate, respiratory rhythm characteristics, respiratory recovery characteristics, and respiratory signal quality.

[0021] In one specific implementation, the rPPG signal can be obtained using POS, CHROM, ICA, PCA, PhysNet, or a combination thereof, and the respiratory waveform and respiratory rate can be derived from the amplitude modulation, frequency modulation, or respiratory sinus arrhythmia characteristics of the pulse wave.

[0022] Further, step S5 specifically involves inputting the / swallow / state message published by the laryngeal motion detection node, the / swallow / respiration message published by the rPPG breathing node, and the / feeding / arm_status message published by the feeding node into the risk fusion and safety gating node, and aligning them according to header.stamp using the precise or approximate time synchronization mechanism of message_filters; the time alignment serves as the basic data processing before multimodal fusion, and the aligned laryngeal motion features, breathing features, head and face posture status, and signal quality of each channel are then entered into the quality-aware dynamic risk fusion module, and the fusion weight is dynamically determined based on the signal quality of each channel.

[0023] Furthermore, step S6 specifically involves obtaining motion risk based on relative laryngeal displacement, relative velocity, and local vertical optical flow, and obtaining respiratory risk based on respiratory rate, respiratory signal quality, and respiratory recovery characteristics; and weighting and fusing motion risk and respiratory risk according to dynamic fusion weights to obtain a comprehensive risk score and map it to the corresponding risk level.

[0024] Furthermore, step S7 specifically involves the risk fusion and safety gating node issuing ALLOW_FEED (allow feeding), PAUSE_FEED (pause feeding), or STOP_FEED (stop feeding) safety gating messages to / swallow / safety_gate. When feeding is allowed, the feeding node continues normal operation; when feeding is paused, the feeding node stops the utensils from being pushed towards the mouth and waits for a risk reassessment; when feeding is stopped, the current feeding process is stopped, and the node executes actions such as withdrawing to a safe position, prohibiting the delivery of the next bite of food, triggering an audible and visual alarm, and recording an event log according to a preset safety policy. An independent hard-wired emergency stop device is set up in parallel with the ROS2 software link as the last level of safety protection.

[0025] The beneficial effects of this invention are: This invention generates multiple candidate regions of interest for the throat by using facial key points and human posture key points, and uses multiple indicators such as depth effectiveness, skin color prior, position rationality and time sequence continuity to improve the localization stability of the target throat region of interest under different head and face postures.

[0026] This invention obtains relative motion information by depth difference between the target laryngeal region of interest and the reference region, reducing interference from the patient's overall body movement, bed movement, and changes in the relative position of the patient and the camera on the detection of local laryngeal motion; the system relies only on a non-contact RGB-D camera and facial RGB video, reducing the impact on patient comfort and nursing operations.

[0027] This invention provides multidimensional visual evidence for small-amplitude local laryngeal movements through joint analysis of relative displacement, relative velocity, and local vertical optical flow. Various motion, breathing, and control states are conveyed using time-stamped ROS2 topics to avoid mixing display data with computational data.

[0028] This invention dynamically adjusts the fusion weights of different modalities by evaluating the signal quality of the laryngeal motion sensing channel and the respiratory sensing channel, thereby reducing the impact of low-quality sensing channels on the overall risk assessment. It can also be combined with ROI pruning, sliding window incremental calculation, and node parallelization to meet real-time control requirements.

[0029] This invention outputs three-state safety gating signals—allow feeding, pause feeding, and stop feeding—based on a comprehensive risk score and risk level, forming a closed-loop control with the feeding actuator; at the same time, it retains an independent hard-wired emergency stop as the last level of hardware safety protection in parallel with the software safety gating. Attached Figure Description

[0030] Figure 1 This is a system hardware architecture diagram of the present invention.

[0031] Figure 2 This is a flowchart of the multimodal swallowing risk fusion and feeding safety gating of the present invention.

[0032] Figure 3 This is a flowchart of the throat candidate region generation, relative depth difference, and local optical flow motion detection process of the present invention.

[0033] Figure 4 This is a schematic diagram of the ROS2 topic interface and its integration with the same machine in this invention.

[0034] Figure 5 This is the system performance verification process and performance index diagram of the present invention.

[0035] Figure 6 This is a deployment diagram of the lateral laryngeal swallowing monitoring camera of the present invention.

[0036] Figure 7 This is a deployment diagram of the front-facing face detection camera of the present invention.

[0037] Figure 8 This is a schematic diagram of the Orbbec 335L image sensor and its bedside deployment according to the present invention.

[0038] Figure 9 This is a scene diagram of the integrated dual-camera multimodal early warning and feeding interruption of the present invention.

[0039] 1. RGB-D camera; 2. Industrial computer or edge computing unit; 21. ROS2 node group; 211. RGB-D acquisition node; 212. Laryngeal motion detection node; 213. rPPG breathing node; 214. Risk fusion and safety gating node; 215. Visualization node; 216. Feeding control node; 3. Feeding robotic arm; 4. Alarm unit; 5. Independent emergency stop unit; 6. Camera bracket; 7. Bedridden patient; 8. Time synchronization module; 9. Safety gate output interface. Detailed Implementation

[0040] In the following description, specific details such as particular system structures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0041] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0042] It should also be understood that the terminology used in this application specification is for the purpose of describing particular examples only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0043] The following is in conjunction with the appendix to this application specification. Figure 1-9 The technical solutions described in this application are clearly and completely illustrated. Obviously, the described examples are only a part of the examples in this application, not all of them. Based on the examples in this application, all other examples obtained by a person skilled in the art without inventive effort are within the scope of protection of this application.

[0044] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific examples disclosed below.

[0045] Implementation Method 1 This embodiment provides a multimodal physiological signal aspiration / choking real-time early warning and emergency feeding interruption system for bedridden feeding. The system includes an RGB-D camera 1, a camera bracket 6, an industrial control computer 2, a feeding robotic arm 3, an alarm unit 4, and an independent emergency stop unit 5. The RGB-D camera 1 is mounted on a camera bracket 6 next to the bed, with the optical axis of the RGB-D camera 1 facing the face, jaw and throat area of ​​the bedridden patient 7. The industrial control computer 2 runs Ubuntu 22.04 and ROS2 Humble, and internally deploys RGB-D acquisition node 211, laryngeal movement detection node 212, rPPG breathing node 213, risk fusion and safety gating node 214, visualization node 215 and feeding control node 216. The RGB-D acquisition node 211 acquires RGB frames and depth frames through pyorbbecsdk, OpenCV or equivalent SDK, and encapsulates the camera intrinsic parameters, extrinsic parameters, frame number and acquisition timestamp into ROS2 messages. The laryngeal motion detection node 212 subscribes to RGB-D data, generates and scores the target laryngeal region of interest based on facial key points and human posture key points, and obtains relative displacement, relative velocity, vertical optical flow, motion risk and motion signal quality by combining reference region depth difference and local image optical flow, and publishes / swallow / state. The rPPG respiratory node 213 subscribes to RGB images or facial ROI features, extracts rPPG signals based on the temporal color changes of the facial skin region, obtains respiratory rhythm features, respiratory recovery features and respiratory signal quality, and publishes / swallow / respiration. The risk fusion and security gate node 214 subscribes to / swallow / state, / swallow / respiration and / feeding / arm_status, completes data alignment before multimodal fusion through the time synchronization module 8, dynamically determines the fusion weight based on the signal quality of each channel, obtains the comprehensive risk score and risk level, and then publishes / swallow / safety_gate; The visualization node 215 is used to display RGB images, depth images, target throat region of interest, reference region, throat relative displacement, throat relative velocity, local optical flow features, rPPG waveform, respiratory features, comprehensive risk score, risk level, and safety gating status; this node does not provide input to the risk fusion and safety gating node 214 to prevent interface refresh, image scaling, or rendering delay from affecting the computation link; The feeding control node 216 controls the feeding robotic arm 3 to execute the logic of allowing feeding, pausing feeding, or stopping feeding according to the safety door output interface 9. The independent emergency stop unit 5 acts directly on the feeding robot arm 3 through a hard-wired circuit, while simultaneously driving the alarm unit 4, and is set in parallel with the ROS2 software link.

[0046] like Figure 1As shown, the system in this embodiment includes an RGB-D camera 1, a camera bracket 6, an industrial computer 2, a feeding robotic arm 3, an alarm unit 4, and an independent emergency stop unit 5. The RGB-D camera 1 is mounted on the bedside bracket 6, with its optical axis facing the face, jaw, and throat area of ​​the bedridden patient 7. The industrial computer 2 runs Ubuntu 22.04 and ROS2 Humble, and internally deploys an RGB-D acquisition node 211, a laryngeal motion detection node 212, an rPPG breathing node 213, a risk fusion and safety gating node 214, a visualization node 215, and a feeding control node 216.

[0047] RGB-D acquisition node 211 acquires RGB and depth frames via pyorbbecsdk, OpenCV, or an equivalent SDK, and encapsulates camera intrinsic and extrinsic parameters, frame numbers, and acquisition timestamps into ROS2 messages. Laryngeal motion detection node 212 publishes ` / swallow / state`, and rPPG respiration node 213 publishes ` / swallow / respiration`. Risk fusion and safety gating node 214 completes data alignment before multimodal fusion via time synchronization module 8, dynamically determines fusion weights based on the signal quality of each channel, and obtains a comprehensive risk score and risk level, subsequently publishing ` / swallow / safetygate`. Feeding control node 216 executes actions to allow feeding, pause feeding, or stop feeding based on the safety gate output interface 9.

[0048] Visualization node 215 displays RGB images, depth images, target throat region of interest, reference region, relative throat displacement, relative throat velocity, local optical flow characteristics, rPPG waveform, respiratory characteristics, comprehensive risk score, risk level, and safety gating status. This node does not provide input to the risk fusion and safety gating node 214. The independent emergency stop unit 5 acts directly on the feeding robot arm 3 or its actuator via hardwired circuitry, and is configured in parallel with the ROS2 software link.

[0049]

[0050] To improve patient safety, this invention incorporates the following anomaly handling logic. When a single sensing channel experiences signal quality degradation due to invalid RGB-D depth, loss of the target laryngeal region of interest, invisibility of the facial region, decreased rPPG signal-to-noise ratio, or other factors, the risk fusion and safety gating node 214 reduces the dynamic fusion weight of that channel in the comprehensive risk calculation and writes a reason code into the reason_code field. When multiple sensing channels fail to provide valid information that meets preset quality requirements, or when the comprehensive risk score reaches the stop-feeding condition, a stop-feeding state is generated. When the comprehensive risk is in the intermediate range requiring further observation, a pause-feeding state is generated and awaits reassessment. When the comprehensive risk meets safety conditions, a allow-feeding state is generated.

[0051] Upon receiving a paused feeding status, feeding control node 216 halts the advancement of the feeding utensils towards the patient's mouth and awaits a risk reassessment. Upon receiving a stopped feeding status, it stops the current feeding process and, according to a preset safety strategy, withdraws the end effector to a safe position and prohibits the delivery of the next bite of food. Alarm unit 4 issues an audible and visual alert and simultaneously writes RGB-D segments, laryngeal movement characteristics, respiratory characteristics, comprehensive risk score, risk level, robotic arm status, and timestamps into the event log for subsequent analysis.

[0052] The independent emergency stop unit 5 is not implemented via ROS2 communication, but is instead connected to the feeding robot's driver or power control circuit via hardwired connection. When the caregiver presses the emergency stop button, the actuator's power is cut off or it enters a safe stop state. This independent emergency stop, in conjunction with the software safety gate, forms a two-layer safety mechanism.

[0053] Implementation Method 2 This embodiment provides a multimodal physiological signal aspiration / choking real-time early warning and emergency feeding interruption method for bedridden feeding. The method uses the multimodal physiological signal aspiration / choking real-time early warning and emergency feeding interruption system for bedridden feeding as described in Embodiment 1. The method includes the following steps: Step S1: Collect color images, depth images, and corresponding timestamp data of the object being fed, and perform time-series synchronization processing.

[0054] Step S2: Generate multiple candidate regions of interest for the throat based on facial key points and human pose key points. Score the candidate regions based on depth validity, skin color prior, position rationality and temporal continuity, and determine the target throat region of interest.

[0055] Step S3: Calculate the relative depth, relative displacement, and relative velocity of the throat based on the depth difference between the region of interest and the reference region of the target throat, and obtain the vertical motion features of the throat by combining the optical flow of the local image.

[0056] Step S4: Extract rPPG signals based on the temporal color changes of the facial skin region, and obtain respiratory rhythm features, respiratory recovery features, and corresponding respiratory signal quality.

[0057] Step S5: Acquire laryngeal motion features, respiratory features, head and face posture, and signal quality of each sensing channel, and dynamically determine the corresponding fusion weights based on the signal quality of each sensing channel.

[0058] Step S6: Perform weighted fusion of motion risk and respiratory risk according to dynamic fusion weights to obtain a comprehensive risk score, and determine the risk level based on the comprehensive risk score.

[0059] Step S7: Generate a feeding safety gate signal based on the comprehensive risk score or risk level, and control the feeding robot to perform operations such as allowing feeding, pausing feeding, or stopping feeding.

[0060] Furthermore, step S1 specifically involves acquiring color image frames and depth image frames captured by the RGB-D camera, encapsulating each frame into a ROS2 message with a header.stamp timestamp, and performing precise or approximate time synchronization; the RGB-D camera is set on a bedside bracket so that its field of view covers the patient's face, jaw region, and throat region.

[0061] Furthermore, step S2 specifically involves performing camera intrinsic parameter calibration, depth-color image alignment, and invalid depth removal; generating multiple candidate regions of interest for the throat based on facial key points and human pose key points, and scoring them according to depth information, effective depth ratio, skin color prior, location rationality, and temporal continuity, selecting the region with the highest comprehensive score or that meets the preset scoring conditions as the target throat region of interest.

[0062] Further, step S3 specifically involves setting a reference region with a predetermined spatial relationship to the target throat region of interest, obtaining the representative depth D_throat(t) of the target throat region and the representative depth D_ref(t) of the reference region, calculating the relative depth D_rel(t), and obtaining the relative displacement disp(t) and relative velocity vel(t) of the throat based on the resting baseline and the time interval between adjacent frames; performing local image optical flow estimation within the target throat region to obtain the local vertical optical flow flow(t), thereby obtaining the local motion characteristics or motion risk of the throat.

[0063] Furthermore, step S4 specifically involves calculating the RGB color mean trajectory within the region of interest on the facial skin, performing color normalization, illumination change suppression, and motion artifact suppression, extracting the rPPG signal based on the temporal color changes in the facial skin region, and obtaining the respiratory waveform, respiratory rate, respiratory rhythm characteristics, respiratory recovery characteristics, and respiratory signal quality.

[0064] In one specific implementation, the rPPG signal can be obtained using POS, CHROM, ICA, PCA, PhysNet, or a combination thereof, and the respiratory waveform and respiratory rate can be derived from the amplitude modulation, frequency modulation, or respiratory sinus arrhythmia characteristics of the pulse wave.

[0065] Further, step S5 specifically involves inputting the / swallow / state message published by the laryngeal motion detection node, the / swallow / respiration message published by the rPPG breathing node, and the / feeding / arm_status message published by the feeding node into the risk fusion and safety gating node, and aligning them according to header.stamp using the precise or approximate time synchronization mechanism of message_filters; the time alignment serves as the basic data processing before multimodal fusion, and the aligned laryngeal motion features, breathing features, head and face posture status, and signal quality of each channel are then entered into the quality-aware dynamic risk fusion module, and the fusion weight is dynamically determined based on the signal quality of each channel.

[0066] Furthermore, step S6 specifically involves obtaining motion risk based on relative laryngeal displacement, relative velocity, and local vertical optical flow, and obtaining respiratory risk based on respiratory rate, respiratory signal quality, and respiratory recovery characteristics; and weighting and fusing motion risk and respiratory risk according to dynamic fusion weights to obtain a comprehensive risk score and map it to the corresponding risk level.

[0067] Furthermore, step S7 specifically involves the risk fusion and safety gating node issuing safety gating messages to / swallow / safety_gate to allow feeding, pause feeding, or stop feeding. When feeding is allowed, the feeding node continues normal operation; when feeding is paused, the feeding node stops pushing the utensils towards the mouth and waits for risk reassessment; when feeding is stopped, the current feeding process is stopped, and the node executes actions such as withdrawing to a safe position, prohibiting the delivery of the next bite of food, triggering an audible and visual alarm, and recording an event log according to a preset safety policy. An independent hard-wired emergency stop device is set up in parallel with the ROS2 software link as the last level of safety protection.

[0068] Furthermore, in one system deployment method, the industrial PC 2 runs Ubuntu 22.04 and ROS2 Humble, and deploys pyorbbecsdk, OpenCV, MediaPipe or an equivalent facial keypoint detection framework, rPPG processing components, and corresponding ROS2 function packages. The system adopts a unified message timestamp writing rule; for multi-camera or multi-node inputs, each sensor message includes a header.stamp field and is generated by the same industrial PC clock; cross-machine deployment can synchronize the system clock via NTP or PTP.

[0069] The system can display RGB images, depth maps, and facial regions in real time through a baseline capture procedure, and verify that the throat region is within the camera's field of view. The computational topics include the original frame, coordinates of the target throat region of interest, coordinates of the reference region, depth quality, and timestamp. The display topics are only used for debugging the interface, thus maintaining the separation of the display and computational links.

[0070] Furthermore, during the throat motion feature extraction process, the RGB-D camera is calibrated, and depth-color image alignment and invalid depth removal are performed. Multiple candidate regions of interest for the throat are generated based on facial key points and human pose key points. The comprehensive score S of the candidate regions is expressed as:

[0071] Where S_depth represents the depth information related score, S_valid represents the effective depth ratio related score, S_skin represents the skin color prior related score, S_pos represents the candidate region location rationality score, S_time represents the candidate region time series continuity score, and a1 to a5 are the corresponding weight coefficients.

[0072] Select the target throat region of interest based on the comprehensive score, and set up a reference region with a predetermined spatial relationship with the target throat region of interest.

[0073] Let D_throat(t) represent the representative depth of the target throat region of interest at time t, and let D_ref(t) represent the representative depth of the reference region at time t.

[0074] The representative depth can be obtained by using the median of the effective depth values ​​within the region, the weighted average based on pixel confidence and spatial distance, or other robust statistical methods; at the same time, the quality q_m of the laryngeal motion sensing channel can be evaluated based on the effective depth ratio, regional stability, and temporal continuity.

[0075] The relative depth D_rel(t) between the region of interest and the reference region of the target throat is expressed as:

[0076] By using depth difference between the laryngeal region and the reference region, the impact of overall patient movement, bed movement, and changes in the overall relative position of the patient and the camera on the detection of local laryngeal movement is reduced.

[0077] Let B represent the relative depth baseline at rest, k represent the scale transformation coefficient, and Δt represent the time interval between adjacent moments. The relative throat displacement disp(t) is expressed as:

[0078]

[0079] Multiple image feature points are selected within the region of interest of the target larynx. The motion of these feature points between consecutive image frames is calculated using the Lucas-Kanade local optical flow method or other equivalent local optical flow estimation methods. The vertical motion components of multiple effective feature points are statistically analyzed, and the median or other robust statistics are used to characterize the local vertical motion intensity flow(t) of the larynx. Based on the relative displacement disp(t), relative velocity vel(t), and local vertical optical flow flow(t) of the larynx, joint analysis is performed to obtain swallowing-related local laryngeal motion characteristics or motion risk scores.

[0080] Furthermore, in the rPPG respiratory rhythm estimation process, the rPPG signal is extracted based on the temporal color changes of the facial skin region in the RGB image. Let the mean RGB value of the facial skin region in frame t be C(t) = [R(t), G(t), B(t)]^T, which is obtained after sliding window normalization:

[0081] Where Lr is the length of the rPPG window, preferably 8-30 s. For the POS method, the normalized color signal can be projected onto a plane orthogonal to the skin color direction to obtain two projection components S1(t) and S2(t), which are then combined according to the standard deviation ratio to form the pulse wave b(t):

[0082]

[0083] For the CHROM method, a chromaticity combination in the color space that is orthogonal to the illumination change can be used; for the PhysNet method, continuous facial ROI image blocks can be input into a spatiotemporal convolutional network to output rPPG waveforms. After obtaining the pulse wave b(t), the dominant cardiac frequency is extracted by bandpass filtering and spectral peak selection, and the respiratory waveform r(t) is estimated by pulse wave amplitude envelope, pulse interval variation, or respiratory sinus arrhythmia features.

[0084] The respiratory rate RR(t) can be calculated from the dominant frequency fr of the respiratory waveform in the range of 5-40 breaths / min:

[0085] The respiratory phase φ(t) can be obtained by Hilbert transform or zero-crossing method:

[0086] Where H{r(t)} represents the Hilbert transform of r(t). The respiratory signal quality q_r is calculated based on the signal spectrum energy ratio, facial region stability, skin color pixel ratio, and motion amplitude. The respiratory feature sensing module outputs the / swallow / respiration message, which includes at least the respiratory waveform r(t), respiratory rate RR(t), respiratory rhythm characteristics, respiratory recovery characteristics, respiratory signal quality q_r, and header.stamp timestamp.

[0087] Furthermore, during time synchronization and system performance verification, data can be collected under conditions such as rest, drinking, swallowing food, simulated choking, slight head tilting, different lighting conditions, and partial occlusion. Each data segment simultaneously records RGB-D frames, the target larynx region of interest, the reference region, local laryngeal motion features, rPPG facial region, respiratory waveform, feeding robotic arm status, and manual annotations.

[0088] Fusion node 214 uses message filters to synchronize / swallow / state, / swallow / respiration, and / feeding / armstatus. When using approximate time synchronization, let the three message timestamps be ts, tr, and ta, if the following conditions are met:

[0089] If all three types of messages are considered eligible for the same fusion callback, then timed-out messages are discarded or the data at that moment is marked as out of sync. Δtmax can be set based on whether the deployment is on the same machine or multiple machines. The collected data can be aligned according to a unified timestamp and used for system verification; after time alignment, the multimodal features enter the quality-aware dynamic risk fusion module for further processing. Time synchronization is the basic data processing before multimodal fusion, and the core of risk fusion is dynamic weight allocation based on signal quality.

[0090] Furthermore, in an optional implementation, the relative displacement of the throat, relative velocity, local vertical optical flow, and their time series can be input into a support vector machine, random forest, LightGBM, 1D-CNN, TCN, GRU, LSTM, Transformer, or other equivalent analysis models for further analysis.

[0091] The above optional analysis is only used to make auxiliary judgments based on the local motion characteristics of the larynx, and does not constitute the technical steps necessary for the present invention to realize laryngeal motion detection, quality perception dynamic fusion and three-state safety gating.

[0092] The quality q_m of the laryngeal motion sensing channel can be evaluated based on the effective depth ratio of the target laryngeal region of interest, regional stability, effective feature point ratio of local optical flow, and temporal continuity.

[0093] The quality q_r of the respiratory sensing channel can be evaluated based on the rPPG spectral energy ratio, facial skin region stability, skin color pixel ratio, illumination changes, and motion amplitude.

[0094] When the quality of any sensing channel decreases, the dynamic fusion weight of that channel in the comprehensive risk calculation is reduced accordingly to avoid low-quality sensing channels having too much impact on the comprehensive risk assessment.

[0095] Furthermore, during the quality-perception dynamic risk fusion process, the risk fusion and safety gating node 214 receives throat movement characteristics, respiratory characteristics, head and face posture status, signal quality of each sensing channel, and the status of the feeding robotic arm. After the time synchronization module completes alignment, the nodes calculate motion risk and respiratory risk respectively, and dynamically allocate fusion weights according to the channel quality.

[0096] The motion risk score s_m is obtained based on the relative displacement, relative velocity, and local vertical optical flow of the throat, and can be expressed as:

[0097] The respiratory risk score s_r is obtained based on the respiratory rate resp_rate, respiratory signal quality resp_quality, and respiratory recovery feature recovery_feature, and can be expressed as: s_r = g(resp_rate, resp_quality, recovery_feature).

[0098] The dynamic fusion weights are determined based on the quality q_m of the laryngeal motion sensing channel and the quality q_r of the respiratory sensing channel; α_m and α_r are the corresponding basic weight coefficients, and ε is a stabilizing term to prevent the denominator from being zero. The dynamic weights are expressed as:

[0099]

[0100]

[0101] The comprehensive risk score R is mapped to a corresponding risk level based on a preset risk threshold, and a three-state safety gating signal is generated based on the comprehensive risk score or risk level. When the signal quality of a certain sensing channel deteriorates, its dynamic fusion weight is reduced; when multiple sensing channels fail to provide effective information that meets preset quality requirements, or when the comprehensive risk score reaches the stop-feeding condition, a stop-feeding state is generated; when the comprehensive risk is within an intermediate risk range requiring further observation, a pause-feeding state is generated and reassessment is awaited; when the comprehensive risk meets safety conditions, a allow-feeding state is generated. Specific risk thresholds can be preset according to the application scenario.

[0102] Furthermore, in the process of real-time integration and robust processing, such as Figure 4 As shown, each computing node can run on the same industrial computer 2 and use a unified DOMAINID. The RGB-D acquisition node 211 writes camera frames into a finite-length queue, the throat motion detection node 212 and the rPPG breathing node 213 use sliding window incremental updates, and the risk fusion and safety gating node 214 only processes the latest synchronization messages. The refresh rate of the visualization node 215 can be lower than the computing frequency to avoid the graphics display occupying control link resources.

[0103] To ensure real-time performance, the system can employ techniques such as throat and facial region cropping, increasing the safety gate refresh rate only during the feeding phase, performing inter-frame tracking of facial key point detection results, using FP16 or INT8 inference for optional analysis models, setting ROS2 QoS parameters to small queues and real-time priority, and avoiding cross-network transmission when deployed on the same machine. The end-to-end latency Ttotal can be expressed as: Ttotal = Tcap + Tpre + Tmotion + TrPPG + Tsync + Tfusion + Tpub Where Tcap is the acquisition latency, Tpre is the preprocessing latency, Tmotion is the laryngeal motion detection calculation latency, TrPPG is the respiratory estimation latency, Tsync is the synchronization waiting latency, Tfusion is the quality-aware dynamic risk fusion calculation latency, and Tpub is the message publishing and feeding node receiving latency.

[0104] Robustness processing includes: color normalization and brightness stability assessment during illumination changes; updating the target throat region of interest using facial keypoints, human pose keypoints, and temporal continuity during head rotation; reducing the dynamic fusion weight of a single channel when occlusion causes a decrease in the quality of that channel; reducing the weight of the motion perception channel when the proportion of invalid depth maps increases; reducing the weight of the breathing perception channel when the facial skin area is unstable or illumination changes are significant; and generating a stop-feeding state when multiple channels fail to meet preset quality requirements.

[0105] Furthermore, during system performance verification, ROS2 nodes related to turning over, feeding, and throat movement detection were integrated onto the same industrial computer 2, unifying the tf2 coordinate system and timestamps to verify the closed-loop process from turning posture adjustment and feeding action execution to multimodal risk fusion and three-state safety gating. For example... Figure 5 As shown, system performance verification may include multi-subject data acquisition, manual truth labeling, perception channel index evaluation, risk fusion and security gating index evaluation, end-to-end delay measurement, and failure safety testing.

[0106] Evaluation metrics may include accuracy (Acc) for laryngeal regional motion recognition, recall, false positive rate (FPR), recall rate for risk fusion and safety gating, mean absolute error of respiratory rate (MAERR), end-to-end latency (Ttotal), three-state safety gating response time, and confusion matrix. These can be defined as follows:

[0107]

[0108]

[0109] Wherein, TP represents the number of correctly identified risks or laryngeal local motion events, TN represents the number of correctly output feeding permission states, FP represents the number of false alarms, FN represents the number of missed alarms, RRi represents the system's estimated respiratory rate, and RRi^gt represents the reference true value. The relevant statistical results are used to verify the effectiveness of target laryngeal region of interest localization, local motion perception, respiratory perception, quality perception dynamic risk fusion, three-state safety gating, and independent hardware emergency stop linkage.

[0110]

[0111] Alternative Solution 1: RGB-D Device Replacement. RGB-D Camera 1 can be replaced with any structured light, binocular, or ToF depth camera capable of simultaneously outputting RGB images and depth maps; alternatively, two or more RGB-D cameras can be used for multi-view fusion to improve throat visibility in scenes with head deflection or occlusion. For different cameras, the camera's intrinsic parameters, depth scale factor, extrinsic parameters, spatial relationship between the target throat region and the reference region, and representative depth calculation parameters can be adjusted accordingly.

[0112] Alternative Solution 2: Replacement of the throat local motion feature extraction method. Local optical flow can be achieved using Lucas-Kanade optical flow or other equivalent methods capable of estimating local motion in consecutive image frames; representative depths of the target throat region and the reference region can be achieved using median, weighted mean, or other robust statistical methods. Any alternative implementation that can obtain throat local motion features based on the relative changes between the target throat region and the reference region is acceptable. These motion features can also be input into machine learning or time-series analysis models for further analysis, but this analysis is not considered a core or essential technical step.

[0113] Alternative Option 3: rPPG Algorithm Replacement. The rPPG respiratory node 213 can employ POS, CHROM, ICA, PCA, Green channel, PhysNet, DeepPhys, rPPGNet, or combinations thereof to extract the rPPG signal based on the temporal color changes of the facial skin region. Respiratory estimation can utilize pulse wave amplitude envelope, respiratory sinus arrhythmia, spectral peak values, phase tracking, or equivalent methods. All algorithms must output the respiratory waveform, respiratory rate, respiratory rhythm characteristics, respiratory recovery characteristics, and respiratory signal quality.

[0114] Alternative Solution 4: Fusion Method Replacement. Risk fusion and security gating node 214 can be implemented using rule-weighted fusion, decision-level weighted fusion, feature-level splicing, Bayesian risk fusion, Hidden Markov Model, temporal attention network, or a rule-model hybrid approach. Regardless of the fusion method used, a comprehensive risk score or risk level can be obtained based on the characteristics and signal quality of each sensing channel, and a security gating state can be generated to allow feeding, pause feeding, or stop feeding.

[0115] Alternative Option 5: Deployment Method Replacement. This example prioritizes intra-machine ROS2 deployment to reduce synchronization and transmission latency. Multi-machine ROS2 deployment can also be used when the computational load is high or the devices are geographically dispersed. In multi-machine deployment, clock synchronization is achieved via NTP or PTP, message interconnection is maintained through a unified DOMAINID, QoS policy, and tf2 coordinate system, and Δtmax and security gate refresh cycles are adjusted based on network latency.

[0116] Alternative Option Six: Feeding Actuator Replacement. The feeding robotic arm 3 can be replaced with a linear propulsion mechanism, a cutlery holder, an automatic drinking device, or other feeding actuators. As long as the actuator can subscribe to or receive the safety gate output interface 9 and execute actions such as allowing continued feeding, pausing propulsion, or stopping delivery according to the safety gate control state, and performing a withdrawal operation according to a preset safety strategy when feeding is stopped, it can be implemented in conjunction with the sensing and fusion method of the present invention.

[0117] Figure 1 The diagram illustrates the relationship between the RGB-D camera 1, camera bracket 6, industrial computer 2, feeding robotic arm 3, alarm unit 4, and independent emergency stop unit 5. The RGB-D camera 1 faces the bedridden patient 7, acquiring facial RGB images and laryngeal depth maps, and connects to the industrial computer 2 via USB3 or GigE link. The industrial computer 2 deploys a ROS2 node group 21, which performs acquisition, laryngeal motion detection, rPPG respiratory estimation, quality-perceived dynamic risk fusion, and feeding control. The risk fusion and safety gating node outputs a three-state safety gating signal to the feeding robotic arm 3, simultaneously driving the alarm unit 4; the independent emergency stop unit 5 directly cuts off the power to the actuator without relying on a software link.

[0118] Figure 2 Starting with RGB-D data acquisition and time synchronization, the process is divided into a laryngeal motion detection branch and an rPPG respiratory sensing branch. The laryngeal motion detection branch outputs the relative displacement and velocity of the larynx, local vertical optical flow, motion risk, and motion signal quality; the respiratory sensing branch outputs respiratory rhythm characteristics, respiratory recovery characteristics, and respiratory signal quality. The risk fusion module dynamically determines the fusion weights based on the signal quality of each channel, performs weighted fusion of motion risk and respiratory risk, and outputs a comprehensive risk score, risk level, and safety gating signals for allowing feeding, pausing feeding, or stopping feeding, thereby driving the feeding actuator to form a closed-loop control.

[0119] Figure 3 Starting with RGB-D frame input, camera calibration, and depth-color alignment, multiple candidate regions of interest (ROIs) for the throat are first generated based on facial and human pose keypoints. These candidate regions are then scored based on depth validity, skin color prior, location rationality, and temporal continuity to determine the target throat ROI. A reference region is then determined, and the relative depth of the throat is obtained by the depth difference between the target throat region and the reference region. Relative displacement and relative velocity are further calculated. Simultaneously, local image optical flow estimation is performed within the target throat region to obtain the vertical motion characteristics and motion signal quality of the throat.

[0120] Figure 4 The diagram shows that the RGB-D acquisition node, laryngeal motion detection node, rPPG breathing node, risk fusion and safety gating node, visualization node, and feeding control node all operate within the same industrial computer 2, using a unified DOMAINID, header.stamp, and tf2 coordinate system. / swallow / state, / swallow / respiration, and / feeding / armstatus are accessed by the risk fusion and safety gating node via message filters; time synchronization serves as the foundational data processing before fusion, and the node dynamically assigns weights based on channel signal quality and publishes / swallow / safetygate. The visualization node only subscribes to tuning views and waveforms and does not provide input to the control link.

[0121] Figure 5 The process of validating the system's performance under multiple subjects, different postures, and varying lighting and occlusion conditions is illustrated: First, manually labeled or reference true values ​​are established. Then, the laryngeal motion sensing channel, rPPG respiratory sensing channel, quality-perceived dynamic risk fusion, and three-state safety gating output are evaluated separately. Finally, accuracy, recall, false alarm rate, confusion matrix, respiratory rate error, end-to-end latency, and safety gating response time are statistically analyzed. These results are used to verify the real-time performance, robustness, and fail-safety capabilities of this invention.

[0122] Figure 6The image shows an RGB-D image sensor aligned with the patient's larynx, lower jaw, and anterior neck region via a bedside bracket or close-range mounting structure when the patient is in a semi-recumbent position with their back raised. This arrangement enhances the ability to observe the relative depth changes and local vertical optical flow between the target larynx region of interest and the reference region. It is suitable for extracting relative displacement, relative velocity, and local vertical motion features of the larynx, and can reduce the influence of obstruction caused by pillows, blankets, or the jaw from a forward-facing perspective.

[0123] Figure 7 The image sensor is shown mounted on a bedside stand or pole, with the lens facing the face and upper body of the bedridden patient. This setup is primarily used to acquire RGB image sequences of the face and depth information from directly in front, supporting rPPG respiratory rhythm estimation, head posture monitoring, and stable tracking of the facial region of interest (ROI), while ensuring that the patient does not need to wear a contact sensor.

[0124] Figure 8 The left side shows the external structure of the bar-shaped RGB-D camera, and the right side shows the application method of this type of image sensor when mounted on a bedside bracket and aimed at the face of a bedridden patient. This figure is used to illustrate the physical shape, mounting orientation, imaging field of view, and spatial relationship between the preferred image sensor of the present invention and the patient's facial area.

[0125] Figure 9 The diagram illustrates the overall relationship between a bedridden patient, a turning and care bed, a forward-facing face monitoring camera, a lateral laryngeal monitoring camera, a feeding robotic arm, an industrial control computer, and an alarm unit. This scenario demonstrates the invention's closed-loop operation, from bedside non-contact sensing and dynamic risk fusion based on quality perception, to the feeding robotic arm's actions of allowing feeding, pausing or stopping feeding, and recording alarms. It also showcases the parallel implementation of independent hardware emergency stop and software safety gating.

Claims

1. A multimodal physiological signal aspiration / choking real-time early warning and emergency feeding interruption system for bedridden feeding, characterized in that, The system includes an RGB-D camera (1), a camera bracket (6), an industrial computer (2), a feeding robotic arm (3), an alarm unit (4), and an independent emergency stop unit (5). The RGB-D camera (1) is mounted on a camera bracket (6) next to the bed, with the optical axis of the RGB-D camera (1) facing the face, jaw and throat area of ​​the bedridden patient (7); The industrial control computer (2) is internally equipped with an RGB-D acquisition node (211), a laryngeal movement detection node (212), an rPPG breathing node (213), a risk fusion and safety gating node (214), a visualization node (215), and a feeding control node (216). The RGB-D acquisition node (211) acquires RGB frames and depth frames, and encapsulates the camera intrinsic parameters, extrinsic parameters, frame number and acquisition timestamp into ROS2 messages; The laryngeal motion detection node (212) acquires facial key points and human posture key points based on RGB-D data, generates multiple laryngeal candidate regions and performs candidate region screening, extracts the relative depth, relative displacement and relative velocity of the laryngeal region based on the depth difference between the target laryngeal region and the reference region, and obtains the vertical motion features of the laryngeal region by combining local image optical flow. The rPPG respiratory node (213) extracts rPPG signals based on the temporal color changes of the facial skin region and obtains respiratory rhythm characteristics and respiratory signal quality; The risk fusion and safety gating node (214) acquires laryngeal movement characteristics, respiratory characteristics, head and face posture status and signal quality of each sensing channel, performs dynamic weight allocation based on the signal quality of each sensing channel and calculates a comprehensive risk score, determines the risk level based on the comprehensive risk score, and generates a corresponding feeding safety gating signal. The visualization node (215) is used to display RGB images, depth images, throat region of interest, reference region, throat relative motion features, rPPG waveforms, risk scores, risk levels, and safety gating status; The feeding control node (216) controls the feeding robotic arm (3) to perform feeding permission, feeding pause or feeding stop operations according to the feeding safety gate signal; The independent emergency stop unit (5) acts directly on the feeding robot arm (3) through a hard-wired circuit, while simultaneously driving the alarm unit (4) and setting it in parallel with the ROS2 software link.

2. A method for real-time early warning and emergency feeding interruption based on multimodal physiological signals of aspiration / choking for bedridden feeding, characterized in that, The method uses the multimodal physiological signal aspiration / choking real-time early warning and emergency feeding interruption system for bedridden feeding as described in claim 1, and the method includes the following steps: Step S1: Collect color images, depth images, and corresponding timestamp data of the object being fed, and perform time-series synchronization processing; Step S2: Generate multiple candidate regions of interest for the throat based on facial key points and human posture key points, score the candidate regions of interest, and determine the target region of interest for the throat; Step S3: Calculate the relative depth, relative displacement, and relative velocity of the throat based on the depth difference between the target throat region of interest and the reference region, and extract the vertical motion features of the throat by combining the optical flow of the local image; Step S4: Extract rPPG signals based on the temporal color changes of the facial skin region to obtain respiratory rhythm features and corresponding respiratory signal quality; Step S5: Acquire laryngeal movement features, respiratory features, head and face posture status and signal quality of each sensing channel, and determine the corresponding dynamic fusion weights based on the signal quality of each sensing channel; Step S6: Weight the laryngeal movement risk and respiratory risk according to the dynamic fusion weight to obtain a comprehensive risk score, and determine the risk level according to the comprehensive risk score; Step S7: Generate a feeding safety gating signal based on the comprehensive risk score or risk level, and control the feeding robot to perform operations such as allowing feeding, pausing feeding, or stopping feeding, so as to realize real-time early warning of aspiration / choking and emergency feeding interruption for multimodal physiological signals of bedridden feeding.

3. The method according to claim 2, characterized in that, Specifically, step S1 involves acquiring color image frames and depth image frames captured by the RGB-D camera, and encapsulating each frame into a ROS2 message with a header.stamp timestamp; the RGB-D camera is mounted on a bedside stand so that its field of view covers the patient's face, jaw region, and throat region.

4. The method according to claim 2, characterized in that, Specifically, step S2 involves performing camera intrinsic parameter calibration, depth-color image alignment, and invalid depth removal. Multiple candidate regions of interest for the throat are generated based on facial key points and human pose key points. The candidate regions are scored according to their depth validity, skin color prior, location rationality and temporal sequence continuity, and the target throat region of interest is selected based on the scoring results.

5. The method according to claim 2, characterized in that, Specifically, step S3 involves obtaining the throat region depth value within the target throat region of interest and obtaining the reference region depth value within a reference region that has a predetermined spatial relationship with the target throat region of interest. The difference between the throat region depth value and the reference region depth value is calculated as the throat relative depth. The relative displacement of the throat is calculated based on the relative depth baseline at rest, and the relative velocity of the throat is calculated based on the relative depth changes and time intervals between adjacent moments.

6. The method according to claim 2, characterized in that, Step S3 further includes selecting multiple image feature points within the target larynx region of interest, estimating the motion of the image feature points between consecutive image frames using local image optical flow, and obtaining the vertical motion intensity of the larynx based on the vertical displacement statistics of the multiple image feature points; and performing joint constraints based on the relative displacement, relative velocity, and vertical motion intensity of the larynx to obtain swallowing-related local larynx motion features.

7. The method according to claim 2, characterized in that, Specifically, step S5 involves determining the region of interest (ROI) of facial skin from the facial RGB image sequence, obtaining the temporal color changes of the ROI, extracting the rPPG signal based on the temporal color changes, obtaining respiratory rhythm features and respiratory signal quality based on the rPPG signal, and obtaining the pulse wave signal using POS, CHROM, ICA, PCA, PhysNet, or a combination thereof, and deriving the respiratory waveform, respiratory rate, and respiratory phase from the amplitude modulation, frequency modulation, or respiratory sinus arrhythmia features of the pulse wave.

8. The method according to claim 2, characterized in that, Specifically, step S5 involves acquiring the motion signal quality corresponding to the laryngeal motion features and the respiratory signal quality corresponding to the respiratory features, and dynamically determining the fusion weights corresponding to the motion risk and respiratory risk based on the motion signal quality and respiratory signal quality, so that the perception channel with higher signal quality obtains a higher fusion weight and the perception channel with lower signal quality obtains a lower fusion weight.

9. The method according to claim 2, characterized in that, Specifically, step S6 involves obtaining a motion risk score based on the relative displacement of the larynx, the relative velocity of the larynx, and the local vertical optical flow; obtaining a respiratory risk score based on the respiratory rhythm, respiratory signal quality, and respiratory recovery characteristics; calculating a comprehensive risk score based on the motion risk score, the respiratory risk score, and their corresponding dynamic fusion weights; and mapping the comprehensive risk score to the corresponding risk level based on a preset risk threshold.

10. The method according to claim 2, characterized in that, Specifically, step S8 involves the risk fusion and safety gating node generating a safety gating signal that allows feeding, pauses feeding, or stops feeding based on a comprehensive risk score or risk level. In the allowed feeding state, the feeding robotic arm continues the current feeding action. In the paused feeding state, the feeding utensils are stopped from being pushed into the mouth and the system waits for a risk reassessment. In the stopped feeding state, the current feeding action is stopped, and the end effector is retracted to a safe position, the next food delivery is stopped, and an alarm is triggered according to a preset safety strategy. An independent hard-wired emergency stop device is set up in parallel with the ROS2 software link as the last level of safety protection.

Images