Safety visual monitoring feedback system based on eating of the elderly

Abstract

This invention relates to the field of intelligent health monitoring technology and discloses a visual monitoring and feedback system for safe eating in the elderly. The system includes a data acquisition module, a data processing and feature analysis module, a risk assessment and decision-making module, and an alarm and feedback output module. The system acquires multimodal data through visual capture, physiological signal perception, and ambient light detection. It utilizes deep learning and feature fusion technology to analyze eating behavior and physiological state, identify potential risks such as difficulty swallowing and aspiration, and provide real-time alarm information through multiple channels. This invention can improve the safety of eating for the elderly, reduce accidental risks, and adapt to complex environments, thus possessing high practical value.

Description

Technical Field

[0001] This invention belongs to the field of intelligent health monitoring technology, specifically a safety visual monitoring and feedback system for elderly people eating. Background Technology

[0002] With the increasing aging of the population, the health and safety of the elderly are receiving growing attention. Eating is a vital activity for survival, but for some elderly people, it can be accompanied by risks such as difficulty swallowing, aspiration, and choking, which can even lead to life-threatening conditions like suffocation or aspiration pneumonia. Existing visual monitoring systems mostly rely on cameras to capture the elderly's eating actions (such as mouth opening and chewing frequency). However, in practical applications, the core risks to the elderly's eating safety (such as food residue in the airway during swallowing, hidden aspiration, and excessively large food pieces due to insufficient chewing strength) often occur inside the mouth or throat, and visual signals have limitations in monitoring these situations. Furthermore, changes in lighting in home settings (such as dim lighting at night) and facial obstruction when the elderly are bending over to eat further affect the reliability of visual recognition.

[0003] In recent years, the development of multimodal sensing technology has provided new ideas for solving the above problems. By combining multiple data sources such as visual perception, physiological parameter perception, and environmental perception, it is possible to more comprehensively capture the behavioral characteristics and physiological changes of the elderly during eating. However, current multimodal monitoring systems still face many challenges: first, how to achieve stable visual perception under complex lighting conditions; second, how to accurately acquire the physiological parameters of the elderly through non-contact methods; and third, how to efficiently fuse and deeply analyze multimodal data to achieve accurate assessment and timely early warning of potential risks.

[0004] Therefore, there is an urgent need for a multimodal perception-based food safety monitoring system for the elderly that can operate stably in complex environments and provide real-time feedback through intelligent analysis, thereby effectively reducing the risks during the elderly's eating process. Summary of the Invention

[0005] This invention relates to a multimodal monitoring and feedback system for the safety of elderly people eating, comprising a data acquisition module, a data processing and feature analysis module, a risk assessment and decision-making module, and an alarm and feedback output module. The data acquisition module acquires visual information, physiological signal information, and ambient light information of the elderly person in the eating area. The data processing and feature analysis module is connected to the data acquisition module via a wired or wireless communication protocol and is used to preprocess the information acquired by the data acquisition module and extract multimodal features. The risk assessment and decision-making module is communicatively connected to the data processing and feature analysis module and is used to integrate and analyze the multimodal features to identify potential risks. The alarm and feedback output module is communicatively connected to the risk assessment and decision-making module and is used to generate real-time alarms based on the risk assessment results and output feedback information through multiple channels. Data interaction between the modules adopts the industrial Ethernet communication protocol to ensure efficient and stable data transmission and command execution.

[0006] Preferably, the data acquisition module includes a visual capture unit, a physiological signal sensing unit, and an ambient light detection unit. The visual capture unit is used to acquire color images and depth information of the elderly person in the eating area; the physiological signal sensing unit is used to non-contactly acquire micro-vibration signals in the neck and chest areas of the elderly person, as well as surface temperature signals in the oral cavity area; the ambient light detection unit is used to monitor the light intensity in the eating area in real time and transmit the light intensity value to the data processing and feature analysis module to support the dynamic adjustment of the visual algorithm.

[0007] Preferably, the visual capture unit consists of at least one wide-angle depth-of-field camera, which is mounted above or slightly in front of the elderly person's eating area. The camera has a horizontal field of view of not less than 90°, a vertical field of view of not less than 60°, a depth measurement range of 0.5 meters to 4 meters, a depth resolution of less than ±2 millimeters at a distance of 2 meters, and an image acquisition frame rate of 35 frames per second. The wide-angle depth-of-field camera has a built-in infrared auxiliary light source, enabling stable depth measurement capabilities under low-light conditions, thereby improving the system's adaptability to complex lighting environments.

[0008] Preferably, the physiological signal sensing unit includes an ultrasonic sensor and a thermal imaging sensor. The ultrasonic sensor is used to transmit and receive ultrasonic signals to acquire micro-motion information of the neck and chest regions of the elderly. Its installation position is optimized to ensure that the ultrasonic beam can stably cover the target area and avoid frequent obstruction by hands or tableware. The thermal imaging sensor is used to non-contactly measure the surface temperature of the oral cavity region of the elderly. Its installation position is precisely calibrated to ensure continuous and stable capture of temperature changes in the oral cavity region.

[0009] Preferably, the ultrasonic sensor operates at a frequency of 40kHz to 50kHz, with a detection accuracy of less than 1 mm and an angular resolution of ±2°. This device analyzes the time delay and phase changes of the reflected signal to extract respiratory rate, micro-movements on the body surface caused by heartbeat, and neck muscle vibration characteristics related to swallowing. The ultrasonic sensor has a sampling frequency of no less than 250Hz, enabling it to capture the short-duration, high-frequency vibration patterns unique to swallowing movements.

[0010] Preferably, the thermal imaging sensor has a temperature measurement range of 25℃ to 50℃, a temperature measurement accuracy of ±0.3℃, and a response time of less than 80 milliseconds. The thermal imaging sensor captures the energy distribution of radiation radiated from the skin surface of the oral cavity area and converts it into a temperature distribution map. Combined with an internal algorithm, it eliminates the influence of ambient temperature fluctuations on the measurement results, thereby improving the stability and accuracy of temperature measurement.

[0011] Preferably, the data processing and feature analysis module includes a visual feature analysis unit, a physiological signal analysis unit, and a multimodal synchronization unit. The visual feature analysis unit extracts relevant features of the elderly's eating behavior from the images and depth information acquired by the visual capture unit using a machine learning model. The physiological signal analysis unit performs signal filtering and spectral analysis on the ultrasound signals and thermal imaging data acquired by the physiological signal sensing unit to extract physiological features. The multimodal synchronization unit employs a high-precision timestamp mechanism to uniformly label the data streams generated by the visual capture unit and the physiological signal sensing unit, and performs time alignment of data from different sensors using interpolation algorithms or nearest neighbor matching methods, providing an accurate temporal basis for subsequent fusion analysis.

[0012] Preferably, the visual feature analysis unit employs a deep learning model based on convolutional neural networks and attention mechanisms to achieve posture estimation, facial key point localization, tableware and food segmentation, and oral cavity activity analysis for the elderly. The features extracted by the visual feature analysis unit from image and depth information include: calculating chewing frequency and force by analyzing the vertical movement frequency and amplitude of mandibular key points; when the mandibular movement frequency is detected to be below a set threshold or the movement amplitude is significantly reduced, it is considered a potential indicator of chewing difficulty; judging the continuity of the food delivery process by tracking the relative positional relationship and contact events between tableware and the oral cavity; when food is detected to remain at the edge of the oral cavity for a long time or when the contact between tableware and the oral cavity is abnormally interrupted, it is considered an abnormal eating behavior; assessing the safety of the eating posture by analyzing the relative angle between the head and trunk; when the head is detected to be excessively tilted forward or backward, it is considered an unsafe posture that increases the risk of aspiration; determining whether there is food spillage or retention by segmenting and identifying food residue in and around the oral cavity; and analyzing the movement trajectory and amplitude of the neck region using optical flow as auxiliary evidence of swallowing actions.

[0013] Preferably, the physiological signal analysis unit performs spectral analysis on the ultrasound signal after filtering and denoising to extract micro-motion features to distinguish vibrations caused by breathing, heartbeat, and swallowing. The features extracted by the physiological signal analysis unit include: identifying high-frequency vibration patterns specific to swallowing movements through time-frequency analysis of ultrasound signals in the neck region; when multiple high-frequency peaks or significantly reduced vibration amplitude are detected during a single swallowing movement, this is considered an indicator of swallowing difficulty; calculating respiratory rate and depth in real time through spectral analysis of ultrasound signals in the chest region; when a sudden increase in respiratory rate or abnormal rhythm is detected, this is considered a stress response after aspiration or coughing; calculating heart rate variability indicators through analysis of micro-motion signals caused by heartbeat; when high-frequency components decrease or low-frequency components increase, this is considered indirect evidence of autonomic nervous system stress response; and monitoring the rate of change of oral cavity surface temperature in real time through time-series analysis of thermal imaging data; when the rate of temperature decrease significantly decreases or remains close to ambient temperature for an extended period, this is considered evidence of interrupted eating behavior.

[0014] Preferably, the risk assessment and decision-making module includes a feature fusion unit, a deep learning assessment unit, and a risk level classification unit. The feature fusion unit employs a multi-level fusion strategy, concatenating or fusing the visual feature vectors output by the data processing and feature parsing modules with the physiological signal feature vectors using an attention mechanism. Through weighted averaging or adaptive learning, appropriate weights are assigned to different modal features to enhance the model's robustness and decision accuracy. The deep learning evaluation unit uses a time-series model based on a recurrent neural network architecture (such as Long Short-Term Memory (LSTM) or gated recurrent units). It accepts the fused feature sequence output by the feature fusion unit as input, models the behavioral and physiological changes of elderly individuals during eating, and outputs the probability values ​​of various risk events occurring at the current moment by learning multimodal feature patterns of different risk events. Risk events include dysphagia, aspiration, choking, excessively large food boluses, and interrupted eating. The risk level classification unit, based on the risk probabilities output by the deep learning evaluation unit and combined with preset thresholds and weighting rules, classifies risk events into at least one of the following categories: low risk, medium risk, high risk, and emergency risk. The judgment logic is configured based on clinical medical recommendations and large-scale data analysis results, and can be personalized according to the individual health status of elderly individuals.

[0015] Preferably, the alarm and feedback output module includes an alarm generation unit and a multi-channel feedback unit. The alarm generation unit generates real-time alarm information based on the output of the risk assessment and decision-making module; the multi-channel feedback unit provides feedback information to users or caregivers through various means such as sound, light, and screen display. The alarm generation unit and the multi-channel feedback unit are connected via a real-time communication protocol to ensure that alarm information can be transmitted quickly and trigger the corresponding feedback mechanism.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention enables real-time monitoring and feedback of the safety of elderly people eating, ensuring a safer and more reliable eating process. Throughout the process, the various modules work collaboratively, fully utilizing the advantages of multimodal sensing technology to overcome the limitations of a single visual monitoring system, providing comprehensive technical protection for the safety of elderly people eating. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention.

[0018] Figure 2 This is a schematic diagram of the data acquisition module.

[0019] Figure 3 This is a schematic diagram of the visual capture unit.

[0020] Figure 4 This is a schematic diagram illustrating the working principle of the physiological signal sensing unit.

[0021] Figure 5 This is a schematic diagram of the output method of the alarm and feedback output module. Detailed Implementation

[0022] This invention relates to a multimodal monitoring and feedback system for food safety in the elderly, the specific implementation of which is described in conjunction with the appendix. Figure 1 To be continued Figure 5 Please provide a detailed explanation. For example... Figure 1 As shown, the system includes a data acquisition module, a data processing and feature analysis module, a risk assessment and decision-making module, and an alarm and feedback output module. These modules communicate efficiently and stably via an industrial Ethernet communication protocol to achieve data transmission and command execution. The operating principle and process of the system will be fully described below with reference to specific embodiments.

[0023] In a preferred implementation, the data acquisition module is the foundation of the entire system, and its components are as follows: Figure 2 As shown, it includes a visual capture unit, a physiological signal sensing unit, and an ambient light detection unit. The visual capture unit consists of at least one wide-angle depth-of-field camera, and the installation position of the wide-angle depth-of-field camera is as shown. Figure 3As shown, the camera can be positioned above or to the side of the elderly person's eating area, with a horizontal field of view of no less than 90° and a vertical field of view of no less than 60°. The depth measurement range is 0.5 meters to 4 meters, with a depth resolution of less than ±2 millimeters at a distance of 2 meters. The image acquisition frame rate is 35 frames per second. The wide-angle depth-of-field camera has a built-in infrared auxiliary light source, providing stable depth measurement capabilities under low-light conditions, thereby improving the system's adaptability in complex lighting environments. The visual capture unit transmits the acquired color images and depth information to the data processing and feature analysis module via wired or wireless communication protocols.

[0024] In a preferred embodiment, the physiological signal sensing unit includes an ultrasonic sensing device and a thermal imaging sensor, and its working principle is as follows: Figure 4 As shown. The ultrasonic sensor operates at a frequency of 40kHz to 50kHz, with a detection accuracy of less than 1 mm and an angular resolution of ±2°. It extracts respiratory rate, micro-movements on the body surface caused by heartbeat, and neck muscle vibration characteristics related to swallowing by analyzing the time delay and phase changes of the reflected signal. The sampling frequency is no less than 250Hz. The thermal imaging sensor has a temperature measurement range of 25℃ to 50℃, a temperature measurement accuracy of ±0.3℃, and a response time of less than 80 milliseconds. It captures the energy distribution of radiation radiated from the skin surface in the oral cavity area and converts it into a temperature distribution map. An internal algorithm is then used to eliminate the influence of ambient temperature fluctuations on the measurement results.

[0025] In a preferred embodiment, the ultrasonic sensor and thermal imaging sensor are respectively installed near the elderly person's neck and chest areas, as well as the oral cavity area, to ensure the stability and accuracy of signal acquisition. An ambient light detection unit monitors the light intensity in the eating area in real time and transmits the light intensity value to the data processing and feature analysis module to support dynamic adjustments of the visual algorithm. These three parts are connected to the data processing and feature analysis module via a hardware interface, forming the overall structure of the data acquisition module.

[0026] The data processing and feature analysis module is responsible for preprocessing the information acquired by the data acquisition module and extracting multimodal features. Internally, it includes a visual feature analysis unit, a physiological signal analysis unit, and a multimodal synchronization unit. The visual feature analysis unit employs a deep learning model based on convolutional neural networks and attention mechanisms to extract relevant features of elderly people's eating behavior from the images and depth information acquired by the visual capture unit. Specifically, the visual feature analysis unit first analyzes the frequency and amplitude of mandibular key points in the vertical direction using pose estimation and facial landmark localization techniques to calculate chewing frequency and force. When the mandibular movement frequency is detected to be lower than a set threshold or the movement amplitude is significantly reduced, it is considered a potential indicator of chewing difficulties. Second, by segmenting and identifying the positional relationship between utensils and food, it tracks the relative positional relationship and contact events between utensils and the oral cavity area to determine the continuity of the food delivery process. When food is detected to remain at the edge of the oral cavity for a long time or when the contact between utensils and the oral cavity is abnormally interrupted, it is considered abnormal eating behavior. Third, it analyzes the movement trajectory and amplitude of the neck area using optical flow as auxiliary evidence of swallowing actions, and simultaneously segments and identifies food residue in and around the oral cavity area to determine whether there is food spillage or retention.

[0027] In one preferred implementation, the physiological signal analysis unit filters and denoises the ultrasound signal before performing spectral analysis to extract micro-motion features and distinguish vibrations caused by breathing, heartbeat, and swallowing. By performing time-frequency analysis on the ultrasound signal in the neck region, it identifies high-frequency vibration patterns unique to swallowing movements. The presence of multiple high-frequency peaks or a significant decrease in vibration amplitude during a single swallowing action is considered an indicator of dysphagia. Spectral analysis of the ultrasound signal in the chest region calculates respiratory rate and depth in real time. A sudden increase in respiratory rate or abnormal rhythm is considered a stress response following aspiration or choking.

[0028] In a preferred implementation, heart rate variability is calculated by analyzing the micro-motion signals caused by heartbeats. A decrease in high-frequency components or an increase in low-frequency components is considered indirect evidence of stress response in the autonomic nervous system. Furthermore, by performing time-series analysis on thermal imaging data, the rate of change in the surface temperature of the oral cavity is monitored in real time. A significant decrease in the rate of temperature decrease or prolonged proximity to ambient temperature is considered evidence of interrupted eating behavior. The multimodal synchronization unit employs a high-precision timestamp mechanism to uniformly label the data streams generated by the visual capture unit and the physiological signal sensing unit. It also uses interpolation algorithms or nearest neighbor matching to time-align data from different sensors, providing an accurate temporal basis for subsequent fusion analysis. The visual feature analysis unit and the physiological signal analysis unit transmit the extracted feature vectors to the multimodal synchronization unit via an internal bus, completing the functions of the data processing and feature analysis modules.

[0029] The risk assessment and decision-making module communicates with the data processing and feature analysis module, and is responsible for integrating and analyzing multimodal features to identify potential risks. The risk assessment and decision-making module includes a feature fusion unit, a deep learning evaluation unit, and a risk level classification unit. The feature fusion unit employs a multi-level fusion strategy, concatenating or fusing the visual feature vectors output by the data processing and feature analysis module with physiological signal feature vectors using an attention mechanism. Through weighted averaging or adaptive learning, appropriate weights are assigned to different modal features to enhance the model's robustness and decision-making accuracy.

[0030] As a preferred implementation, the deep learning evaluation unit employs a time-series model based on a recurrent neural network architecture. It accepts the fused feature sequence output by the feature fusion unit as input to model the behavioral and physiological changes of elderly individuals during eating. By learning multimodal feature patterns of different risk events, it outputs the probability values ​​of various risk events occurring at the current moment. Risk events include dysphagia, aspiration, choking, excessively large food boluses, and interrupted eating. The risk level classification unit, based on the risk probabilities output by the deep learning evaluation unit and combined with preset thresholds and weighting rules, classifies risk events into at least one category: low risk, medium risk, high risk, and emergency risk. The judgment logic is configured based on clinical medical recommendations and large-scale data analysis results, and can be personalized according to the individual health condition of the elderly individual. The risk assessment and decision-making module transmits the risk assessment results to the alarm and feedback output module via an internal communication interface.

[0031] In a preferred implementation, the alarm and feedback output module is communicatively connected to the risk assessment and decision-making module, responsible for generating real-time alarms based on the risk assessment results and outputting feedback information through multiple channels. The alarm and feedback output module includes an alarm generation unit and a multi-channel feedback unit. The alarm generation unit generates real-time alarm information based on the output results of the risk assessment and decision-making module. The multi-channel feedback unit provides feedback information to users or caregivers through various means such as sound, lights, and screen displays.

[0032] In a preferred implementation, the alarm generation unit and the multi-channel feedback unit are connected via a real-time communication protocol to ensure that alarm information can be transmitted quickly and trigger the corresponding feedback mechanism. For example... Figure 5 As shown, the specific output forms of the multi-channel feedback unit include voice prompts, flashing lights, and screen displays of text or image information. Voice prompts play alarm content through a speaker, flashing lights use LEDs to change color and adjust flashing frequency, and the screen display shows the risk type and corresponding suggestions on an LCD screen. The alarm and feedback output module achieves comprehensive monitoring and timely feedback on the elderly's eating safety through these multiple methods.

[0033] In a practical application scenario, assuming an elderly person is dining in a designated area, a wide-angle depth-of-field camera captures the posture of their head and torso, as well as the positional relationship between their utensils and food, in real time. Ultrasonic sensors and thermal imaging sensors collect micro-vibration signals from the neck and chest regions, and surface temperature signals from the oral cavity, respectively. An ambient light detection unit monitors the light intensity in the dining area and transmits the data to a data processing and feature analysis module. This module extracts relevant features through visual feature analysis and physiological signal analysis units, and performs time alignment of the data using a multimodal synchronization unit. Subsequently, a risk assessment and decision-making module integrates and analyzes the multimodal features, identifies potential risks, and generates risk levels. Finally, an alarm and feedback output module generates alarms based on the risk levels and provides feedback to caregivers through sound, light, and screen displays. For example, when the system detects difficulty swallowing in an elderly person, the alarm and feedback output module immediately provides a voice prompt, "Please be aware of the risk of difficulty swallowing," and flashes a red LED to alert caregivers to take appropriate measures. Through these steps, the entire system achieves real-time monitoring and feedback on the elderly person's dining safety, ensuring a safer and more reliable dining experience.

[0034] In practical applications, assuming an elderly person is dining in the designated area, the system monitors and provides feedback according to a pre-set workflow. First, a wide-angle depth-of-field camera is installed above or slightly in front of the elderly person's dining area. Its horizontal field of view is no less than 90°, its vertical field of view is no less than 60°, and its depth measurement range is 0.5 meters to 4 meters, ensuring comprehensive coverage of the elderly person's head, torso, and the positional relationship between tableware and food. The wide-angle depth-of-field camera acquires color images and depth information at a frequency of 35 frames per second and provides stable depth measurement capabilities under low-light conditions through a built-in infrared auxiliary light source. Meanwhile, the ambient light detection unit monitors the light intensity in the dining area in real time and transmits the light intensity value to the data processing and feature analysis module for dynamically adjusting the parameters of the visual algorithm, thereby improving adaptability in complex lighting environments.

[0035] Subsequently, an ultrasonic sensor and a thermal imaging sensor acquired micro-vibration signals from the neck and chest regions of the elderly patient, as well as surface temperature signals from the oral cavity region. The ultrasonic sensor operates at a frequency of 40kHz to 50kHz, with a detection accuracy of less than 1 mm and an angular resolution of ±2°. By analyzing the time delay and phase changes of the reflected signals, it extracts the micro-movements on the body surface caused by respiratory rate and heartbeat, as well as the vibration characteristics of neck muscles related to swallowing. The sampling frequency is no less than 250Hz to ensure the capture of short-duration, high-frequency vibration patterns unique to swallowing. The thermal imaging sensor has a temperature measurement range of 25℃ to 50℃, a measurement accuracy of ±0.3℃, and a response time of less than 80 milliseconds. It captures the energy distribution of radiation radiated from the skin surface of the oral cavity region and converts it into a temperature distribution map. Combined with an internal algorithm, it eliminates the influence of ambient temperature fluctuations on the measurement results, thereby stably monitoring temperature changes in the oral cavity region.

[0036] After receiving the aforementioned multimodal data, the data processing and feature analysis module first processes the images and depth information acquired by the wide-angle depth-of-field camera through the visual feature analysis unit. The visual feature analysis unit employs a deep learning model based on convolutional neural networks and attention mechanisms to extract relevant features of the elderly's eating behavior from the images. For example, by analyzing the frequency and amplitude of mandibular key points in the vertical direction, chewing frequency and force are calculated. When the mandibular movement frequency is detected to be below a set threshold or the movement amplitude is significantly reduced, it is considered a potential indicator of chewing difficulties. Simultaneously, by segmenting and identifying the positional relationship between utensils and food, the relative positional relationship and contact events between utensils and the oral cavity are tracked to determine the continuity of the food delivery process. If food is detected to remain at the edge of the oral cavity for an extended period or if the contact between utensils and the oral cavity is abnormally interrupted, it is considered abnormal eating behavior. Furthermore, optical flow analysis of the movement trajectory and amplitude in the neck region serves as auxiliary evidence of swallowing actions, and by segmenting and identifying food residue in and around the oral cavity, it is determined whether food spillage or retention exists.

[0037] The physiological signal analysis unit processes ultrasound signals and thermal imaging data. First, after filtering and denoising the ultrasound signals, spectral analysis is performed to extract micro-motion features to distinguish vibrations caused by breathing, heartbeat, and swallowing. Time-frequency analysis of ultrasound signals from the neck region identifies high-frequency vibration patterns specific to swallowing movements. Multiple high-frequency peaks or significantly reduced vibration amplitude during a single swallowing action are considered indicators of dysphagia. Spectral analysis of ultrasound signals from the chest region calculates respiratory rate and depth in real time. A sudden increase in respiratory rate or abnormal rhythm is considered a stress response following aspiration or coughing. Analysis of micro-motion signals caused by heartbeats calculates heart rate variability. A decrease in high-frequency components or an increase in low-frequency components is considered indirect evidence of autonomic nervous system stress response. Furthermore, time-series analysis of thermal imaging data monitors the rate of change in oral cavity surface temperature in real time. A significantly reduced rate of temperature decrease or prolonged proximity to ambient temperature is considered evidence of interrupted eating behavior.

[0038] The multimodal synchronization unit employs a high-precision timestamp mechanism to uniformly mark the data streams generated by the visual capture unit and the physiological signal sensing unit. It also uses interpolation algorithms or nearest neighbor matching to time-align data from different sensors, providing an accurate temporal basis for subsequent fusion analysis. The visual feature parsing unit and the physiological signal parsing unit transmit the extracted feature vectors to the multimodal synchronization unit via an internal bus, completing the data processing and feature parsing modules.

[0039] After receiving multimodal feature data from the data processing and feature analysis module, the risk assessment and decision-making module first employs a multi-level fusion strategy through the feature fusion unit. This involves concatenating visual feature vectors with physiological signal feature vectors or fusing them using an attention mechanism. Appropriate weights are assigned to different modal features through weighted averaging or adaptive learning to enhance the model's robustness and decision accuracy. The deep learning evaluation unit uses a time-series model based on a recurrent neural network architecture. It receives the fused feature sequence output from the feature fusion unit as input, models the behavioral and physiological changes of elderly individuals during eating, and outputs the probability values ​​of various risk events occurring at the current moment by learning multimodal feature patterns of different risk events. Risk events include difficulty swallowing, aspiration, choking, excessively large food boluses, and interrupted eating. The risk level classification unit, based on the risk probabilities output by the deep learning evaluation unit and combined with preset thresholds and weighting rules, classifies risk events into at least one of the following categories: low risk, medium risk, high risk, and emergency risk. The judgment logic is configured based on clinical medical recommendations and large-scale data analysis results, and can be personalized according to the individual health condition of the elderly person.

[0040] Finally, the alarm and feedback output module generates real-time alarms based on the output of the risk assessment and decision-making module and outputs feedback information through multiple channels. The alarm generation unit generates real-time alarm information based on the output of the risk assessment and decision-making module, and the multi-channel feedback unit provides feedback information to users or caregivers through various means such as sound, light, and screen display. For example, when the system detects that an elderly person is experiencing difficulty swallowing, the alarm and feedback output module immediately provides a voice prompt, "Please be aware of the risk of difficulty swallowing," and flashes a red LED to remind caregivers to take appropriate measures. Simultaneously, the LCD screen displays the risk type and corresponding response suggestions, ensuring that caregivers can quickly understand the risk situation and take appropriate action.

[0041] Through the above steps, the system achieves real-time monitoring and feedback of the elderly's eating safety, ensuring a safer and more reliable eating process. Throughout the process, the various modules work collaboratively, fully utilizing the advantages of multimodal sensing technology to overcome the limitations of a single visual monitoring system, providing comprehensive technical support for the elderly's eating safety.

[0042] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0043] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A safety visual monitoring feedback system based on eating of the elderly, characterized by, include The data acquisition module is used to acquire color images and depth information of the elderly in the eating area, micro-vibration signals in the neck and chest areas, surface temperature signals in the oral cavity area, and ambient light intensity. The data processing and feature parsing module is communicatively connected to the data acquisition module and is used to preprocess the data acquired by the data acquisition module and extract multimodal features. The risk assessment and decision-making module is communicatively connected to the data processing and feature parsing module, and is used to integrate and analyze the multimodal features to identify potential risks; as well as An alarm and feedback output module is communicatively connected to the risk assessment and decision-making module, and is used to generate real-time alarms based on the risk assessment results of the risk assessment and decision-making module and output feedback information through multiple channels; The data acquisition module, the data processing and feature analysis module, the risk assessment and decision-making module, and the alarm and feedback output module interact with each other via the industrial Ethernet communication protocol.

2. The safety visual monitoring feedback system based on eating of the elderly according to claim 1, characterized in that, The data acquisition module includes The visual capture unit, consisting of at least one wide-angle depth-of-field camera, is used to acquire color images and depth information of the elderly in the eating area; The physiological signal sensing unit, including an ultrasonic sensor and a thermal imaging sensor, is used to acquire micro-vibration signals in the neck and chest regions of elderly people and surface temperature signals in the oral cavity region in a non-contact manner. as well as An ambient light detection unit is used to monitor the light intensity of the feeding area in real time and transmit the light intensity value to the data processing and feature analysis module.

3. A safety visual monitoring feedback system based on eating of the elderly according to claim 2, characterized in that, The wide-angle depth-of-field camera is installed above or to the side front of the elderly's eating area, with a horizontal field of view of not less than 90°, a vertical field of view of not less than 60°, a depth measurement range of 0.5 meters to 4 meters, a depth resolution of less than ±2 millimeters at a distance of 2 meters, and an image acquisition frame rate of 35 frames per second. The wide-angle depth-of-field camera has a built-in infrared auxiliary light source, which enables it to provide stable depth measurement capabilities under low-light conditions.

4. The safety visual monitoring feedback system based on eating of the elderly according to claim 2, characterized in that, The ultrasonic sensing device operates at a frequency of 40kHz to 50kHz, has a detection accuracy of less than 1 mm, an angular resolution of ±2°, and a sampling frequency of not less than 250Hz. The ultrasonic sensing device analyzes the time delay and phase changes of the reflected signal to extract the micro-movements on the body surface caused by respiratory rate and heartbeat, as well as the vibration characteristics of neck muscles related to swallowing.

5. The safety visual monitoring feedback system based on eating of the elderly according to claim 2, characterized in that, The thermal imaging sensor has a temperature measurement range of 25℃ to 50℃, a temperature measurement accuracy of ±0.3℃, and a response time of less than 80 milliseconds. The thermal imaging sensor captures the energy distribution of radiation radiated from the skin surface in the oral cavity area and converts it into a temperature distribution map, and combines it with an internal algorithm to eliminate the influence of ambient temperature fluctuations on the measurement results.

6. A safety visual monitoring and feedback system for elderly people eating, as described in claim 1, is characterized in that, The data processing and feature parsing module includes A visual feature parsing unit is used to extract relevant features of the elderly’s eating behavior from the color image and depth information acquired by the visual capture unit. The physiological signal analysis unit is used to perform signal filtering and spectrum analysis on the ultrasound signals and thermal imaging data obtained from the physiological signal sensing unit to extract physiological features; as well as The multimodal synchronization unit employs a high-precision timestamp mechanism to uniformly mark the data streams generated by the visual capture unit and the physiological signal sensing unit, and performs time alignment of data from different sensors through interpolation algorithms or nearest neighbor matching.

7. A safety visual monitoring and feedback system for elderly people eating, as described in claim 6, is characterized in that... The visual feature analysis unit uses a deep learning model based on convolutional neural networks and attention mechanisms to extract features such as the frequency and amplitude of jaw movements, the relative positional relationship between tableware and the oral cavity, the relative angle between the head and the torso, the food residue in the oral cavity, and the movement trajectory and amplitude of the neck region.

8. A safety visual monitoring and feedback system for elderly people eating, as described in claim 6, is characterized in that... The physiological signal analysis unit performs spectrum analysis on the ultrasound signal after filtering and denoising, and extracts the high-frequency vibration mode of swallowing action, respiratory frequency and depth, heart rate variability index, and the rate of change of oral cavity surface temperature as features.

9. A safety visual monitoring and feedback system for elderly people eating, as described in claim 1, is characterized in that, The risk assessment and decision-making module includes The feature fusion unit is used to concatenate the visual feature vectors output by the data processing and feature parsing module with the physiological signal feature vectors or fuse them using an attention mechanism. The deep learning evaluation unit adopts a time-series model based on a recurrent neural network architecture, accepts the fused feature sequence output by the feature fusion unit as input, and outputs the probability values ​​of various risk events occurring at the current moment. as well as The risk level classification unit is used to classify risk events into at least one of the following categories: low risk, medium risk, high risk, and emergency risk, based on the risk probability output by the deep learning evaluation unit and in combination with preset thresholds and weighting rules.

10. A safety visual monitoring and feedback system for elderly people eating, as described in claim 1, characterized in that, The alarm and feedback output module includes An alarm generation unit is used to generate real-time alarm information based on the output results of the risk assessment and decision-making module. as well as The multi-channel feedback unit provides feedback information to users or caregivers through sound, light, and screen display. The alarm generation unit and the multi-channel feedback unit are connected via a real-time communication protocol.

Images