Night safety management method and equipment for nursing home

Abstract

The invention discloses a nursing home night safety management method and equipment, and the method and equipment are applied to an autonomous patrol robot, the autonomous patrol robot comprises an edge calculation unit and a multi-mode sensor, and the method comprises the steps: when the autonomous patrol robot patrols to a monitoring distance of a door, the edge calculation unit calculates the monitoring distance of the door; the edge calculation unit performs outdoor non-contact monitoring through a multi-modal sensor and acquires multi-modal sensing data; determining physical sign data of a human body according to the multi-modal sensing data; and carrying out association analysis on the physical sign data of the human body to obtain physical sign deviation information of the human body so as to carry out risk prediction on the old. Night safety management can be carried out on the nursing home more efficiently, and risk early warning can be carried out quickly and accurately.

Description

Technical Field

[0001] This application relates to the field of safety management technology, and in particular to a method and equipment for nighttime safety management in nursing homes. Background Technology

[0002] Currently, the demand for beds and care in elderly care facilities is constantly increasing. Especially at night, traditional manual patrol and monitoring systems have many drawbacks:

[0003] Insufficient human resources: Night patrols require a large number of people, but nursing staff are limited by factors such as work intensity and night fatigue, which can easily lead to missed detections.

[0004] Many monitoring blind spots: Fixed camera and mattress monitoring solutions have problems such as limited coverage and high false alarm rate in actual use.

[0005] Response delay: After capturing abnormal events, traditional monitoring equipment often requires a certain amount of time for signal transmission to the central control platform for judgment or manual judgment, which may result in a failure to issue timely warnings and increase the risk to the elderly. Summary of the Invention

[0006] This application provides a method and equipment for nighttime security management in nursing homes, which addresses the problems of high manpower costs, high false alarm rates, and response delays in nighttime security management in nursing homes.

[0007] The embodiments of this application adopt the following technical solutions:

[0008] On one hand, this application provides a nighttime security management method for nursing homes, applied to an autonomous patrol robot. The autonomous patrol robot includes an edge computing unit and multimodal sensors. The method includes: when the autonomous patrol robot patrols to the monitoring distance of the door, the edge computing unit performs non-contact monitoring outside the door through the multimodal sensors and collects multimodal sensing data; based on the multimodal sensing data, it determines the human body's vital signs data; and it performs correlation analysis on the human body's vital signs data to obtain the human body's vital signs deviation information, so as to predict the risk of the elderly.

[0009] In one example, when the multimodal sensor is a millimeter-wave radar, determining the vital signs data of a human body specifically includes: analyzing the signal strength at different distances using FFT distance analysis to lock the chest cavity fluctuation signal; extracting the chest cavity displacement signal by converting the phase change of the chest cavity fluctuation signal into physical displacement; separating the respiratory and heartbeat signals from the chest cavity displacement signal according to a preset separation frequency range; and obtaining the elderly person's respiratory rate and heartbeat rate by finding the maximum spectral peak in the respiratory signal frequency band and heartbeat signal frequency band respectively, so as to determine the temporal characteristics of the elderly person's respiratory rate and heartbeat rate.

[0010] In one example, when the multimodal sensor is an infrared thermal imaging sensor, the determination of human vital signs data specifically includes: performing background subtraction on the thermal imaging image sequence and extracting the hot spot region of each frame image;

[0011] The average temperature of the neck region of the hot spot in each frame of the image is calculated to obtain the body surface temperature of the human body; the area and position of the hot spot region are calculated to determine the largest connected region; based on the largest connected region, the centroid of the human hot spot and the bounding box of the human hot spot in each frame of the image are obtained to determine the trajectory features and aspect ratio temporal features of the hot spot centroid; the skeleton is extracted from the largest connected region to obtain the human body posture position in each frame of the image to determine the temporal features of the human body's joint kinematics.

[0012] In one example, when the multimodal sensor is a ToF depth camera, determining the human body's vital signs data specifically includes: modeling a 3D skeleton based on the depth value of each pixel; determining the centroid coordinates of the human skeleton based on the coordinates of key points in the 3D skeleton; determining the boundary of the support area using the coordinates of the foot joints; calculating the minimum distance between the centroid projection and the boundary of the support area based on the centroid coordinates and the boundary of the support area to determine the temporal features of the centroid projection; and extracting the temporal features of key angles of the skeleton based on the coordinates of key points in the 3D human skeleton.

[0013] In one example, the correlation analysis of the human body's vital sign data to obtain information on deviations in vital sign data for risk prediction in the elderly specifically includes: time synchronization and normalization of the human body's vital sign data; sliding window segmentation of the normalized vital sign data to obtain a time window sequence of vital sign data; inputting the time window sequence into a pre-trained LSTM model to obtain the human body's vital sign deviation behavior events and risk warning levels; and making risk predictions for the elderly based on the vital sign deviation behavior events and risk warning levels.

[0014] In one example, the step of predicting the risk of the elderly based on the behavioral deviation event and the risk warning level specifically includes: retrieving the number of occurrences of the behavioral deviation event from the elderly's historical warning records; when the number of occurrences is greater than a preset threshold, retrieving the compensation coefficient corresponding to the number of occurrences in the compensation mapping table; the compensation coefficient is greater than 1; and compensating the risk warning level of the behavioral deviation event based on the compensation coefficient.

[0015] In one example, the method further includes: training an LSTM model based on the correlation between sample vital signs data; the correlation includes: conditions for determining abnormal respiratory rate or heart rate include: respiratory rate greater than a preset respiratory rate threshold or heart rate greater than a preset heart rate threshold; conditions for determining abnormal body temperature include: body temperature higher than a preset body temperature threshold; events for determining bed exit events include at least one of the following: the first distance of the hot spot centroid deviating from the bed boundary exceeds a first preset distance threshold and the deviation time exceeds a preset duration, the second distance of the support area boundary deviating from the bed boundary exceeds a second preset distance threshold and the deviation time exceeds the preset duration; conditions for determining fall events include at least one of the following: the descent rate of the hot spot centroid exceeds a preset velocity threshold, the centroid projection is outside the support area boundary, and the skeletal centroid... The conditions for determining a rolling over event include at least one of the following: the hot spot centroid moves laterally within the bed plane beyond a preset movement threshold and the change in the aspect ratio of the hot spot boundary exceeds a preset boundary change threshold; the hot spot centroid moves laterally within the bed plane beyond a preset movement threshold and the change in the hot spot centroid exceeds a preset centroid change threshold; the conditions for determining a fall from bed event include a sudden change in position and a drastic change in posture; the conditions for a sudden change in position include the hot spot centroid deviating from the bed area boundary at a speed greater than a preset speed threshold, and partially or completely falling into the area outside the bed; the conditions for a drastic change in posture include the conditions for a fall event.

[0016] In one example, the method further includes: updating the occupancy grid map of the patrol environment using the SLAM built into the autonomous patrol robot; constructing a coverage grid corresponding to the occupancy grid map; recording whether each grid has been patrolled in the coverage grid; when a moving object is detected, performing path replanning according to a dynamic window algorithm to avoid the moving object; after avoiding the moving object, checking the missed areas of the coverage grid and determining the center point of the missed area as the next patrol target.

[0017] In one example, the method further includes: mapping the behavioral events of deviating vital signs to risk warning levels to obtain multi-device linkage response information for the elderly; the multiple devices include a warning device carried by a caregiver, an emergency call device in the bed area, and a central control platform; the warning device, emergency call device, and autonomous patrol robot are networked via ZigBee, and the autonomous patrol robot and the central control platform are networked via LoRaWAN; and warning responses are given to the corresponding devices based on the multi-device linkage response information and the networking method.

[0018] On the other hand, embodiments of this application provide a nighttime security management device for nursing homes, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-described nighttime security management methods for nursing homes.

[0019] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0020] An autonomous patrol robot system capable of integrating multiple non-contact vital sign monitoring technologies, multi-sensor fusion, and edge intelligent decision-making algorithms.

[0021] Based on this, mobile non-contact life monitoring systems can provide more efficient and accurate early warnings while reducing reliance on manual patrols.

[0022] Multimodal correlation analysis using multimodal sensing technology fills the gaps in traditional fixed monitoring solutions, improves early warning accuracy, and enables high-precision vital sign monitoring without opening doors, while reducing monitoring interference.

[0023] Data is processed in real time by edge computing units, and the model continuously optimizes itself to meet the requirements of elderly care institutions for efficient early warning. Attached Figure Description

[0024] To more clearly illustrate the technical solution of this application, some embodiments of this application will be described in detail below with reference to the accompanying drawings, in which:

[0025] Figure 1 A flowchart illustrating a nighttime security management method for nursing homes, provided as an embodiment of this application;

[0026] Figure 2 This is a schematic diagram of the structure of a nighttime security management device for a nursing home, provided as an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0029] Figure 1 This is a flowchart illustrating a nighttime security management method for nursing homes, provided as an embodiment of this application. Certain input parameters or intermediate results in this process allow for manual intervention and adjustment to help improve accuracy.

[0030] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a central control platform server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using the edge computing unit built into the autonomous patrol robot as an example.

[0031] In some embodiments of this application, the nighttime autonomous patrol robot system has the advantages of high efficiency, intelligence, low interference, and high privacy protection. An autonomous patrol robot system integrating multiple non-contact vital sign monitoring technologies, multi-sensor fusion, and edge intelligent decision-making algorithms is designed. The system can be specifically described as follows:

[0032] Autonomous navigation module: It uses UWB and laser SLAM fusion positioning technology to achieve accurate positioning, and combines dynamic obstacle avoidance algorithm to ensure that the autonomous patrol robot completes patrol tasks with an efficient path in complex environment.

[0033] It should be noted that the autonomous patrol robot is equipped with a high-performance LiDAR and UWB module, and the hardware system achieves data acquisition and fusion through a high-speed data bus.

[0034] Multimodal sensing module: Integrates multiple sensors to achieve comprehensive monitoring of the elderly's vital signs and behaviors through data fusion technology. For example, sensors may include millimeter-wave radar, infrared thermal imaging, and ToF depth vision.

[0035] Edge intelligent decision-making module: Deploy edge computing units locally on the autonomous patrol robot to analyze monitoring data in real time through deep learning algorithms (such as LSTM) and continuously optimize the risk prediction model using federated learning technology.

[0036] Multi-device linkage communication module: Adopting LoRaWAN and ZigBee dual-mode communication, it enables seamless connection between autonomous patrol robots and bed equipment (such as smart mattresses), emergency call systems and central control platforms, ensuring the timely transmission of early warning information.

[0037] Energy management module: Equipped with dual-battery hot-swappable modules and wireless fast charging technology, it supports long-term (e.g., up to 8 hours) continuous patrol and achieves optimal energy allocation through an intelligent energy management system.

[0038] Low-interference design module: By adopting preset decibel (e.g., 25 decibels) silent drive and vibration suppression technology, the system operating noise is significantly lower than most existing service robots (generally higher than 30 decibels), ensuring a quiet environment during nighttime monitoring and ensuring that the robot will not disturb the elderly during patrols.

[0039] It should be noted that the autonomous patrol robot has a built-in dedicated clock synchronization chip to ensure that the data from multiple sensors are synchronized in time and space at the millisecond level, providing an accurate data foundation for subsequent risk prediction models.

[0040] The external design of the autonomous patrol robot can be as follows:

[0041] 1. The overall structure and dimensions are designed with full consideration of the characteristics of the indoor environment of the nursing home, as detailed below:

[0042] Size design: The autonomous patrol robot can be 120 cm tall, and the sensor module has a height adjustment capability of ±15 cm, which can adapt to the door frame height of different floors and facilitate monitoring the status of different beds.

[0043] Chassis Design: The chassis diameter can be set to 55 cm. It adopts a Mecanum wheel design to achieve omnidirectional movement and is equipped with a suspension and shock absorption system to adapt to complex ground surfaces such as thresholds and carpets, ensuring stable operation during patrols.

[0044] 2. The materials and appearance / workmanship meet hygiene requirements, as detailed below:

[0045] Body Material and Safety Standards: The upper part of the body is made of cylindrical antibacterial silicone material, which complies with GB 15979-2002 standard.

[0046] Antibacterial and stain-resistant: Antibacterial silicone material not only has good antibacterial properties, but is also easy to clean and disinfect, meeting the hygiene requirements of nursing homes.

[0047] Privacy protection: The casing design has no visible camera holes, and the sensor area is covered with a frosted black transparent material, effectively preventing privacy leaks.

[0048] 3. Human-computer interaction design can meet the requirements of friendliness and convenience, as detailed below:

[0049] Touchscreen interface: The top waterproof touchscreen is for caregivers to use only after unlocking with their fingerprints. The interface is simple and intuitive, and supports alarm record query and equipment status monitoring.

[0050] Status indicator light: The bottom ring LED status light emits a breathing blue light when working, and automatically switches to a low-frequency red light when an alarm is triggered, providing caregivers with intuitive feedback on the equipment status.

[0051] Overall appearance: A rounded body design and a simple color scheme can be set so that the robot's appearance is both in line with the warm environment of elderly care facilities and has a high-tech feel.

[0052] 4. The equipment's protection level and environmental adaptability fully consider the application requirements in different environments, as detailed below:

[0053] IP54 protection rating: The entire unit has an IP54 protection rating, which is effective in preventing dust and water damage. For example, it can adapt to working environments with temperatures of 16-28℃ and humidity of 20%-80%.

[0054] Environmental adaptability: In practical applications, the robot can adjust its working mode in a timely manner through internal temperature sensors to ensure stable operation in both high and low temperature environments.

[0055] Based on this, the following methods can be applied to the aforementioned autonomous patrol robots for nighttime security management in nursing homes:

[0056] Figure 1 The process includes the following steps:

[0057] S101: When the autonomous patrol robot patrols to the monitoring distance of the door, the edge computing unit performs non-contact monitoring outside the door through multimodal sensors and collects multimodal sensing data.

[0058] It should be noted that nursing home rooms are usually equipped with viewing windows. When the door is closed, the bed area can be scanned through the viewing window.

[0059] In some embodiments of this application, multimodal sensors are illustrated using millimeter-wave radar, infrared thermal imaging sensors, and ToF depth cameras as examples.

[0060] In some embodiments of this application, a 60GHz millimeter-wave radar can accurately detect respiratory and heartbeat signals in an undisturbed manner in elderly individuals. The working process of the millimeter-wave radar is as follows:

[0061] The transmitter inside the millimeter-wave radar generates high-frequency millimeter-wave signals, which are then radiated towards the target in the form of electromagnetic waves through the antenna. In other words, when used for human body monitoring, the emitted millimeter waves will propagate into the space inside the doorway.

[0062] When millimeter waves encounter a target (such as the human body), they will be reflected and scattered, and some of the electromagnetic waves will return to the radar receiver.

[0063] The receiver of the millimeter-wave radar captures the reflected millimeter-wave signals through the antenna, converts them into electrical signals, amplifies and filters the received electrical signals, and uploads the processed echo signal sequence to the edge computing unit.

[0064] In some embodiments of this application, infrared thermal imaging technology is used for body movement and temperature monitoring. The operation of the infrared thermal imaging sensor is as follows:

[0065] A top-mounted infrared thermal imaging camera can be used to capture continuous thermal images at night, with a frame rate of approximately 10 FPS and a resolution selectable from 8×8 to 32×32 pixels, to balance privacy and monitoring details.

[0066] Infrared thermal imaging sensors receive infrared radiation from various points on an object via an infrared focal plane array (FPA). Each pixel corresponds to an infrared detector, which converts the radiation intensity into an electrical signal. This signal is then processed to generate a thermal image, in which each pixel corresponds to the surface temperature of the target. The thermal image sequence is then uploaded to the edge computing unit.

[0067] In some embodiments of this application, a ToF depth camera can be used to construct a three-dimensional posture model of an elderly person's limbs, that is, to construct a 3D human body model using depth information. The working process of the ToF depth camera is as follows:

[0068] The Time-of-Flight (ToF) depth camera calculates the distance of each pixel using time-of-flight, outputting a depth image (each pixel value representing the target distance) and point cloud data (converting the depth map into a 3D point cloud using camera intrinsics). The depth image sequence and corresponding point cloud data sequence are then uploaded to the edge computing unit.

[0069] S102: Determine the human body's vital signs data based on the multimodal sensing data.

[0070] In some embodiments of this application, the extraction of heartbeat and breathing features is used as an example to illustrate the extraction process for millimeter-wave radar sensing data sequences. The general extraction process is as follows:

[0071] First, the signal intensity at different distances is analyzed using FFT distance analysis to pinpoint the chest cavity fluctuation signal.

[0072] Then, the chest cavity displacement signal is extracted by converting the phase change of the chest cavity fluctuation signal into physical displacement;

[0073] Then, based on the preset separation frequency range, the respiratory and heartbeat signals are separated from the chest displacement signal;

[0074] Finally, by finding the maximum spectral peak in the respiratory signal frequency band or the heartbeat signal frequency band, the respiratory rate or heartbeat rate of the elderly person is obtained, so as to determine the temporal characteristics of the elderly person's respiratory rate and heartbeat rate.

[0075] The extraction process includes signal preprocessing (locking the chest cavity fluctuation signal), phase extraction and micro-motion detection, separation of respiratory and heartbeat signals, and spectrum analysis and peak detection, as follows:

[0076] 1. The signal preprocessing process includes FFT analysis, clutter suppression, and dynamic component enhancement, as detailed below:

[0077] FFT analysis:

[0078] First, a range-based FFT is performed on each chirp (frequency-modulated pulse) to convert the time-domain signal to the range domain. Finally, the target's range cell is locked (i.e., the chest rise / fall signal is locked), eliminating other range interference, and used as the object for subsequent data analysis. It should be noted that the millimeter-wave radar transmits an FMCW signal. Then, the receiver acquires the baseband I / Q signals via an A / D converter.

[0079] It should be noted that pulse noise removal is performed on the IF signal, and DC components and background clutter are removed to reduce the impact of low-frequency drift on respiratory detection.

[0080] For example, a millimeter-wave radar with a detection range of 0.5-5 meters might show strong reflection from a target at 2.3 meters using range FFT. The time-domain signal for this range cell is extracted, and the reflection components from static objects (such as walls) are removed.

[0081] Clutter suppression and dynamic component enhancement:

[0082] High-pass filtering or adaptive filtering (such as RLS) is used to suppress static target echoes, focusing attention on time-varying micro-motion signals and improving the response sensitivity to small thoracic amplitudes.

[0083] For example, by using an RLS filter and referencing the static channel signal as a noise reference source, background clutter can be suppressed in real time, increasing the amplitude of the chest cavity fluctuation signal from 0.01V to 0.1V (a 10-fold increase in signal-to-noise ratio).

[0084] 2. The phase extraction and micro-motion detection process includes phase demodulation and millimeter-wave displacement-phase mapping.

[0085] For example, if a phase change of 0.2 rad is detected, λ = 5 mm, the chest cavity displacement Δd = (0.2 × 5 mm) / (4π) ≈ 0.08 mm (typical heartbeat displacement).

[0086] 3. The process of separating respiratory and heartbeat signals includes bandpass / filter scheme design and advanced separation algorithms, as detailed below:

[0087] Bandpass / filter design:

[0088] The preset separation frequency range can be set as follows: for respiration, the typical respiratory frequency band can be set to 0.1–0.6Hz (6–36 breaths / minute), and low-frequency components can be extracted using digital IIR or FIR filters; for heartbeat, the heart rate frequency band can be set to 0.8–4Hz (48–240 beats / minute), and mid-to-high frequency components can be extracted using high-pass or band-pass filters.

[0089] Advanced Separation Algorithms:

[0090] DR-MUSIC: By combining RLS adaptive filtering and MUSIC spectrum estimation, it effectively suppresses respiratory harmonic interference and achieves signal decomposition of both. That is, it dynamically suppresses the respiratory fundamental frequency through RLS adaptive filtering and estimates the heartbeat spectrum through the MUSIC algorithm, thus avoiding respiratory harmonic interference.

[0091] Wavelet decomposition / EMD: By performing wavelet packet decomposition or empirical mode decomposition on the phase sequence, the respiratory and heart rate components are separated by frequency bands and their respective time series are reconstructed.

[0092] For example, the heartbeat in the original mixed signal is drowned out by respiratory harmonics (such as the second harmonic of breathing at 1.2 Hz). DR-MUSIC identifies the true heartbeat spectrum peak at 1.8 Hz (accuracy ±0.1 Hz).

[0093] 4. The spectrum analysis and peak detection process includes FFT / power spectral density, peak tracking, and smoothing, as detailed below:

[0094] FFT / Power Spectral Density:

[0095] The power spectral density (PSD) is estimated by performing a short-time FFT or Welch method on the filtered signal block. The maximum spectral peak is found in the frequency band corresponding to breathing and heartbeat, and the corresponding frequency is output as breathing / heart rate.

[0096] Peak tracking and smoothing:

[0097] To avoid jumps caused by transient noise, Kalman filtering or moving average smoothing can be applied to the detected values ​​in the frequency or time domain to ensure the stability of continuous measurements.

[0098] For example, a certain frame of signal PSD shows significant spectral peaks at 0.28Hz (breathing) and 1.15Hz (heartbeat), and after moving average, outputs a stable value for 10 consecutive seconds.

[0099] Based on this, we can finally obtain the respiratory rate change sequence and the heart rate change sequence.

[0100] In summary, millimeter-wave radar can accurately detect respiratory and heartbeat signals in elderly individuals without interference, achieving a monitoring accuracy of ±0.5 breaths per minute. Furthermore, it possesses penetrating power, allowing detection even through wooden doors when they are closed, ensuring comprehensive and accurate data collection. After rapid processing, the radar signals can provide immediate feedback on abnormal conditions, offering timely evidence and real-time response for early warning mechanisms, such as in cases of fainting in the elderly.

[0101] In some embodiments of this application, infrared thermal imaging sensing data are used as examples to illustrate the extraction process, taking body surface temperature and body motion characteristics (such as the centroid of human hot spots, aspect ratio of the human body, joint velocity, acceleration, and other joint kinematic timing characteristics).

[0102] First, background subtraction is performed on the thermal imaging image sequence to extract the high-temperature hot spot region of each frame.

[0103] It should be noted that end-to-end learning can be performed using a 3D convolutional neural network to automatically extract dynamic hotspot features.

[0104] Then, the average temperature of the neck region of the hot spot in each frame of the image is calculated to obtain the body surface temperature of the human body.

[0105] For example, a pixel box of a preset size is extracted from the neck region of the hot spot, and the surface temperature of each frame of the image is obtained by calculating the average temperature of the pixel box region. The average surface temperature of each frame of the image is then calculated.

[0106] Then, the largest connected region is determined by calculating the area and location of the hot spot region.

[0107] Then, based on the maximum connected region, the centroid of the human hot spot and the bounding box of the human hot spot are obtained for each frame of the image to determine the trajectory features of the hot spot centroid and the aspect ratio temporal features.

[0108] Then, the skeleton is extracted from the largest connected region to obtain the human pose position of each frame image, so as to determine the temporal characteristics of human joint kinematics.

[0109] The extraction process includes hot spot detection and temperature distribution analysis, as well as skeleton and human posture extraction, as detailed below:

[0110] 1. Hot spot detection and temperature distribution analysis

[0111] Baseline background modeling (such as GMM or average background) is performed on each frame of infrared thermal imaging image and the difference is made with the current frame to extract high-temperature hot spots (human heat source areas).

[0112] Then, connected component analysis is performed on the hot spot region to extract the largest connected region as the human body heat source, and the centroid position, area and boundary length and width of the human body are calculated for subsequent behavior discrimination.

[0113] For example, a thermal imager mounted on the ceiling of a nursing home room detected two heat sources: the primary target, occupying 15% of the area and with its centroid located in the center of the bed (identified as a bedridden elderly person); and the secondary target, occupying only 3% of the area and with its centroid located in the corner (identified as a heater). The system automatically locked onto the primary target and began tracking it.

[0114] It should be noted that Kalman filtering can also be applied to smooth the hot spot centroid trajectory.

[0115] 2. Skeleton and Human Posture Extraction

[0116] A deep learning-based skeleton extraction method is used to detect human bodies in thermal imaging images (e.g., using AlphaPose or YOLOv7POSE), and then outputs the coordinates of 17 joints for fine-grained pose analysis. These 17 joint coordinates include: head feature points (nose, eyes, ears), torso features (shoulders, hips), and limb joints (elbows, knees, hands, feet).

[0117] Then, the joints are connected to form a dynamic skeletal model, which can reflect changes in body posture in real time.

[0118] Finally, the joint coordinates are normalized, and the kinematic characteristics such as joint velocity and acceleration are calculated.

[0119] It should be noted that thermal imaging images are more accurate in predicting different torso positions, therefore, thermal imaging is used to predict kinematic characteristics.

[0120] In summary, infrared thermal imaging can accurately capture changes in the surface temperature of an elderly person's body and analyze abnormal body temperatures. Furthermore, by combining body temperature distribution with thermal imaging images, the system can help determine if an elderly person is at risk of getting out of bed, falling, or falling out of bed. In addition, the infrared thermal imaging images are blurred to avoid exposing personal privacy information while still retaining sufficient behavioral analysis data.

[0121] In some embodiments of this application, for ToF sensing data, the distance between the center of gravity projection and the boundary of the support area and the body motion characteristics are used as examples for illustration. The general extraction process is as follows:

[0122] First, a 3D skeleton model is created based on the depth value of each pixel.

[0123] Then, based on the coordinates of the key points of the three-dimensional skeleton, the coordinates of the centroid of the human skeleton are determined.

[0124] Then, the boundaries of the support area are determined using the coordinates of the foot joints.

[0125] Then, based on the coordinates of the centroid of the human skeleton and the boundary of the support area, the minimum distance between the centroid projection and the boundary of the support area is calculated to determine the temporal characteristics of the centroid projection.

[0126] In addition, based on the coordinates of key points of the human three-dimensional skeleton, temporal features of key angles of the skeleton are extracted.

[0127] 1. Calculation of the centroid of a 3D skeleton

[0128] First, obtain the position sequence of key points of the human 3D skeleton (such as head, shoulders, hips, knees, ankles, etc.).

[0129] Then, according to the mass distribution of the human body linkage (such as the mass ratio of each standard body part corresponding to each joint), the position of the center of mass of each joint is calculated.

[0130] Then, the coordinates of the overall body's center of mass are obtained by taking a mass-weighted average of all the center of mass locations.

[0131] 2. Centroid Projection and Supporting Polygon Analysis

[0132] First, the three-dimensional centroid is projected onto the horizontal plane of the ground to obtain the projection point.

[0133] Then, the boundaries of the support area are determined by the ToF foot joints.

[0134] Then, the minimum distance from the centroid projection point to the boundary of the support region is calculated to obtain the centroid projection time series characteristics.

[0135] 3. Key Angle Feature Extraction

[0136] Based on the key point coordinates of the human 3D skeleton (such as head, shoulder, hip, knee, ankle, etc.), key angles (such as knee flexion and extension, trunk forward tilt angle, spinal curvature, trunk-to-ground angle, shoulder-hip rotation angle, etc.) are extracted, and biomechanical features are tracked.

[0137] For example, a typical scenario could be when a bedridden elderly person sits up: the head joint rises 30 centimeters, the distance between the shoulder and hip joints shortens, and the skeletal angle changes from 180° when lying flat to 110° when sitting. Or, for example, if the torso tilt angle is greater than 15 degrees, a fall forward tilt event can be determined.

[0138] S103: Perform correlation analysis on the vital sign data of the human body to obtain information on deviations in the vital sign data, so as to predict the risk of the elderly.

[0139] In some embodiments of this application, based on the characteristics of the vital signs data extracted in this application, a deep learning model based on LSTM (Long Short-Term Memory Network) is used to achieve long-term memory and trend prediction of the elderly's behavioral patterns through time series data analysis.

[0140] Based on this, the process of correlation analysis of human vital sign data is as follows:

[0141] First, the human body's vital signs data are synchronized and normalized over time.

[0142] Then, the normalized vital signs data are divided into sliding window segments to obtain the time window sequence of the vital signs data.

[0143] Then, the time window sequence is input into the pre-trained LSTM model to obtain the human body's deviation from behavioral events and risk warning levels.

[0144] Finally, risk prediction is conducted for the elderly based on deviations from human physical signs, behavioral events, and risk warning levels.

[0145] In summary, LSTM networks can process complex time-series data, perform correlation analysis on parameters such as respiration, heart rate, and body movement, and promptly capture abnormal patterns.

[0146] In some embodiments of this application, it is necessary to pre-mine the correlation relationships and pre-train an LSTM model based on the correlation relationships between sample vital sign data. The correlation relationships can be as follows:

[0147] The conditions for determining abnormal respiratory rate or heart rate include: respiratory rate greater than a preset respiratory rate threshold or heart rate greater than a preset heart rate threshold.

[0148] The criteria for determining abnormal body temperature include: body temperature exceeding a preset body temperature threshold.

[0149] The conditions for determining an off-bed event include at least one of the following: the first distance by which the hot spot centroid deviates from the boundary of the bed area exceeds a first preset distance threshold and the deviation time exceeds a preset duration; the second distance by which the boundary of the support area deviates from the boundary of the bed area exceeds a second preset distance threshold and the deviation time exceeds a preset duration; and the descent height of the ankle joint is lower than a preset height threshold.

[0150] The conditions for determining a fall event include at least one of the following: the rate of descent of the hot spot centroid height exceeds a preset speed threshold; the rate of change of the hot spot aspect ratio exceeds a set speed threshold; the projection of the skeletal centroid is within the boundary of the support area and the projection distance is less than a preset distance threshold; the projection of the centroid is outside the boundary of the support area; the key angles of the skeletal frame do not conform to the preset angle range and the rate of increase of the kinematic values ​​of the joints is greater than a preset increase speed threshold; the skeletal centroid height decreases and the forward tilt angle of the torso is greater than a preset forward tilt angle threshold (e.g., 45 degrees) and the duration is greater than a preset duration threshold (e.g., 0.5 seconds).

[0151] The conditions for determining a turning-over event include at least one of the following: the hot spot centroid moves laterally within the bed plane beyond a preset movement threshold and for a preset duration; the hot spot centroid moves laterally within the bed plane beyond a preset movement threshold and the change in the hot spot aspect ratio exceeds a preset boundary change threshold; or the hot spot centroid moves laterally within the bed plane beyond a preset movement threshold and the change in the hot spot centroid exceeds a preset centroid change threshold.

[0152] For example, a thermal spot centroid shift of ≥15cm can effectively correspond to a complete rollover and can be fused with skeletal data to further reduce false alarms.

[0153] The core criteria for determining a fall from bed: A fall from bed is generally understood as an unexpected and sudden event of falling off the bed. It is essentially a fall that occurs in the bed area or at the edge of the bed. Therefore, the criteria should combine the characteristics of leaving the bed (change of position) and falling (drastic change of posture).

[0154] In other words, the positional conditions are similar to leaving the bed but more sudden (for example, the center of mass moves out of the bed boundary in a short period of time), and the conditions for drastic changes in posture are similar to falling.

[0155] Therefore, the conditions for determining a bed fall incident include a sudden change in position and a drastic change in posture.

[0156] Among them, the conditions for drastic position changes include the hot spot centroid deviating from the boundary of the bed area at a speed greater than a preset speed threshold and falling partially or completely into the area outside the bed.

[0157] The conditions for drastic changes in posture include at least one of the following: the rate of decrease in the height of the hot spot's center of mass exceeds a preset speed threshold; the rate of change in the aspect ratio of the hot spot exceeds a set speed threshold; the projection of the center of mass is within the boundary of the support area and the projection distance is less than a preset distance threshold; the projection of the center of mass is outside the boundary of the support area; the key angles of the skeleton do not conform to the preset angle range and the rate of increase in the kinematic values ​​of the joints is greater than a preset increase speed threshold; the height of the skeleton's center of mass decreases and the forward tilt angle of the torso is greater than a preset forward tilt angle threshold.

[0158] For example, in thermal imaging: the height of the hot spot centroid decreases at a rate exceeding a preset speed threshold (e.g., a decrease rate greater than 1 m / s) within a very short time (e.g., 0.2-0.5 seconds).

[0159] The aspect ratio of the hot spot (reflecting body posture, such as from lying / sitting to stretching) changes drastically at the same time and the rate of change exceeds the threshold.

[0160] ToF skeleton level: The overall center of mass height of the skeleton drops more than a preset height threshold in a very short time (e.g., 0.2-0.5 seconds) (e.g., a drop of more than 40 cm).

[0161] The torso tilt angle increases sharply at the same time and exceeds the preset angle threshold (e.g., greater than 45 degrees).

[0162] A sudden, involuntary change in joint angle characteristics (such as hip and knee angles) exceeding a threshold was detected.

[0163] If the projection point of the skeletal centroid is outside the boundary of the support area (indicating that the body has lost support) or although it is inside the boundary, the projection distance is extremely small (indicating that the support is unstable and the body is on the verge of falling).

[0164] It should be noted that the duration of the above-mentioned drastic posture changes usually needs to be sustained for a very short preset duration (e.g., >0.3 seconds) to eliminate noise.

[0165] In one example, an elderly person attempted to sit up and get out of bed, but suddenly lost their balance. The system detected that the center of mass of the hot spot instantly moved out of the center of the bed and crossed the bed boundary. At the same time, the height of the hot spot center of mass dropped by 50 centimeters in 0.4 seconds (at a speed greater than 1.25 meters per second), and the shape of the hot spot (length-to-width ratio) rapidly elongated from an approximately square (sitting posture) (body extended and falling). The system determined this to be a fall from the bed.

[0166] In another example, an elderly person accidentally rolled off the bed while turning over. The ToF sensor detected that the projection of the body's overall center of mass rapidly moved from inside the bed to the area outside the bed. During this movement, the torso's forward tilt angle increased sharply from 20 degrees to 70 degrees within 0.3 seconds, while the overall center of mass height decreased by 55 centimeters within 0.5 seconds. Additionally, the ankle height instantly rose and then dropped sharply (reflecting the feet leaving the ground and the body falling). The system determined this to be a fall from the bed event.

[0167] In another example, an elderly person sitting on the edge of the bed leaned forward excessively, causing them to fall forward. The system detected that the center of mass projection point was briefly located near the bed boundary, while the torso's forward tilt angle instantaneously exceeded 50 degrees, and the center of mass projection point was clearly outside the boundary of the foot support area (the body's center of gravity exceeded the range that the feet could support). The system determined this to be an impending or ongoing fall from the bed (forward fall).

[0168] It is important to note that the key characteristic of a fall from bed is a rapid, involuntary, and uncontrolled change in posture and a fall from the bed or its edge. It combines the features of leaving the bed area with a posture consistent with a fall, and the process is extremely rapid and violent, unlike a smooth exit from the bed.

[0169] Based on this, the process of training the LSTM model is as follows:

[0170] First, obtain event labels for sample feature data and sample vital signs data. Sample vital signs data include respiratory rate time series features, heart rate time series features, centroid trajectory features, aspect ratio time series features, joint kinematic time series features, centroid projection time series features, and skeletal key angle time series features, etc.

[0171] It should be noted that manually or semi-automatically labeling events such as getting out of bed, falling, and falling from bed provides supervised labels for subsequent model training.

[0172] Then, the sample feature data is time-synchronized, normalized, and segmented using a sliding window.

[0173] It should be noted that the sequences are aligned at the same sampling frequency and standardized (zero mean, unit variance) to eliminate the influence of different units.

[0174] Finally, based on the sample feature data of the time window sequence and the corresponding event labels, the initial LSTM model architecture is trained to obtain the LSTM model.

[0175] It should be noted that the LSTM model architecture can be designed as follows:

[0176] 1. Multiple Input Multiple Output Structure: The three-dimensional vector of each time step is simultaneously input into the LSTM unit, and the hidden layer state retains information in both the time dimension and the feature dimension, capturing the correlation between variables.

[0177] 2. Enhanced Association Modeling Through Attention Mechanisms: Adding an attention layer to the LSTM allows the model to explicitly focus on the interactions between different parameters at different time points. ATT-LSTM filters out irrelevant information through learnable attention weights, significantly improving the accuracy of multivariate prediction and association analysis.

[0178] 3. Correlation analysis and anomaly detection

[0179] Multi-step prediction: Predicts the next m steps using the past n steps. Outliers are identified when the difference between the prediction and the true value exceeds the tolerance range.

[0180] Cyclic correlation measurement: During training or inference, Pearson correlation coefficients of different channels in the hidden layer output can be calculated to analyze the real-time coupling between respiration and heartbeat / body movement. If an unreasonable relationship such as cardiac arrest with continued respiration is detected, further alerts can be issued.

[0181] Clustering preprocessing removes noisy features that are irrelevant to the detection of the main anomaly, thus improving the overall analysis results.

[0182] It should be noted that the LSTM model training design can be as follows:

[0183] 1. Training Strategies

[0184] Training data includes abnormal state data and normal state data. Training the model with normal state data can optimize loss functions such as mean squared error (MSE) or log-likelihood.

[0185] Hierarchical attention or federated learning architectures can be introduced to further enhance the ability to learn the correlation patterns of different data sources.

[0186] 2. Evaluation Indicators

[0187] Reconstruction / prediction error distribution: ROC-AUC, Precision@k, Recall, etc. are used to measure anomaly detection performance.

[0188] Cross-variable consistency: Calculate the covariance of errors of each signal in multivariate prediction to ensure that when the model captures an anomaly in one signal, the errors of other related signals also increase synchronously, thus verifying the effectiveness of the correlation analysis.

[0189] In summary, the risk prediction model has been validated by a large amount of experimental data, and its performance is as follows:

[0190] Prediction accuracy: Through joint analysis of historical and real-time vital signs data, the system achieves an accuracy of over 92% in predicting bed falls, with an AUC value of 0.89.

[0191] Improved response time: After adopting this system, the detection time of high-risk events has been shortened from the traditional 15 minutes to less than 3 minutes, effectively reducing the risk level after an accident occurs.

[0192] Experimental data shows that the model achieves an accuracy rate of over 92% in predicting the risk of bed falls, with an AUC (area under the curve) of 0.89, demonstrating high reliability in early warning. Furthermore, it possesses adaptive learning capabilities, meaning the model has a self-optimizing function; as the amount of data increases, it continuously improves the accuracy of risk prediction through federated learning techniques.

[0193] In some embodiments of this application, considering the strong regularity and repetitiveness of elderly people's behavioral patterns, such as the occurrence of multiple falls from bed, this application utilizes this characteristic to improve the accuracy of risk prediction.

[0194] Based on this, the frequency of behavioral events was retrieved from the elderly person's historical warning records.

[0195] When the number of occurrences exceeds a preset threshold, the compensation coefficient corresponding to the number of occurrences is retrieved from the compensation mapping table; the compensation coefficient is greater than 1.

[0196] The risk warning level of a behavioral event is compensated based on a compensation coefficient to obtain the behavioral event and its risk level for the elderly. The compensation process can involve multiplying the compensation coefficient by the risk warning coefficient of the risk warning level, and then converting the compensated risk warning coefficient into a risk warning level.

[0197] In summary, non-contact vital sign monitoring allows for the real-time and accurate collection of key data such as the elderly's breathing, heartbeat, and body movement without opening the door.

[0198] Highly efficient risk prediction: Based on LSTM neural networks and federated learning technology, it can promptly and accurately identify high-risk events such as falls from beds and falls.

[0199] In some embodiments of this application, a fusion positioning technology of UWB (Ultra-Wideband) and laser SLAM (Simultaneous Localization and Mapping) is employed, enabling high-precision positioning with an error of less than a few centimeters (e.g., 3 centimeters) in the complex environment of nursing homes. Its core technological advantages are:

[0200] High-precision positioning: UWB signals have good stability when penetrating obstacles such as walls and furniture, while laser SLAM provides auxiliary correction by building an environmental map in real time, which significantly improves the overall positioning accuracy.

[0201] Dynamic obstacle avoidance: With the help of dynamic obstacle avoidance algorithms, the robot can quickly adjust its route to avoid collision risks when it encounters elderly people walking or other moving obstacles.

[0202] Real-time feedback: Location data is updated in real time, and patrol routes are dynamically adjusted according to the current environment to ensure that every area can be fully monitored.

[0203] Based on this, the patrol path is dynamically adjusted according to the current environment, and the real-time environmental perception and coverage map generation process can be as follows:

[0204] First, the occupancy grid map of the patrol environment is updated using the SLAM built into the autonomous patrol robot.

[0205] Then, construct the covering grid corresponding to the occupancy grid map. Within the covering grid, record whether each grid has been patrolled.

[0206] Then, when a moving object is detected, path replanning is performed according to the dynamic window algorithm to avoid the moving object.

[0207] After avoiding moving objects, check the missed areas of the covered grid and identify the center point of the missed area as the next patrol target.

[0208] Based on this, the specific process of the above scheme can be as follows:

[0209] 1. Real-time environmental perception design scheme

[0210] First, the autonomous robot continuously builds and updates the occupancy grid map of the environment using laser SLAM or visual SLAM, while maintaining a corresponding coverage grid to record whether each grid has been patrolled. Uncovered areas are marked as pending patrol in the coverage map.

[0211] Each time new sensor data is acquired, the system immediately updates the occupied and covered grids and maps newly detected dynamic obstacles into the grids in real time, providing accurate input for subsequent path replanning.

[0212] 2. Complete coverage path planning and design scheme

[0213] Initial path generation: The entire patrol area is decomposed using Boustrophedon or plowshare-style segmentation, dividing the space into several connected sub-regions, and generating plowshare-style paths within each sub-region to ensure that the initial paths are non-overlapping and fully covered.

[0214] Dynamic goal selection and task allocation:

[0215] Dynamic target selection: Real-time scanning of leading points in the coverage grid, i.e., the grids immediately adjacent to the covered / uncovered boundaries, and using them as the next patrol target to avoid missing any corners.

[0216] Online task allocation mechanism: If multiple robots are deployed, K-means clustering and dual optimization can be combined to divide the area to be covered into multiple task units, and then each robot can be assigned the nearest and uncovered sub-region, which can improve the efficiency of multi-robot collaboration and further avoid path conflicts.

[0217] Based on this, the online path replanning and dynamic obstacle avoidance methods are as follows:

[0218] Local path replanning: When a new dynamic obstacle (such as a caregiver or mobile device) appears in front, the robot triggers a dynamic window algorithm (such as DWA) or an improved RRT in the local grid to generate an obstacle avoidance path in real time, bypass the obstacle, and then return to the global coverage trajectory.

[0219] Scrolling window prediction: The robot continuously maintains a scrolling window, predicts the short-term movement trajectory of obstacles within the area, and adjusts its speed and direction in advance based on the prediction results to further improve the obstacle avoidance success rate.

[0220] It should be noted that a hierarchical hybrid decision-making structure can also be adopted: the global layer first generates the coverage path, and then the local layer schedules deep learning or reinforcement learning models in real time to perform fine-grained obstacle avoidance and path fine-tuning. For example, LSTM and federated learning models jointly make decisions on dynamic obstacles and coverage progress at the local edge unit to ensure that coverage efficiency is not significantly affected during obstacle avoidance.

[0221] 4. Coverage loop and integrity guarantee

[0222] After each obstacle avoidance or brief deviation, the robot will backtrack and cover the grid, select the next priority target in the current patrol area, and ensure that no grid is missed. The system will end the patrol task when all grids have been visited.

[0223] In summary, by continuously comparing covered and uncovered states, and combining the dual optimization goals of minimizing distance and maximizing coverage area, the robot can achieve a balance between dynamic obstacle avoidance and full coverage, ultimately ensuring that every room, corridor, and other area is fully monitored at night.

[0224] In some embodiments of this application, LoRaWAN and ZigBee dual-mode communication protocols are supported. For example, the communication network is composed as follows:

[0225] The alert and warning devices carried by caregivers (such as AR glasses and wristbands), the emergency call devices in the bed area (such as bedside alarms and smart mattress alarms), and the autonomous patrol robots are networked via ZigBee. The autonomous patrol robots are networked with the central control platform via LoRaWAN.

[0226] In summary, low-latency communication: dual-mode communication ensures that the data transmission delay between devices is less than 1.2 seconds, and the warning information can be transmitted to the caregiver terminal in the shortest possible time.

[0227] Cross-device collaboration: The robot seamlessly integrates with the smart mattress, emergency call system, and central control platform to jointly build a complete risk warning network.

[0228] Remote monitoring: The central control platform aggregates and analyzes real-time monitoring data from the entire hospital through the data center, guiding caregivers to take rapid emergency measures in critical moments.

[0229] It should be noted that in the early warning system, different risk levels correspond to different emergency response strategies, and the response process is as follows:

[0230] First, behavioral events and risk warning levels are mapped to obtain multi-device linkage response information for the elderly.

[0231] Then, based on the multi-device linkage response information and networking method, early warning responses are issued to the corresponding devices.

[0232] For example, the emergency response strategy is as follows:

[0233] Low-risk events (minor anomalies): Detailed status information will be pushed to caregivers via AR glasses to remind them to pay attention.

[0234] Medium-risk events (when obvious abnormalities are detected but have not yet posed a direct threat): The system will simultaneously activate bedside warning reminders and caregiver wristband alarms to urge caregivers to arrive at the scene in a timely manner.

[0235] High-risk events (for situations where abnormal data is obvious and the risk warning confidence level reaches 90% or higher): The system will immediately trigger multiple alarm mechanisms and record the full-cycle data for further analysis and model optimization.

[0236] In some embodiments of this application, to address the need for long-duration nighttime patrols, this system employs dual-battery hot-swappable technology, as detailed below:

[0237] Continuous power supply: Even if a single battery is depleted, the system can automatically switch to a backup battery to ensure uninterrupted monitoring.

[0238] Wireless fast charging technology: The charging efficiency can reach up to 85%, supporting the completion of charging tasks in a short time, thereby reducing system downtime.

[0239] Intelligent energy management: By monitoring the energy consumption of each module in real time, the system can automatically adjust the working status of devices such as radar and edge computing units to achieve optimal energy allocation.

[0240] In some embodiments of this application, data protection is achieved through federated learning. The federated learning model plays a crucial role in data privacy protection, as detailed below:

[0241] Distributed data storage: Different elderly care institutions can process data locally, avoiding the centralized transmission of sensitive information to the cloud.

[0242] Collaborative modeling: Various elderly care institutions can participate in model training together to optimize the global model while protecting data privacy.

[0243] Model updates: Federated learning not only improves the accuracy of prediction models, but also ensures the data security of all participants. When updating and iterating the model, historical data can be used for training, and the more data the model processes, the higher the recognition rate.

[0244] It should be noted that after the robot deploys the model, AI benchmark tests can be performed to test the model's maximum computing power.

[0245] It should be noted that, although the embodiments in this application are based on... Figure 1 Steps S101 to S103 will be described sequentially, but this does not mean that steps S101 to S103 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1 The order in which steps S101 to S103 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S101 to S103 can be appropriately adjusted according to actual needs.

[0246] pass Figure 1 The method enables the development of an autonomous patrol robot system that integrates multiple non-contact vital sign monitoring technologies, multi-sensor fusion, and edge intelligent decision-making algorithms.

[0247] Based on this, the system detects breathing and heartbeat signals using millimeter-wave radar, captures body temperature and movement information using infrared thermal imaging, and constructs a three-dimensional human posture model using a ToF depth camera, thereby enabling real-time monitoring and prediction of risks such as falls and bed falls for the elderly without opening the door.

[0248] The robot can be equipped with a lightweight AI model. In the process of multimodal predictive analysis, the system also adopts a risk prediction model based on LSTM neural network and federated learning to conduct in-depth analysis of real-time sensor data and the elderly’s historical behavior patterns, and finally achieve a high-precision and high-response early warning mechanism.

[0249] Federated learning risk prediction model: While ensuring the data privacy of each institution, it significantly improves the accuracy of risk prediction and the overall robustness of the system through distributed data collaborative training.

[0250] In addition, all vital sign detection data are anonymized to fully consider privacy protection and data security. Under the premise of ensuring privacy and security, all detection data are linked with the central control system through LoRaWAN and ZigBee dual-mode communication technology to achieve graded response and emergency handling.

[0251] In addition, the robot is equipped with a high-performance LiDAR and UWB module. The hardware system achieves data acquisition and fusion through a high-speed data bus, and the built-in dedicated clock synchronization chip ensures that the data from multiple sensors are synchronized in time and space at the millisecond level.

[0252] In addition, by adopting preset decibel silent drive and vibration suppression technology, the system's operating noise is significantly lower than that of most service robots on the market.

[0253] In addition, the dual-battery hot-swappable design allows the system to automatically switch to the backup battery even if one battery is depleted, ensuring uninterrupted monitoring.

[0254] Based on this, the advantages of the proposed solution are as follows:

[0255] Mobile, non-contact life monitoring systems can provide more efficient and accurate early warnings while reducing reliance on manual patrols.

[0256] Data fusion: Multi-sensor data fusion technology comprehensively considers millimeter wave, infrared and ToF data, thereby filling the gaps in traditional monitoring solutions and effectively avoiding measurement deviations that may occur with a single sensor in dynamic scenarios.

[0257] Non-invasive detection: Sensor data is collected without contact outside the door. The elderly are unaware of the patrol robot's monitoring process, which reduces interference during risk monitoring. At the same time, the elderly do not need to wear the device, eliminating the discomfort of wearing the device.

[0258] The model's predictions are accurate: Experimental data shows that in a 30-day simulation test, the system's false alarm rate for bed falls was only 2.1 times per thousand beds per night, and the missed alarm rate was controlled within 0.3% (confidence level 95%), which is significantly better than the false alarm rate of 5.8 times per thousand beds per night of the traditional camera solution.

[0259] Real-time dynamic optimization: Data is processed in real time through an edge computing platform, and the model continuously optimizes itself to meet the requirements of elderly care institutions for efficient early warning.

[0260] Low-interference design and energy consumption control: To ensure that the robot will not disturb the elderly during patrols, the system has been finely tuned in both hardware and software design.

[0261] In summary, by employing autonomous patrols, non-contact monitoring, and data anonymization (traditional monitoring methods carry the risk of privacy leaks), nighttime monitoring becomes more intelligent, less disruptive, and possesses high privacy protection capabilities. This not only effectively reduces the nighttime manpower costs for nursing homes but also significantly improves monitoring accuracy and security, providing solid technical support for building a new model of smart elderly care.

[0262] To demonstrate the system's practical application capabilities more intuitively, the following example, using nighttime monitoring of a nursing home bed in room 203, illustrates the implementation process in detail:

[0263] Patrol initiated:

[0264] At 8:00 PM, the autonomous patrol robot activated its autonomous navigation system and entered the area outside room 203 according to the preset patrol route.

[0265] The sensor module is activated, and various sensors begin to collect data and upload the real-time data to the edge computing platform for preliminary processing.

[0266] Data acquisition and real-time monitoring:

[0267] Infrared thermal imaging detection: Through the thermal imaging system, the temperature distribution of the bed and surrounding environment is monitored in real time, and it was found that part of the elderly person's upper body extended 30 centimeters beyond the edge of the bed.

[0268] Millimeter-wave radar detection: An abnormal respiratory rate was detected in the elderly person, with a recorded data of 22 breaths per minute, which is far higher than the normal level.

[0269] ToF depth camera modeling: Constructing a 3D human posture model, analysis revealed that the elderly person's skeletal center of gravity offset rate exceeded 60%, indicating a significant risk of imbalance.

[0270] Risk assessment and early warning triggering:

[0271] The edge computing platform used an LSTM risk prediction model to calculate a risk warning confidence level of 93% for this event.

[0272] The system automatically sends high-risk alarm information to the central control platform and the caregiver's wristband, and also alerts the caregiver through the bedside soft light. Upon receiving the information, the caregiver quickly rushes to the scene to provide assistance.

[0273] Data recording and model feedback:

[0274] The system records data for the entire cycle, including patrol routes, data from various sensors, and the timeliness of early warning responses, providing ample data support for subsequent model optimization.

[0275] The data from this incident was anonymized and used in federated learning training to continuously improve the overall system's early warning accuracy.

[0276] Multi-scenario extended applications:

[0277] In addition to the aforementioned nighttime bed fall warning system, the system can also be applied to the following scenarios:

[0278] Fall risk monitoring: By monitoring changes in posture and shifts in center of gravity in real time during the night or when elderly people are active, the risk of falls can be identified in a timely manner to ensure the safety of the elderly.

[0279] Comprehensive health status assessment: Long-term data recording and analysis can provide personalized health management plans for the elderly, such as sleep quality assessment and early warning of abnormal breathing and heart rate, providing data basis for subsequent medical intervention.

[0280] Emergency response coordination: The system can be linked with in-hospital emergency equipment and remote monitoring centers to achieve multi-level response to abnormal events, form a rapid closed loop, and shorten the time for handling accidents.

[0281] Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0282] Figure 2 A schematic diagram of a nighttime security management device for nursing homes provided in this application embodiment includes:

[0283] At least one processor; and,

[0284] A memory communicatively connected to the at least one processor; wherein,

[0285] The memory stores instructions that can be executed by the at least one processor, which, when executed, enable the at least one processor to perform any of the above-described methods for nighttime security management in nursing homes.

[0286] Some embodiments of this application provide a non-volatile computer storage medium for nighttime security management in nursing homes, which stores computer-executable instructions capable of executing any of the above-described methods for nighttime security management in nursing homes.

[0287] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0288] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0289] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0290] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0291] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0292] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0293] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0294] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0295] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0296] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0297] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the technical principles of this application should fall within the protection scope of this application.

Claims

1. A method for nighttime security management in nursing homes, characterized in that, Applied to autonomous patrol robots, which include edge computing units and multimodal sensors, the method includes: When the autonomous patrol robot patrols to the monitoring distance of the door, the edge computing unit performs non-contact monitoring outside the door through multimodal sensors and collects multimodal sensing data. Based on the multimodal sensing data, determine the human body's vital signs data; Correlation analysis is performed on the vital sign data of the human body to obtain information on deviations in vital signs, so as to predict the risk of the elderly.

2. The method according to claim 1, characterized in that, When the multimodal sensor is millimeter-wave radar, the determination of human vital signs data specifically includes: By analyzing the signal intensity at different distances using FFT distance analysis, the chest cavity fluctuation signal can be identified. The chest cavity displacement signal is extracted by converting the phase change of the chest cavity fluctuation signal into physical displacement. Based on a preset separation frequency range, respiratory and heartbeat signals are separated from the chest displacement signal; By finding the maximum spectral peaks in the respiratory signal frequency band and the heartbeat signal frequency band respectively, the respiratory rate and heartbeat rate of the elderly are obtained, so as to determine the temporal characteristics of the elderly's respiratory rate and heartbeat rate.

3. The method according to claim 1, characterized in that, When the multimodal sensor is an infrared thermal imaging sensor, the determination of human vital signs data specifically includes: Background subtraction is performed on the thermal imaging image sequence to extract the hot spot region of each frame; The average temperature of the neck region of the hot spot is calculated in each frame of the image to obtain the body surface temperature of the human body. The maximum connected region is determined by calculating the area and location of the hot spot region; Based on the maximum connected region, the human hot spot centroid and human hot spot bounding box of each frame image are obtained to determine the hot spot centroid trajectory features and aspect ratio temporal features. Skeleton extraction is performed on the largest connected region to obtain the human pose position of each frame image, so as to determine the temporal characteristics of human joint kinematics.

4. The method according to claim 1, characterized in that, When the multimodal sensor is a ToF depth camera, determining the human body's vital signs specifically includes: 3D skeleton modeling is performed based on the depth value of each pixel; Determine the centroid coordinates of the human skeleton based on the coordinates of the key points of the 3D skeleton; The boundaries of the support area are determined by the coordinates of the foot joints; Based on the centroid coordinates of the human skeleton and the boundary of the support region, calculate the minimum distance between the centroid projection and the boundary of the support region to determine the temporal characteristics of the centroid projection. Based on the coordinates of key points of the human 3D skeleton, extract the temporal features of key angles of the skeleton.

5. The method according to claim 1, characterized in that, The correlation analysis of the human body's vital sign data to obtain deviation information of the human body's vital signs for risk prediction for the elderly specifically includes: The vital signs data of the human body are synchronized and normalized over time; The normalized vital sign data are divided into time window sequences by sliding window segmentation. The time window sequence is input into a pre-trained LSTM model to obtain the human body’s deviance behavior events and risk warning levels; Based on the behavioral events and risk warning levels associated with the aforementioned deviations in vital signs, risk prediction is performed on the elderly individuals.

6. The method according to claim 5, characterized in that, The risk prediction for the elderly based on the behavioral events deviating from the vital signs and the risk warning level specifically includes: The number of times the aforementioned deviating behavioral events occurred was retrieved from the elderly person's historical warning records; When the number of occurrences exceeds a preset threshold, the compensation coefficient corresponding to the number of occurrences is retrieved from the compensation mapping table; the compensation coefficient is greater than 1. The risk warning level of the behavioral deviation event is compensated based on the compensation coefficient.

7. The method according to claim 5, characterized in that, The method further includes: The LSTM model is trained based on the correlation between the sample vital signs data; The associations include: The conditions for determining abnormal respiratory rate or heart rate include: respiratory rate greater than a preset respiratory rate threshold or heart rate greater than a preset heart rate threshold; The criteria for determining abnormal body temperature include: body temperature exceeding a preset body temperature threshold; The events that determine the bed departure event include at least one of the following: the hot spot centroid deviates from the bed boundary by a first distance exceeding a first preset distance threshold and the deviation time exceeds a preset duration; the support area boundary deviates from the bed boundary by a second distance exceeding a second preset distance threshold and the deviation time exceeds the preset duration. The conditions for determining a fall event include at least one of the following: the rate of descent of the hot spot centroid height exceeds a preset speed threshold; the centroid projection is outside the boundary of the support area; the skeletal centroid height decreases and the torso tilt angle is greater than a preset tilt angle threshold and the duration is greater than a preset duration threshold; the key angles of the skeletal frame do not conform to a preset angle range and the rate of increase of the kinematic values ​​of the joints is greater than a preset increase rate threshold. The criteria for determining a turning-over event include at least one of the following: the hot spot centroid moves laterally within the bed plane beyond a preset movement threshold and the change in the aspect ratio of the hot spot boundary exceeds a preset boundary change threshold; or the hot spot centroid moves laterally within the bed plane beyond a preset movement threshold and the change in the hot spot centroid exceeds a preset centroid change threshold. The conditions for determining a fall from the bed include a sudden change in position and a drastic change in posture; the conditions for a sudden change in position include the hot spot centroid deviating from the boundary of the bed area at a speed greater than a preset speed threshold, and partially or completely falling into the area outside the bed; the conditions for a drastic change in posture include the conditions for the fall event.

8. The method according to claim 1, characterized in that, The method further includes: The occupancy grid map of the patrol environment is updated using the SLAM built into the autonomous patrol robot. Construct the coverage grid corresponding to the occupancy grid map; record in the coverage grid whether each grid has been patrolled; When a moving object is detected, path replanning is performed according to the dynamic window algorithm to avoid the moving object; After avoiding the moving object, check the missed areas of the coverage grid and determine the center point of the missed area as the next patrol target.

9. The method according to claim 1, characterized in that, The method further includes: The system maps behavioral events that deviate from vital signs to risk warning levels, thereby obtaining multi-device linkage response information for the elderly person. The multi-devices include a warning device carried by the caregiver, an emergency call device in the bed area, and a central control platform. The warning device, emergency call device, and autonomous patrol robot are networked via ZigBee, and the autonomous patrol robot and the central control platform are networked via LoRaWAN. Based on the multi-device linkage response information and networking method, early warning responses are issued to the corresponding devices.

10. A nighttime security management device for nursing homes, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the nighttime security management method for nursing homes as described in any one of claims 1-9.

Images