Mobile health service device following system based on multi-mode perception

Abstract

The invention discloses a mobile health service device following system based on multi-mode perception, which realizes a cross-platform health service function through the design of a modular perception unit (millimeter wave radar / vision / laser radar) and a universal control interface. The innovativeness of the system is as follows: 1) a multi-device adaptive architecture is adopted, the same system can be deployed on a service robot, an intelligent wheelchair, a medical carrier and the like, and the hardware reuse rate is greater than or equal to 80%; 2) physical sign-motion cooperative control: dynamically adjusting algorithm parameters according to different carrier characteristics (for example, the noise reduction intensity in a wheelchair mode is improved by 30%); (3) emergency service expansion: the carriers form the Internet of Things (such as wheelchair-carrier-robot linkage), and support the relay transportation of first-aid medicines, the system overcomes the technical bottlenecks of strong equipment specificity and single function in the traditional scheme, and in the actual measurement of the pension community, the cross-equipment cooperation efficiency is obviously improved, and the first-aid response time is shortened to several minutes.

Description

Technical Field

[0001] The present invention belongs to the technical field of health service robots, and specifically relates to a robot system (as well as intelligent wheelchairs, intelligent vehicles, and other mobile devices) that integrates multimodal perception, dynamic following, and health monitoring functions. By fusing multi-source data such as millimeter-wave radar, visual sensors, and lidar, it achieves high-precision positioning of human targets, real-time monitoring of vital signs, and adaptive following control. The system is suitable for personalized health services in scenarios such as home elderly care and medical care. Background Art

[0002] Existing technical limitations in the field of health service robots and mobile health service devices: 1) No integrated vital signs monitoring: For example, patent CN202510184375.6 uses a lidar + vision navigation solution. Although it improves environmental adaptability, it does not integrate vital signs monitoring functions and cannot meet health service needs; 2) Insufficient comfort in human-machine collaboration: Patent CN2025103237214 achieves medication management through voice interaction, but lacks dynamic perception of human posture and physiological state, resulting in stiff follow-up movements and a poor interactive experience. 3) Separation of positioning and obstacle avoidance: For example, patent CN202211689013 achieves tracking based on the fusion of UWB and lidar, but does not solve the problem of target loss in complex environments (such as multi-person interference or occlusion scenes).

[0003] The technical breakthrough direction of the present invention is: 1) Multimodal perception fusion: Integrating millimeter-wave radar (breathing / heartbeat monitoring), vision (skeletal point tracking), and lidar (environmental modeling) to achieve integrated positioning, obstacle avoidance, and health monitoring; 2) Dynamic following strategy: Adjust the following distance (0.3-1.5m) based on vital signs data (such as respiratory rate), balancing safety and service response speed; 3) Universal control interface design to realize cross-platform health service functions. Summary of the Invention

[0004] This invention discloses a mobile health service device tracking system based on multimodal perception. This system utilizes modular sensing units (millimeter-wave radar / visual / lidar) and a universal control interface design to implement cross-platform health service capabilities. The system utilizes a multi-device adaptability architecture: the same system can be deployed on service robots, smart wheelchairs, medical vehicles, and more, with a hardware reuse rate of ≥80%. The following example uses a tracking system deployed on a service robot as an example.

[0005] The hardware system includes sensor modules and a multimodal navigation system. The sensor modules include dual millimeter-wave radar, millimeter-wave radar + vision, combination A millimeter-wave radar + vision, combination B millimeter-wave + lidar, and combination C millimeter-wave radar + vision + lidar. 1) Dual millimeter wave radar module ( Figure 2 ): High-position radar (1.2m height): 60GHz frequency band, beam width ±15°, focusing on respiratory and heartbeat detection in the chest and abdomen (accuracy: respiration ±0.2 times / min, heart rate ±3bpm); Low-position radar (0.5m height): 24GHz frequency band, wide beam (±60°) to achieve leg movement tracking and obstacle detection (positioning accuracy ±5cm); 2) Combination A (millimeter wave + vision) Figure 3 Implementation method: Millimeter-wave radar provides preliminary human body positioning (accuracy ±10cm), RGB-D camera performs face / clothing feature recognition (YOLOv5s model, recognition delay <200ms), data fusion: maps millimeter-wave coordinates to the visual coordinate system (calibration error <5mm); 3) Combination B (millimeter wave + lidar) Figure 4 Implementation method: LiDAR constructs a point cloud map of the environment (16 lines, scanning frequency 10Hz), while millimeter-wave radar extracts human micro-motion features (chest displacement caused by breathing ±2mm), and the ICP algorithm is used to associate dynamic human targets with the static map. 4) Combination C (millimeter wave + vision + lidar) Figure 5 Implementation method: LiDAR provides the spatial distribution of obstacles, the visual system identifies human skeleton points (OpenPose model), and the millimeter-wave radar verifies the target's vital signs. These three sources of data are used to make decisions based on DS evidence theory to ultimately follow the target.

[0006] The multimodal navigation system includes an inertial measurement unit (IMU) and a fisheye camera: 1) Inertial Measurement Unit (IMU): 9-axis attitude solution (heading angle error <1°); 2) Fisheye camera: 150° ultra-wide-angle field of view, used for rapid face re-identification.

[0007] The hybrid perception control strategy of the health service robot following system is reflected in the multi-sensor weight distribution mechanism and cross-modal calibration method: 1) Multi-sensor weighting mechanism: In sufficient light, visual sensors are weighted up to 70%. At night or in hazy conditions, millimeter-wave sensors are prioritized (weight > 80%). LiDAR is used for obstacle avoidance in densely populated areas (with the highest priority for safety distance calculation). 2) Cross-modal calibration method ( Figure 6): Design a dedicated calibration target (including reflective marking points + millimeter wave reflector) to automatically complete the temporal and spatial parameter calibration (online calibration takes <30 seconds).

[0008] The calibration method software system of the health service robot following system includes: 1) Vital sign-motion decoupling algorithm: A Kalman filter is used to separate respiratory waveforms from walking vibration noise (improving the signal-to-noise ratio by 12dB) and dynamically adjust the sampling frequency (1Hz at rest → 10Hz at walking). 2) Smart follow controller close mode: maintains a dynamic distance of 0.8-1.2m (PID control response time <0.3s); standby mode: the radar frequency is reduced to 0.1Hz on duty, and it automatically wakes up when it detects the owner moving >2m.

[0009] Taking dual millimeter-wave radar sensing as an example, the data fusion process is explained: 1) Spatial alignment: Establishing the dual radar coordinate system conversion matrix through the calibration plate (error < 2mm); Time synchronization: Aligning the data acquisition timestamps with the hardware trigger signal (deviation < 1ms); 2) Target Locking: The low-position radar identifies human gait characteristics (step length, swing frequency) and generates a motion vector. The high-position radar extracts the chest cavity waveform and combines it with IMU data to compensate for the effects of body tilt. 3) Mode switching logic: Close following trigger, detecting that the owner moves continuously for >3 seconds and the speed is <1m / s, and the breathing rate is within the resting range (12-20 times / min); emergency standby trigger conditions, heart rate continuously >120bpm for more than 1 minute, and detection of the owner falling (acceleration >3g and posture angle >45°).

[0010] The abnormal warning protocol for health service functions includes: 1) Level 1 warning (abnormal breathing): Voice prompt "Shortness of breath detected, please sit down and rest"; 2) Level 2 warning (abnormal heart rate): Flashing warning (red LED pulse frequency 2Hz) and shortening the following distance to 0.5m; Level 3 warning (fall); 3) Automatically dial emergency contacts and start video surveillance feedback.

[0011] The data visualization solution for the health service robot following system specifically includes: 1) Generate daily health reports: including walking steps (low-level radar count), average heart rate, and abnormal breathing events; 2) Comparison of cloud data: Deviation analysis is performed with historical baseline values (±10%).

[0012] The following system of the present invention is modularly designed and can be adapted to the following smart mobile devices: 1) Service robots: home elderly care robots, hospital logistics robots, etc.; 2) Smart wheelchair: An electric wheelchair with autonomous navigation that supports patient rehabilitation training; 3) Intelligent vehicles: cargo transport vehicles, guided tour vehicles, etc., with expanded health monitoring functions (e.g., simultaneous monitoring of temperature and humidity when transporting emergency medicines); 4) Other mobile devices: smart scooters with integrated health service modules, cleaning robots, etc.

[0013] In order to adapt to different smart mobile devices, this system has been designed for universal technology: 1) Hardware interface standardization: Provide CAN bus / UART / Ethernet multi-protocol interfaces to adapt to control systems of different mobile platforms; 2) Removable perception module: The millimeter-wave radar and vision module adopt a quick-release structure (replacement time ≤ 3 minutes); 3) Dynamic parameter configuration: Automatically load the preset control strategy according to the device type (e.g. the maximum speed limit in wheelchair mode is 1.0m / s). BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 : Cross-device system architecture.

[0015] Figure 2 : Dual millimeter-wave radar architecture.

[0016] Figure 3 : Millimeter-wave radar + vision (combination A).

[0017] Figure 4 : Millimeter wave + lidar (combination B).

[0018] Figure 5 : Millimeter wave + vision + lidar (combination C).

[0019] Figure 6 : Multimodal calibration process (general). DETAILED DESCRIPTION

[0020] Example 1: Daily Following and Health Monitoring. Scenario description: When an elderly person walks in the living room, the robot automatically enters the close-fitting mode and moves synchronously with the elderly person while maintaining a distance of 1m. 1) Low-position radar real-time tracking of gait (step length 65cm±3cm); 2) High-position radar monitoring of respiratory rate (16 breaths / min → fluctuation range ±2); 3) When the sofa obstacle is detected, the robot automatically moves around it (path planning takes less than 0.5 seconds); 4) Data recording and generation of exercise reports: walking distance 352m, average heart rate 78bpm; abnormal breathing rhythm (3 consecutive breathing intervals >4 seconds), triggering a level 1 warning.

[0021] Example 2: Emergency Response. Abnormal Detection: 1) The high-position radar captured a sudden change in chest acceleration (4.2g); 2) The low-position radar detected an attitude angle of 52° (lasting for 3 seconds without recovery); 3) IMU data verification: Z-axis acceleration remains at 0 (determined to be a fall); Execution Agreement: 1) Activate emergency standby mode and shorten the distance to 0.3m (for easier rescue); 2) Voice call: "Fall detected! Family members have been notified and video recording has been initiated"; 3) Fisheye camera captures live footage (H.265 encoding, bit rate 1.5Mbps).

[0022] Example 3: Millimeter wave + vision hybrid tracking (supermarket scene). Implementation process: 1) The millimeter-wave radar locks the owner's approximate position (accuracy ±0.5m) under the cover of the shelf; 2) The visual system accurately locates the image based on the preset clothing features (red top); 3) Generate a follow-up path after data fusion (avoiding dynamic obstacles of the shopping cart); Measured data: The positioning success rate in obstructed environments is 92% (only 65% for a pure millimeter wave solution); the probability of mistakenly following others is reduced to 3% (18% for a pure vision solution).

[0023] Example 4: Trimodal emergency obstacle avoidance (fire and smoke scenario). Implementation process: 1) The visual system fails due to smoke (visibility <1m); 2) LiDAR detection of the spatial structure of the escape route; 3) Millimeter-wave radar continuously monitors the owner's vital signs (blood oxygen saturation estimation); Performance indicators: Positioning accuracy remains at ±0.3m (traditional laser solutions have an error of >1m in smoke); vital sign monitoring is uninterrupted (detection of elevated CO concentration triggers an alarm).

[0024] Example 5: Smart wheelchair rehabilitation tracking (hospital scenario). Implementation process: 1) Millimeter-wave radar monitors the patient's respiratory rate (with an accuracy of ±0.3 breaths / min even when restrained by a chest strap); 2) The visual system recognizes the therapist's gesture commands (stop / accelerate / turn); 3) Wheelchair speed limit module: automatically adjusts according to the patient's heart rate (reduced to 0.5m / s when the heart rate is >100bpm); Data recording: Patients' exercise rehabilitation time increased by 40% (traditional solutions are limited by the need for manual monitoring), and the false trigger rate was reduced.

[0025] Example 6: Smart vehicle drug transportation (from pharmacy to ward). Multi-device linkage: 1) Vehicle A departs from the pharmacy, and uses LiDAR to build a high-precision map of the hospital (update frequency 1Hz). 2) Robot B receives the medicine in the ward area and completes meter-level handover positioning with vehicle A via UWB; 3) Wheelchair C provides last-meter delivery (entering the restricted area of the ward); Performance indicators: cross-device positioning error <0.2m, high drug temperature control qualification rate.

Claims

1. A mobile health service device tracking system based on multimodal perception, characterized by Suitable for service robots, smart wheelchairs, smart vehicles and other mobile devices, including: a. A multimodal perception module (including at least a millimeter-wave radar and another sensor, wherein the other sensor is selected from one or more of a visual sensor, a laser radar, and a UWB positioning module); b. Vital sign-motion decoupling algorithm, used to eliminate the interference of the carrier's own motion on vital sign detection; c. Dynamic following controller, which adaptively adjusts following parameters according to device type, including real-time adjustment of following distance (0.3-2.0m) and speed (0-1.5m / s); d. Standardized hardware interface (supports CAN bus / UART / Ethernet protocols) to achieve plug-and-play with different mobile platforms.

2. The system according to claim 1, characterized in that The multimodal perception module includes: 1) The first millimeter-wave radar (60GHz frequency band, installation height 1.2-1.5m, beam width ±15°, used for chest rise and fall monitoring); 2) Second millimeter-wave radar (24GHz frequency band, installation height 0.4-0.8m, beam width ±60°, for leg motion tracking).

3. The system according to claim 1, characterized in that The multimodal perception module is a combination of millimeter-wave radar and visual sensor, wherein: 1) The visual sensor includes an RGB-D camera (resolution ≥1280×720, frame rate ≥30fps) and an infrared fill light module (wavelength 850nm); 2) Data fusion uses a spatiotemporal alignment algorithm to map millimeter wave coordinates to the visual coordinate system (residual ≤ 5 mm).

4. The system according to claim 1, characterized in that The multimodal perception module is a combination of millimeter wave radar and laser radar, wherein: 1) The laser radar is a 16-line solid-state radar (horizontal field of view 360°, vertical field of view ≥30°) used to construct obstacle point cloud maps; 2) Millimeter-wave radar array (≥3 transceiver units) is used for respiratory signal extraction and motion compensation.

5. The system according to claim 1, characterized in that The multimodal perception module is a three-modal combination of millimeter wave radar, visual sensor and laser radar, and its data processing method includes: 1) Fusion of multi-source data using DS evidence theory (confidence threshold ≥ 0.85); 2) Establish a priority obstacle avoidance strategy: human body > dynamic obstacles > static obstacles.

6. The system according to claim 1, when applied to a smart wheelchair, further comprising: 1) The vital sign monitoring module is integrated into the inside of the wheelchair armrest (≤30cm from the user's chest); 2) Emergency braking protocol: Automatically cuts off the power supply to the drive motor when an abnormal heart rate (>120 bpm for 30 seconds) is detected; 3) Rehabilitation training mode: adjust the power intensity according to the breathing rate (0-50N gradient output).

7. The system according to claim 1, when applied to an intelligent vehicle, further comprising: 1) The cargo area is equipped with a constant temperature chamber for medicines (temperature control range 4-25°C, accuracy ±1°C); 2) Medical supplies transportation priority algorithm: Identify the urgency of medicines (normal / urgent / super urgent) based on RFID tags; 3) Path planning dynamically avoids medical restricted areas (operating room / ICU area).

8. The system according to claim 1, characterized in that The physical sign-motion decoupling algorithm includes: 1) Pre-store a library of typical motion noise (acceleration characteristics of walking, climbing stairs, and turning); 2) Dynamically select noise templates for waveform cancellation based on IMU data; 3) Respiratory signal reconstruction error ≤ 0.5 times / min.

9. The system according to claim 1, characterized in that The dynamic following controller includes: a safety distance calculation model: (where k is the device type coefficient: robot k=1.0, wheelchair k=1.2, vehicle k=0.8); emergency braking acceleration ≥ -3m / s².

10. The system according to claim 1, characterized in that Also includes: 1) General SDK interface, providing a vehicle type identification API (returning the robot / wheelchair / vehicle classification code); 2) Cloud-based policy management platform, supporting remote updates of follow-up parameters and health monitoring thresholds.

Images