Fall monitoring system and method based on combination of radar and TOF

Abstract

The invention provides a falling monitoring system and method based on combination of radar and TOF, and belongs to the technical field of action recognition and data processing. The method comprises the steps of determining a target monitoring range and a target monitoring person; acquiring historical image monitoring data of the target monitored person; determining an arrangement scheme of the radar and the TOF sensor based on the target monitoring range and the historical image monitoring data; judging whether the target monitored person falls down or not based on the radar monitoring data and the TOF sensing data; and when it is judged that the target person falls down and does not return to a normal state within a preset time period, sending a warning signal to related personnel of the target monitored person. The system comprises a historical image monitoring data acquisition unit, a radar sensing module, a TOF sensing module, a fall detection unit and a warning unit. According to the technical scheme of the invention, tumble detection based on the personalized features of the target person can be realized while the privacy of the user is ensured, and the detection precision is high.

Description

Technical Field

[0001] The present invention belongs to the field of motion recognition and data processing monitoring technology, and in particular relates to a radar combined with TOF fall monitoring system and method, a computer-readable storage medium for implementing the method, a computer program product, and a related mobile terminal. Background Art

[0002] Many elderly people living alone often experience falls in their daily lives. Unable to get up and seek help on their own, and with no one nearby to spot them, they often miss the best time to seek medical attention, ultimately leading to serious consequences. Some elderly people, after being down for a long time, may not receive timely medical attention for their injured parts, leading to worsening wound infections and even the risk of amputation. For some elderly people who already suffer from cardiovascular and cerebrovascular diseases, emotional stress and physical reactions after a fall can trigger life-threatening conditions such as myocardial infarction and cerebral infarction, endangering their lives.

[0003] In order to protect the health of these elderly people and provide them with good treatment services, it is very necessary to conduct a series of reliable health monitoring activities for them. Fall detection is one of the most important health monitoring methods.

[0004] In the related technologies, fall detection mainly includes detection based on visual image acquisition, detection based on infrared camera, detection based on millimeter wave radar, etc. For example, the Chinese invention patent publication CN117368909A proposes a millimeter wave radar based on Vision Transformer.

[0005] Human fall recognition method, a dynamic target monitoring method based on millimeter-wave radar proposed in CN112859067A, etc.

[0006] Fall detection methods based on visual images are highly accurate and avoid misjudgment, but they are privacy-sensitive. For example, bathrooms and bedrooms are high-risk areas for falls, but installing cameras is difficult. Fall detection methods based on infrared cameras offer more accurate posture detection, but installation costs are high, and their appearance generally resembles a camera, which can easily lead to privacy concerns. Millimeter-wave radar-based detection methods require a frequency band of 60 GHz or higher, a multi-transmitter and multi-receiver antenna design, and high-computing MCU processing to ensure accuracy, which is very costly. Summary of the Invention

[0007] In response to the above technical problems, the present invention proposes a radar combined with TOF fall monitoring system and method, a computer-readable storage medium for implementing the method, a computer program product and a related mobile terminal.

[0008] In a first aspect of the present invention, a radar combined with TOF fall detection method is proposed, the method comprising:

[0009] Determine the target monitoring scope and target monitoring personnel;

[0010] Obtain historical image monitoring data of the target monitored person;

[0011] Determining a layout plan of radar and TOF sensors based on the target monitoring range and the historical image monitoring data;

[0012] Determine whether the target person has fallen based on radar monitoring data and TOF sensor data;

[0013] When it is determined that the target person has fallen and has not returned to normal within a preset time period, a warning signal is sent to relevant personnel of the target monitored person.

[0014] The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range;

[0015] The step of obtaining historical image monitoring data of a target monitored person includes:

[0016] Based on the image sensor arranged before executing the fall monitoring method, historical image monitoring data of the target monitored person in the first monitoring range is obtained, and the historical monitoring data includes walking posture data and static posture data of the target monitored person.

[0017] The determination of the arrangement of radar and TOF sensors based on the target monitoring range and the historical image monitoring data specifically includes:

[0018] Determining the number of radars and TOF sensors to be deployed based on the target monitoring range;

[0019] Determining the layout positions of the radar and the TOF sensor based on the historical image monitoring data;

[0020] After the radar and the TOF sensor are arranged within the target monitoring range based on the arrangement quantity and the arrangement position, the state parameters of the radar and the TOF sensor are calibrated based on the historical image monitoring data.

[0021] The method of determining whether the target monitored person has fallen based on the radar monitoring data and the TOF sensor data specifically includes:

[0022] Determining whether there is a target person in the monitoring range based on the first radar monitoring data, and if so, determining the height of the target person based on at least one TOF sensor data;

[0023] When the height of the target person is lower than a preset threshold, it is determined whether the target person may fall based on multiple TOF data; if so, the second radar monitoring data is obtained, the movement amplitude of the target person is monitored based on the second radar monitoring data, and whether the target monitored person falls is determined based on the movement amplitude.

[0024] The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range;

[0025] When the target person enters the second monitoring range, it is determined whether the target person falls based on the radar monitoring data and the TOF sensing data.

[0026] The method of determining whether the target monitored person has fallen based on the radar monitoring data and the TOF sensor data specifically includes:

[0027] Based on the first radar monitoring data, it is determined whether there is only a target monitoring person in the monitoring range. If so, the height of the target person is determined based on at least one TOF sensing data.

[0028] In a second aspect of the present invention, a radar combined with TOF fall monitoring system is also proposed, the system comprising a historical image monitoring data acquisition unit, a radar sensor module, a TOF sensor module, a fall detection unit, and a warning unit;

[0029] After determining the target monitoring range and the target monitoring person, the historical image monitoring data acquisition unit acquires the historical image monitoring data of the target monitoring person;

[0030] Determining the layout of the radar sensor module and the TOF sensor module based on the target monitoring range and the historical image monitoring data;

[0031] The fall detection unit determines whether the target person has fallen based on the radar monitoring data and TOF sensor data;

[0032] When the fall detection unit determines that the target person has fallen and has not returned to normal within a preset time period, the warning unit sends a warning signal to the relevant personnel of the target monitored person;

[0033] The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range; an image sensor is installed in the first monitoring range.

[0034] The arrangement scheme of the radar sensor module and the TOF sensor module is determined based on the target monitoring range and the historical image monitoring data, specifically including:

[0035] The image sensor is not installed in the second monitoring range.

[0036] Determining the number of radars and TOF sensors to be arranged based on the second monitoring range;

[0037] Determining, based on the historical image monitoring data, the layout positions of the radar and the TOF sensor in the second monitoring range;

[0038] After the radar and TOF sensor are arranged within the second monitoring range based on the arrangement quantity and the arrangement position, the state parameters of the radar and TOF sensor are calibrated based on the historical image monitoring data.

[0039] In a second aspect of the present invention, an electronic device is also proposed, which includes a processor and a memory, wherein the memory stores computer-executable program instructions; when the program instructions are executed by the processor, the aforementioned radar combined with TOF fall monitoring method is implemented.

[0040] The electronic device can also be implemented as a human-computer interaction device including an external data interface; the external data interface can be connected to at least one computer-readable storage medium, and the computer-readable storage medium stores computer program code. When the computer program code is transferred to the memory, it is executed by the processor to implement all steps of the aforementioned radar combined with TOF fall monitoring method.

[0041] Based on the same inventive concept, the technical solution of the present invention also provides a computer medium, which stores a computer program. When the computer program is executed, all or part of the steps of the above-mentioned radar combined with TOF fall monitoring method are implemented.

[0042] In the third aspect of the present invention, a mobile terminal is also proposed, which includes a human-computer interaction unit, characterized in that the human-computer interaction unit is used to receive the warning signal issued by the radar combined with TOF fall monitoring method described in the first aspect, and the warning signal is used to prompt the target person to fall within the target monitoring range.

[0043] This invention proposes a low-cost, highly accurate fall detection solution. This solution utilizes a low-cost radar paired with a multi-zone time-of-flight sensor. The solution uses historical image monitoring data to calibrate the radar and TOF sensor parameters, enabling fall detection based on the individual characteristics of the target person. This solution significantly outperforms radar-only detection while protecting user privacy. Its specific advantages and implementation principles are further detailed in the detailed examples section, along with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a schematic diagram of the main flow of a radar combined with TOF fall detection method according to an embodiment of the present invention;

[0046] Figure 2 It is implemented using a computer program Figure 1 A schematic flow chart of the method;

[0047] Figure 3 Is implemented Figure 1 A schematic diagram of a layout scenario of the method;

[0048] Figure 4 FIG1 is a schematic diagram showing the hardware unit composition of a radar combined with TOF fall monitoring system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In the specific implementation of this application, if the embodiments of the relevant technical solutions involve user-related data, when the embodiments of this application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0050] Before introducing the specific embodiments of the present invention, continuing with the introduction of the background technology, we first introduce the various fall detection schemes existing in the prior art and their defects, thereby introducing the improvement motivation of the present invention and further understanding the advantages of the technical solution of this application.

[0051] In the existing technology, fall detection technology mainly includes detection based on visual image acquisition, detection based on infrared camera, detection based on millimeter wave radar and other solutions.

[0052] Detection technology based on visual image acquisition: This technology uses a camera to capture human movements in real time and uses deep learning algorithms (such as YOLOv8 and OpenPose) to analyze changes in human posture within video frames to identify falls. For example, YOLOv8, combined with a posture estimation model, first detects the body's position and then extracts key points (such as shoulders and knees) to identify posture anomalies.

[0053] Its advantage lies in its high accuracy, which allows it to precisely capture changes in posture during a fall, such as the displacement of key points from standing to falling. In scenes with sufficient lighting and a simple background, the accuracy can reach over 95%. However, its significant drawback is that the visual camera must capture visible images of the human body, which can easily lead to privacy disputes. This is especially true in private places (such as bedrooms and bathrooms). For example, bathrooms and bedrooms are high-risk areas for falls, but installing cameras is difficult. Furthermore, deep learning models (such as 3DCNN) require a large amount of computing resources, and real-time detection may place high demands on the performance of edge devices.

[0054] Infrared camera-based detection technology: Infrared sensors capture human thermal radiation, generating thermal images and identifying falls by analyzing temperature distribution changes. For example, an improved Alphapose algorithm combined with the YOLOv5s object detection network can detect abnormal posture based on key points of the human skeleton in infrared images.

[0055] This solution works without visible light and is suitable for nighttime or dimly lit scenes (such as elderly people's bedrooms), with an accuracy rate of over 95%. However, this solution requires precise calibration of the sensor position and angle to ensure coverage of the monitoring area, and the equipment cost is relatively high (for example, the H60 infrared waterproof camera costs over 1,000 yuan). In addition, its appearance generally looks like a camera, which can easily lead to the perception that privacy issues are violated.

[0056] Detection technology based on millimeter-wave radar: It emits electromagnetic waves in the millimeter-wave frequency band, obtains human body distance, speed and angle information by analyzing the time difference (TOF) and Doppler frequency shift of the reflected signal, and uses the micro-Doppler effect to identify falls.

[0057] This solution only outputs human motion parameters (such as position and speed), without images or video, thus protecting user privacy. However, in environments with multiple fast-moving objects or high background noise (such as multiple people simultaneously), signal processing complexity increases, potentially affecting real-time performance. To ensure accuracy, a frequency band of 60 GHz or higher, combined with a multi-transmitter, multi-receiver antenna design and a high-performance MCU, requires high processing power, which is very costly.

[0058] To this end, an embodiment of the present invention proposes a fall detection method that can ensure user privacy while implementing a fall detection method based on personalized features of a target person and ensure high detection accuracy.

[0059] See also Figure 1 , Figure 1 A schematic diagram illustrating the main flow of a radar combined with TOF fall monitoring method according to an embodiment of the present invention is shown. Figure 1 Five flow charts are shown, including five main implementation steps.

[0060] For the sake of simplicity, Figure 1 The five flow charts shown are numbered S1-S5 (step numbers are omitted in the accompanying drawings):

[0061] S1: Determine the target monitoring range and target monitoring person;

[0062] S2: Obtain historical image monitoring data of the target monitored person;

[0063] S3: Determine the layout of the radar and TOF sensor based on the target monitoring range and the historical image monitoring data;

[0064] S4: Determine whether the target person has fallen based on the radar monitoring data and TOF sensor data;

[0065] S5: When it is determined that the target person has fallen and has not returned to normal within a preset time period, a warning signal is sent to relevant personnel of the target monitored person.

[0066] Figure 1 The method can be automatically performed by a computer program, Figure 2 The relevant program flow chart is shown and described in computer pseudo code language as follows:

[0067] start;

[0068] Step 1: Determine the target monitoring range and target monitoring person;

[0069] Step 2: Obtain historical image monitoring data of the target monitored person;

[0070] Step 3: Determine the layout of radar and TOF sensors based on the target monitoring range and the historical image monitoring data, and then arrange the radar and TOF sensors.

[0071] Real-time collection of radar monitoring data and TOF sensing data;

[0072] Step 4: Determine whether the target person has fallen based on the monitoring data and sensor data. If yes, proceed to step 5; otherwise, return to the step of "real-time acquisition of radar monitoring data and TOF sensor data";

[0073] Step 5: Start the preset time period (e.g., 30-second countdown);

[0074] Determine whether the target person returns to normal within the preset time period. If so, return to the step of "real-time acquisition of radar monitoring data and TOF sensor data";

[0075] If not, go to step 6;

[0076] Step 6: Send a warning signal to relevant personnel;

[0077] Finish.

[0078] Figure 3 Show implementation Figure 1 The schematic diagram of the layout scene of the method includes a three-dimensional space schematic diagram and a plane layout schematic diagram from a bird's-eye view.

[0079] Figure 3 Shown in a certain indoor living space Figure 1 Schematic diagram of the fall monitoring method. The indoor living space includes a living room area and multiple room areas.

[0080] The living room area and multiple room areas constitute the target monitoring range of the embodiment of the present invention; the elderly who engage in daily activities in the target monitoring range constitute the target monitoring persons of the embodiment of the present invention.

[0081] In summary, the target monitoring range in the embodiment of the present invention includes a first monitoring range A and a plurality of second monitoring ranges B1 - B3 except the first monitoring range.

[0082] by Figure 3 For example, the first monitoring range may be a living room area; multiple second monitoring ranges may be multiple room areas, including bedrooms, kitchens, bathrooms, etc.;

[0083] It's understandable that for home safety, at least one image sensor is typically installed in the living room, covering the entire living room area. This image sensor can be an intelligent one, capable of identifying people moving within the living room, the number of people, and whether their postures are abnormal. Installing a visual image sensor in the living room is generally acceptable to most users. Therefore, within the living room, gesture recognition can be performed using this intelligent image sensor. For example, when a person enters the living room, accurate fall detection can be achieved using the aforementioned detection technology based on visual image acquisition.

[0084] However, installing visual image acquisition sensors in multiple second monitoring ranges outside the first monitoring range can easily be perceived by users as an invasion of privacy. Therefore, the first improvement of the present invention is to not install visual image sensors in the second monitoring ranges; instead, only visual intelligent image sensors for fall detection are installed in the first monitoring range.

[0085] Correspondingly, radar and TOF sensors need to be installed within the second monitoring range.

[0086] In the embodiments of the present invention, radar and TOF sensor can also be referred to as radar sensor module and TOF sensor module. A radar sensor module includes multiple radars, and a TOF can include multiple TOF sensors.

[0087] The radar sensor module is composed of a low-frequency radar; preferably, the embodiment of the present invention adopts a 24 GHz low-cost radar.

[0088] Millimeter-wave radars commonly used in related technologies typically operate in the 30-60 GHz range, using a frequency band of 60 GHz or higher. These radars require multiple transmit and receive antennas and a high-performance MCU, resulting in high costs. The 24 GHz radars preferred in this invention offer significant advantages, costing only 1 / 10 to 1 / 20 of 60 GHz radars.

[0089] The TOF sensing module consists of multiple TOF sensors that can communicate with each other.

[0090] TOF itself is the abbreviation of Time of Flight. It is a three-dimensional imaging technology that obtains the distance to the target person by measuring the time from the emission of a light pulse to its reflection by the target person and reception by the sensor.

[0091] Accordingly, the TOF sensor involved in the embodiments of the present invention calculates the distance between two points by emitting a light signal (typically near-infrared light or laser light, with common wavelengths of 850nm and 940nm) and using the time it takes for photons to travel between the two points. Based on the time difference between the signal being emitted and its reflection off an object and returning to the sensor, the distances measured by all pixels form a depth map.

[0092] Because low-frequency radar and TOF sensors are required in the second monitoring range for accurate fall detection, the modules must be installed in locations where human movement can be effectively detected. This ensures that the low-frequency radar and TOF sensor monitoring range covers areas where people may move and fall, such as the center of the living room and near toilets and showers. Furthermore, the sensors must be installed away from electrical equipment, vents, metal objects, and other areas that may interfere with the signal. This is because low-frequency radar signals may be affected by electromagnetic interference, while TOF sensors may be affected by metal reflections, affecting measurement accuracy.

[0093] As a general rule, the installation height should be determined based on the specific product characteristics and monitoring scenario. Generally speaking, low-frequency radars can be installed approximately 2-3 meters above the ground to achieve a wider monitoring range. If TOF sensors are used for auxiliary detection, they can be installed at a lower position, such as approximately 1.5 meters above the ground, to more accurately detect motion changes at close range.

[0094] However, each target person has different height, gait and walking posture (such as walking length, walking speed, bending habits), and the spatial structure of each target monitoring range is also different; if the relevant sensor modules are installed only according to the general principles mentioned above, the monitoring effect will be greatly reduced.

[0095] Specifically, another improvement of the present invention is to obtain historical image monitoring data of the target monitoring person; determine the layout of the radar and TOF sensor based on the target monitoring range and the historical image monitoring data;

[0096] Specifically, the number of radars and TOF sensors to be deployed is determined based on the target monitoring range;

[0097] Determining the layout positions of the radar and the TOF sensor based on the historical image monitoring data;

[0098] After the radar and the TOF sensor are arranged within the target monitoring range based on the arrangement quantity and the arrangement position, the state parameters of the radar and the TOF sensor are calibrated based on the historical image monitoring data.

[0099] As a specific example, the step of obtaining historical image monitoring data of a target monitored person includes:

[0100] Based on the visible image sensor arranged before executing the fall monitoring method, historical visible image monitoring data of the target monitored person in the first monitoring range is obtained, and the historical monitoring data includes walking posture data and static posture data of the target monitored person.

[0101] For example, based on the historical visual image monitoring data set collected by the visual intelligent image sensor arranged in the living room (first monitoring range), the walking posture data and the static posture data of the target monitored person are screened out; the walking posture data include walking height, walking step length, walking speed, bending angle / bending-standing average duration value, etc.; the static posture data include standing height, average standing static duration, sitting height, average sitting duration, average sitting-standing switching time, etc.

[0102] Preferably, the visual image monitoring data set screens out relevant visual image data of the target monitored person and further includes position coordinate values ​​(relative height values) of key skeleton points (such as shoulders and hips) of the human body.

[0103] Determining the number of radars and TOF sensors to be arranged based on the target monitoring range, specifically, determining the number of radars and TOF sensors to be arranged based on the number of target monitoring ranges, the area of ​​each target monitoring range, the duration of the target person in each target monitoring range, the frequency of entry and exit, etc.;

[0104] As a general principle, the target monitoring range only includes the second monitoring range (i.e., excluding the living room, which is already equipped with a visual intelligent image sensor); each second monitoring range is equipped with at least one low-frequency radar and one TOF sensor;

[0105] On this basis, the number of low-frequency radars and TOF sensors arranged in each second monitoring range can increase in positive correlation (for example, increase in positive proportion) with the duration of the target person in each second target monitoring range, the frequency of entry and exit, the area of ​​each second target monitoring range, etc.; for example, the larger the area of ​​each target monitoring range, the greater the number of low-frequency radars and TOF sensors, etc.

[0106] On this basis, the layout positions of the radar and TOF sensors within the monitoring range of each target can also be specifically determined. For example, the layout heights of the radar and TOF sensors can be further determined based on the standing height and sitting height data in the static posture data, and the moving height data in the walking posture data.

[0107] Taking the shower room (restroom, toilet) as an example, in one example, the target person's head height is about 1.69 meters when standing, so the low-frequency radar is installed on the ceiling near the toilet / shower area at a height of 2.5 meters; it covers the head height when standing (1.6-1.9 meters), and monitors high-fall areas such as toilets and shower areas through a horizontal field of view (such as 100°); this position can also penetrate water vapor and steam, and stably detect falls in humid environments with an accuracy rate of ≥97%; the ToF sensor is installed at a height of 1.5 meters on the side wall of the toilet, covering the head height when sitting down (0.8-1.2 meters), combined with close-range detection capabilities (0.5-3 meters) to monitor toileting and getting up movements, and detect lying postures after falls (supine, prone, etc.), with a false alarm rate of less than 3%.

[0108] The radars and TOF sensors installed according to the above configuration quantity and configuration position are generally still in the initial factory setting state. This factory setting state is usually set according to the universal target person characteristics derived from general statistical principles, and may not be compatible with the target person characteristics within the current monitoring range and the monitoring environment itself.

[0109] Therefore, the present invention is next introduced to an improved point. After the radar and TOF sensor are arranged within the target monitoring range based on the arrangement quantity and the arrangement position, the state parameters of the radar and TOF sensor are corrected based on the historical image monitoring data.

[0110] In a specific example, the process of correcting the relevant state parameters includes:

[0111] Height: Assume that historical image monitoring data shows that the average height of an individual walking normally is 1.75 meters (this value can be calculated by converting the coordinates of key skeletal points, such as the head vertex, relative to the ground). If the radar and TOF sensor's initial detection height range is expected to be 1.7-1.8 meters for this individual's walking state, if the actual height values ​​in a large amount of walking data deviate significantly from this range, correction is required. For example, if the radar-detected height data for this individual while walking is consistently concentrated between 1.6-1.7 meters, there may be a deviation in the radar's installation angle or height. The installation height or vertical angle of the radar needs to be adjusted appropriately to ensure that the detected height matches the average height value in the historical data.

[0112] Walking stride length: According to historical image monitoring data statistics, the average walking stride length of this individual is 0.7 meters (calculated by the distance of the position change of the key skeleton points of the foot on the same side in two consecutive frames of images). If the TOF sensor calculates the stride length based on the distance change detected when detecting the individual's walking movement and it is significantly different from 0.7 meters, for example, long-term detection results show a stride length of 0.5 meters. This may mean that there is a problem with the detection accuracy of the TOF sensor and it needs to be recalibrated. By setting a standard reflector at a specific distance and detecting it with the TOF sensor, the internal parameters of the TOF sensor, such as the time measurement accuracy parameters, can be adjusted based on the deviation between the known distance and the detection distance so that it can accurately detect the walking stride length.

[0113] Travel speed: Historical image monitoring data indicates that the average speed of this individual in normal walking conditions is 1.2 m / s (calculated by the displacement of key skeletal points at the center of gravity of the human body and the time intervals between consecutive frames of images). If the speed calculated by the radar using the Doppler shift of the reflected signal differs significantly from 1.2 m / s, such as a continuous detection result of 0.8 m / s, the radar's Doppler shift measurement module should be inspected and its frequency response parameters may need to be recalibrated. By emitting a standard target moving at a known speed, the radar's detected speed can be compared with the actual speed, and the relevant parameters can be adjusted to ensure that the radar can accurately detect travel speed.

[0114] Bending Angle / Average Bending-Standing Duration: Assume that historical data indicates that an individual's average bending angle during daily activities is 45° (calculated by the relative positional changes of key skeletal points at the shoulder and hip), and the average bending-standing duration is 5 seconds. If the bending angle data detected by the TOF sensor deviates significantly from 45°, or the bending-standing duration determined by the radar in combination with the TOF sensor data differs significantly from 5 seconds—for example, if the detected bending angle is consistently around 30° and the bending-standing duration is 8 seconds—the TOF sensor may need to be recalibrated to improve its accuracy in detecting distance to objects at different angles. This can be done by simulating a standard model with different bending angles, allowing the TOF sensor to perform measurements and adjusting internal algorithm parameters based on the deviation between the measured results and the actual angle. For radar, the signal processing algorithm for changes in human posture may need to be optimized to more accurately coordinate with the TOF sensor to determine the duration of movements such as bending and standing.

[0115] Standing height: Historical image monitoring data shows that the individual's standing height is 1.78 meters (determined by the relative height of the head vertex above the ground). If the standing height detected by the radar and TOF sensor is inconsistent with 1.78 meters, for example, 1.75 meters for the radar and 1.82 meters for the TOF sensor, first check the correct installation position and angle of the radar and TOF sensor, and check for any obstructions or other influencing factors. If the installation is correct, the radar can be calibrated by adjusting its distance measurement calibration parameters and performing multiple measurements using a standard object of known height. The radar's distance measurement accuracy can be calibrated based on the deviation between the measured results and the actual height. The TOF sensor can be calibrated to verify the accuracy of the time of flight measurement circuitry used to transmit and receive optical signals. By comparing the time with a standard time source, the relevant circuit parameters can be adjusted to accurately measure standing height.

[0116] Average Standing Still Duration: Based on historical data, the average standing still duration for this individual is 30 minutes. If the standing still duration detected by the radar and TOF sensor differs significantly from 30 minutes, for example, only 10 minutes, the sensor's threshold for determining stillness may be improperly set and needs to be adjusted. For example, the radar can adjust its signal fluctuation threshold for determining the target person's stillness. Specifically, when the fluctuation of the reflected signal from the target person is less than a certain value within a certain period of time, the target person is considered to be still. This threshold can be optimized based on historical data on signal fluctuations during standing still. The TOF sensor can adjust its sensitivity threshold for distance changes. When the detected distance change within a certain period of time is less than a certain value, the target person is considered to be still. This threshold can be adjusted appropriately based on historical data.

[0117] Sitting height: Historical image monitoring data indicates that the individual's sitting height is 0.9 meters (determined by the relative positions of key skeletal points of the hips and head when sitting). If the sitting height detected by the radar and TOF sensor deviates from 0.9 meters, for example, 0.85 meters for the radar and 0.95 meters for the TOF sensor, the radar's accuracy during close-range detection may be affected by interference from the surrounding environment, such as signal reflection from nearby metal objects. This interference source needs to be identified and eliminated, and the parameters for close-range detection need to be recalibrated. For the TOF sensor, the linearity parameters for close-range measurement can be recalibrated. Using close-range standard targets of varying heights for detection, the linearity parameters can be adjusted based on the deviation between the detection result and the actual height to ensure accurate seated height detection.

[0118] Average sitting duration: Assume that historical data indicates an average sitting duration of 45 minutes for this individual. If the sensor-detected average sitting duration differs significantly from 45 minutes, such as 60 minutes, this may indicate a problem with the sensor's judgment of the end of the sitting state. For systems combining radar and TOF sensors, the algorithm for detecting the transition from sitting to standing needs to be optimized. For example, by combining positional changes of multiple key skeletal points, and incorporating information such as angle changes rather than just distance changes, this can more accurately determine the end of the sitting state and thus calibrate the average sitting duration detection result.

[0119] Average sit-to-stand transition time: Historical image monitoring data indicates that the average sit-to-stand transition time for this individual is 3 seconds (derived by detecting the position change time of key skeletal points, such as the hips and knees, during the sit-to-stand transition). If the average sit-to-stand transition time detected by radar and TOF sensors differs from 3 seconds, for example, 5 seconds, it is necessary to check whether the sensors are sufficiently responsive to rapid changes in human posture. For TOF sensors, it may be necessary to optimize their signal processing algorithms to reduce data processing latency, enabling them to more quickly and accurately capture changes in distance and posture during the sit-to-stand transition. For radar, the tracking algorithm parameters for fast-moving target detection can be adjusted to improve tracking accuracy for rapid changes in human posture, thereby accurately detecting the average sit-to-stand transition time.

[0120] After the above-mentioned state parameter correction, the fall detection process can be started, that is, judging whether the target monitored person has fallen based on the radar monitoring data and TOF sensor data;

[0121] Specifically, determining whether there is a target person in the monitoring range based on the first radar monitoring data, and if so, determining the height of the target person based on at least one TOF sensor data;

[0122] When the height of the target person is lower than a preset threshold, it is determined whether the target person may fall based on multiple TOF data; if so, the second radar monitoring data is obtained, the movement amplitude of the target person is monitored based on the second radar monitoring data, and whether the target monitored person falls is determined based on the movement amplitude.

[0123] It should be understood that the terms "first radar monitoring data" and "second radar monitoring data" here simply represent two sets of radar monitoring data at different points in time and do not represent the data's source. In other words, the "first radar monitoring data" and "second radar monitoring data" could be two sets of radar monitoring data collected by the same low-frequency radar at two different time points (one before and one after), or two sets of radar monitoring data collected by two different low-frequency radars at two different time points (one before and one after). The same applies to multiple TOF data sets.

[0124] As mentioned above, when the target person is in the living room area (the first target monitoring range), relevant fall monitoring can be performed through the visual intelligent image sensor. Therefore, when the target person enters the second monitoring range, it is judged whether the target monitored person falls based on the radar monitoring data and TOF sensing data.

[0125] In one possible scenario, the target person may already be accompanied by someone. In this case, fall monitoring and warning are not necessary. Fall monitoring and warning are only necessary when the target person is alone. Therefore, the determination of whether the target person has fallen based on radar monitoring data and TOF sensor data specifically includes:

[0126] Based on the first radar monitoring data, it is determined whether there is only a target monitored person in the monitoring range. If so, the height of the target person is determined based on at least one TOF sensor data; when the height of the target person is lower than a preset threshold, it is determined based on multiple TOF data whether the target person may fall; if so, the second radar monitoring data is obtained, the movement amplitude of the target person is monitored based on the second radar monitoring data, and whether the target monitored person falls based on the movement amplitude.

[0127] As a management and early warning means, when it is determined that the target person has fallen and has not returned to normal within a preset time period, a warning signal is sent to the relevant personnel of the target monitored person; the warning signal is used to remind the relevant personnel that the target person has fallen within the target monitoring range so that the relevant personnel can take necessary rescue measures.

[0128] As a more specific introduction to the principle, a specific scenario of the above method embodiment of the present invention is described as follows:

[0129] When the target monitoring person does not exist in the monitoring range, it is determined based on the first radar monitoring data that the target monitoring person does not exist in the monitoring range (is empty);

[0130] At this time, although there is no target person in the target monitoring range, there are still other objects, such as sofas, tables and chairs, beds, home appliances, floors, etc.; the basic parameters of the current environment (such as floor height, bed height, table and chair height) can be learned and recorded based on the radar monitoring data;

[0131] Next, when the target person enters the monitoring range, the radar monitoring data can detect the entry of the target person; at the same time, due to the entry of the target person, the height of the relevant position will also change, which further confirms that the target person has entered the current monitoring range;

[0132] At this time, if the target person is detected to enter the current monitoring range, but the height of the relevant position has not changed significantly, the judgment results include: the target person is lying / sitting down (normal sitting, normal sleeping) or falling down (abnormal situation);

[0133] While lying down and falling down have different posture parameters, as well as differences in the amplitude and duration of movement after lying down or falling, if the differences are not significant, simply using radar data can lead to misjudgments. Therefore, the technical solution of the present invention requires combining TOF sensor monitoring data to further confirm whether a fall has actually occurred. More significantly, a comprehensive judgment based on TOF sensor data and radar monitoring data can be made from the outset, thus avoiding misjudgments and improving the success rate.

[0134] exist Figure 1-Figure 3 Based on the method embodiment, Figure 4 A schematic diagram illustrating the hardware unit composition of a radar combined with TOF fall monitoring system according to an embodiment of the present invention is shown.

[0135] Figure 4 The system embodiment can be used to perform Figure 1 The method.

[0136] Specifically, Figure 4 A radar combined with TOF fall monitoring system is shown, which includes a historical image monitoring data acquisition unit, a radar sensor module, a TOF sensor module, a fall detection unit, and a warning unit;

[0137] After determining the target monitoring range and the target monitoring person, the historical image monitoring data acquisition unit acquires the historical image monitoring data of the target monitoring person;

[0138] Determining the layout of the radar sensor module and the TOF sensor module based on the target monitoring range and the historical image monitoring data;

[0139] The fall detection unit determines whether the target person has fallen based on the radar monitoring data and TOF sensor data;

[0140] When the fall detection unit determines that the target person has fallen and has not returned to normal within a preset time period, the warning unit sends a warning signal to the relevant personnel of the target monitored person;

[0141] The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range; an image sensor is installed in the first monitoring range.

[0142] The arrangement scheme of the radar sensor module and the TOF sensor module is determined based on the target monitoring range and the historical image monitoring data, specifically including:

[0143] The image sensor is not installed in the second monitoring range.

[0144] Determining the number of radars and TOF sensors to be arranged based on the second monitoring range;

[0145] Determining, based on the historical image monitoring data, the layout positions of the radar and the TOF sensor in the second monitoring range;

[0146] After the radar and TOF sensor are arranged within the second monitoring range based on the arrangement quantity and the arrangement position, the state parameters of the radar and TOF sensor are calibrated based on the historical image monitoring data.

[0147] Optionally, as a preferred embodiment, the radar sensor module and the TOF sensor module can also be integrated products, that is, integrated into a combined product, and the main installation method is to directly install them on the ceiling.

[0148] Of course, when implementing the technical solution of the present invention, the radar sensor module and the TOF sensor module can also be separate modular units, which can be installed separately or in combination according to actual needs. The present invention does not make specific limitations on this.

[0149] Although not shown in the accompanying drawings, preferably, further product embodiments may include an electronic device comprising a memory and one or more processors. The memory stores one or more application programs, and the one or more application programs are adapted to be executed by the one or more processors for executing the aforementioned radar combined with TOF fall detection method.

[0150] Although not shown in the drawings, more embodiments further include a computer-readable storage medium storing a computer program. When the computer program is executed, the aforementioned steps of the radar combined with TOF fall monitoring method are implemented.

[0151] It can be understood that the system, product, device, medium embodiments and method implementations correspond to each other and can reference each other. Their principles are similar or the same, so they will not be repeated.

[0152] For other technologies, principles, algorithms or models not elaborated in detail in this application, please refer to the existing technology.

[0153] In traditional fall detection solutions, some radar sensors have high costs due to their high technical complexity and strict manufacturing process requirements, which to a certain extent limits their large-scale promotion and application. The present invention uses a 24GHz low-cost radar, which costs only one-tenth to one-twentieth of the common 60GHz radar. This cost advantage greatly reduces the hardware procurement cost when deploying fall detection systems on a large scale. For example, in scenarios such as nursing homes and communities that require extensive coverage of fall detection equipment, the use of the low-cost radar of the present invention can achieve monitoring coverage in more areas with a limited budget, allowing more elderly people in need to benefit. At the same time, for home users, the lower equipment cost also reduces the economic threshold for them to install fall detection equipment for the elderly, making it affordable for more families and increasing the popularity of fall detection equipment.

[0154] The technical solution of the present invention uses a multi-area TOF sensor to broaden the detection range and improve the precision of detection. The TOF sensor obtains the distance information of the target object by measuring the flight time of the light pulse. Its measurement accuracy is high, especially when measuring at close range. In the fall detection scenario, TOF sensors in different areas can monitor the human body from multiple angles at the same time. When a human body falls, the distance between various parts of the body and the sensor will change rapidly. The multi-area TOF sensor can capture these subtle distance changes in time and transmit the data to the system for analysis. This multi-angle, all-round monitoring method greatly reduces the detection blind spots caused by complex changes in human posture or occlusion, making the system's judgment of falling actions more accurate.

[0155] In the solution of the present invention, multi-zone TOF sensors and radars work together to make up for the shortcomings of pure radar detection. TOF sensors can provide more detailed close-range human movement information, and combined with the long-range, large-area monitoring capabilities of radar, form a more comprehensive and reliable monitoring system. In multi-person scenarios, TOF sensors can assist radars in distinguishing different individuals by accurately identifying close-range human targets, reducing misjudgments caused by signal interference; for easily confused daily movements and falls, by comprehensively analyzing the distance change details obtained by the TOF sensor and the overall movement trends monitored by the radar, it is possible to more accurately determine whether a fall has occurred, greatly improving the reliability of the solution in complex scenarios.

[0156] In addition, the historical image monitoring data collected by the traditional visual intelligent image sensor installed in the living room contains rich human posture information, such as the walking height value, walking step length, walking speed, bending angle / bending-standing average duration value in the walking posture data, and the standing height, standing average static duration, sitting height, sitting average duration, sitting-standing average switching duration, etc. in the static posture data. The technical solution of the present invention uses this data to calibrate the state parameters of the radar and TOF sensors, which can make the sensors better adapt to the personalized characteristics of the target person. For example, through the statistics of the long-term historical image monitoring data of an individual, it is known that the average walking height value of the individual during normal walking is 1.75 meters. If the height data of the individual initially detected by the radar and TOF sensors during walking deviates from this value for a long time, the detection parameters of the sensor are adjusted through correction to make it more suitable for the actual situation of the individual, thereby improving the accuracy of the fall detection of the individual. This parameter correction based on personalized characteristics effectively avoids misjudgments and missed judgments caused by the general parameter settings of the sensor not being consistent with the actual situation of the individual, significantly improving the accuracy of detection.

[0157] More importantly, this solution uses radar and time-of-flight sensors for fall detection, fundamentally different from camera-based visual image monitoring solutions. Radar senses the position and motion of target objects by transmitting and receiving radio waves, while time-of-flight sensors use light pulses to measure distance. Neither involves collecting user images. In areas where privacy is paramount, such as bedrooms and bathrooms in homes, this non-visual monitoring technology can effectively avoid the privacy risks associated with image acquisition, providing users with greater peace of mind when using fall detection equipment.

[0158] Even when using historical image monitoring data to calibrate sensor state parameters to achieve fall detection based on the personalized characteristics of the target person, this solution also focuses on protecting user privacy. During the processing of the historical image monitoring data used, only key information related to human posture is extracted, such as the position coordinate values ​​(relative height values) of key human skeleton points (such as shoulders and hips), and does not involve the retention and use of sensitive information such as user facial features and personal identification. This refined data processing method maximizes the protection of user privacy while ensuring the accuracy of fall detection, making the solution more feasible and acceptable in practical applications.

[0159] The foregoing has shown and described the method embodiments and system of the present invention, but it is understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A radar combined with TOF fall monitoring method, characterized in that: The method comprises: Determine the target monitoring scope and target monitoring personnel; Obtain historical image monitoring data of the target monitored person; Determining a layout plan of radar and TOF sensors based on the target monitoring range and the historical image monitoring data; Determine whether the target person has fallen based on radar monitoring data and TOF sensor data; When it is determined that the target person has fallen and has not returned to normal within a preset time period, a warning signal is sent to relevant personnel of the target monitored person.

2. The radar combined with TOF fall monitoring method according to claim 1, characterized in that: The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range; The step of obtaining historical image monitoring data of a target monitored person includes: Based on the image sensor arranged before executing the fall monitoring method, historical image monitoring data of the target monitored person in the first monitoring range is obtained, and the historical monitoring data includes walking posture data and static posture data of the target monitored person.

3. The radar combined with TOF fall monitoring method according to claim 1, characterized in that: The determination of the arrangement of radar and TOF sensors based on the target monitoring range and the historical image monitoring data specifically includes: Determining the number of radars and TOF sensors to be deployed based on the target monitoring range; Determining the layout positions of the radar and the TOF sensor based on the historical image monitoring data; After the radar and the TOF sensor are arranged within the target monitoring range based on the arrangement quantity and the arrangement position, the state parameters of the radar and the TOF sensor are calibrated based on the historical image monitoring data.

4. The radar combined with TOF fall monitoring method according to claim 1, characterized in that: The method of determining whether the target monitored person has fallen based on the radar monitoring data and the TOF sensor data specifically includes: Determining whether there is a target person in the monitoring range based on the first radar monitoring data, and if so, determining the height of the target person based on at least one TOF sensor data; When the height of the target person is lower than a preset threshold, it is determined whether the target person may fall based on multiple TOF data; if so, the second radar monitoring data is obtained, the movement amplitude of the target person is monitored based on the second radar monitoring data, and whether the target monitored person falls is determined based on the movement amplitude.

5. The radar combined with TOF fall monitoring method according to claim 1, characterized in that: The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range; When the target person enters the second monitoring range, it is determined whether the target person falls based on the radar monitoring data and the TOF sensing data.

6. The radar combined with TOF fall monitoring method according to claim 4, characterized in that: The method of determining whether the target monitored person has fallen based on the radar monitoring data and the TOF sensor data specifically includes: Based on the first radar monitoring data, it is determined whether there is only a target monitoring person in the monitoring range. If so, the height of the target person is determined based on at least one TOF sensing data.

7. A radar combined with time-of-flight fall monitoring system, comprising a historical image monitoring data acquisition unit, a radar sensor module, a time-of-flight sensor module, a fall detection unit, and an alarm unit; Its characteristics are: After determining the target monitoring range and the target monitoring person, the historical image monitoring data acquisition unit acquires the historical image monitoring data of the target monitoring person; Determining the layout of the radar sensor module and the TOF sensor module based on the target monitoring range and the historical image monitoring data; The fall detection unit determines whether the target person has fallen based on the radar monitoring data and TOF sensor data; When the fall detection unit determines that the target person has fallen and has not returned to normal within a preset time period, the warning unit sends a warning signal to the relevant personnel of the target monitored person; The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range; an image sensor is installed in the first monitoring range.

8. The radar combined with TOF fall monitoring system according to claim 7, characterized in that: The arrangement scheme of the radar sensor module and the TOF sensor module is determined based on the target monitoring range and the historical image monitoring data, specifically including: Determining the number of radars and TOF sensors to be arranged based on the second monitoring range; Determining, based on the historical image monitoring data, the layout positions of the radar and the TOF sensor in the second monitoring range; After the radar and TOF sensor are arranged within the second monitoring range based on the arrangement quantity and the arrangement position, the state parameters of the radar and TOF sensor are calibrated based on the historical image monitoring data.

9. The radar combined with TOF fall monitoring system according to claim 7, characterized in that: The image sensor is not installed in the second monitoring range.

10. A mobile terminal comprising a human-computer interaction unit, characterized in that: The human-computer interaction unit is used to receive the warning signal issued by the radar combined with TOF fall monitoring method according to any one of claims 1 to 6, and the warning signal is used to prompt the target person to fall within the target monitoring range.

Images