Fall monitoring system and method using radar combined with TOF

Abstract

The application provides a fall monitoring system and method combining radar and TOF, and belongs to the technical field of action recognition and data processing. The method comprises the following steps: determining a target monitoring range and a target monitoring person; obtaining historical image monitoring data of the target monitoring person; determining a layout scheme of a radar and a TOF sensor based on the target monitoring range and the historical image monitoring data; judging whether the target monitoring person falls based on radar monitoring data and TOF sensing data; and sending a warning signal to the relevant personnel of the target monitoring person when it is judged that the target person has a falling phenomenon and has not recovered to a normal state within a preset time period. The system comprises a historical image monitoring data acquisition unit, a radar sensing module, a TOF sensing module, a fall detection unit and a warning unit. The technical scheme of the application can realize fall detection based on the individual characteristics of the target person while ensuring user privacy, and has high detection accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of action recognition and data processing monitoring, and particularly relates to a radar combined with TOF fall monitoring system and method, computer readable storage medium for implementing the method, computer program product and related mobile terminal. BACKGROUND

[0002] Many elderly people in daily life once inadvertently fall down, due to inability to help themselves, and no one around timely discovery, often delay the best opportunity to send medical treatment, eventually lead to serious consequences. Some old people because of a long time fall to the ground, the injured part of the body cannot be treated in time, wound infection and deterioration, even face the risk of amputation; some old people who have cardiovascular and cerebrovascular diseases may induce heart attack, cerebral infarction and other fatal diseases due to emotional tension, body stress reaction, etc., endangering life safety.

[0003] In order to protect the health of these old people and provide them with good treatment services, it is necessary to carry out a series of reliable health monitoring activities for them. Fall detection is an important part of various health monitoring.

[0004] In related technologies, fall detection mainly includes detection based on visual image acquisition, detection based on infrared camera, detection based on millimeter wave radar, etc. For example, the millimeter wave radar old person fall identification method based on Vision Transformer proposed in Chinese invention patent publication CN117368909A, the monitoring method of dynamic target based on millimeter wave radar proposed in CN112859067A.

[0005]

[0006] The fall detection method based on visual image has high accuracy and will not misjudge, but the privacy is not friendly, for example, bathroom and bedroom are high-risk areas of falling, but it is difficult to accept to install a camera; the fall detection method based on infrared camera has accurate posture detection, but the installation cost is high, and its appearance generally looks like a camera, which is easy to feel that privacy is violated; in the detection method based on millimeter wave radar, in order to ensure accuracy, 60GHz or above frequency band is needed to cooperate with multi-transmitting and multi-receiving antenna design, large power MCU processing, and the cost is high. SUMMARY

[0007] In view of the above technical problems, the present application provides a radar combined with TOF fall monitoring system and method, computer readable storage medium for implementing the method, computer program product and related mobile terminal.

[0008] In the first aspect of the present application, a radar combined with TOF fall monitoring method is provided, which comprises:​

[0009] determining a target monitoring range and a target monitoring person;

[0010] acquiring historical image monitoring data of the target monitoring person;

[0011] determining a placement scheme of a radar and a TOF sensor based on the target monitoring range and the historical image monitoring data;

[0012] judging whether the target monitoring person falls based on radar monitoring data and TOF sensing data;

[0013] when it is judged that the target person has a falling phenomenon and has not recovered to a normal state within a preset time period, sending a warning signal to a related person of the target monitoring person.

[0014] the target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range;

[0015] the acquiring of the historical image monitoring data of the target monitoring person includes:

[0016] acquiring historical image monitoring data of the target monitoring person in the first monitoring range based on an image sensor arranged before the execution of the fall monitoring method, the historical monitoring data including walking posture data and static posture data of the target monitoring person.

[0017] the determining of the placement scheme of the radar and the TOF sensor based on the target monitoring range and the historical image monitoring data specifically includes:

[0018] determining the number of placements of the radar and the TOF sensor based on the target monitoring range;

[0019] determining the placement positions of the radar and the TOF sensor based on the historical image monitoring data;

[0020] after the radar and the TOF sensor are arranged in the target monitoring range based on the number of placements and the placement positions, correcting the state parameters of the radar and the TOF sensor based on the historical image monitoring data.

[0021] the judging of whether the target monitoring person falls based on the radar monitoring data and the TOF sensing data specifically includes:

[0022] judging whether there is a target monitoring person in the monitoring range based on first radar monitoring data, and if so, judging the height of the target person based on at least one TOF sensing data;

[0023] When the height of the target person is lower than a preset threshold, it is determined whether the target person is likely to fall based on the plurality of TOF data; if so, second radar monitoring data is acquired, the motion amplitude of the target person is monitored based on the second radar monitoring data, and it is determined whether the target monitoring person falls based on the motion amplitude.

[0024] The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range.

[0025] When the target person enters the second monitoring range, it is determined whether the target monitoring person falls based on radar monitoring data and TOF sensing data.

[0026] The determination of whether the target monitoring person falls based on radar monitoring data and TOF sensing data specifically includes:

[0027] Based on the first radar monitoring data, it is determined whether the monitoring range only has the target monitoring person, and if so, the height of the target person is determined based on at least one TOF sensing data.

[0028] In a second aspect of the present application, a fall monitoring system combining radar and TOF is also provided, which includes a historical image monitoring data acquisition unit, a radar sensing module, a TOF sensing module, a fall detection unit, and a warning unit.

[0029] After the target monitoring range and the target monitoring person are determined, the historical image monitoring data acquisition unit acquires historical image monitoring data of the target monitoring person.

[0030] Based on the target monitoring range and the historical image monitoring data, a layout scheme of the radar sensing module and the TOF sensing module is determined.

[0031] The fall detection unit determines whether the target monitoring person falls based on radar monitoring data and TOF sensing data.

[0032] When the fall detection unit determines that the target person has a fall phenomenon and has not recovered to a normal state within a preset time period, the warning unit sends a warning signal to the relevant personnel of the target monitoring person.

[0033] The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range; an image sensor is installed in the first monitoring range.

[0034] The determination of the layout scheme of the radar sensing module and the TOF sensing module based on the target monitoring range and the historical image monitoring data specifically includes:

[0035] The image sensor is not installed in the second monitoring range.

[0036] determining a number of arrangements of the radar and TOF sensors based on the second monitoring range;

[0037] determining a position of arrangement of the radar and TOF sensors in the second monitoring range based on the historical image monitoring data;

[0038] after the radar and TOF sensors are arranged in the second monitoring range based on the number of arrangements and the position of arrangement, correcting state parameters of the radar and TOF sensors based on the historical image monitoring data.

[0039] In a second aspect of the present application, an electronic device is provided, which comprises a processor and a memory storing computer executable program instructions; when the program instructions are executed by the processor, the aforementioned method for fall detection by radar combined with TOF is implemented.

[0040] The electronic device can also be implemented as a human-computer interaction device comprising an external data interface; the external data interface can access at least one computer readable storage medium, which stores computer program code; when the computer program code is transferred to the memory and executed by the processor, all steps of the aforementioned method for fall detection by radar combined with TOF are implemented.

[0041] Based on the same inventive concept, the present application also provides a computer medium, which stores a computer program; when the computer program is executed, all or part of the steps of the aforementioned method for fall detection by radar combined with TOF are implemented.

[0042] In a third aspect of the present application, a mobile terminal is provided, which comprises a human-computer interaction unit, and the human-computer interaction unit is configured to receive the warning signal issued by the method for fall detection by radar combined with TOF according to the first aspect, and the warning signal is used to prompt the target person to fall in the target monitoring range.

[0043] The present application provides a low-cost and high-accuracy fall detection scheme, which uses low-cost radar combined with multi-region TOF sensors, and corrects the state parameters of the radar and TOF sensors using historical image monitoring data, to realize fall detection based on the individual characteristics of the target person. The reliability of the scheme is significantly better than pure radar detection while protecting user privacy, and its specific advantages and implementation principles will be further described in detail in the specific embodiments combined with the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0045] Figure 1 is a main flow diagram of a radar combined with TOF fall detection method according to an embodiment of the present application;

[0046] Figure 2 is a computer program for implementing Figure 1 the method;

[0047] Figure 3 is a layout scene diagram for implementing Figure 1 the method;

[0048] Figure 4 is a hardware unit composition diagram of a radar combined with TOF fall detection system according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the detailed description of the present application, if the embodiments of the related technical solutions involve user-related data, when the embodiments of the present application are applied to specific products or technologies, the user's permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0050] Before introducing the specific embodiments of the present application, continuing the introduction of the prior art, first introduce the existing multiple fall detection schemes and their defects in the prior art, so as to lead to the improvement motivation of the present application, and then better understand the advantages of the technical solutions of the present application.

[0051] In the prior art, fall detection technology mainly includes detection based on visual image acquisition, detection based on infrared camera, detection based on millimeter wave radar and other schemes.

[0052] Detection technology based on visual image acquisition: through a camera to capture human body action in real time, using deep learning algorithm (such as YOLOv8, OpenPose) to analyze the human body posture change in video frame, to identify the fall behavior. For example, YOLOv8 combined with pose estimation model, first detects the position of human body, and then extracts key points (such as shoulders, knees) to judge the posture abnormality.

[0053] Its advantages are high accuracy, which can accurately capture the posture change when falling, such as the key point displacement of the human body from standing to falling, and the accuracy can reach more than 95% in a scene with sufficient light and simple background. However, its obvious defect is that the visible camera needs to shoot the visible human body image, which is easy to cause privacy disputes, especially in private places (such as bedroom, bathroom), the privacy is very unfriendly, for example, bathroom, bedroom is a high incidence area of falling, but it is difficult to accept to install a camera; at the same time, the deep learning model (such as 3DCNN) needs a large amount of computing resources, and real-time detection may have high requirements for the performance of edge devices.

[0054] The detection technology based on infrared camera: using infrared sensor to capture human body heat radiation, generating thermal image, identifying falling by analyzing temperature distribution change. For example, the improved Alphapose algorithm combined with YOLOv5s target detection network, judging posture anomaly through human skeleton key points in infrared image.

[0055] In this scheme, it can work without visible light, suitable for night or dim light scene (such as old people's bedroom), and the accuracy can reach more than 95%; however, this scheme needs to accurately calibrate the position and angle of the sensor to ensure the coverage of the monitoring area, and the cost of the device is higher (such as H60 infrared waterproof camera, the unit price is more than 1000 yuan); in addition, its appearance is still the appearance of a camera in general, which is easy to feel that the privacy is violated.

[0056] The detection technology based on millimeter wave radar: transmitting millimeter wave frequency electromagnetic wave, obtaining human body distance, speed and angle information by analyzing time difference (TOF) and Doppler frequency shift of reflected signal, and identifying falling action combined with micro Doppler effect.

[0057] In this scheme, only human motion parameters (such as position, speed) are output, without image or video, which can protect user privacy; however, in the environment with multiple moving targets or strong background noise (such as multiple people moving at the same time), the signal processing complexity increases, which may affect the real-time performance. In order to ensure the accuracy, 60GHz or above frequency band is needed to cooperate with multi-transmitting and multi-receiving antenna design, high-power MCU processing, and high cost.

[0058] Therefore, the embodiment of the present application proposes a falling detection method based on the individual characteristics of the target person, which can ensure the privacy of the user and ensure high detection accuracy.

[0059] Reference Figure 1 , Figure 1 The main flowchart of a radar combined with TOF falling monitoring method according to an embodiment of the present application is shown. Figure 1 Five flowcharts are shown, including five main implementation steps.

[0060] For brevity of description, the following description is givenFigure 1 The five flowcharts shown are numbered S1-S5 (step numbers are omitted in the figure):

[0061] S1: Determine the target monitoring range and the target monitoring person;

[0062] S2: Obtain historical image monitoring data of the target monitoring person;

[0063] S3: Determine the arrangement scheme of the radar and TOF sensor based on the target monitoring range and the historical image monitoring data;

[0064] S4: Determine whether the target monitoring person has fallen based on the radar monitoring data and TOF sensor data;

[0065] S5: When it is determined that the target person has a falling phenomenon and has not recovered to a normal state within a preset time period, an alarm signal is sent to the relevant personnel of the target monitoring person.

[0066] Figure 1 The method can be automatically executed by a computer program, Figure 2 The relevant program flowchart is shown, described in computer pseudo-code language as follows:

[0067] Start;

[0068] Step 1: Determine the target monitoring range and the target monitoring person;

[0069] Step 2: Obtain historical image monitoring data of the target monitoring person;

[0070] Step 3: Determine the arrangement scheme of the radar and TOF sensor based on the target monitoring range and the historical image monitoring data, and then arrange the radar and TOF sensor,

[0071] Real-time collection of radar monitoring data and TOF sensor data;

[0072] Step 4: Determine whether the target person has fallen based on the monitoring data and sensor data. If yes, go to step 5; otherwise, return to the step of real-time collection of radar monitoring data and TOF sensor data;

[0073] Step 5: Start a preset time period timer (e.g., 30-second countdown);

[0074] Determine whether the target person has recovered to a normal state within the preset time period. If yes, return to the step of real-time collection of radar monitoring data and TOF sensor data;

[0075] If no, go to step 6;

[0076] Step 6: Send an alarm signal to the relevant personnel;

[0077] End.

[0078] Figure 3 showing the implementation of Figure 1 a layout scene diagram of the method, including a stereoscopic space diagram and a planar layout diagram from a bird's eye view.

[0079] Figure 3 showing the implementation of Figure 1 a schematic diagram of the fall monitoring method. The indoor living space includes a living room area and multiple room areas.

[0080] The living room area and the multiple room areas constitute the target monitoring range of the embodiment of the application; the elderly who are active in the target monitoring range constitute the target monitoring person of the embodiment of the application.

[0081] As a summary, the target monitoring range of the embodiment of the application includes a first monitoring range A and multiple second monitoring ranges B1-B3 other than the first monitoring range.

[0082] For example, the first monitoring range can be the living room area; the multiple second monitoring ranges can be the multiple room areas, including the bedroom, the kitchen, the bathroom, etc. Figure 3

[0083] It can be understood that, for home safety, at least one image sensor is usually installed in the living room area, which can cover the entire living room activity range. The image sensor can be a smart image sensor, which can identify the active person, the number of persons, whether the posture of the person is abnormal, etc. in the living room range. The installation of a visual image sensor in the living room is usually acceptable to most users. Therefore, in the living room range, the posture of the person can be identified based on the smart image sensor. For example, when the person enters the living room range, accurate fall monitoring is realized based on the aforementioned detection technology based on visual image acquisition.

[0084] However, in the multiple second monitoring ranges other than the first monitoring range, the installation of a visual image acquisition sensor is easy to be felt by the user as an invasion of privacy. At this time, the first improvement of the application is that the visual image sensor is not installed in the second monitoring range; only a visual smart image sensor is installed in the first monitoring range for fall detection.

[0085] Correspondingly, in the second monitoring range, a radar and a TOF sensor need to be installed.

[0086] In the embodiment of the application, the radar and the TOF sensor can also be referred to as a radar sensing module and a TOF sensing module. One radar sensing module includes multiple radars, and one TOF can include multiple TOF sensors. ​

[0087] The radar sensing module is composed of a low-frequency radar; preferably, the embodiment of the present application adopts a 24GHz low-cost radar.

[0088] The millimeter wave radar generally used in the related art is usually above 30-60GHz, and a 60GHz or above frequency band is used, a multi-transmitting and multi-receiving antenna design is used, a large computing power MCU is used for processing, and the cost is very high. The cost of the 24GHz radar preferably used in the present application is only 1 / 10-1 / 20 of that of the 60GHz radar, and the advantage is obvious.

[0089] The TOF sensing module is composed of a plurality of TOF sensors that can communicate with each other.

[0090] TOF is the abbreviation of Time of Flight, which is a three-dimensional imaging technology for acquiring the distance of a target person by measuring the time of a light pulse from transmission to reflection by the target person and reception by the sensor.

[0091] Correspondingly, the TOF sensor involved in the embodiment of the present application calculates the distance between two points by transmitting a light signal (usually near-infrared light or laser, and the common wavelength is 850nm and 940nm) and using the time required for the photon to propagate between the two points. Based on the time difference between the signal transmission and its return to the sensor after being reflected by the object, the distances measured by all pixel points constitute a depth map.

[0092] Since low-frequency radar and TOF sensors need to be installed in the second monitoring range to achieve accurate fall monitoring, the related modules need to be installed in a position that can effectively detect human motion, so that the monitoring range of the low-frequency radar and the TOF sensor can cover the area where the personnel may move and fall, such as the center position of the living room, the toilet and the shower area, etc. At the same time, it needs to be away from electrical equipment, air vents, metal objects and other places that may interfere with the signal. Because the low-frequency radar signal may be affected by electromagnetic interference, and the TOF sensor may be affected by metal reflection to affect the measurement accuracy.

[0093] As a general principle, the installation height is determined according to the specific product characteristics and monitoring scene. Generally, the low-frequency radar can be installed at a position about 2-3 meters away from the ground to obtain a wider monitoring range; if the TOF sensor is used for auxiliary detection, it can be installed at a lower position, such as about 1.5 meters away from the ground, so as to more accurately detect the motion changes at a close distance.

[0094] However, the height, gait and marching posture (such as marching step, marching speed, bending habit) of each target person are different, and the spatial structure of each target monitoring range is also different; if the relevant sensing module is installed only according to the foregoing general principle, the monitoring effect will be greatly reduced.

[0095] Specifically, another improvement of the present application is that: historical image monitoring data of the target monitoring person is acquired; and the arrangement scheme of the radar and the TOF sensor is determined based on the target monitoring range and the historical image monitoring data;

[0096] Specifically, the arrangement number of the radar and the TOF sensor is determined based on the target monitoring range;

[0097] The arrangement position of the radar and the TOF sensor is determined based on the historical image monitoring data;

[0098] After the radar and the TOF sensor are arranged in the target monitoring range based on the arrangement number and the arrangement position, the state parameters of the radar and the TOF sensor are corrected based on the historical image monitoring data.

[0099] As a specific example, the historical image monitoring data of the target monitoring person is acquired, including:

[0100] The historical visual image monitoring data of the target monitoring person in the first monitoring range is acquired based on the visual image sensor arranged before the fall monitoring method is performed, and the historical monitoring data includes walking posture data and static posture data of the target monitoring person.

[0101] For example, the walking posture data and the static posture data of the target monitoring person are screened out from the historical visual image monitoring data set collected by the visual smart image sensor arranged in the living room (first monitoring range); the walking posture data includes walking height value, walking step length, walking speed, bending angle, and bending- standing average duration value; and the static posture data includes standing height, standing average static duration, sitting height, sitting average duration, and sitting- standing average switching duration.

[0102] Preferably, the relevant visual image data of the target monitoring person screened out from the visual image monitoring data set further includes position coordinate value (relative height value) of human key skeleton points (such as shoulder and hip).

[0103] The arrangement number of the radar and the TOF sensor is determined based on the target monitoring range, specifically: the arrangement number of the radar and the TOF sensor is determined based on the number of target monitoring ranges, and the area of each target monitoring range, the duration of the target person in each target monitoring range, and the in-out frequency.

[0104] As a general principle, the target monitoring range only includes the second monitoring range (i.e. does not include the living room, which has been configured with a visual smart image sensor); and each second monitoring range is configured with at least one low-frequency radar and one TOF sensor;

[0105] On this basis, the number of low-frequency radars and TOF sensors arranged in each second monitoring range can be increased in positive correlation (for example, in direct proportion) with the duration of the target person in each second target monitoring range, the frequency of entry and exit, the area of each second target monitoring range, and the like. For example, the larger the area of each target monitoring range, the more low-frequency radars and TOF sensors, and the like.

[0106] On this basis, the arrangement position of radars and TOF sensors in each target monitoring range can also be determined in detail, for example, based on standing height data in static posture data, sitting height data, based on walking height data in walking posture data, and the like, to further determine the arrangement height of radars and TOF sensors.

[0107] Taking a bathroom (hand washing room, toilet) as an example, in one example, the head height of a target person standing is about 1.69 meters, and therefore a low-frequency radar is installed at a position close to the toilet / shower area on the ceiling at a height of 2.5 meters; the standing head height (1.6-1.9 meters) is covered, and at the same time, the toilet, shower area, and other high-fall areas are monitored through a horizontal field of view angle (such as 100°); this position can also penetrate water vapor and steam, and stably detect the falling action in a humid environment, with an accuracy of ≥97%; and a TOF sensor is installed at a height of 1.5 meters on the side wall of the toilet, covering the sitting head height (0.8-1.2 meters), and combining the close-range detection capability (0.5-3 meters) to monitor the actions of defecation and getting up, and to detect the lying posture (flat lying, prone lying, and the like) after falling, with a false alarm rate of less than 3%.

[0108] The radars and TOF sensors installed according to the above configuration number and configuration position are generally in an initial factory setting state. This factory setting state is usually set according to general statistical principles and universal target person characteristics, and may not be suitable for the target person characteristics in the current monitoring range and the monitoring environment itself.

[0109] Therefore, the next step is to enter the improvement point of the present application, that is, after arranging the radars and TOF sensors in the target monitoring range based on the arrangement number and the arrangement position, the state parameters of the radars and TOF sensors are corrected based on the historical image monitoring data.

[0110] In one specific example, the process of related state parameter correction includes:

[0111] Walking height value: Assuming that through historical image monitoring data statistics, the walking height value of a certain individual in normal walking state is 1.75 meters on average (this value can be calculated by the position coordinate value of the key skeleton point of the human body such as the top of the head relative to the ground). If the initial detection height range set by the radar and the TOF sensor is expected to be 1.7-1.8 meters for the walking state of the individual, when it is found that the walking height value in a large number of actual walking data deviates from this range more, it needs to be corrected. For example, if the height data of the individual walking detected by the radar is long-term concentrated in 1.6-1.7 meters, it may be that the installation angle or height of the radar is deviated, and the installation height needs to be adjusted or the vertical angle of the radar needs to be recalibrated, so that the detected walking height value is consistent with the average walking height value in the historical data.

[0112] Walking step length: According to historical image monitoring data statistics, the walking step length of the individual is 0.7 meters on average (calculated by the distance change of the key skeleton point of the same side foot in two consecutive images). If the TOF sensor detects the walking action of the individual, and the step length calculated according to the detected distance change is significantly different from 0.7 meters, such as the long-term detection result shows that the step length is 0.5 meters. This may mean that the detection accuracy of the TOF sensor has a problem, and it needs to be recalibrated. The standard reflective plate can be set at a certain distance, and the TOF sensor is used to detect it. According to the deviation between the known distance and the detected distance, the internal parameters of the TOF sensor are adjusted, such as the time measurement accuracy parameter, so that it can accurately detect the walking step length.

[0113] Walking speed: From historical image monitoring data, it is analyzed that the walking speed of the individual in normal walking state is 1.2 m / s on average (calculated by the displacement of the key skeleton point of the human body and the time interval in multiple consecutive images). If the radar calculates the walking speed by detecting the Doppler frequency shift of the reflected signal, and the difference is obvious, such as the continuous detection result is 0.8 m / s. At this time, the Doppler frequency shift measurement module of the radar needs to be checked, and the frequency response parameters of the module may need to be recalibrated. By comparing the speed detected by the radar with the actual speed of the standard target moving at a known speed, the relevant parameters are adjusted, so that the radar can accurately detect the walking speed.

[0114] Bending angle / Bending-stand average duration value: Assuming historical data shows that the individual's average bending angle during daily activities is 45° (calculated by the relative position change of the shoulder and hip key skeleton points), and the average duration of bending-stand is 5 seconds. If the bending angle data detected by the TOF sensor deviates greatly from 45°, or the duration of bending-stand judged by the radar combined with the TOF sensor data is significantly different from 5 seconds, such as the detected bending angle is long-term around 30°, and the duration of bending-stand is 8 seconds. For the TOF sensor, it may need to recalibrate its accuracy in detecting the distance of objects at different angles. This can be done by simulating standard models with different bending angles and allowing the TOF sensor to detect them. According to the deviation between the detection results and the true angle, adjust the internal algorithm parameters. For the radar, it may need to optimize its signal processing algorithm during the change of human posture, so that it can more accurately judge the duration of actions such as bending-stand with the TOF sensor.

[0115] Standing height: Historical image monitoring data shows that the individual's standing height is 1.78 meters (determined by the relative height value of the head vertex to the ground). If the height detected by the radar and TOF sensor when the individual is standing does not match 1.78 meters, for example, the radar detects 1.75 meters, and the TOF sensor detects 1.82 meters. First, check whether the installation position and angle of the radar and TOF sensor are correct, and whether there are influencing factors such as obstruction. If the installation is correct, for the radar, it can adjust the calibration parameters of its distance measurement, use a standard object with known height to measure multiple times, and calibrate the distance measurement accuracy of the radar according to the deviation between the measurement results and the actual height. For the TOF sensor, it can calibrate its internal time-of-flight measurement circuit, check whether the time accuracy of its transmitted and received light signals is accurate, compare it with a standard time source, and adjust the relevant circuit parameters so that it can accurately measure the standing height.

[0116] Standing average still duration: According to historical data, the individual's standing average still duration is 30 minutes. If the standing still duration detected by the radar and TOF sensor is significantly different from 30 minutes, such as the detection result is only 10 minutes. It may be that the sensor's judgment threshold for the still state is not reasonable and needs to be adjusted. For example, the radar can adjust the signal fluctuation threshold for judging the still state of the target person, that is, when the target person's reflected signal fluctuation is less than a certain value within a certain time, it is determined to be in a still state. According to the signal fluctuation during standing still in historical data, optimize the threshold. The TOF sensor can adjust its sensitivity threshold for distance changes. When the detected distance change is less than a certain value within a period of time, it is determined to be in a still state, and it is adjusted reasonably according to historical data.

[0117] Sitting height: historical image monitoring data shows that the sitting height of the individual is 0.9 meters (determined by the relative positions of the hip and head key skeleton points when sitting). If the sitting height detected by the radar and TOF sensor deviates from 0.9 meters, such as 0.85 meters detected by the radar and 0.95 meters detected by the TOF sensor. For the radar, it may be that its accuracy in close-range detection is affected by environmental interference, such as nearby metal objects reflecting signals, etc., which needs to be investigated and interference sources eliminated, while recalibrating the parameters for close-range detection. For the TOF sensor, the linearity parameter for close-range measurement can be recalibrated by using different height close-range standard target objects for detection, and adjusting the linearity parameter according to the deviation between the detection result and the actual height, so that the sitting height detection is accurate.

[0118] Sitting average duration: assuming that the historical data shows that the average sitting duration of the individual is 45 minutes. If the average sitting duration detected by the sensor deviates significantly from 45 minutes, such as 60 minutes. This may be a problem with the sensor's judgment of the end of the sitting state. For a system combining radar and TOF sensors, the detection algorithm for the transition from sitting to standing needs to be optimized. For example, by adding joint judgment of the position changes of multiple key skeleton points, not just relying on distance changes, but also combining angle changes, to more accurately determine the end of the sitting state, and thus correct the detection result of the average sitting duration.

[0119] Sitting-stand average switching duration: historical image monitoring data shows that the average sitting-stand switching duration of the individual is 3 seconds (determined by detecting the position change time of key skeleton points such as the hip and knee during the sitting to standing process). If the average sitting-stand switching duration detected by the radar and TOF sensor does not match 3 seconds, such as 5 seconds. It needs to be checked whether the sensor's detection response speed is sufficient during the rapid change of human posture. For the TOF sensor, it may need to optimize its signal processing algorithm to reduce data processing delay, so that it can capture the distance and posture changes in the sitting-stand process more quickly and accurately. For the radar, the tracking algorithm parameters for detecting fast-moving targets can be adjusted to improve the tracking accuracy of rapid changes in human posture, so as to accurately detect the average sitting-stand switching duration.

[0120] After the above state parameter correction, the fall monitoring process can begin, i.e. based on radar monitoring data and TOF sensor data to determine whether the target monitoring person has fallen;

[0121] Specifically, based on the first radar monitoring data to determine whether there is a target monitoring person in the monitoring range, if so, based on at least one TOF sensor data to determine the height of the target person;

[0122] When the height of the target person is lower than a preset threshold, whether the target person is likely to fall is determined based on multiple TOF data; if yes, second radar monitoring data is acquired, the action amplitude of the target person is monitored based on the second radar monitoring data, and whether the target monitoring person falls is determined based on the action amplitude.

[0123] It should be understood that the "first radar monitoring data" and "second radar monitoring data" herein are only used to represent two groups of radar monitoring data at different time points, and do not represent data sources. That is, the "first radar monitoring data" and "second radar monitoring data" can be two groups of radar monitoring data collected by the same low-frequency radar at two different time nodes (one after the other), or two groups of radar monitoring data collected by two different low-frequency radars at two different time nodes (one after the other). The same applies to the multiple TOF data.

[0124] As mentioned above, when the target person is in the living room range (the first target monitoring range), relevant fall monitoring can be performed through the visual intelligent image sensor, and therefore, when the target person enters the second monitoring range, whether the target monitoring person falls is determined based on the radar monitoring data and the TOF sensing data.

[0125] In a possible case, the target person can be accompanied by someone, in which case fall monitoring and early warning are not required; only when the target person is in a solitary situation, fall monitoring and early warning are required, and therefore, whether the target monitoring person falls is determined based on the radar monitoring data and the TOF sensing data, specifically including:

[0126] Whether the target monitoring person is the only one in the monitoring range is determined based on the first radar monitoring data, if yes, the height of the target person is determined based on at least one TOF sensing data; when the height of the target person is lower than a preset threshold, whether the target person is likely to fall is determined based on multiple TOF data; if yes, second radar monitoring data is acquired, the action amplitude of the target person is monitored based on the second radar monitoring data, and whether the target monitoring person falls is determined based on the action amplitude.

[0127] As a management and early warning means, when it is determined that the target person has a falling phenomenon and has not recovered to a normal state within a preset time period, a warning signal is sent to the relevant personnel of the target monitoring person; the warning signal is used to prompt the relevant personnel that the target person falls in the target monitoring range, so that the relevant personnel can take necessary rescue measures.

[0128] As a more specific and principle introduction, a specific scene of the above-mentioned method embodiment of the application is introduced as follows:

[0129] When the target monitoring range does not exist the target monitoring person, it is based on the first radar monitoring data to judge that the target monitoring range does not exist the target monitoring person (empty);

[0130] At this time, although the target monitoring range does not exist the target monitoring person, but still exist other objects, such as sofa, table and chair, bed, home appliances, floor, etc.;It can be based on the radar monitoring data to learn the basic parameters of the current environment (such as floor height value, bed height value, table and chair height value) and record them;

[0131] Next, when the target monitoring person enters the monitoring range, the radar monitoring data can monitor the target person entering. At the same time, due to the target monitoring person entering, the height of the relevant position will also change, so as to further confirm that the target person enters the current monitoring range;

[0132] At this time, if the target person is monitored to enter the current monitoring range, but the height of the relevant position does not change significantly, the judgment result includes that the target person lies down / sits down (normal sitting, normal sleeping) or falls (abnormal situation);

[0133] Although the posture parameters of lying down and falling down are different, and the motion amplitude and duration after lying down or falling down are different, but in the case of no significant difference, the use of radar data alone may lead to misjudgment. Therefore, the technical scheme of the present application needs to combine the monitoring data of TOF sensor to further confirm whether there is a real fall. More significantly, the comprehensive judgment can be made based on the TOF sensing data and the radar monitoring data at the beginning, so as to avoid misjudgment and improve the success rate.

[0134] In the method embodiment, Figures 1-3 On the basis of the method embodiment, Figure 4 The hardware unit composition schematic diagram of the radar combined with TOF fall monitoring system of one embodiment of the present application is shown.

[0135] Figure 4 The system embodiment can be used to execute Figure 1 The method.

[0136] Specifically, Figure 4 The radar combined with TOF fall monitoring system shown includes a historical image monitoring data acquisition unit, a radar sensing module, a TOF sensing module, a fall detection unit and a warning unit.

[0137] After determining the target monitoring range and the target monitoring person, the historical image monitoring data acquisition unit acquires the historical image monitoring data of the target monitoring person.

[0138] Based on the target monitoring range and the historical image monitoring data, the arrangement scheme of the radar sensing module and the TOF sensing module is determined.

[0139] The fall detection unit determines whether the target monitored person falls based on the radar monitoring data and the TOF sensing data;

[0140] When the fall detection unit determines that the target person has a falling phenomenon and has not recovered to a normal state within a preset time period, the warning unit sends a warning signal to the related personnel of the target monitored person;

[0141] The target monitoring range includes a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range; an image sensor is installed in the first monitoring range.

[0142] The arrangement scheme of the radar sensing module and the TOF sensing module is determined based on the target monitoring range and the historical image monitoring data, specifically including:

[0143] The image sensor is not installed in the second monitoring range.

[0144] The arrangement number of the radar and TOF sensors is determined based on the second monitoring range;

[0145] The arrangement position of the radar and TOF sensors in the second monitoring range is determined based on the historical image monitoring data;

[0146] After the radar and TOF sensors are arranged in the second monitoring range based on the arrangement number and the arrangement position, the state parameters of the radar and TOF sensors are corrected based on the historical image monitoring data.

[0147] Optionally, as a preferred embodiment, the radar sensing module and the TOF sensing module can also be an integrated product, i.e., integrated into a combined product, and the main installation method is to be directly installed on the ceiling.

[0148] Of course, when implementing the technical solutions of the present application, the radar sensing module and the TOF sensing module can also be separate module units, which can be installed separately or combined according to actual needs. The present application does not make specific limitations on this.

[0149] Although not shown in the drawings, preferably, more product embodiments can also be an electronic device, which includes a memory and one or more processors. The memory stores one or more application programs, and the one or more application programs are adapted to be executed by the one or more processors to implement the aforementioned radar combined TOF fall monitoring method.

[0150] Although not shown in the drawings, more embodiments also include a computer readable storage medium storing a computer program, when the computer program is executed, the aforementioned radar fall detection method steps are implemented.

[0151] It can be understood that the system, product, device, medium embodiments and the method embodiments correspond to each other, and can be mutually referred to, and the principles are similar or the same, and therefore will not be repeated.

[0152] Other technologies, principles, algorithms or models not expanded in detail in the present application can refer to prior art.

[0153] In the traditional fall detection scheme, part of the radar sensor has high technical complexity and strict manufacturing process requirements, resulting in high cost, which limits its large-scale popularization and application to some extent. The present application selects a 24GHz low-cost radar, which has a cost of only one tenth to one twentieth of the commonly used 60GHz radar. This cost advantage greatly reduces the hardware procurement cost when deploying a large-scale fall detection system. For example, in nursing homes, communities and other scenes that need to widely cover fall detection devices, using the low-cost radar of the present application, more areas can be monitored under the condition of limited budget, so that more elderly people in need can benefit. At the same time, for family users, the lower cost of the device also reduces the economic threshold for them to install fall detection devices for the elderly, so that more families can afford it, improving the popularization of fall detection devices.

[0154] In the technical scheme of the present application, multi-region TOF sensors are used to widen the detection range and improve the detection accuracy. The TOF sensor obtains the distance information of the target object by measuring the flight time of the light pulse, and has high measurement accuracy, especially in short distance measurement. In the fall detection scene, different regions of the TOF sensor can monitor the human body from multiple angles at the same time. When the human body falls, the distance between the body parts and the sensor will change rapidly, and the multi-region TOF sensor can capture these subtle distance changes in time and transmit the data to the system for analysis. This multi-angle, all-around monitoring method greatly reduces the detection blind area caused by complex body posture changes or occlusion, making the system's judgment of the fall action more accurate.

[0155] In the scheme of the present application, the multi-region TOF sensor cooperates with the radar to make up for the shortcomings of pure radar detection. The TOF sensor can provide more detailed information of human body motion in close range, combined with the long-distance and large-area monitoring capability of the radar, forming a more comprehensive and reliable monitoring system. In a multi-person scene, the TOF sensor can assist the radar in distinguishing different individuals through accurate identification of close-range human targets, reducing misjudgment caused by signal interference; for daily actions and fall actions that are easy to confuse, through comprehensive analysis of the distance change details obtained by the TOF sensor and the overall motion trend monitored by the radar, it can more accurately judge whether a fall event has occurred, greatly improving the reliability of the scheme in complex scenes.

[0156] In addition, the historical image monitoring data collected by the traditional living room installed visual intelligent image sensor contains rich human posture information, such as the walking height value, walking step length, walking speed, bending angle / bending- standing average duration value in the walking posture data, and standing height, standing average static duration, sitting height, sitting average duration, sitting- standing average switching duration, etc. in the static posture data. The technical scheme of the present application corrects the state parameters of the radar and the TOF sensor using these data, which can make the sensor better adapt to the individual characteristics of the target person. For example, through long-term statistical analysis of the historical image monitoring data of a certain individual, it is found that the walking height value of the individual is 1.75 meters on average when walking normally. If the height data detected by the radar and the TOF sensor deviates from this value for a long time, the detection parameters of the sensor can be adjusted through correction to make it more consistent with the actual situation of the individual, thereby improving the accuracy of fall detection of the individual. This parameter correction based on individual characteristics effectively avoids misjudgment and missed judgment caused by the mismatch between the general parameter setting of the sensor and the actual situation of the individual, and significantly improves the accuracy of detection.

[0157] More importantly, the present scheme uses radar and TOF sensor for fall detection, which is fundamentally different from the visual image monitoring scheme based on camera. The radar detects the position and motion state of the target object by emitting and receiving radio waves, and the TOF sensor measures the distance by using light pulses, neither of which involves the collection of user image information. In some places with extremely high privacy requirements, such as bedrooms and bathrooms in the home, this non-visual monitoring technology can effectively avoid the risk of privacy leakage caused by image collection, making users feel more secure when using fall detection equipment.

[0158] Even in the process of state parameter correction of the sensor by using historical image monitoring data to realize fall detection based on the individual characteristics of the target person, the scheme focuses on the protection of user privacy. The historical image monitoring data used only extracts key information related to human posture during processing, such as the position coordinate values (relative height values) of key human skeleton points (such as shoulders and hips), and does not involve the retention and use of sensitive information such as user facial features and personal identification. This fine processing method of data not only ensures the accuracy of fall detection, but also maximally protects the privacy of users, making the scheme more feasible and acceptable in actual application.

[0159] The foregoing has shown and described the method embodiments and systems of the present application, but it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A fall monitoring method using radar combined with TOF, characterized by, The method comprises: determining a target monitoring range and a target monitoring person; acquiring historical image monitoring data of the target monitoring person; determining a placement scheme of a radar and a TOF sensor based on the target monitoring range and the historical image monitoring data; judging whether the target monitoring person falls based on radar monitoring data and TOF sensing data; when it is judged that the target person has a falling phenomenon and has not recovered to a normal state within a preset time period, sending a warning signal to a related person of the target monitoring person; the judgment of whether the target monitoring person falls based on the radar monitoring data and the TOF sensing data specifically comprises: judging whether there is a target monitoring person in the monitoring range based on first radar monitoring data, and if so, judging the height of the target person based on at least one TOF sensing data; when the height of the target person is lower than a preset threshold, judging whether the target person is likely to fall based on multiple TOF data; if so, acquiring second radar monitoring data, monitoring the motion amplitude of the target person based on the second radar monitoring data, and judging whether the target monitoring person falls based on the motion amplitude.

2. The radar combined TOF fall monitoring method of claim 1, wherein the target monitoring range comprises a first monitoring range and multiple second monitoring ranges other than the first monitoring range; the acquisition of the historical image monitoring data of the target monitoring person comprises: acquiring historical image monitoring data of the target monitoring person in the first monitoring range based on an image sensor arranged before the execution of the fall monitoring method, the historical image monitoring data comprising walking posture data and static posture data of the target monitoring person.

3. The radar combined TOF fall monitoring method of claim 1, wherein the determination of the placement scheme of the radar and the TOF sensor based on the target monitoring range and the historical image monitoring data specifically comprises: determining the number of placements of the radar and the TOF sensor based on the target monitoring range; determining the placement positions of the radar and the TOF sensor based on the historical image monitoring data; after the radar and the TOF sensor are arranged in the target monitoring range based on the number of placements and the placement positions, correcting the state parameters of the radar and the TOF sensor based on the historical image monitoring data.

4. The fall monitoring method using radar combined with TOF as claimed in claim 1, wherein, the target monitoring range comprises a first monitoring range and multiple second monitoring ranges other than the first monitoring range; when the target person enters the second monitoring range, judging whether the target monitoring person falls based on radar monitoring data and TOF sensing data.

5. The radar combined TOF fall monitoring method of claim 1, wherein the judgment of whether the target monitoring person falls based on the radar monitoring data and the TOF sensing data specifically comprises: judging whether there is only a target monitoring person in the monitoring range based on first radar monitoring data, and if so, judging the height of the target person based on at least one TOF sensing data.

6. A fall monitoring system combining radar and TOF, the system comprising a historical image monitoring data acquisition unit, a radar sensing module, a TOF sensing module, a fall detection unit and an alarm unit; characterized in that: after determining a target monitoring range and a target monitoring person, the historical image monitoring data acquisition unit acquires historical image monitoring data of the target monitoring person; a layout scheme of the radar sensing module and the TOF sensing module is determined based on the target monitoring range and the historical image monitoring data; the fall detection unit determines whether the target monitoring person falls based on radar monitoring data and TOF sensing data; when the fall detection unit determines that the target person has a falling phenomenon and has not recovered to a normal state within a preset time period, the alarm unit sends an alarm signal to a related person of the target monitoring person; wherein the target monitoring range comprises a first monitoring range and a plurality of second monitoring ranges other than the first monitoring range; an image sensor is installed in the first monitoring range; the fall detection unit determines whether the target monitoring person falls based on radar monitoring data and TOF sensing data, specifically comprising: determining whether there is a target monitoring person in the monitoring range based on first radar monitoring data, and if so, determining the height of the target person based on at least one TOF sensing data; when the height of the target person is lower than a preset threshold, determining whether the target person is likely to fall based on a plurality of TOF data; if so, acquiring second radar monitoring data, monitoring the motion amplitude of the target person based on the second radar monitoring data, and determining whether the target monitoring person falls based on the motion amplitude.

7. The fall monitoring system combining radar and TOF according to claim 6, characterized in that: the layout scheme of the radar sensing module and the TOF sensing module is determined based on the target monitoring range and the historical image monitoring data, specifically comprising: determining the number of arrangements of radar and TOF sensors based on the second monitoring range; determining the arrangement positions of radar and TOF sensors in the second monitoring range based on the historical image monitoring data; after arranging the radar and TOF sensors in the second monitoring range based on the number of arrangements and the arrangement positions, correcting the state parameters of the radar and TOF sensors based on the historical image monitoring data.

8. The fall monitoring system combining radar and TOF according to claim 7, characterized in that: the image sensor is not installed in the second monitoring range.

9. A mobile terminal comprising a man-machine interaction unit, characterized in that the human-computer interaction unit is used to receive the alarm signal issued by the fall monitoring method combining radar and TOF according to any one of claims 1-5, and the alarm signal is used to prompt the target person to fall in the target monitoring range.

Images