Fall detection method, apparatus, device, and medium

Abstract

This invention discloses a fall detection method, apparatus, device, and medium. The method includes: determining target noise point detection information corresponding to the target radar detection information of a target person based on personnel radar detection information detected by millimeter-wave radar and noise point detection information detected by an acoustic detection device; determining the detection mode of the target person based on the target radar detection information, and determining target detection information from the target radar detection information and target noise point detection information based on the detection mode; wherein the detection mode includes an obstruction mode, a stationary mode, and a motion mode; and detecting the fall state of the target person based on the target detection information. This invention uses a combination of millimeter-wave radar and acoustic detection for fall detection, enabling continuous fall state detection even when the target person is obstructed by objects indoors, improving the accuracy and efficiency of fall detection; and since both millimeter waves and acoustic waves are invisible waves, user privacy can be effectively protected.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor detection technology, and in particular to a fall detection method, apparatus, device, and medium. Background Technology

[0002] Fall detection is a technology that analyzes human movement patterns to determine whether a fall has occurred. It is primarily used for monitoring vulnerable groups such as the elderly and patients to prevent accidents.

[0003] Currently, common fall detection methods mainly include video-based methods and wearable device-based methods. Millimeter-wave radar-based fall detection methods also exist. However, in indoor environments, obstructions from walls, furniture, and other objects can affect the accuracy of detecting falls using millimeter-wave radar-based or video-based methods. Furthermore, video-based methods raise privacy concerns, and wearable device-based methods require the user to wear the device, presenting certain limitations. Summary of the Invention

[0004] This invention provides a fall detection method, apparatus, device, and medium to address the issues of low fall detection accuracy in indoor environments with obstructions and the protection of personal privacy during fall detection.

[0005] According to one aspect of the present invention, a fall detection method is provided, wherein a sensor fusion device is installed in a target detection area located indoors, the sensor fusion device including a millimeter-wave radar and an acoustic wave detection device, the acoustic wave detection device including an acoustic wave transmitting module and an acoustic wave receiving module, the method comprising:

[0006] Based on the personnel radar detection information detected by millimeter-wave radar and the noise point detection information detected by the acoustic detection device, the target noise point detection information corresponding to the target personnel's target radar detection information is determined; wherein, the personnel radar detection information includes candidate radar detection information of at least one candidate personnel; the noise point detection information includes candidate noise point detection information of at least one candidate target;

[0007] The detection mode of the target personnel is determined based on the target radar detection information, and the target detection information is determined from the target radar detection information and the target noise point detection information based on the detection mode; wherein, the detection mode includes occlusion mode, stationary mode and moving mode;

[0008] The fall status of the target person is detected based on the target detection information.

[0009] According to another aspect of the present invention, a fall detection device is provided, wherein a sensor fusion device is installed in a target detection area located indoors, the sensor fusion device including a millimeter-wave radar and an acoustic wave detection device, the acoustic wave detection device including an acoustic wave transmitting module and an acoustic wave receiving module, the device comprising:

[0010] A multi-sensor matching module is used to determine target noise point detection information corresponding to the target radar detection information of the target personnel based on personnel radar detection information detected by millimeter-wave radar and noise point detection information detected by acoustic detection device; wherein, the personnel radar detection information includes candidate radar detection information of at least one candidate personnel; and the noise point detection information includes candidate noise point detection information of at least one candidate target.

[0011] The target detection information determination module is used to determine the detection mode of the target personnel based on the target radar detection information, and to determine the target detection information from the target radar detection information and the target noise point detection information based on the detection mode; wherein, the detection mode includes occlusion mode, stationary mode and moving mode;

[0012] The fall detection module is used to detect the fall status of the target person based on the target detection information.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the fall detection method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the fall detection method according to any embodiment of the present invention.

[0018] The technical solution of this invention combines millimeter-wave radar and acoustic detection for fall detection. It fuses millimeter-wave radar and acoustic detection according to the detection pattern of personnel movement, so that the fall status can be continuously detected even when the target person is blocked by objects indoors. Furthermore, since both millimeter waves and acoustic waves are invisible waves, user privacy can be effectively protected, and the accuracy and efficiency of fall detection can be improved without increasing the number of hardware devices or increasing computational complexity.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a fall detection method provided according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart of another fall detection method provided by an embodiment of the present invention;

[0023] Figure 3 This is a flowchart of another fall detection method provided by an embodiment of the present invention;

[0024] Figure 4 This is a flowchart of another fall detection method provided by an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of a fall detection device according to an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the fall detection method of this invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "candidate," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Figure 1 The present invention provides a flowchart of a fall detection method. This embodiment is applicable to the detection of falls in indoor environments. A sensor fusion device is installed in the target detection area located indoors. The sensor fusion device includes a millimeter-wave radar and an acoustic detection device. The method can be executed by a fall detection device, which can be implemented in hardware and / or software. The fall detection device can be configured in a server with computing power.

[0030] The size of the target detection area is determined based on the detection area of ​​the millimeter-wave radar. Therefore, in order to maximize the target detection area, the recommended installation angle of the millimeter-wave radar is determined.

[0031] Specifically, millimeter-wave radar can be installed in three ways: top-mounted, wall-mounted, and corner-mounted. Top-mounted refers to installation on the ceiling of an indoor space, wall-mounted refers to installation on the four walls of an indoor space, and corner-mounted refers to installation at the corner between two walls of an indoor space. The recommended installation angle for top-mounted radar is 90 degrees, while the recommended installation angle for wall-mounted and corner-mounted radar is 20 degrees. The configurable installation height for wall-mounted and corner-mounted radar is 2-3 meters, while the configurable installation height for top-mounted radar is 2-4.5 meters.

[0032] The size of the target detection region is determined according to the following formula:

[0033] If it is a top-mounted installation, the target detection area can be represented by a circle:

[0034]

[0035] Where (x, y) are the coordinates of the point at the edge of the target detection area, h represents the installation height of the millimeter-wave radar, h1 represents the reference height value, which is determined based on the experience value of the detection area exploration during scene layout, for example, set to 1.1, and a and b are determined based on the azimuth and elevation angles of the millimeter-wave radar, where a is half of the azimuth angle and b is half of the elevation angle.

[0036] If the installation is wall-mounted or corner-mounted, the target detection area can be represented by a sphere: x 2 +y 2 +(h-h1) 2 =h2 2 ;

[0037] Where (x, y) are the coordinates of the point on the edge of the target detection area, h represents the installation height of the millimeter-wave radar, h1 represents the reference height value, which is determined based on the experience value of the detection area exploration during scene layout, for example, set to 1.1, and h2 is determined based on the experience value of the detection area exploration during scene layout, for example, set to 11.

[0038] like Figure 1 As shown, the method includes:

[0039] S110. Based on the personnel radar detection information detected by the millimeter-wave radar and the noise point detection information detected by the acoustic detection device, determine the target noise point detection information corresponding to the target personnel's target radar detection information.

[0040] Millimeter-wave radar refers to radar that operates in the millimeter-wave band. It detects target information within a target detection area by emitting millimeter waves and identifies the targets. Millimeter-wave radar can also acquire radar detection information of personnel corresponding to the detection area within the target detection area. Acoustic wave detection devices, on the other hand, determine candidate noise points within a target detection area by emitting sound waves and receiving sound wave echoes. Because there may be continuously generating noise objects within the target detection area, such as refrigerator noise or outdoor noise, the acoustic wave information acquired by acoustic wave detection devices cannot determine whether the candidate noise points correspond to personnel or non-personnel.

[0041] The personnel radar detection information includes candidate radar detection information for at least one candidate person; the noise point detection information includes candidate noise point detection information for at least one candidate target. The candidate radar detection information included in the personnel radar detection information corresponds to the detected personnel. If there is one detected person within the target detection area, the personnel radar detection information includes candidate radar detection information for one candidate person, and this candidate person is the target personnel. If there are at least two detected personnel within the target detection area, the personnel radar detection information includes candidate radar detection information for at least two candidate personnel, and each candidate person is identified as the target personnel. The candidate targets corresponding to the candidate noise point detection information in the noise point detection information can be either candidate personnel or non-candidate personnel, i.e., other objects; therefore, matching detection information collected by different sensors is required.

[0042] Specifically, the radar attribute information of each target person is determined based on the personnel radar detection information detected by millimeter-wave radar, such as radar altitude information; and the corresponding acoustic attribute information, such as acoustic height information, is determined based on the noise point detection information detected by the acoustic detection device. The target noise point detection information corresponding to the target person's radar detection information is then determined based on the matching relationship between the radar attribute information and the acoustic attribute information. For example, the personnel radar detection information includes the radar detection information of one target person; the radar altitude information of the target person is determined based on this radar detection information; the acoustic height information of each candidate target is determined based on the candidate noise point detection information of each candidate target; and the candidate target whose acoustic height information is closest to the radar altitude information is identified as the target person.

[0043] Optionally, in order to improve the accuracy of noise detection information of personnel through sound wave detection device, continuous sound sources on the person's body, such as the upper body, including breathing sounds and heartbeat sounds, are detected, and non-continuous noise sources in the target detection area are screened to reduce the impact of sudden transient noise on fall detection.

[0044] S120. Determine the detection mode of the target personnel based on the target radar detection information, and determine the target detection information from the target radar detection information and the target noise point detection information based on the detection mode.

[0045] The detection mode is used to determine the detection criteria for a target person's fall state. The detection mode is determined based on whether the target person is in motion within the target detection area and whether they can be simultaneously detected by the millimeter-wave radar and acoustic detection device in the sensor fusion device. Specifically, the detection modes include an occlusion mode, a stationary mode, and a motion mode. The occlusion mode represents the occlusion status of the target person. For example, if the target person is obstructed by an obstacle in the target detection area, resulting in the loss of all or part of the target radar detection information, then the target person is in an occlusion mode. The motion mode and stationary mode represent the movement speed of the target person. For example, if the target person's movement speed is greater than a preset speed threshold, then the target person is considered to be in a motion mode; otherwise, the target person is considered to be in a stationary mode. The preset speed threshold can be set to a small value, and the specific value is not limited in this embodiment of the invention.

[0046] Specifically, based on the target radar detection information, it is determined whether the trajectory position of the target person has changed. If it has not changed, the target person is determined to be in stationary mode; if it has changed, the target person is determined to be in motion mode. Based on the missing target radar detection information, if at a certain moment the number of point clouds in the target radar detection information corresponding to the target person is less than the number of point clouds in the previous frame by a preset percentage, or if the target radar detection information corresponding to the target person is not detected at a certain moment, the target person is determined to be in occlusion mode.

[0047] Because millimeter-wave radar and acoustic detection devices have different characteristics—millimeter-wave radar has strong penetration and high resolution for detecting moving targets, while acoustic detection devices can still detect people obscured by obstacles—the advantages of different sensors are combined, and the detection information from different sensors is selected for fall detection based on the target person's detection mode. If the target person is in an obscured mode, the target radar detection information is inaccurate, and target noise point information needs to be combined as the basis for fall detection; that is, part of the target radar detection information and target noise point detection information are used as the target detection information. If the target person is in a moving mode and not in an obscured mode, the target radar detection information can accurately detect the target person's behavior, and the target radar detection information is used as the target detection information. If the target person is in a stationary mode, the target radar detection information has a certain error in the accuracy of detecting stationary objects; therefore, target noise point detection information is used as the target detection information.

[0048] For example, if the target person is in motion, the target radar detection information from the millimeter-wave radar is used as the primary target detection information, and the target noise point detection information from the acoustic detection device is used as the secondary target detection information; if the target person is stationary, the target noise point detection information from the acoustic detection device is used as the primary target detection information, and the target radar detection information from the millimeter-wave radar is used as the secondary target detection information. Fusion weights are set for the primary and secondary target detection information, and the primary and secondary target detection information are fused to obtain the final target detection information. The fusion weight of the primary target detection information is greater than the fusion weight of the secondary target detection information.

[0049] For example, after matching the detection information from millimeter-wave radar and acoustic detection devices, if the millimeter-wave radar cannot detect the target radar detection information of the target person, but the target noise point detection information that matches the target radar detection information still exists, it indicates that the target person is obstructed within the target detection area. In this case, the target noise point detection information is determined as the target detection information, and the fall status of the target person is directly detected based on the target noise point detection information. Since the target person is not obstructed when initially entering the target detection area, the target radar detection information of the target person can be obtained, and the corresponding target noise point detection information can be determined based on the target radar detection information. As the target person moves within the target detection area, the target person may be obstructed by other objects in the target detection area. In this case, the millimeter-wave radar cannot obtain the target radar detection information of the target person, but according to the pre-determined correspondence between radar and acoustic waves, the target noise point detection information can still be obtained through the acoustic detection device, and the fall status of the target person can continue to be determined through the target noise point detection information.

[0050] This embodiment supplements target information with multi-dimensional information, enriching the target information and enabling a more accurate determination of whether a fall event has occurred, thus improving the accuracy of fall detection.

[0051] S130. Detect the fall status of the target personnel based on the target detection information.

[0052] The height of the target person is determined based on target detection information obtained from sensors adapted to the current scene. The fall status of the target person is then determined based on changes in their height information. For example, if the target person's height decreases according to the target detection information, and the decrease exceeds a preset height threshold, then the target person is determined to be in a fall state.

[0053] For example, if there are at least two target personnel, the corresponding target detection information is determined based on the status information of each target personnel, and the fall status of each target personnel is judged separately based on their respective target detection information.

[0054] The technical solution of this invention combines millimeter-wave radar and acoustic detection for fall detection. It fuses millimeter-wave radar and acoustic detection according to the detection pattern of personnel movement, so that the fall status can be continuously detected even when the target person is blocked by objects indoors. Furthermore, since both millimeter waves and acoustic waves are invisible waves, user privacy can be effectively protected, and the accuracy and efficiency of fall detection can be improved without increasing the number of hardware devices or increasing computational complexity.

[0055] Figure 2This is a flowchart of a fall detection method provided by an embodiment of the present invention. This embodiment further refines the matching of sensor detection information in the above embodiments. Figure 2 As shown, the method includes:

[0056] S210. Determine the number of people in the target detection area based on personnel radar detection information.

[0057] Millimeter-wave radar has the characteristics of strong penetration and high resolution in detecting moving targets. Therefore, based on the detection information of millimeter-wave radar, it can be determined whether someone has entered the target detection area. If someone has entered, the radar detection information can be used to determine the personnel radar detection information and the number of personnel included in the personnel radar detection information, that is, to determine the number of candidate personnel included in the personnel radar detection information.

[0058] S220. Determine the radar tracking distance information of the target personnel based on personnel radar detection information, and determine the acoustic tracking distance information of all candidate targets based on noise point detection information.

[0059] The radar tracking distance information for a target personnel refers to the continuous tracking distance between the target personnel and the radar, determined based on radar detection information. For example, this can be determined based on the round-trip time (RTT) of the millimeter waves emitted by the radar. The acoustic tracking distance information for a candidate target refers to the continuous tracking distance between the candidate target and the acoustic detection device, determined based on candidate noise point detection information. For example, the acoustic detection device emits sound waves and measures the time from emission to reception of the reflected wave; this time is called the round-trip time (RTT). The acoustic distance information is calculated as: sound wave propagation speed × round-trip time / 2. The acoustic tracking distance information is obtained by continuously updating the noise point detection information and then determining the acoustic tracking distance information for each candidate target based on the updated noise point detection information.

[0060] Specifically, the millimeter-wave radar continuously updates the personnel radar detection information and determines the radar tracking distance information of the target personnel based on the updated personnel radar detection information. For example, if there is only one person, the personnel radar detection information is the target personnel radar detection information, and the radar tracking distance information of the target personnel can be determined directly based on the personnel radar detection information. If there are at least two people, each candidate person is identified as the target personnel, and the radar tracking distance information of the target personnel is determined sequentially based on the target personnel radar detection information in the personnel radar detection information. The acoustic detection device continuously updates the noise point detection information and determines the acoustic tracking distance information of each candidate target based on the updated noise point detection information.

[0061] In one feasible embodiment, prior to S220, the method further includes:

[0062] When the millimeter-wave radar detects no personnel radar detection information within the target detection area, determine the fixed noise information detected by the acoustic detection device;

[0063] The fixed noise information in the noise point detection information is removed to obtain the updated noise point detection information.

[0064] Because there is interference from fixed noise in the target detection area, such as the continuous noise emitted by household appliances, which may be mistakenly detected as candidate targets, in order to improve the accuracy of the noise point detection information of candidate targets obtained by the acoustic detection device and improve the matching efficiency between target personnel and candidate targets, the fixed noise in the noise point detection information is removed.

[0065] Specifically, millimeter-wave radar has the characteristics of strong penetration and high resolution in detecting moving targets. Based on the relatively accurate results of millimeter-wave radar in judging whether there are people or not in the target detection area, when the millimeter-wave radar determines that there are no people in the target detection area, the acoustic detection device collects the sound information of the target detection area when there are no people as fixed noise information for subsequent noise filtering.

[0066] S230. Based on the number of personnel, the radar tracking distance information of the target personnel and the acoustic tracking distance information of all candidate targets are tracked and matched to determine the acoustic tracking distance information of the target personnel that matches the radar tracking distance information of the target personnel.

[0067] Since both millimeter-wave radar and acoustic detection devices are integrated into the sensor fusion device, the radar tracking distance information and acoustic tracking distance information of the same target are the same or close. Based on this feature, the radar tracking distance information of the target personnel and the acoustic tracking distance information of all candidate targets are tracked and matched. If there is only one candidate target whose acoustic tracking distance information is the same as the radar tracking distance information of the target personnel, or the difference is less than the difference threshold, then the acoustic tracking distance information of that target is determined to be the target acoustic tracking distance information.

[0068] In one feasible embodiment, S230 includes:

[0069] Candidate targets whose acoustic tracking range information does not match the radar tracking range information of the target personnel are eliminated to obtain the remaining candidate targets;

[0070] If the number of remaining candidate targets is greater than 1, the remaining candidate targets will be continuously tracked to obtain the updated acoustic tracking distance information of the remaining candidate targets, as well as the updated radar tracking distance information of the target personnel at the corresponding time.

[0071] Candidate targets whose updated acoustic tracking range information does not match the updated radar tracking range information of the target personnel at the corresponding time are excluded until the number of remaining candidate targets is 1, which is then used as a reference target.

[0072] The acoustic wave detection device transmits acoustic wave signals to the location of the reference target and determines the reference acoustic distance information corresponding to the reference target and the reference radar distance information of the target personnel at the corresponding time.

[0073] If the reference acoustic range information and the reference radar range information match, then it is determined that the target acoustic tracking range information of the reference target matches the radar tracking range information of the target personnel.

[0074] Based on the number of personnel, each target personnel detected by the millimeter-wave radar is tracked and matched to determine the target acoustic tracking distance information that matches the radar tracking distance information of each target personnel.

[0075] Specifically, if the difference between the acoustic tracking distance information of the candidate target and the radar tracking distance information of the target personnel is greater than the difference threshold, it is determined that the acoustic tracking distance information of the candidate target and the radar tracking distance information of the target personnel do not match; if the difference between the acoustic tracking distance information of the candidate target and the radar tracking distance information of the target personnel is less than or equal to the difference threshold, it is determined that the acoustic tracking distance information of the candidate target and the radar tracking distance information of the target personnel match.

[0076] Specifically, if there is only one person, the radar tracking distance information of the target person is matched with the acoustic tracking distance information of all candidate targets. Candidate targets whose acoustic tracking distance information is different from the radar tracking distance information of the target person or whose difference is greater than the difference threshold are excluded. If the number of remaining candidate targets after exclusion is not 1, the system continues to wait for the target person to move and then matches the updated radar tracking distance information of the target person with the updated acoustic tracking distance information of the remaining candidate targets again until all redundant noise points in the candidate targets are excluded, leaving only the noise point detection information of one reference target.

[0077] After determining the noise point detection information of the reference target corresponding to the target personnel, the sound wave transmitting module in the sound wave detection device transmits a sound wave signal to the position corresponding to the noise point detection information of the reference target. The sound wave receiving module determines the reference sound wave distance information corresponding to the reference target and the reference radar distance information of the target personnel at the corresponding time. If the reference sound wave distance information and the reference radar distance information match, the target sound wave tracking distance information of the reference target matches the radar tracking distance information of the target personnel. If the reference sound wave distance information and the reference radar distance information do not match, the target sound wave tracking distance information of the reference target does not match the radar tracking distance information of the target personnel. The above steps are then repeated to re-determine the target sound wave tracking distance information that matches the radar tracking distance information of the target personnel.

[0078] After determining the reference target, the acoustic wave detection module actively emits acoustic waves to check whether the acoustic wave tracking distance information of the reference target matches the radar tracking distance. This improves the accuracy of reference target determination and, in turn, improves the accuracy of matching noise point detection information with radar detection information.

[0079] If there are at least two people, then distance tracking and matching will be performed on the two target personnel in the personnel radar detection information. The two target personnel can be matched simultaneously or sequentially, without restriction on the matching order. The distance tracking and matching method for each target personnel is the same as described above, and will not be repeated here.

[0080] The results of millimeter-wave radar and acoustic detection are fused: because the speaker and microphone in the acoustic detection device are integrated with the millimeter-wave radar, both can be considered to share the same origin, and both use polar coordinate systems. Target correlation can be effectively established through simple correction. By fusing the results of millimeter-wave radar and acoustic detection, the target's position and motion patterns can be determined from multiple dimensions, thereby improving detection accuracy and efficiency. Even when the target position detected by millimeter-wave radar is obscured, it can still be identified using the results of acoustic detection.

[0081] S240. Determine the correspondence between the target noise point detection information corresponding to the target acoustic wave tracking distance information and the target radar detection information of the target personnel.

[0082] If the target noise point corresponding to the target acoustic tracking distance information that matches the radar tracking distance information of the target personnel matches the target personnel, then the target noise point detection information of the target noise point corresponds to the target radar detection information of the target personnel. That is, the target noise point detection information of the target noise point is the acoustic detection information of the target personnel.

[0083] S250. Determine the detection mode of the target personnel based on the target radar detection information, and determine the target detection information from the target radar detection information and the target noise point detection information based on the detection mode.

[0084] The detection modes include occlusion mode, stationary mode, and motion mode.

[0085] S260. Detect the fall status of the target personnel based on the target detection information.

[0086] The technical solution of this invention improves the accuracy of determining the fall status of a target person based on the target noise point detection information by matching radar tracking distance information and acoustic tracking distance information.

[0087] Figure 3 This is a flowchart of a fall detection method provided by an embodiment of the present invention. This embodiment further refines the fall detection based on target detection information when the target person is in an occluded mode as described in the above embodiment. Figure 3 As shown, the method includes:

[0088] S310. Based on the personnel radar detection information detected by the millimeter-wave radar and the noise point detection information detected by the acoustic detection device, determine the target noise point detection information corresponding to the target personnel's target radar detection information.

[0089] S320. If the target radar detection information of the lost target personnel is determined, then the target personnel are determined to be in an obstruction mode, and the target radar detection information and target noise point detection information of the target before the loss of tracking are determined as the target detection information for a preset number of frames.

[0090] Specifically, the system determines whether target radar detection information for a target person has been lost based on the missing point cloud data in the target radar detection information. If, at a certain moment, the number of point clouds in the target radar detection information corresponding to a target person is less than a preset percentage compared to the previous frame, or if the target radar detection information corresponding to a target person is not detected at a certain moment, then the target person is determined to be in an occlusion mode.

[0091] For example, since the target detection area is located indoors, such as in a home, there will definitely be various objects in the home. The placement of these objects is limited by factors such as indoor space, size, and ease of use. This will cause the target person to be blocked by indoor objects or other obstacles when they are moving around indoors. The body parts of the target person that are blocked by obstacles will lose the corresponding target radar detection information.

[0092] If the target person is determined to be in an occlusion mode, then the target radar detection information and target noise point detection information prior to the tracking loss are used as the basis for judging the target person's fall state, i.e., as the target detection information. For example, the moment when the target radar detection information of the target person is missing or lost is called the loss moment. The target radar detection information of the frame before the loss moment and all target noise point detection information before and after the loss moment are used as the target detection information; or, to improve the accuracy of the target radar detection information, the target radar detection information of the two frames before the loss moment and all target noise point detection information before and after the loss moment are used as the target detection information. The preset number of frames corresponding to the target radar detection information can be determined according to the needs of the detection scenario and is not limited here.

[0093] S330: Determine the detection angle, position information, and radar distance information before obstruction of the target personnel based on the target radar detection information of the previous frame before tracking loss.

[0094] The target radar detection information includes the target personnel's detection angle, position, and distance information. The distance information from the target radar detection information of the frame before tracking was lost is used as the distance information of the target personnel before they were obscured.

[0095] S340. Determine the target person's height before obstruction based on the detection angle and radar distance information before obstruction, and determine the target person's fall status based on the height information.

[0096] Among them, the activity height information is used to represent the height of the target person before obstruction. For example, if the target radar detection information can represent the overall outline of the target person, then the activity height information is the height of the top of the target person's head from the ground before being obstructed.

[0097] Specifically, a straight line is drawn connecting the millimeter-wave radar's position and the highest point of the target person, forming the hypotenuse. A perpendicular line is then drawn from the millimeter-wave radar's position to the ground, forming the right-angled side. The highest point of the target person is then perpendicularly connected to this right-angled side, forming the other right-angled side. This, along with the hypotenuse and the two right-angled sides, forms a right triangle. The detection angle of the target person is the angle subtended by the right-angled side parallel to the ground within this triangle. The radar distance before obstruction is the length of the hypotenuse. Based on the relationship between the angles and sides of the right triangle, the length of the right-angled side perpendicular to the ground is determined according to the detection angle and the radar distance before obstruction. The installation height of the millimeter-wave radar is pre-determined, and the difference between the installation height and the length of the right-angled side is used to obtain the target person's activity height information.

[0098] Since the activity height information represents the highest point of the target person's overall silhouette, it is used to determine whether the target person was in a falling state before the occlusion. Specifically, if the activity height information is greater than a first height threshold, it is determined that the target person is not in a falling state; if it is less than or equal to the first height threshold, it is determined that the target person is in a falling state, triggering an alarm. The first height threshold is determined based on the target person's height, and in this embodiment of the invention, the specific value is not limited; for example, the first height threshold is set to one meter.

[0099] S350. If it is determined from the activity height information that the target person is not in a fall state, then the fall state of the target person shall be detected based on the target noise point detection information.

[0100] If the activity height information determines that the target person is in a fall state, an alarm is triggered directly; if the activity height information determines that the target person is in a normal state, the target noise point detection information after radar tracking loss is used to determine whether the target person is in a fall state. This ensures that when the millimeter-wave radar cannot perform normal fall detection on the target person in the obstruction mode, the target noise point detection information obtained by the acoustic detection device can still continue to detect the target person's fall, thus improving the accuracy of fall detection.

[0101] In one feasible embodiment, before detecting the fall status of the target person based on the target noise point detection information, the method further includes:

[0102] Based on the location information, determine the target noise point detection information corresponding to the target radar detection information of the previous frame that was lost, and determine the acoustic distance information of the target personnel before obstruction based on the target noise point detection information of the previous frame; determine the fall status of the target personnel based on the sensor detection difference between the radar distance information before obstruction and the acoustic distance information before obstruction.

[0103] Based on the correspondence between the target noise point detection information and the target radar detection information of the target personnel, the target noise point detection information corresponding to the target radar detection information of the previous frame before tracking loss is determined as the target noise point detection information of the previous frame before loss, and the acoustic distance information of the target personnel before obstruction is determined from the noise point detection information.

[0104] Under normal circumstances, since the radar distance information and the acoustic distance information before the obstruction are matched, the difference between the radar distance information and the acoustic distance information before the obstruction should be within a certain range. If it exceeds a certain range, it is considered that an abnormality has occurred in the multi-sensor detection and matching. In order to ensure the safety of the target personnel, this abnormality is considered that the target personnel are in a fall state, and an alarm is triggered.

[0105] Specifically, the absolute value of the difference between the radar distance information and the acoustic distance information before the obstruction is used as the sensor detection gap. If the sensor detection gap is greater than the preset gap value, it is determined that the target person is in a fallen state; otherwise, it is determined that the target person is in a normal state.

[0106] Optionally, if it is determined that the target person is not in a fall state based on both the activity height information and the sensor detection gap, then the fall state of the target person can be detected based on the target noise point detection information.

[0107] In one feasible embodiment, detecting the fall status of a target person based on target noise point detection information includes:

[0108] The initial lost acoustic wave distance is determined based on the target noise point detection information after tracking loss, and the updated lost acoustic wave distance is determined based on the subsequent updated target noise point detection information.

[0109] If the difference between the updated lost sound wave distance of a consecutive first preset number of frames and the initial lost sound wave distance is greater than a first preset difference threshold, then the target person is determined to be in a fallen state; or if the difference between the updated lost sound wave distance of a consecutive second preset number of frames and the updated lost sound wave distance of the corresponding previous frame is greater than a second preset difference threshold, then the target person is determined to be in a fallen state.

[0110] Specifically, the distance information obtained from the target noise point detection information of the frame after radar tracking loss is determined as the initial lost acoustic distance D1, and the distance information obtained from the subsequent updated target noise point detection information is determined as the updated lost distance D2. For example, after obtaining the target noise point detection information of the second frame after radar tracking loss, the distance obtained from the target noise point detection information of that frame is D2, and after obtaining the target noise point detection information of the third frame after radar tracking loss, the distance obtained from the target noise point detection information of that frame is the updated D2.

[0111] If the difference between the updated lost acoustic distance and the initial lost acoustic distance in multiple consecutive frames is greater than a first preset difference threshold, then the target person is determined to be in a fallen state. For example, starting from the target noise point detection information in the third frame after radar tracking loss, and continuing to the target noise point detection information in the sixth frame after radar tracking loss, if the difference between the updated lost acoustic distance and the initial lost acoustic distance in each frame is greater than the first preset difference threshold, then the target person is determined to be in a fallen state. The first preset number of frames is four frames, and the first preset difference threshold is set to 0.5m. This can be adjusted according to the actual scenario, and the specific value is not limited here.

[0112] Alternatively, if the difference between the updated lost acoustic distance in multiple consecutive frames and the updated lost acoustic distance in the corresponding previous frame is greater than a second preset difference threshold, then the target person is determined to be in a fallen state. For example, if the updated lost acoustic distance corresponding to the target noise point detection information in the third frame after radar tracking loss is greater than the updated lost acoustic distance corresponding to the second frame after radar tracking loss, and the updated lost acoustic distance corresponding to the target noise point detection information in the fourth frame after radar tracking loss is greater than the updated lost acoustic distance corresponding to the third frame after radar tracking loss, and so on, if this condition is met for four consecutive frames starting from the third frame, then the target person is determined to be in a fallen state. Here, the second preset number of frames is four frames, and the second preset difference threshold is less than the first preset difference threshold, set to 0.1m. This can be adjusted according to the actual scenario, and the specific value is not limited here.

[0113] This embodiment focuses on scenarios of sudden and slow falls, and detects the fall status of the target person, thereby improving the accuracy of fall detection.

[0114] The solution of this invention determines the fall status of a target person by analyzing the target radar detection information and target noise point detection information before and after the target person is obscured by other objects indoors. This achieves the goal of continuously and accurately detecting the fall status even after the target person's radar detection information is lost due to obstruction, thereby improving the universality of fall detection.

[0115] Figure 4 This is a flowchart of a fall detection method provided by an embodiment of the present invention. This embodiment further refines the fall detection based on target detection information when the target person is in motion or stationary mode as described in the above embodiments. Figure 4 As shown, the method includes:

[0116] S410. Based on the personnel radar detection information detected by the millimeter-wave radar and the noise point detection information detected by the acoustic detection device, determine the target noise point detection information corresponding to the target personnel's target radar detection information.

[0117] S420. If the target personnel are determined to be stationary based on the target radar detection information, then the target noise point detection information is determined to be the target detection information; if the target personnel are determined to be moving based on the target radar detection information, then the target radar detection information is determined to be the target detection information.

[0118] To avoid interference between detection information from different sensors, when the target person is in motion, the radar detection information is used as the target detection information to avoid interference from other noise points. When the target person is stationary, the accuracy of the radar detection information decreases significantly due to the weakened radar reflection signal; therefore, the target noise point detection information is used as the target detection information in this case.

[0119] S430. Determine the height information of the target personnel based on the target detection information.

[0120] If the target detection information is target radar detection information, the overall outline information of the target personnel can be determined based on the target radar detection information, and then the height information of the target personnel can be determined based on the overall outline information; and as the target personnel move and act, the target radar detection information and the height information of the target personnel can be continuously updated.

[0121] If the target detection information is target noise point detection information, it is acquired through a sound wave detection device. For example, the sound wave detection device includes a sound transmitter and multiple microphones. The sound signals captured by the microphone array fluctuate over time. By analyzing the waveform changes of these signals, it can be determined whether a continuous noise point (e.g., breathing or heartbeat) exists. The height information of the noise point above the ground is determined based on its location. Since the target noise point detection information is the location of a continuously occurring sound, such as breathing, the location of the noise point is the nose of the target person. The height information of the target person determined based on the target noise point detection information is the height of the target person's nose above the ground. The target noise point detection information and the height information of the target person are continuously updated as they move and perform actions.

[0122] S440. If the duration for which the height information meets the fall threshold condition is greater than a preset time threshold, determine the movement displacement of the target person based on the target detection information.

[0123] The fall threshold condition is determined based on the specific content of the height information and is used to represent the height of the falling target. Specifically, if the height information is the current frame height of the target, the fall threshold condition is that the height information is less than a preset height threshold. This preset height threshold is determined based on the height of the target in the fall state and can be determined according to the actual scenario, such as based on the average height of active people in the target detection area. If the height information is the height change information of the target, the fall threshold condition is that the falling height information of the target is greater than a preset height change threshold. This preset height change threshold is determined based on the height of the target in the fall state and can be determined according to the actual scenario, such as based on the average height of active people in the target detection area.

[0124] Specifically, when the height information is the current frame height information of the target person, if the target detection information determines that the height information of the target person is less than the preset height threshold, and the duration of the height being less than the preset height threshold is greater than the preset time threshold, then the height of the target person is determined to meet the suspected fall state. The target person's movement displacement is then determined based on the target detection information, so as to further determine the fall state of the target person based on the movement position of the target person.

[0125] When the height information is the height change information of the target person, if the target detection information determines that the descent height change of the target person compared to the previous frame is greater than a preset height change threshold, and the duration of this descent height change is greater than a preset time threshold, then the target person's height is determined to meet the suspected fall condition. The target detection information is then used to determine the target person's movement displacement to further determine the fall condition based on the target person's movement position. For example, if the target person's height in the previous frame is 1.6m, the current frame's height is 0.9m, the corresponding descent height change in the current frame is 0.7m, and the preset height change threshold is 0.6m, then the current frame's height information meets the fall threshold condition. The next frame's height is then assessed. If, in the next preset number of frames, the target person's height and the descent height change of the current frame continuously meet the fall threshold condition, then the target person's height is determined to meet the suspected fall condition.

[0126] The determination that the height information of the target person meets the fall threshold condition is based on whether a suspected fall has occurred in the vertical direction. The target person's displacement refers to the horizontal distance moved after a suspected fall occurs in the vertical direction. Specifically, if the duration for which the height information meets the fall threshold condition is greater than a preset time threshold, the trajectory displacement of the target person starting from a preset trajectory height is determined as the target person's displacement. For example, the horizontal trajectory displacement of the target person starting from a trajectory height of 1.3 meters is determined as the target person's displacement. The value of the preset trajectory height can be determined based on the target person's normal activity height, and there is no restriction on the specific value here.

[0127] S450. If the displacement is greater than the preset displacement threshold, the target person is determined to be in a fall state.

[0128] Since the displacement represents the horizontal distance traveled by the target person, if the target person exhibits suspected fall behavior simultaneously in both the vertical and horizontal directions, it is determined that the target person is in a fall state, and an alarm is triggered to notify other personnel. The preset movement threshold value can be determined based on the actual scenario and is not limited here; for example, the preset movement threshold could be set to 0.5m.

[0129] In one feasible embodiment, prior to S440, the method further includes: determining the posture information of the target person based on the target radar detection information; and determining the corresponding fall threshold condition and the preset displacement threshold based on the posture information.

[0130] The target person's posture information includes standing and sitting postures. Since the target person may fall in different postures, the movement distance corresponding to the suspected fall behavior in the vertical and horizontal directions is different in different postures. Therefore, it is necessary to determine the corresponding fall threshold conditions and preset displacement thresholds based on the target person's posture information.

[0131] Specifically, if the target person's posture information indicates a standing posture, and the duration for which the height information meets the first fall threshold condition is greater than a preset time threshold, the target person's movement displacement is determined based on the target detection information; if the movement displacement is greater than the first preset displacement threshold, the target person is determined to be in a fall state. If the target person's posture information indicates a sitting posture, and the duration for which the height information meets the second fall threshold condition is greater than a preset time threshold, the target person's movement displacement is determined based on the target detection information; if the movement displacement is greater than the second preset displacement threshold, the target person is determined to be in a fall state. Here, the first fall threshold condition is greater than the second fall threshold condition, and the first preset displacement threshold is greater than the second preset displacement threshold. For example, if the target person's posture information is standing, and the duration of the height change information being greater than 0.9m is greater than a preset time threshold, the target person's movement displacement is determined based on the target detection information; if the movement displacement is greater than 0.6m, the target person is determined to be in a falling state. If the target person's posture information is sitting, and the duration of the height change information being greater than 0.6m is greater than a preset time threshold, the target person's movement displacement is determined based on the target detection information; if the movement displacement is greater than 0.5m, the target person is determined to be in a falling state.

[0132] Different fall assessments are made based on the different postures of the target personnel, which improves the accuracy of fall detection for the target personnel.

[0133] In one feasible embodiment, the method further includes:

[0134] If the target person is determined to be in a lying position based on the target detection information, then the sleep state of the target person is judged based on the target noise point detection information.

[0135] If the target person's posture information is sitting, and it is determined that the target person is in a fallen state, then it is determined that the target person is in a lying position.

[0136] If the person is in a lying position, the millimeter-wave radar detection operation is stopped, and the sound wave emitted by the acoustic detection device is controlled to calculate the angle and distance of the sound source in the lying area based on the time delay difference. By adjusting the phase difference between the array microphones, the sound is focused on the area where the sound source is located, i.e., the lying area. Based on the sound wave information obtained after focusing, frequency feature identification is performed to determine the person's sleep state.

[0137] Specifically, if the target person is in a sitting position and located in a predetermined lying area, and it is determined that the target person has fallen, then the target person is in a lying position. Therefore, in this case, the target person's sleep state is determined. If it is determined that the target person is in a sleeping state, a signal is sent to the millimeter-wave radar to stop the alarm.

[0138] Specifically, when a target person sits on the bed from a standing position, it triggers a suspected fall in the vertical direction while in a standing position, but the horizontal displacement does not meet the fall criteria. However, when the target person changes from sitting on the bed to lying on the bed, it triggers both the vertical and horizontal fall criteria while in a sitting position, thus determining that the target person is currently in a fall state. To prevent false alarms, the fall state while in a sitting position is further assessed to determine whether the target person is in a lying position.

[0139] Once in a lying position, the sound waves emitted by the acoustic detection device calculate the angle and distance of the sound source based on the time delay difference. By adjusting the phase difference between the array microphones, the sound can be focused on the lying area. This helps to accurately locate the sound in a multi-source environment (e.g., a room with other background noise sources). The acoustic detection device analyzes specific frequency characteristics generated by breathing, heartbeat, and other movements to determine whether the person is asleep and to monitor their sleep state. Alternatively, other devices can be used to determine whether the person is asleep, such as heart rate monitoring devices; however, there are no restrictions on the other devices used for monitoring sleep.

[0140] The technical solution of this invention improves the accuracy of fall detection by combining the height information and movement displacement of the target person to determine the fall status.

[0141] Figure 5 This is a schematic diagram of a fall detection device provided in an embodiment of the present invention. A sensor fusion device is installed in an indoor target detection area. The sensor fusion device includes a millimeter-wave radar and a sound wave detection device. The sound wave detection device includes a sound wave transmitting module and a sound wave receiving module, as shown below. Figure 5 As shown, the device includes:

[0142] The multi-sensor matching module 510 is used to determine target noise point detection information corresponding to the target radar detection information of the target personnel based on the personnel radar detection information detected by the millimeter-wave radar and the noise point detection information detected by the acoustic detection device; wherein, the personnel radar detection information includes candidate radar detection information of at least one candidate personnel; and the noise point detection information includes candidate noise point detection information of at least one candidate target.

[0143] The target detection information determination module 520 is used to determine the detection mode of the target personnel based on the target radar detection information, and to determine the target detection information from the target radar detection information and the target noise point detection information based on the detection mode; wherein, the detection mode includes an occlusion mode, a stationary mode, and a moving mode;

[0144] The fall detection module 530 is used to detect the fall status of the target person based on the target detection information.

[0145] The technical solution of this invention combines millimeter-wave radar and acoustic detection for fall detection. It fuses millimeter-wave radar and acoustic detection according to the detection pattern of personnel movement, so that the fall status can be continuously detected even when the target person is blocked by objects indoors. Furthermore, since both millimeter waves and acoustic waves are invisible waves, user privacy can be effectively protected, and the accuracy and efficiency of fall detection can be improved without increasing the number of hardware devices or increasing computational complexity.

[0146] Optional, multi-sensor matching module, including:

[0147] A personnel quantity determination unit is used to determine the number of personnel in the target detection area based on the personnel radar detection information;

[0148] The tracking distance determination unit is used to determine the radar tracking distance information of the target personnel based on the personnel radar detection information, and to determine the acoustic tracking distance information of all candidate targets based on the noise point detection information.

[0149] The distance matching unit is used to perform tracking and matching on the radar tracking distance information of the target personnel and the acoustic tracking distance information of all candidate targets based on the number of personnel, and to determine the target acoustic tracking distance information that matches the radar tracking distance information of the target personnel.

[0150] The detection information matching unit is used to determine whether the target noise point detection information corresponding to the target acoustic wave tracking distance information corresponds to the target radar detection information of the target personnel.

[0151] Optionally, the device also includes a fixed noise removal module, used specifically for: before determining the target noise point detection information corresponding to the target radar detection information of the target personnel, to:

[0152] If the millimeter-wave radar detects no personnel radar detection information within the target detection area, determine the fixed noise information detected by the acoustic detection device;

[0153] The fixed noise information in the noise point detection information is removed to obtain the updated noise point detection information.

[0154] Optional, distance matching unit, specifically used for:

[0155] Candidate targets whose acoustic tracking range information does not match the radar tracking range information of the target personnel are excluded to obtain the remaining candidate targets;

[0156] If the number of remaining candidate targets is greater than 1, the remaining candidate targets are continuously tracked to obtain the updated acoustic tracking distance information of the remaining candidate targets and the updated radar tracking distance information of the target personnel at the corresponding time.

[0157] Candidate targets whose updated acoustic tracking distance information does not match the updated radar tracking distance information of the target personnel at the corresponding time are excluded until the number of remaining candidate targets is 1, which is then used as a reference target.

[0158] The acoustic wave detection device transmits an acoustic wave signal to the location of the reference target and determines the reference acoustic distance information corresponding to the reference target and the reference radar distance information of the target personnel at the corresponding time.

[0159] If the reference acoustic distance information and the reference radar distance information match, then it is determined that the target acoustic tracking distance information of the reference target matches the radar tracking distance information of the target personnel.

[0160] Based on the number of personnel, each target personnel detected by the millimeter-wave radar is tracked and matched to determine the target acoustic tracking distance information that matches the radar tracking distance information of each target personnel.

[0161] Optionally, the target detection information determination module includes an occlusion mode determination unit, specifically used for:

[0162] If it is determined that the target radar detection information of the lost target person is lost, then it is determined that the target person is in an obstruction mode, and the target radar detection information of a preset number of frames before the loss of tracking and the target noise point detection information are determined as the target detection information;

[0163] Accordingly, the fall detection module includes:

[0164] The radar-related information determination unit is used to determine the detection angle, position information, and radar distance information before obstruction of the target personnel based on the target radar detection information of the previous frame before tracking loss.

[0165] The activity height fall detection unit is used to determine the activity height information of the target person before the obstruction based on the detection angle of the target person and the radar distance information before the obstruction, and to determine the fall state of the target person based on the activity height information;

[0166] The acoustic fall detection unit is used to detect the fall status of the target person based on the target noise point detection information if it is determined from the activity height information that the target person is not in a fall state.

[0167] Optional, acoustic fall detection unit, specifically used for:

[0168] The initial lost acoustic wave distance is determined based on the target noise point detection information after tracking loss, and the updated lost acoustic wave distance is determined based on the subsequent updated target noise point detection information.

[0169] If the difference between the updated lost sound wave distance in a first preset number of consecutive frames and the initial lost sound wave distance is greater than a first preset difference threshold, then the target person is determined to be in a fallen state; or if the difference between the updated lost sound wave distance in a second preset number of consecutive frames and the updated lost sound wave distance in the corresponding previous frame is greater than a second preset difference threshold, then the target person is determined to be in a fallen state.

[0170] Optionally, the target detection information determination module includes a motion and stationary mode determination unit, specifically used for:

[0171] If the target personnel are determined to be in a stationary mode based on the target radar detection information, then the target noise point detection information is determined to be the target detection information.

[0172] If the target personnel are determined to be in motion mode based on the target radar detection information, then the target radar detection information is identified as target detection information.

[0173] Correspondingly, the fall detection module is specifically used for:

[0174] The height information of the target person is determined based on the target detection information;

[0175] If the duration for which the height information satisfies the fall threshold condition is greater than a preset time threshold, the movement displacement of the target person is determined based on the target detection information.

[0176] If the displacement is greater than a preset displacement threshold, then the target person is determined to be in a fallen state.

[0177] Optionally, the device further includes a sleep state determination module, specifically used for:

[0178] If the target person is determined to be in a bedridden state based on the target detection information, then the sleep state of the target person is judged based on the target noise point detection information.

[0179] The fall detection device provided in this embodiment of the invention can execute the fall detection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0180] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations and do not violate public order and good morals.

[0181] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0182] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0183] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0184] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0185] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as fall detection methods.

[0186] In some embodiments, the fall detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the fall detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the fall detection method by any other suitable means (e.g., by means of firmware).

[0187] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific reference products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0189] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0191] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or computing systems that include switching components (e.g., application servers), or computing systems that include front-end components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such back-end, switching, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0192] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0193] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0194] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A fall detection method, characterized in that, A sensor fusion device is installed in an indoor target detection area. The sensor fusion device includes a millimeter-wave radar and a sound wave detection device. The sound wave detection device includes a sound wave transmitting module and a sound wave receiving module. The method includes: Based on the personnel radar detection information detected by millimeter-wave radar and the noise point detection information detected by the acoustic detection device, the target noise point detection information corresponding to the target personnel's target radar detection information is determined; wherein, the personnel radar detection information includes candidate radar detection information of at least one candidate personnel; the noise point detection information includes candidate noise point detection information of at least one candidate target; The detection mode of the target personnel is determined based on the target radar detection information, and the target detection information is determined from the target radar detection information and the target noise point detection information based on the detection mode; wherein, the detection mode includes occlusion mode, stationary mode and moving mode; The fall status of the target person is detected based on the target detection information.

2. The method according to claim 1, characterized in that, Based on the personnel radar detection information detected by millimeter-wave radar and the noise point detection information detected by the acoustic detection device, the target noise point detection information corresponding to the target personnel's target radar detection information is determined, including: The number of people in the target detection area is determined based on the personnel radar detection information; The radar tracking distance information of the target personnel is determined based on the personnel radar detection information, and the acoustic tracking distance information of all candidate targets is determined based on the noise point detection information. Based on the number of personnel, the radar tracking distance information of the target personnel and the acoustic tracking distance information of all candidate targets are tracked and matched respectively to determine the acoustic tracking distance information of the target personnel that matches the radar tracking distance information of the target personnel. The target noise point detection information corresponding to the target acoustic wave tracking distance information is determined to correspond with the target radar detection information of the target personnel.

3. The method according to claim 1 or 2, characterized in that, Before determining the target noise point detection information corresponding to the target radar detection information of the target personnel, the method further includes: If the millimeter-wave radar detects no personnel radar detection information within the target detection area, determine the fixed noise information detected by the acoustic detection device; The fixed noise information in the noise point detection information is removed to obtain the updated noise point detection information.

4. The method according to claim 2, characterized in that, Based on the number of personnel, the radar tracking range information of the target personnel and the acoustic tracking range information of all candidate targets are tracked and matched to determine the acoustic tracking range information of the target that matches the radar tracking range information of the target personnel, including: Candidate targets whose acoustic tracking distance information does not match the radar tracking distance information of the target personnel are excluded to obtain the remaining candidate targets; If the number of remaining candidate targets is greater than 1, the remaining candidate targets are continuously tracked to obtain the updated acoustic tracking distance information of the remaining candidate targets and the updated radar tracking distance information of the target personnel at the corresponding time. Candidate targets whose updated acoustic tracking distance information does not match the updated radar tracking distance information of the target personnel at the corresponding time are excluded until the number of remaining candidate targets is 1, which is then used as a reference target. The acoustic wave detection device transmits an acoustic wave signal to the location of the reference target and determines the reference acoustic distance information corresponding to the reference target and the reference radar distance information of the target personnel at the corresponding time. If the reference acoustic distance information and the reference radar distance information match, then it is determined that the target acoustic tracking distance information of the reference target matches the radar tracking distance information of the target personnel. Based on the number of personnel, each target personnel detected by the millimeter-wave radar is tracked and matched to determine the target acoustic tracking distance information that matches the radar tracking distance information of each target personnel.

5. The method according to claim 1, characterized in that, The detection mode of the target personnel is determined based on the target radar detection information, and the target detection information is determined from the target radar detection information and the target noise point detection information based on the detection mode, including: If it is determined that the target radar detection information of the lost target person is lost, then it is determined that the target person is in an obstruction mode, and the target radar detection information of a preset number of frames before the loss of tracking and the target noise point detection information are determined as the target detection information; Accordingly, the fall status of the target person is detected based on the target detection information, including: The detection angle, position information, and radar distance information before obstruction of the target personnel are determined based on the target radar detection information of the previous frame before tracking loss. The target person's height information before being blocked is determined based on the detection angle and the radar distance information before the blockage, and the target person's fall status is determined based on the height information. If it is determined based on the activity height information that the target person is not in a fall state, then the fall state of the target person is detected based on the target noise point detection information.

6. The method according to claim 5, characterized in that, Detecting the fall status of the target person based on the target noise point detection information includes: The initial lost acoustic wave distance is determined based on the target noise point detection information after tracking loss, and the updated lost acoustic wave distance is determined based on the subsequent updated target noise point detection information. If the difference between the updated lost sound wave distance in a first preset number of consecutive frames and the initial lost sound wave distance is greater than a first preset difference threshold, then the target person is determined to be in a fallen state; or if the difference between the updated lost sound wave distance in a second preset number of consecutive frames and the updated lost sound wave distance in the corresponding previous frame is greater than a second preset difference threshold, then the target person is determined to be in a fallen state.

7. The method according to claim 1, characterized in that, The detection mode of the target personnel is determined based on the target radar detection information, and the target detection information is determined from the target radar detection information and the target noise point detection information based on the detection mode, including: If the target personnel are determined to be in a stationary mode based on the target radar detection information, then the target noise point detection information is determined to be the target detection information. If the target personnel are determined to be in motion mode based on the target radar detection information, then the target radar detection information is identified as target detection information. Accordingly, the fall status of the target person is detected based on the target detection information, including: The height information of the target person is determined based on the target detection information; If the duration for which the height information satisfies the fall threshold condition is greater than a preset time threshold, the movement displacement of the target person is determined based on the target detection information. If the displacement is greater than a preset displacement threshold, then the target person is determined to be in a fallen state.

8. A fall detection device, characterized in that, A sensor fusion device is installed in an indoor target detection area. The sensor fusion device includes a millimeter-wave radar and a sound wave detection device. The sound wave detection device includes a sound wave transmitting module and a sound wave receiving module. The device comprises: A multi-sensor matching module is used to determine target noise point detection information corresponding to the target radar detection information of the target personnel based on personnel radar detection information detected by millimeter-wave radar and noise point detection information detected by acoustic detection device; wherein, the personnel radar detection information includes candidate radar detection information of at least one candidate personnel; and the noise point detection information includes candidate noise point detection information of at least one candidate target. The target detection information determination module is used to determine the detection mode of the target personnel based on the target radar detection information, and to determine the target detection information from the target radar detection information and the target noise point detection information based on the detection mode; wherein, the detection mode includes occlusion mode, stationary mode and moving mode; The fall detection module is used to detect the fall status of the target person based on the target detection information.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the fall detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the fall detection method according to any one of claims 1-7.

Images