A fall warning method and system based on deep learning

By combining millimeter-wave radar and environmental vision sensors in fall detection, the body parameters of the target person are entered and the corresponding model is activated, the comprehensive problem of indoor-outdoor detection in the existing technology is solved, and personalized fall detection and continuous evolution are achieved. Improve detection accuracy and adaptability.

CN118609215BActive Publication Date: 2024-11-12KUNSHAN FIRST PEOPLES HOSPITAL
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410757589.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-13
Publication Date
2024-11-12
Estimated Expiration
2044-06-13

AI Technical Summary

Technical Problem

The existing fall detection technology is usually aimed at indoor or outdoor environments alone, and fails to effectively consider the comprehensive situation of indoor-outdoor activities of the target person. The general detection scheme leads to high detection complexity and poor adaptability, and fails to target specific target persons. Make personalized settings.

Method used

Using deep learning-based fall warning method, by using millimeter wave radar and environmental vision sensor indoors, the body parameters of the target person are entered, the center of mass coordinates are obtained and the corresponding fall calculation model is activated; using environmental vision sensor outdoors, iterating through video streams The detection model is used for fall detection, and the warning signal and image frames are continuously updated as training sample sets to achieve personalized customization and environmental adaptation.

Benefits of technology

It improves the accuracy and adaptability of fall detection, reduces the training complexity of deep learning models, and realizes personalized fall detection and continuous evolution for designated target characters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118609215B_ABST
    Figure CN118609215B_ABST
Patent Text Reader

Abstract

The present invention proposes a fall warning method and system based on deep learning, which belongs to the technical field of fall detection and image recognition. The method includes inputting the body parameters of the monitored target person, obtaining the actual center of mass coordinates based on the body parameters, using the first radar fall estimation model to perform indoor fall phenomenon detection, and using the second video stream iterative detection model to perform outdoor fall phenomenon, and continuously updating the training sample set to update and train the second video stream iterative detection model. The system includes a first millimeter-wave radar set in an indoor environment, a second environmental vision sensor, and a third environmental vision sensor set in an outdoor environment. The present invention realizes the full-process fall detection of target persons with known features indoors and outdoors, reduces the complexity of deep learning model training, and can realize the adaptive training and update of deep learning image detection models, and improves the adaptability of the model to different environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of fall detection and image recognition, and in particular to a fall warning method and system based on deep learning, a computer-readable storage medium for implementing the method, a computer program product, and an electronic device. Background Art

[0002] Accidental falls are one of the important factors causing injuries to the elderly. In order to prevent safety risks from the source, it is necessary to detect abnormal behaviors of elderly people living alone in real time. At present, fall detection algorithms can be divided into two categories: fall detection algorithms based on non-computer vision and fall detection algorithms based on computer vision. In addition, existing technologies already use radar and cameras to capture human postures and use convolutional neural networks to determine whether a fall has occurred. These are usually fall detection methods based on deep learning and multi-sensor fusion.

[0003] Fall detection systems based on environmental sensors usually embed non-video sensors such as wireless sensor networks, infrared sensors, sound sensors, pressure sensors, Doppler radars, etc. in the human activity area to collect human motion and posture information, and then determine whether a fall has occurred. This type of method is usually suitable for fall detection of the elderly in indoor areas; when the elderly are outdoors (such as open courtyards, recuperation area squares, etc.), fall detection based on computer vision is a better alternative, which uses human skeleton sequence extraction and posture estimation methods to detect fall behavior.

[0004] However, the fall detection solutions in the prior art usually only target indoors or outdoors and adopt corresponding visual or non-visual solutions, without considering the comprehensive situation of the target person's indoor-outdoor activities; at the same time, the fall detection solutions given in the prior art are universal, that is, fall detection for all people, and each detection needs to be run from the beginning and all processes are executed for all people within the monitoring range, without considering the limited characteristics and personalized settings of the target person's own characteristics during actual home fall detection, resulting in high complexity and poor adaptability of the fall detection and warning process based on deep learning. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a fall warning method and system based on deep learning, a computer-readable storage medium, a computer program product and an electronic device for implementing the method.

[0006] In a first aspect of the present invention, a fall warning method based on deep learning is proposed, the method comprising the following steps:

[0007] S100: inputting the body parameters of the monitored target person, and obtaining the actual center of mass coordinates based on the body parameters;

[0008] S200: When the monitored target person is active in the indoor environment, first radar acquisition data is acquired through the first millimeter wave radar, and a second image monitoring frame is acquired through the second environment vision sensor;

[0009] S300: When the radar centroid coordinates of the monitored target person determined based on the first radar collected data match the actual centroid coordinates, starting a first radar fall estimation model;

[0010] S400: When the calculation result of the first radar fall calculation model is that an indoor fall phenomenon occurs, an indoor fall warning signal is issued, and a second image monitoring frame in a preset time period before and after the indoor fall phenomenon occurs is added to a training sample set;

[0011] S500: training a second video stream iterative detection model based on the training sample set;

[0012] S600: When the monitored target person is active in an outdoor environment, a third image monitoring frame is obtained through a third environmental visual sensor, and the video centroid coordinates of the monitored target person are determined based on the third image monitoring frame;

[0013] S700: When the video centroid coordinates match the actual centroid coordinates, starting a second video stream iteration detection model;

[0014] S800: When the detection result of the second video stream iterative detection model is that an outdoor fall occurs, an outdoor fall warning signal is issued.

[0015] When the detection result in step S800 is that an outdoor fall phenomenon occurs, the third image monitoring frame in a preset time period before and after the outdoor fall phenomenon occurs is added to the training sample set, and the process returns to step S500.

[0016] The step S300 determines the radar centroid coordinates of the monitored target person based on the first radar collected data, specifically including:

[0017] Based on the data collected by the first radar, a three-dimensional Cartesian rectangular coordinate system is established;

[0018] The reference centroid clustering algorithm is used to obtain the horizontal coordinates of the human body's centroid, and the centroid height is obtained based on the torso feature algorithm, thereby determining the radar centroid coordinates of the monitored target person.

[0019] The first radar fall estimation model estimates whether the monitored target person has fallen based on the determined radar center of mass coordinates and movement speed of the monitored target person in combination with the CRBENN fall estimation algorithm.

[0020] The second video stream iterative detection model includes YOLOv5 and Ka l man filtering algorithms;

[0021] The step S600 determines the video centroid coordinates of the monitored target person based on the third image monitoring frame, specifically including:

[0022] Perform skeleton point recognition on the video frame to obtain the coordinates of multiple skeleton key points;

[0023] Based on the coordinates of multiple skeleton key points, the coordinates of the video center of mass of the monitored target person are determined.

[0024] The deep learning-based fall warning method described in the first aspect can be automatically implemented through various forms of electronic devices through computer program instructions; the computer program instructions can be stored in storage media of different forms and loaded into computer electronic devices for execution.

[0025] Therefore, in the second aspect of the present invention, a computer-readable storage medium is also provided for storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes a fall warning method based on deep learning as described in the first aspect.

[0026] In the third aspect of the present invention, an electronic device is also proposed, comprising a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory so that the electronic device executes the aforementioned fall warning method based on deep learning.

[0027] In a fourth aspect of the present invention, a computer program product is also proposed, which includes a computer program. When the computer program is executed, the aforementioned fall warning method based on deep learning is implemented.

[0028] Corresponding to the method technical solution, in the fifth aspect of the present invention, a fall warning system based on deep learning is also proposed, which can be used to implement the fall warning method based on deep learning described in the first aspect.

[0029] In terms of system architecture, the system includes a first millimeter-wave radar arranged in an indoor environment, a second environmental vision sensor, and a third environmental vision sensor arranged in an outdoor environment.

[0030] In a more specific application, in order to reduce hardware cost and match the actual environment, the resolution of the third environment vision sensor is greater than the resolution of the second environment vision sensor.

[0031] The system also includes a human body parameter input unit, a warning unit, a first radar fall estimation model and a second video stream iterative detection model;

[0032] The human body parameter input unit inputs the body parameters of the monitored target person, obtains the actual center of mass coordinates based on the body parameters, and sends the actual center of mass coordinates to the first radar fall estimation model and the second video stream iterative detection model;

[0033] Preferably, the human body parameter input unit includes an image scanning unit, which scans the monitored target person to obtain the body parameters of the monitored target person, and the body parameters include height, body proportions, three-dimensional coordinates of bones, and a set of coordinates of key points of bones.

[0034] The first radar fall inference model obtains the first radar collection data through the first millimeter wave radar, determines the radar center of mass coordinates of the monitored target person, and infers whether the monitored target person falls based on the determined radar center of mass coordinates and movement speed of the monitored target person in combination with the CRBENN fall inference algorithm;

[0035] The second video stream iterative detection model obtains a third image monitoring frame through a third environmental visual sensor, performs skeleton point recognition on the third image monitoring frame, and obtains a plurality of skeleton key point coordinates; based on the plurality of skeleton key point coordinates, determines the video centroid coordinates of the monitored target person, and detects whether the monitored target person falls based on YOLOv5 and Ka l man filtering algorithms;

[0036] When the radar centroid coordinates match the actual centroid coordinates, the first radar fall estimation model is started, and when the video centroid coordinates match the actual centroid coordinates, the second video stream iterative detection model is started;

[0037] When the monitored target person falls, the warning unit sends out a warning signal.

[0038] Furthermore, the system also includes an update training module;

[0039] When the estimation result of the first radar fall estimation model is that an indoor fall occurs, an indoor fall warning signal is issued, and a second image monitoring frame in a preset time period before and after the indoor fall occurs is added to the training sample set;

[0040] The update training module trains a second video stream iterative detection model based on the training sample set.

[0041] Furthermore, when the detection result of the second video stream iterative detection model is that an outdoor fall phenomenon occurs, a third image monitoring frame in a preset time period before and after the outdoor fall phenomenon occurs is added to the training sample set.

[0042] Different from the fall detection models or algorithms proposed in the prior art, which are usually general or universal, the fall detection algorithm or model adopted in the present invention realizes personalized customization and startup to a certain extent, that is, the physical parameters of the monitored target person are pre-entered and sent to the fall detection algorithm or model in the corresponding environment, so that the fall detection algorithm or model can carry out detection on the target person of the specified type and reduce the information interference of irrelevant targets; at the same time, the image frames corresponding to the corresponding output results are used as updated training sample sets to continue training the corresponding fall detection algorithm or model, so that the subsequent fall detection process for the target person of the specified features or type continues to evolve, which further improves the detection accuracy while reducing the complexity of deep learning model training and improving the adaptability of the model to different environments.

[0043] Further advantages of the present invention will be further reflected in detail in the specific embodiments section in conjunction with the drawings of the specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 This is a schematic diagram of the main flow of a fall warning method based on deep learning according to an embodiment of the present invention.

[0046] Figure 2 Schematic diagram of data collection and centroid positioning for indoor fall detection using millimeter wave radar

[0047] Figure 3 This is a schematic diagram of the functional unit composition of a fall warning system based on deep learning according to an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of the scenario of indoor and outdoor fall detection implemented by the technical solution of this application

[0049] Figure 5 yes Figure 1 Schematic diagram of an embodiment of a computer flow chart of a fall warning method based on deep learning DETAILED DESCRIPTION

[0050] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments that are the same as the present application. Instead, they are only examples of devices and methods that are the same as some aspects of the present application as detailed in the attached claims.

[0051] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0052] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0053] Figure 1 It is a main flow chart of a fall warning method based on deep learning according to an embodiment of the present invention.

[0054] Figure 1 The method includes steps S100-S800, and each step is specifically performed as follows:

[0055] S100: inputting the body parameters of the monitored target person, and obtaining the actual center of mass coordinates based on the body parameters;

[0056] S200: When the monitored target person is active in the indoor environment, first radar acquisition data is acquired through the first millimeter wave radar, and a second image monitoring frame is acquired through the second environment vision sensor;

[0057] S300: When the radar centroid coordinates of the monitored target person determined based on the first radar collected data match the actual centroid coordinates, starting a first radar fall estimation model;

[0058] S400: When the calculation result of the first radar fall calculation model is that an indoor fall phenomenon occurs, an indoor fall warning signal is issued, and a second image monitoring frame in a preset time period before and after the indoor fall phenomenon occurs is added to a training sample set;

[0059] S500: training a second video stream iterative detection model based on the training sample set;

[0060] S600: When the monitored target person is active in an outdoor environment, a third image monitoring frame is obtained through a third environmental visual sensor, and the video centroid coordinates of the monitored target person are determined based on the third image monitoring frame;

[0061] S700: When the video centroid coordinates match the actual centroid coordinates, starting a second video stream iteration detection model;

[0062] S800: When the detection result of the second video stream iterative detection model is that an outdoor fall occurs, an outdoor fall warning signal is issued.

[0063] Next, Figure 1 The specific implementation process of each step of the embodiment is described in detail as follows:

[0064] S100: inputting the body parameters of the monitored target person, and obtaining the actual center of mass coordinates based on the body parameters;

[0065] In practical applications, the body parameters of the monitored target person are input in advance through the body parameter input unit, the actual center of mass coordinates are obtained based on the body parameters, and the actual center of mass coordinates are sent to the first radar fall estimation model and the second video stream iterative detection model;

[0066] Preferably, the human body parameter input unit includes an image scanning unit, which scans the monitored target person to obtain the body parameters of the monitored target person, and the body parameters include height, body proportions, three-dimensional coordinates of bones, and a set of coordinates of key points of bones.

[0067] This step involves the collection process of user data. However, it is understandable that, according to the subsequent technical solution introduction, the data collected by the present invention are all unrelated to personal identity information (such as height ratio and other data), and do not involve user identity identification; at the same time, before starting the relevant service, a prompt interface, pop-up window or output voice prompt information can be displayed, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that the present application only starts to execute the relevant steps of obtaining user-related data after obtaining the user's confirmation operation on the prompt interface or pop-up window, otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining user-related data are terminated, that is, the user's relevant data is not obtained, and the relevant services cannot be provided.

[0068] Therefore, in the specific implementation of this application, if user-related data is involved, when the embodiment of this application is applied to a specific product or technology, the user's permission or consent must be obtained, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0069] In an environment where elderly people live alone, there is usually only one monitored target person. The subsequent embodiments of this application are based on the example of one monitored target person.

[0070] However, it is understandable that there may be multiple monitored target persons. However, even if there are multiple, they are limited and determined multiple, which is different from the fall detection process of the prior art that is not targeted at a specific person. In other words, before the present application performs fall detection, certain deterministic features of the monitored target person are pre-determined, especially the center of mass coordinate features.

[0071] In an indoor environment, proceed to steps S200-S300:

[0072] S200: When the monitored target person is active in the indoor environment, first radar acquisition data is acquired through the first millimeter wave radar, and a second image monitoring frame is acquired through the second environment vision sensor;

[0073] S300: When the radar centroid coordinates of the monitored target person determined based on the first radar collected data match the actual centroid coordinates, starting a first radar fall estimation model.

[0074] It is understandable that even indoor fall detection is usually only for elderly people living alone. When other non-target people appear in the target range indoors, fall detection is not necessary.

[0075] Therefore, the first radar fall estimation model is activated only when the radar center of mass coordinates of the monitored target person determined based on the first radar acquisition data matches the actual center of mass coordinates.

[0076] During indoor millimeter-wave radar detection, millimeter waves reflected from objects such as walls, sofas, and bathroom cabinets can easily interfere with the sensor and cause errors in the system's detection. Therefore, it is necessary to use an environmental clutter filtering algorithm to remove the object echo signal.

[0077] Since the area of ​​walls, wardrobes and other objects is larger than other targets, the amount of millimeter-wave information reflected back is large, resulting in a large number of point clouds collected by the millimeter-wave radar, which is the main source of system interference. At this time, they are excluded by artificially calibrating the detection area, and the millimeter-wave echo signals outside the area can no longer be processed, thereby fundamentally eliminating their adverse effects on detection.

[0078] It is worth noting that after being processed by the regional isolation method, there are still other small and medium-sized interference sources in the detection area, which can be processed using the static clutter removal algorithm, that is, taking the average value of the millimeter-wave radar echo signal in the environment as an estimate of the environmental clutter, and removing this value from the original echo signal to achieve a filtering effect.

[0079] The first radar fall inference model adopted in the embodiment of the present application obtains the first radar collection data through the first millimeter-wave radar, determines the radar center of mass coordinates of the monitored target person, and infers whether the monitored target person falls based on the determined radar center of mass coordinates and movement speed of the monitored target person in combination with the CRBENN fall inference algorithm.

[0080] Next, combine Figure 2 The schematic diagram of data collection and centroid positioning when using millimeter wave radar for indoor fall detection is shown, and some principles of the above process are introduced.

[0081] In order to reflect the movement of the target person in the detection space, especially the change of height, the following Figure 2 The three-dimensional Cartesian rectangular coordinate system shown in FIG. 1 , in which the ground is selected as the XOY plane, and the millimeter-wave radar mapping point on the ground is the coordinate origin O (0, 0, 0).

[0082] Since the judgment of falling depends on the speed and position of the person, the state vector of the person's centroid at time t is selected as p t ∈R 6*1 ;

[0083] That is, p t =[p tp ,p tv ]=[x t ,y t ,z t ,v x (t),v y (t),v z (t)] T ;

[0084] Where: p t p ,p t v are the center of mass position vector and velocity vector respectively; x t ,y t ,z t are the position coordinates of the center of mass of the person in the xyz direction at time t; v x (t),v y (t),v z (t) are the velocities of the target person’s center of mass in the x, y, and z directions at time t.

[0085] Next, in Figure 3 In the three-dimensional Cartesian rectangular coordinate system shown, the horizontal plane coordinates of the human body's center of mass are obtained using the reference center of mass clustering algorithm, and the center of mass height is obtained based on the torso feature algorithm, thereby determining the center of mass position and movement speed of the target person.

[0086] The reference centroid clustering algorithm first determines the coordinate information of the target person’s centroid mapped to the ground, i.e., p tp The horizontal and vertical coordinates x t ,y t The signal matrix Q after the environmental clutter is filtered out is used t , process the point cloud of the target person. Map the point cloud of the target person to a two-dimensional plane, and use the improved clustering algorithm to determine the target person x t ,y t Information (z t >0 is always true).

[0087] In order to improve the detection efficiency and realize the centroid position identification, the DBSCAN algorithm is improved with reference to the centroid clustering algorithm. A round of cyclic classification is added to the original DBSCAN algorithm, and reference points in densely populated areas are introduced as centroids. At the same time, more noise point clouds can be removed to increase the accuracy of clustering.

[0088] After determining the plane coordinates of the center of mass of the human body, it is necessary to determine p t p Middle t The size of is the height of the target person's centroid. Studies have shown that the chest cavity has more point clouds than the limbs and head, which is the most important millimeter wave reflection part of the human body, which means that the centroid obtained after clustering is mostly located here. Therefore, the coordinate information in the z direction provided by the point cloud can be used to determine p t p Middle t Then, based on the person's position coordinate information, the velocity information of the target person's centroid in the xyz direction can be determined.

[0089] The first radar fall estimation model estimates whether the monitored target person has fallen based on the determined radar center of mass coordinates and movement speed of the monitored target person in combination with the CRBENN fall estimation algorithm.

[0090] Falling behavior is a process action and cannot be judged based on the character's state at a certain moment. Instead, it is necessary to extract the changes in the character's state over a period of time as the basis for judgment.

[0091] First, we need to determine the starting frame of the falling action. b At the beginning, when the height drop of the human body center Δz exceeds 0.5m, t bSet as the moment to start the fall inference algorithm, and then realize the identification of fall actions in simple and complex character actions respectively;

[0092] Secondly, determine the end frame of the falling action. When the character drops to the lowest point, let the current sampling time be t e , represents the end time of the falling action. b to e During this period, four sampling points are selected according to the following formula:

[0093] t i =floor[((t b +(i-1)*(t e -t b ) / 5×10] / 10

[0094] i=2,3,4,5

[0095] Where: floor() is the rounding down function; t i is the i-th sampling moment;

[0096] Take p in =[p tb ,p t2 ,p t3 ,p t4, p t5 ,p te ] is used as the input of CRBENN’s fall inference algorithm, and the algorithm determines whether a fall has occurred.

[0097] Considering the severity of the fall, if the height of the person at a certain moment after the fall is z t Greater than the starting height z t 1 If the value is 0.8 times of the falling pressure, the fall warning will be cancelled.

[0098] Next, enter step S400: when the calculation result of the first radar fall calculation model is that an indoor fall occurs, an indoor fall warning signal is issued, and the second image monitoring frame in a preset time period before and after the indoor fall occurs is added to the training sample set.

[0099] Preferably, the preset time period before and after the fall phenomenon can be [t b ,t e ].

[0100] Through the above steps, the indoor fall detection can be fully realized, and at the same time, a training sample set that is actually fully labeled can be obtained to transfer learning to the video stream iterative model of outdoor fall detection.

[0101] Therefore, the next step is to enter the outdoor detection model training and execution detection stage of steps S500-S800.

[0102] S500: training a second video stream iterative detection model based on the training sample set;

[0103] S600: When the monitored target person is active in an outdoor environment, a third image monitoring frame is obtained through a third environmental visual sensor, and the video centroid coordinates of the monitored target person are determined based on the third image monitoring frame;

[0104] S700: When the video centroid coordinates match the actual centroid coordinates, starting a second video stream iteration detection model;

[0105] S800: When the detection result of the second video stream iterative detection model is that an outdoor fall occurs, an outdoor fall warning signal is issued.

[0106] The improvement of this embodiment over the prior art lies in step S600, that is, starting the second video stream iterative detection model to first determine the video centroid coordinates of the monitored target person based on the third image monitoring frame.

[0107] The step S600 determines the video centroid coordinates of the monitored target person based on the third image monitoring frame, specifically including:

[0108] Perform skeleton point recognition on the video frame to obtain the coordinates of multiple skeleton key points;

[0109] Based on the coordinates of multiple skeleton key points, the coordinates of the video center of mass of the monitored target person are determined.

[0110] Specifically, skeleton key points are also called human body key points, which usually correspond to joint positions with a certain degree of freedom on the human body. Skeleton key points can be used to describe human activities in a very accurate way. In one embodiment, the 18 human body key points detected can be used to generate multiple circumscribed rectangular frames, calculate the center of mass position of each circumscribed rectangular frame, and then weight the multiple center of mass positions to obtain the video center of mass coordinates of the human body.

[0111] As for the second video stream iterative detection model described in steps S700-S800, it can be various existing fall detection models based on computer vision, and the second video stream iterative detection model includes YOLOv5 and Ka l man filtering algorithms; the present invention does not specifically expand on this, and the following prior art has relevant records on this, and its content is introduced as part of the embodiments of the present application:

[0112] [1] Zhao Pu, Wu Yi. Fall detection algorithm for community medical care[J]. Chinese Journal of Medical Physics, 2023, 40(12): 1486-1493.

[0113] [2] Zhang Min, Zhang Huan, Shi Xiaojuan, et al. Design and implementation of fall detection system for elderly patients[J]. Chinese Medical Equipment, 2024, 21(02): 157-161.

[0114] [3] Wang Xin, Yang Xiumei. Personnel fall detection based on YOLOv5s and improved center of mass tracking[J]. Electronic Measurement Technology, 2023, 46(24).

[0115] To achieve Figure 1 The method, Figure 3 This is a schematic diagram of the functional unit composition of a fall warning system based on deep learning according to an embodiment of the present invention.

[0116] exist Figure 3 In the figure, a fall warning system based on deep learning is shown, wherein the system includes a first millimeter-wave radar arranged in an indoor environment, a second environmental vision sensor, and a third environmental vision sensor arranged in an outdoor environment.

[0117] The system also includes a human body parameter input unit, a warning unit, a first radar fall estimation model and a second video stream iterative detection model;

[0118] The human body parameter input unit inputs the body parameters of the monitored target person, obtains the actual center of mass coordinates based on the body parameters, and sends the actual center of mass coordinates to the first radar fall estimation model and the second video stream iterative detection model;

[0119] The first radar fall inference model obtains the first radar collection data through the first millimeter wave radar, determines the radar center of mass coordinates of the monitored target person, and infers whether the monitored target person falls based on the determined radar center of mass coordinates and movement speed of the monitored target person in combination with the CRBENN fall inference algorithm;

[0120] The second video stream iterative detection model obtains a third image monitoring frame through a third environmental visual sensor, performs skeleton point recognition on the third image monitoring frame, and obtains a plurality of skeleton key point coordinates; based on the plurality of skeleton key point coordinates, determines the video centroid coordinates of the monitored target person, and detects whether the monitored target person falls based on YOLOv5 and Ka l man filtering algorithms;

[0121] When the radar centroid coordinates match the actual centroid coordinates, the first radar fall estimation model is started, and when the video centroid coordinates match the actual centroid coordinates, the second video stream iterative detection model is started;

[0122] When the monitored target person falls, the warning unit sends out a warning signal.

[0123] The system also includes an update training module;

[0124] When the estimation result of the first radar fall estimation model is that an indoor fall occurs, an indoor fall warning signal is issued, and a second image monitoring frame in a preset time period before and after the indoor fall occurs is added to the training sample set;

[0125] The update training module trains a second video stream iterative detection model based on the training sample set.

[0126] When the detection result of the second video stream iterative detection model is that an outdoor fall occurs, a third image monitoring frame in a preset time period before and after the outdoor fall occurs is added to the training sample set. Preferably, the human body parameter input unit includes an image scanning unit, which scans the monitored target person to obtain the body parameters of the monitored target person, and the body parameters include height, body proportions, three-dimensional coordinates of bones, and a set of coordinates of key points of bones.

[0127] Figure 3 The deep learning-based fall warning system can be specifically applied to a combination of indoor environments and outdoor environments.

[0128] For specific implementation, please refer to Figure 4 The schematic diagram described.

[0129] exist Figure 4 In the system, millimeter-wave radars and low-resolution cameras are arranged in indoor environments, while high-resolution cameras are arranged in outdoor environments, thereby realizing fall detection and early warning for the target person from indoor to outdoor environments.

[0130] Specifically, the technical solution of the present application can be automatically implemented through various forms of electronic devices through computer program instructions; the computer program instructions can be stored in storage media of different forms and loaded into computer electronic devices for execution.

[0131] Figure 5 A schematic diagram of an embodiment of a computer process of a fall warning method based on deep learning is shown, which specifically includes the following process implementation composed of pseudo code language:

[0132] Step 1: Input the body parameters of the monitored target person, and obtain the actual center of mass coordinates based on the body parameters;

[0133] Step 2: Is the monitored target person active in an indoor environment? If yes, go to step 3;

[0134] Otherwise, go to step 9;

[0135] Step 3: Acquire the first radar collection data through the first millimeter wave radar; determine whether the radar center of mass coordinates of the monitored target person determined based on the first radar collection data match the actual center of mass coordinates?

[0136] If yes, go to step 4;

[0137] Otherwise, return to step 2;

[0138] Step 4: Start the first radar fall estimation model;

[0139] step5: The calculation result of the first radar fall calculation model is that an indoor fall occurred?

[0140] If yes, go to step 6;

[0141] Otherwise, return to step 2;

[0142] Step 6: Send out indoor fall warning signal;

[0143] Step 7: Add the second image monitoring frame of the preset time period before and after the indoor fall phenomenon to the training sample set;

[0144] Step 8: Update and train the second video stream iterative detection model and return to step 2;

[0145] step9: obtain a third image monitoring frame through a third environmental visual sensor, and determine the video center of mass coordinates of the monitored target person based on the third image monitoring frame; do the video center of mass coordinates match the actual center of mass coordinates?

[0146] If yes, go to step 10;

[0147] Otherwise, return to step 2;

[0148] step10: Start the second video stream iterative detection model;

[0149] step11: Is the detection result of the second video stream iterative detection model that an outdoor fall occurred?

[0150] If yes, go to step 12;

[0151] Otherwise, return to step 2;

[0152] step12: Send an outdoor fall warning signal;

[0153] Step 13: Add the third image monitoring frame in the preset time period before and after the outdoor fall phenomenon to the training sample set, and jump to step 8.

[0154] Different from the fall detection models or algorithms proposed in the prior art, which are usually general or universal, the fall detection algorithm or model adopted in the present invention realizes personalized customization and startup to a certain extent, that is, the physical parameters of the monitored target person are pre-entered and sent to the fall detection algorithm or model in the corresponding environment, so that the fall detection algorithm or model can carry out detection on the target person of the specified type and reduce the information interference of irrelevant targets; at the same time, the image frames corresponding to the corresponding output results are used as updated training sample sets to continue training the corresponding fall detection algorithm or model, so that the subsequent fall detection process for the target person of the specified features or type continues to evolve, which further improves the detection accuracy while reducing the complexity of deep learning model training and improving the adaptability of the model to different environments.

[0155] The present invention provides multiple embodiments, each of which can constitute an independent technical solution and may contribute to the prior art and solve corresponding technical problems. However, it should be pointed out that different embodiments can be combined with each other without violating logic; at the same time, each embodiment can solve at least one technical problem, but it is not required that each individual embodiment solves multiple or all technical problems.

[0156] For other technologies, principles, algorithms or models not elaborated in detail in this application (such as CRBENN neural network model algorithm, etc.), please refer to the prior art.

[0157] The foregoing has shown and described the method embodiments and system of the present invention, but it is understandable to those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A fall warning method based on deep learning, characterized in that: The method comprises the following steps: S100: inputting the body parameters of the monitored target person, and obtaining the actual center of mass coordinates based on the body parameters; S200: When the monitored target person is active in the indoor environment, first radar acquisition data is acquired through the first millimeter wave radar, and a second image monitoring frame is acquired through the second environment vision sensor; S300: When the radar centroid coordinates of the monitored target person determined based on the first radar collected data match the actual centroid coordinates, starting the first radar fall estimation model; the first radar fall estimation model estimates whether the monitored target person falls based on the determined radar centroid coordinates and movement speed of the monitored target person in combination with the CRBENN fall estimation algorithm; S400: When the calculation result of the first radar fall calculation model is that an indoor fall phenomenon occurs, an indoor fall warning signal is issued, and a second image monitoring frame in a preset time period before and after the indoor fall phenomenon occurs is added to a training sample set; S500: training a second video stream iterative detection model based on the training sample set; the second video stream iterative detection model includes YOLOv5 and Kalman filtering algorithms; S600: When the monitored target person is active in an outdoor environment, a third image monitoring frame is obtained through a third environmental visual sensor, and the video centroid coordinates of the monitored target person are determined based on the third image monitoring frame; S700: When the video centroid coordinates match the actual centroid coordinates, starting a second video stream iteration detection model; S800: When the detection result of the second video stream iterative detection model is that an outdoor fall phenomenon occurs, an outdoor fall warning signal is issued; The step S600 determines the coordinates of the video centroid of the monitored target person based on the third image monitoring frame, specifically comprising: performing skeleton point recognition on the video frame to obtain coordinates of multiple skeleton key points; Based on the coordinates of multiple skeleton key points, determine the coordinates of the video center of mass of the monitored target person; When the detection result in step S800 is that an outdoor fall phenomenon occurs, the third image monitoring frame in a preset time period before and after the outdoor fall phenomenon occurs is added to the training sample set, and the process returns to step S500.

2. A fall warning method based on deep learning as claimed in claim 1, characterized in that: The step S300 determines the radar centroid coordinates of the monitored target person based on the first radar collected data, specifically including: Based on the data collected by the first radar, a three-dimensional Cartesian rectangular coordinate system is established; The reference centroid clustering algorithm is used to obtain the horizontal coordinates of the human body's centroid, and the centroid height is obtained based on the torso feature algorithm, thereby determining the radar centroid coordinates of the monitored target person.

3. A fall warning system based on deep learning, the system comprising a first millimeter wave radar arranged in an indoor environment, a second environmental vision sensor, and a third environmental vision sensor arranged in an outdoor environment, It is characterized in that The system is used to implement a fall warning method based on deep learning as described in claim 1 or 2; The system also includes a human body parameter input unit, a warning unit, a first radar fall estimation model and a second video stream iterative detection model; The human body parameter input unit inputs the body parameters of the monitored target person, obtains the actual center of mass coordinates based on the body parameters, and sends the actual center of mass coordinates to the first radar fall estimation model and the second video stream iterative detection model; The first radar fall inference model obtains the first radar collection data through the first millimeter wave radar, determines the radar center of mass coordinates of the monitored target person, and infers whether the monitored target person falls based on the determined radar center of mass coordinates and movement speed of the monitored target person in combination with the CRBENN fall inference algorithm; The second video stream iterative detection model obtains a third image monitoring frame through a third environmental visual sensor, performs skeleton point recognition on the third image monitoring frame, and obtains multiple skeleton key point coordinates; based on the multiple skeleton key point coordinates, the video centroid coordinates of the monitored target person are determined, and based on the YOLOv5 and Kalman filter algorithms, whether the monitored target person falls is detected; When the radar centroid coordinates match the actual centroid coordinates, the first radar fall estimation model is started, and when the video centroid coordinates match the actual centroid coordinates, the second video stream iterative detection model is started; When the monitored target person falls, the warning unit sends out a warning signal.

4. A fall warning system based on deep learning as claimed in claim 3, characterized in that: The system also includes an update training module; When the estimation result of the first radar fall estimation model is that an indoor fall occurs, an indoor fall warning signal is issued, and a second image monitoring frame in a preset time period before and after the indoor fall occurs is added to the training sample set; The update training module trains a second video stream iterative detection model based on the training sample set.

5. A fall warning system based on deep learning as claimed in claim 4, characterized in that: When the detection result of the second video stream iterative detection model is that an outdoor fall phenomenon occurs, a third image monitoring frame in a preset time period before and after the outdoor fall phenomenon occurs is added to the training sample set.

6. A fall warning system based on deep learning as claimed in claim 3, characterized in that: The human body parameter input unit includes an image scanning unit, which scans the monitored target person to obtain the body parameters of the monitored target person, and the body parameters include height, body proportions, three-dimensional coordinates of bones, and a set of coordinates of key points of bones.

7. A fall warning system based on deep learning as claimed in claim 3, characterized in that: The resolution of the third environment vision sensor is greater than the resolution of the second environment vision sensor.

Citation Information

Cited By

  • A human body fall risk early warning method, system, device and storage medium

    CN121354282B

  • Scenarized abnormal behavior classification method and system, equipment and storage medium

    CN121412844A