Fall event identification method and device and medium

By combining attention mechanism and posture estimation network's fall event recognition method, the accuracy and resource consumption problems of fall event recognition in the existing technology are solved, and the high accuracy and low complexity fall event recognition is achieved, which is suitable for the elderly. and ward environment.

CN118968566AActive Publication Date: 2024-11-15JILIN UNIV FIRST HOSPITAL +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411449644.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-17
Publication Date
2024-11-15
Estimated Expiration
2044-10-17

AI Technical Summary

Technical Problem

The existing fall event recognition methods have poor accuracy, high false alarm rate, large resource consumption and limitations in application scenarios, especially in the elderly and ward environments, which are difficult to achieve real-time and accurate fall behavior recognition.

Method used

The YOLOv8n object detection model combining channel attention mechanism and spatial attention mechanism is adopted, combined with the GhostNet-OpenPose pose estimation network, and the fall events are identified through the interleaving and comparison and decision tree classification algorithm, and the histogram equalization processing and SE attention module optimization model is used. , reduce the computational complexity and improve the recognition accuracy.

Benefits of technology

The accuracy of falling events recognition is achieved at 95.46%, reducing the number of model parameters and calculation complexity, suitable for resource-constrained environments, taking into account accuracy and recall rates, and reducing the occurrence of accidental injuries.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968566A_ABST
    Figure CN118968566A_ABST
Patent Text Reader

Abstract

The invention relates to a tumble event identification method and device and a medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring an image of a target place by using a sensor; identifying the image by means of a target detection model to obtain a bounding box of a human body and a bounding box of a bed; according to the bounding box of the human body and the bounding box of the bed, the intersection-to-union ratio is calculated; judging whether the intersection-union ratio is smaller than an intersection-union ratio threshold value or not; if the intersection-to-union ratio is smaller than an intersection-to-union ratio threshold value, it is judged that the human body is in an active state; using a GhostNet-OpenPose attitude estimation network to identify skeleton key points of a human body, and outputting coordinates of each skeleton key point; calculating a real-time joint angle of an elbow joint and a real-time joint angle of a knee joint based on the coordinate of each skeleton key point; and inputting the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint into a decision tree tumble classification algorithm, and identifying a tumble event. The fall event can be accurately identified.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a fall event recognition method, device and medium. Background Art

[0002] With the increasing aging of the population, the reality of aging and the problem of elderly care have been put before us. Its importance has become very prominent, and the social and family problems caused by elderly care are even more difficult. For empty-nest elderly people, falls are common and serious emergencies. It is often difficult to detect falls and fainting in this group of people in time. Some elderly people choose nursing homes, but the caregivers have limited energy and are prone to ignore the occurrence of falls. In addition, patients in the ward may not be discovered in time after fainting and falling. If they are not discovered in time, it will cause immeasurable consequences.

[0003] Currently, there are two methods for fall event recognition:

[0004] (1) Fall event recognition method based on environmental sensors: It mainly obtains real-time information about the human body by setting up floor pressure sensors, radars, acoustic sensors, etc. in the self-monitoring area, and analyzes the movement patterns of the human body by integrating multiple sets of data, so as to discover abnormal movement behaviors and determine the occurrence of falls. Although this method is not very invasive, it is sensitive to noise. Environmental sensors have blind spots, high configuration costs, and poor accuracy.

[0005] (2) Fall event recognition method based on wearable devices: The human motion data is collected through the sensors built into the transmission device, and the human motion data is compared with the threshold to determine whether there is a fall event. This method has a high false alarm rate and requires the device to be worn at all times. If you forget to wear it, it will not be possible to identify whether a fall event has occurred. Battery life is also a pain point of this method.

[0006] (3) Fall recognition method based on visual information: This method obtains human motion information from monitoring equipment and determines the possibility of falling by analyzing the collected videos and images. This algorithm does not require carrying any equipment and has a good user experience and high detection accuracy. Compared with fall recognition based on wearable devices, fall recognition based on visual information has less interference and higher accuracy and robustness. Fall feature extraction based on this method is usually divided into traditional methods and deep learning methods. Among them, the feature extraction method based on deep learning can achieve end-to-end learning. Before learning, there is no need to extract image edge, color, spatial position and other information. Useful features can be automatically extracted from raw data through a deep network structure. In terms of fall recognition, deep learning can realize automatic feature extraction and analysis of monitoring videos, reduce the burden of manual monitoring, save labor costs, and extract rich multidimensional features, including posture, movement, environment and other information, which helps to more comprehensively identify fall events. With the continuous development of deep learning in the field of human posture recognition, accurate extraction of human skeleton key points through convolutional neural networks is conducive to effective analysis of human posture. However, most of the algorithms with high accuracy proposed based on this method have complex network structures and cannot meet the real-time requirements. Fall behavior recognition based on surveillance video streams is different from other image analysis tasks. The recognition process must be accurate and fast enough, and the algorithm needs to consider the problems of ordinary security monitoring such as blurry, distorted or noisy images under poor lighting conditions. In addition, the application scenarios of the current common fall recognition methods are relatively limited. When there is furniture for lying down and resting in the environment, when a person is in a lying posture, it is easy to be confused with falling behavior, resulting in false alarms. Summary of the invention

[0007] The present disclosure provides a fall event recognition method, device and medium to solve the above technical problems.

[0008] According to a first aspect of the present disclosure, a fall event recognition method is provided, the method comprising: using a sensor to collect an image of a target place, wherein there are a human body and a bed in the target place; recognizing the image with the help of a target detection model to obtain a bounding box of the human body and a bounding box of the bed, wherein, when constructing the target detection model, a CBAM module combining a channel attention mechanism and a spatial attention mechanism is added to YOLOv8n; calculating an intersection-over-union (IoU) according to the bounding box of the human body and the bounding box of the bed; judging whether the IoU is less than an IoU threshold; if the IoU is less than the IoU threshold, judging that the human body is in an active state; using a GhostNet-OpenPose posture estimation network to identify the skeleton key points of the human body, and outputting the coordinates of each skeleton key point, wherein, when constructing the GhostNet-OpenPose, the feature extraction part is replaced by GhostNet with VGG-19, and OpenPos e multi-branch iterative mechanism only retains the initial stage and one refinement stage, merges part of the convolution structure to calculate the key point heat map and part of the affinity field, and uses the COCO data set as a training set; based on the coordinates of each skeletal key point, calculate the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint; input the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint into a decision tree fall classification algorithm to identify fall events, wherein the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the knee joint are used as training sets to train the decision tree fall classification algorithm, the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the elbow joint are calculated based on the historical data of the coordinates of each skeletal key point output by the GhostNet-OpenPose posture estimation network, and the historical data of the coordinates of the skeletal key point are calculated based on the historical data of the aspect ratio of the bounding box of the human body output by the target detection model.

[0009] In some embodiments, before the image is recognized by means of a target detection model, the method further includes: reading the image frame by frame; adjusting each frame of the image to a preset size; and performing histogram equalization on the adjusted image to complete preprocessing of the image.

[0010] In some embodiments, if the intersection-over-union ratio is greater than or equal to an intersection-over-union ratio threshold, it is determined that the human body is resting in bed.

[0011] In some embodiments, after inputting the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint into the decision tree fall classification algorithm to identify the fall event, it also includes: broadcasting a "fall" alarm message with the help of an alarm system.

[0012] According to a second aspect of the present disclosure, a fall event recognition device is provided, comprising: an image acquisition module, for using a sensor to acquire an image of a target place, wherein the target place contains a human body and a bed; an image recognition module, for recognizing the image with the aid of a target detection model to obtain a bounding box of the human body and a bounding box of the bed, wherein when constructing the target detection model, a CBAM module combining a channel attention mechanism and a spatial attention mechanism is added to YOLOv8n; an IoU calculation module, for calculating an IoU according to the bounding box of the human body and the bounding box of the bed; an IoU judgment module, for judging whether the IoU is less than an IoU threshold; an active state judgment module, for judging that the human body is in an active state if the IoU is less than an IoU threshold; a skeletal key point recognition module, for recognizing skeletal key points of the human body using a GhostNet-OpenPose posture estimation network, and outputting the coordinates of each skeletal key point, wherein when constructing the GhostNet-OpenPose, the feature extraction part is extracted with GhostNet. t replaces VGG-19, the OpenPose multi-branch iterative mechanism only retains the initial stage and one refinement stage, merges some convolution structures to calculate key point heat maps and some affinity fields, and uses the COCO data set as a training set; a real-time joint angle calculation module, used to calculate the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint based on the coordinates of each skeletal key point; a fall event recognition module, used to input the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint into a decision tree fall classification algorithm to identify fall events, wherein the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the knee joint are used as training sets to train the decision tree fall classification algorithm, the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the elbow joint are calculated based on the historical data of the coordinates of each skeletal key point output by the GhostNet-OpenPose posture estimation network, and the historical data of the coordinates of the skeletal key point are calculated based on the historical data of the aspect ratio of the bounding box of the human body output by the target detection model.

[0013] According to a third aspect of the present disclosure, a fall event recognition device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the fall event recognition method as described above based on instructions stored in the memory.

[0014] According to a fourth aspect of the present disclosure, a computer storable medium is provided, on which computer program instructions are stored, and when the instructions are executed by a processor, the fall event recognition method as described above is implemented.

[0015] Beneficial effects of the present disclosure:

[0016] (1) The histogram equalization technology was introduced in the image preprocessing stage to enhance the image contrast, improve the visibility of image details, and effectively improve the image quality, thereby increasing the recognition accuracy of the bed and the human body by nearly three percentage points respectively.

[0017] (2) The introduction of the CBAM module combines the channel attention mechanism and the spatial attention mechanism. The CBAM module can compress the spatial dimension without changing the channel dimension, better focus on the location information of the target, and accurately generate the bounding box of the human body and the bounding box of the bed.

[0018] (3) The original OpenPose key point detection network is optimized and GhoseNet is used as the backbone feature extraction network. It includes technologies such as SE attention module and deep separable convolution, which reduces the number of floating point operations from 8.85GFLOPs to 6.28GFLOPs, effectively reducing the number of parameters and computational complexity of the model, enabling real-time recognition. The recognition accuracy on the COCO key point recognition dataset is improved by 9.5%, making it suitable for deployment and application in resource-constrained environments.

[0019] (4) We selected multiple features that best represent abnormal posture changes of the human body, and adopted a decision tree model that can effectively solve the classification problem. Experimental evaluation proved that the overall recognition accuracy of the model for fall events reached 95.46%, and the response to fall events was more reliable. At the same time, it also achieved a good balance between precision and recall, with the calculated values ​​of the two being 94.87% and 93.54% respectively, which can take into account the recognition of both positive and negative samples.

[0020] As mentioned above, the technical means provided by the present disclosure can be applied to specific indoor places containing beds and human bodies. As a technical means for auxiliary care, it can improve the quality of life of the elderly and specific groups of people and reduce the occurrence of accidental injuries. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings.

[0023] Figure 1 is a flow chart illustrating a method for identifying a fall event according to some embodiments of the present disclosure.

[0024] Figure 2 is an original picture showing a target to be identified according to some embodiments of the present disclosure;

[0025] Figure 32 is a diagram showing the effect of YOLOv8n target recognition combined with the attention mechanism according to some embodiments of the present disclosure.

[0026] Figure 4 is a schematic diagram showing the structure of a GhostNet-OpenPose pose estimation network according to some embodiments of the present disclosure.

[0027] Figure 5 is one of the GhostNet-OpenPose recognition effect diagrams according to some embodiments of the present disclosure.

[0028] Figure 6 2 is a second diagram showing the GhostNet-OpenPose recognition effect according to some embodiments of the present disclosure.

[0029] Figure 7 3 is a diagram showing the effect of GhostNet-OpenPose recognition according to some embodiments of the present disclosure.

[0030] Figure 8 Schematic diagram showing human body activity states according to some embodiments of the present disclosure.

[0031] Fig. 9 Schematic diagram showing a human body falling according to some embodiments of the present disclosure.

[0032] Fig.10 is a schematic diagram showing a human body in resting state according to some embodiments of the present disclosure.

[0033] Fig.11 is a block diagram showing a fall event recognition apparatus according to some embodiments of the present disclosure.

[0034] Fig.12 is a block diagram showing a fall event recognition device according to some other embodiments of the present disclosure.

[0035] Fig.13 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0037] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0038] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0039] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0040] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0041] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0042] At present, with the increasing aging of the population, the reality of aging and the problem of elderly care have been put before us, and its importance has become very prominent. The social and family problems caused by elderly care are even more difficult. For empty-nest elderly people, falls are common and serious emergencies. It is often difficult to detect falls and fainting in this group of people in time. Some elderly people choose nursing homes, but the caregivers have limited energy and are prone to ignore the occurrence of falls. In addition, patients in the ward may not be discovered in time after fainting and falling. If they are not discovered in time, it will cause immeasurable consequences.

[0043] There are two methods for fall event recognition: fall event recognition method based on environmental sensors: mainly by setting up floor pressure sensors, radars, acoustic sensors, etc. in the self-monitoring area to obtain real-time information of the human body, and analyzing the human body movement rules by integrating multiple sets of data, so as to find abnormal movement behavior and determine the occurrence of fall behavior. Although this method is not very invasive, it is sensitive to noise. Environmental sensors have blind spots, high configuration costs, and poor accuracy; fall event recognition method based on wearable devices: collect human motion data through sensors built into the transmission device, compare the human motion data with the threshold, and determine whether there is a fall event. This method has a high false alarm rate and needs to wear the device at all times. If you forget to wear it, you cannot identify whether a fall event has occurred. Battery life is also a pain point of this method. Fall recognition method based on visual information: This method obtains human motion information from the monitoring device, and analyzes the collected videos and images to determine the possibility of falling. This algorithm does not require carrying any equipment, has a good user experience and a high detection accuracy. Compared with fall recognition based on wearable devices, fall recognition based on visual information has less interference and higher accuracy and robustness. The fall feature extraction based on this method is usually divided into traditional methods and deep learning methods. Among them, the feature extraction method based on deep learning can realize end-to-end learning. Before learning, there is no need to extract information such as the edge, color, and spatial position of the image, and useful features can be automatically extracted from the original data through a deep network structure. In terms of fall recognition, deep learning can realize automatic feature extraction and analysis of surveillance videos, reduce the burden of manual monitoring, save labor costs, and extract rich multidimensional features, including posture, action, environment and other information, which helps to more comprehensively identify fall events. With the continuous development of deep learning in the field of human posture recognition, accurate extraction of human skeleton key points through convolutional neural networks is conducive to effective analysis of human posture. However, most of the algorithms with high accuracy proposed based on this method have complex network structures and cannot meet the real-time requirements. Fall behavior recognition based on surveillance video streams is different from other image analysis tasks. The recognition process must be accurate and fast enough, and the algorithm needs to consider the problems of blurred, distorted or noise in ordinary security monitoring when the light conditions are poor. In addition, the application scenarios of the current common fall recognition methods are relatively limited. When there is furniture for lying down and resting in the environment where a person is located, the person's lying posture can easily be confused with falling behavior, resulting in false alarms.

[0044] In view of this, the present disclosure proposes a fall event recognition method, device and medium, and introduces histogram equalization processing technology in the image preprocessing stage, which enhances the contrast of the image, improves the visibility of image details, and effectively improves the image quality, thereby increasing the recognition accuracy of the bed and the human body by nearly three percentage points respectively. The introduction of the CBAM module combines the channel attention mechanism and the spatial attention mechanism. The CBAM module can compress the spatial dimension without changing the channel dimension, better focus on the location information of the target, and accurately generate the bounding box of the human body and the bounding box of the bed. The original OpenPose key point detection network is optimized, and Long GhoseNet is used as the backbone feature extraction network, which includes SE attention module and deep separable convolution and other technologies, reducing the number of floating point operations from 8.85GFLOPs to 6.28GFLOPs, effectively reducing the number of parameters and computational complexity of the model, and realizing real-time recognition. The recognition accuracy on the COCO key point recognition data set is increased by 9.5%, which is suitable for deployment and application in resource-constrained environments. Multiple features that best represent abnormal posture changes of the human body are selected, and a decision tree model that can effectively solve classification problems is adopted. Experimental evaluation proves that the model has an overall recognition accuracy of 95.46% for fall events, and is more reliable in responding to fall events. At the same time, it has achieved a good balance between precision and recall, with the calculated values ​​of 94.87% and 93.54% respectively, which can take into account the recognition of positive and negative samples. As mentioned above, the technical means provided by the present disclosure can be applied to specific indoor places containing beds and human bodies. As a technical means for auxiliary care, it can improve the quality of life of the elderly and specific groups of people and reduce the occurrence of accidental injuries.

[0045] Figure 1 FIG. 4 is a flow chart showing a method for identifying a fall event according to some embodiments of the present disclosure. Figure 1 As shown, the fall event recognition method includes steps 110 to 180.

[0046] In step 110, a sensor is used to collect an image of a target location, wherein a human body and a bed are present in the target location.

[0047] In some embodiments, the method further includes: reading the image frame by frame; adjusting each frame of the image to a preset size; and performing histogram equalization processing on the adjusted image to complete preprocessing of the image.

[0048] In step 120, the image is recognized by means of a target detection model to obtain a bounding box of a human body and a bounding box of a bed, wherein when constructing the target detection model, a CBAM module combining a channel attention mechanism and a spatial attention mechanism is added to YOLOv8n. Figure 2 and Figure 3 shown.

[0049] In step 130, an intersection-over-union ratio is calculated based on the bounding box of the human body and the bounding box of the bed.

[0050] In step 140, it is determined whether the IoU is less than an IoU threshold.

[0051] In some embodiments, if the intersection-over-union ratio is greater than or equal to an intersection-over-union ratio threshold, it is determined that the human body is resting in bed.

[0052] In step 150, if the IoU is less than an IoU threshold, it is determined that the human body is in an active state.

[0053] In step 160, the GhostNet-OpenPose posture estimation network is used to identify the skeleton key points of the human body, and the coordinates of each skeleton key point are output. When constructing the GhostNet-OpenPose, the feature extraction part is replaced by VGG-19 with GhostNet, the OpenPose multi-branch iterative mechanism only retains the initial stage and one refinement stage, and some convolution structures are merged to calculate the key point heat map and some affinity fields, and the COCO data set is used as the training set.

[0054] In step 170, based on the coordinates of each skeleton key point, the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint are calculated.

[0055] In step 180, the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint are input into a decision tree fall classification algorithm to identify fall events, wherein the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the knee joint are used as training sets to train the decision tree fall classification algorithm, the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the elbow joint are calculated based on the historical data of the coordinates of each skeleton key point output by the GhostNet-OpenPose posture estimation network, and the historical data of the coordinates of the skeleton key point are calculated based on the historical data of the aspect ratio of the bounding box of the human body output by the target detection model.

[0056] In some embodiments, the method further includes: broadcasting a "fall" alarm message by means of an alarm system.

[0057] This method uses a target detection network with an attention mechanism to identify the target boundaries of human bodies and beds in the room, and explores the optimal intersection-over-union ratio judgment threshold for human activity areas through experiments. Figure 8 , Fig. 9 , Fig.10 As shown in the figure, the algorithm can measure the overlap between the human body and the bed by calculating the intersection of the two target detection frames and then judge the current activity state of the subject. Figure 8 and Fig. 9 The overlap shown does not reach the set threshold, so it is in the active area. Fig.10 The overlap shown exceeds the set threshold and is in the rest area. Figure 4 As shown in Figure 2, the improved lightweight posture estimation network is used to identify the coordinates of the key points of the human skeleton in the active area. Figure 5 , Figure 6 and Figure 7 As shown, the training set used for training the lightweight posture estimation network is stated as follows: the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the knee joint are used as training sets to train the decision tree fall classification algorithm, the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the elbow joint are calculated based on the historical data of the coordinates of each skeleton key point output by the GhostNet-OpenPose posture estimation network, and the historical data of the coordinates of the skeleton key point are calculated based on the historical data of the aspect ratio of the bounding box of the human body output by the target detection model. If and only if the person is in the activity area and the classifier determines that it is a fall posture, such as Fig. 9 As shown, it is determined that the person has fallen and a "fall" alarm message is sent.

[0058] Histogram equalization processing can enhance image contrast and highlight picture details.

[0059] The intersection-over-union ratio threshold value may be 0.35. The embodiments of the present disclosure are not limited here.

[0060] The advantages of the present invention are as follows: (1) The histogram equalization processing technology is introduced in the image preprocessing stage to enhance the image contrast, improve the visibility of image details, and effectively improve the image quality. The same batch of images are preprocessed with and without histogram equalization, and then the two groups of images are sent to the trained YOLOv8-n bed and human target detection network respectively. The histogram equalization processing can increase the average accuracy of bed and human target recognition in the image by nearly 3 percentage points respectively. Experiments show that the introduction of this method makes the image easier to process and analyze.

[0061] (2) The introduction of the CBAM module combines the channel attention mechanism and the spatial attention mechanism. This module can compress the spatial dimension without changing the channel dimension, better focus on the location information of the target, and accurately generate the bounding boxes of the bed and human targets. The attention mechanism module used in this study is inserted between the Backbone and Neck of YOLOv8-n. The network training uses the COCO dataset, only for the two target categories of Bed and Person. The training uses the original weights for transfer learning. The training results show that the introduction of the CBAM module significantly improves the accuracy of network target recognition.

[0062] (3) The original OpenPose key point recognition network is optimized and GhostNet is used as the backbone feature extraction network, which includes technologies such as SE attention module and deep separable convolution. The number of floating-point operations is reduced from 8.85GFLOPs to 6.28GFLOPs, effectively reducing the number of parameters and computational complexity of the model, enabling real-time recognition. The recognition accuracy on the COCO key point recognition dataset is improved by 9.5%, making it suitable for deployment and application in resource-constrained environments.

[0063] (4) We selected multiple features that best represent abnormal posture changes of the human body, and adopted a decision tree model that can effectively solve the classification problem. Experimental evaluation proved that the model can accurately predict the fall behavior as a whole, with a recognition accuracy of 95.46%, and respond more reliably to the fall event. At the same time, it also achieved a good balance between precision and recall, with the calculated values ​​of the two being 94.87% and 93.54% respectively, which can take into account the recognition of both positive and negative samples.

[0064] The embodiments of the present disclosure have a wide range of applications, and the method is highly versatile and can be applied to indoor places containing beds and people, such as wards, bedrooms, and nursing homes. As auxiliary care, it can improve the quality of life of the elderly and specific groups of people and reduce the occurrence of accidental injuries.

[0065] Fig.11 is a block diagram showing a fall event recognition device according to some embodiments of the present disclosure. Fig.11 As shown, the fall event recognition device 1100 includes an image acquisition module 1110, an image recognition module 1120, an intersection-and-union ratio calculation module 1130, an intersection-and-union ratio judgment module 1140, an activity state judgment module 1150, a skeleton key point recognition module 1160, a real-time joint angle calculation module 1170, and a fall event recognition module 1180.

[0066] The image acquisition module 1110 is configured to use a sensor to acquire an image of a target location, wherein the target location contains a human body and a bed;

[0067] An image recognition module 1120 is configured to recognize the image with the aid of a target detection model to obtain a bounding box of a human body and a bounding box of a bed, wherein when constructing the target detection model, a CBAM module combining a channel attention mechanism and a spatial attention mechanism is added to YOLOv8n;

[0068] The intersection-and-union ratio calculation module 1130 is configured to calculate the intersection-and-union ratio according to the bounding box of the human body and the bounding box of the bed;

[0069] The I / O ratio determination module 1140 is configured to determine whether the I / O ratio is less than an I / O ratio threshold;

[0070] An activity state determination module 1150 is configured to determine that the human body is in an activity state if the intersection-and-union ratio is less than an intersection-and-union ratio threshold;

[0071] The skeleton key point recognition module 1160 is configured to use the GhostNet-OpenPose posture estimation network to recognize the skeleton key points of the human body and output the coordinates of each skeleton key point, wherein when constructing the GhostNet-OpenPose, the feature extraction part is replaced by VGG-19 with GhostNet, the OpenPose multi-branch iterative mechanism only retains the initial stage and one refinement stage, merges some convolution structures to calculate the key point heat map and some affinity fields, and uses the COCO data set as a training set;

[0072] A real-time joint angle calculation module 1170 is configured to calculate the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint based on the coordinates of each skeleton key point;

[0073] The fall event recognition module 1180 is configured to input the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint into a decision tree fall classification algorithm to identify a fall event, wherein the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the knee joint are used as training sets to train the decision tree fall classification algorithm, the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the elbow joint are calculated based on the historical data of the coordinates of each bone key point output by the GhostNet-OpenPose posture estimation network, and the historical data of the coordinates of the bone key point is calculated based on the historical data of the aspect ratio of the bounding box of the human body output by the target detection model.

[0074] In the device of the embodiment of the present disclosure, the histogram equalization processing technology is introduced in the image preprocessing stage, which enhances the contrast of the image, improves the visibility of the image details, and effectively improves the image quality, thereby increasing the recognition accuracy of the bed and the human body by nearly three percentage points respectively. The introduction of the CBAM module combines the channel attention mechanism and the spatial attention mechanism. The CBAM module can compress the spatial dimension without changing the channel dimension, better focus on the location information of the target, and accurately generate the bounding box of the human body and the bounding box of the bed. The original OpenPose key point detection network is optimized, and Long GhoseNet is used as the backbone feature extraction network, which includes SE attention module and deep separable convolution and other technologies, reducing the number of floating point operations from 8.85GFLOPs to 6.28GFLOPs, effectively reducing the number of parameters and computational complexity of the model, and realizing real-time recognition. The recognition accuracy on the COCO key point recognition data set is increased by 9.5%, which is suitable for deployment and application in resource-constrained environments. Multiple features that best represent abnormal posture changes of the human body are selected, and a decision tree model that can effectively solve classification problems is adopted. Experimental evaluation proves that the model has an overall recognition accuracy of 95.46% for fall events, and is more reliable in responding to fall events. At the same time, it has achieved a good balance between precision and recall, with the calculated values ​​of 94.87% and 93.54% respectively, which can take into account the recognition of positive and negative samples. As mentioned above, the technical means provided by the present disclosure can be applied to specific indoor places containing beds and human bodies. As a technical means for auxiliary care, it can improve the quality of life of the elderly and specific groups of people and reduce the occurrence of accidental injuries.

[0075] Fig.12 FIG. 2 is a block diagram showing a fall event recognition device according to some other embodiments of the present disclosure. Fig.12 As shown, the fall event recognition device 1200 includes a memory 1210; and a processor 1220 coupled to the memory 1210. The memory 1210 is used to store instructions for executing the corresponding embodiments of the fall event recognition method. The processor 1220 is configured to execute the fall event recognition method in any of the embodiments of the present disclosure based on the instructions stored in the memory 1210.

[0076] Fig.13 is a block diagram showing a computer system for implementing some embodiments of the present disclosure. Fig.13 As shown, the computer system 1300 may be embodied in the form of a general-purpose computing device. The computer system 1300 includes a memory 1310, a processor 1320, and a bus 1330 that connects various system components.

[0077] The memory 1310 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions for executing at least one corresponding embodiment of the fall event recognition method. The non-volatile storage medium includes, but is not limited to, a disk memory, an optical memory, a flash memory, and the like.

[0078] The processor 1220 can be implemented by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, etc. Discrete hardware components. Accordingly, each module such as the image acquisition module, the image recognition module, the intersection-and-union ratio calculation module, the intersection-and-union ratio judgment module, the active state judgment module, the skeleton key point recognition module, the real-time joint angle calculation module, and the fall event recognition module can be implemented by the central processing unit (CPU) running the instructions in the memory to execute the corresponding steps, or can be implemented by a dedicated circuit to execute the corresponding steps.

[0079] The bus 1330 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0080] The computer system 1300 may also include an input / output interface 1340, a network interface 1350, a storage interface 1360, etc. These interfaces 1340, 1350, 1360, the memory 1310, and the processor 1320 may be connected via a bus 1330. The input / output interface 1340 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 1350 provides a connection interface for various networked devices. The storage interface 1360 provides a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0081] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames can be implemented by computer-readable program instructions.

[0082] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0083] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to work in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0084] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0085] In the present disclosure, the histogram equalization processing technology is introduced in the image preprocessing stage to enhance the contrast of the image, improve the visibility of the image details, effectively improve the image quality, and thus increase the recognition accuracy of the bed and the human body by nearly three percentage points respectively. The introduction of the CBAM module combines the channel attention mechanism and the spatial attention mechanism. The CBAM module can compress the spatial dimension without changing the channel dimension, better focus on the location information of the target, and accurately generate the bounding box of the human body and the bounding box of the bed. The original OpenPose key point detection network is optimized, and Long GhoseNet is used as the backbone feature extraction network, which includes SE attention modules and deep separable convolution and other technologies, reducing the number of floating point operations from 8.85GFLOPs to 6.28GFLOPs, effectively reducing the number of parameters and computational complexity of the model, and realizing real-time recognition. The recognition accuracy on the COCO key point recognition data set is improved by 9.5%, which is suitable for deployment and application in resource-constrained environments. Multiple features that best represent abnormal posture changes of the human body are selected, and a decision tree model that can effectively solve classification problems is adopted. Experimental evaluation proves that the model has an overall recognition accuracy of 95.46% for fall events, and is more reliable in responding to fall events. At the same time, it has achieved a good balance between precision and recall, with the calculated values ​​of 94.87% and 93.54% respectively, which can take into account the recognition of positive and negative samples. As mentioned above, the technical means provided by the present disclosure can be applied to specific indoor places containing beds and human bodies. As a technical means for auxiliary care, it can improve the quality of life of the elderly and specific groups of people and reduce the occurrence of accidental injuries.

[0086] So far, the fall event recognition method, device and medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0087] Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A fall event recognition method, characterized in that: The method comprises: Using a sensor to collect an image of a target location, wherein a human body and a bed are located in the target location; The image is recognized by means of a target detection model to obtain a bounding box of a human body and a bounding box of a bed, wherein a CBAM module combining a channel attention mechanism and a spatial attention mechanism is added to YOLOv8n when constructing the target detection model; Calculating an intersection-over-union ratio according to the bounding box of the human body and the bounding box of the bed; Determine whether the intersection-over-union ratio is less than an intersection-over-union ratio threshold; If the intersection-over-union ratio is less than the intersection-over-union ratio threshold, it is determined that the human body is in an active state; The GhostNet-OpenPose posture estimation network is used to identify the skeleton key points of the human body and output the coordinates of each skeleton key point. When constructing the GhostNet-OpenPose, the feature extraction part is replaced by VGG-19 with GhostNet, the OpenPose multi-branch iterative mechanism only retains the initial stage and a refinement stage, and some convolution structures are merged to calculate the key point heat map and some affinity fields, and the COCO dataset is used as the training set; Based on the coordinates of each bone key point, the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint are calculated; The real-time joint angle of the elbow joint and the real-time joint angle of the knee joint are input into a decision tree fall classification algorithm to identify fall events, wherein the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the knee joint are used as training sets to train the decision tree fall classification algorithm, the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the elbow joint are calculated based on the historical data of the coordinates of each skeleton key point output by the GhostNet-OpenPose posture estimation network, and the historical data of the coordinates of the skeleton key point are calculated based on the historical data of the aspect ratio of the bounding box of the human body output by the target detection model.

2. The fall event recognition method according to claim 1, characterized in that: Before the image is recognized by means of the target detection model, the method further includes: Reading the image frame by frame; Resize each frame to the preset size; The adjusted image is subjected to histogram equalization processing to complete the preprocessing of the image.

3. The fall event recognition method according to claim 1 or 2, characterized in that: If the intersection-and-union ratio is greater than or equal to the intersection-and-union ratio threshold, it is determined that the human body is resting in bed.

4. The fall event recognition method according to claim 1 or 2, characterized in that: After inputting the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint into the decision tree fall classification algorithm to identify the fall event, it also includes: broadcasting a "fall" alarm message by means of an alarm system.

5. A fall event recognition device, characterized in that: include: An image acquisition module, used for acquiring an image of a target location using a sensor, wherein the target location contains a human body and a bed; An image recognition module, used to recognize the image with the aid of a target detection model to obtain a bounding box of a human body and a bounding box of a bed, wherein when constructing the target detection model, a CBAM module combining a channel attention mechanism and a spatial attention mechanism is added to YOLOv8n; An intersection-and-union ratio calculation module, used to calculate the intersection-and-union ratio according to the bounding box of the human body and the bounding box of the bed; An I / O ratio judgment module, used to judge whether the I / O ratio is less than an I / O ratio threshold; An activity state determination module, configured to determine that the human body is in an activity state if the intersection-over-union ratio is less than an intersection-over-union ratio threshold; A skeleton key point recognition module is used to identify the skeleton key points of the human body using the GhostNet-OpenPose posture estimation network and output the coordinates of each skeleton key point. When constructing the GhostNet-OpenPose, the feature extraction part replaces VGG-19 with GhostNet, the OpenPose multi-branch iterative mechanism only retains the initial stage and one refinement stage, merges some convolution structures to calculate the key point heat map and some affinity fields, and uses the COCO data set as a training set; A real-time joint angle calculation module is used to calculate the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint based on the coordinates of each bone key point; A fall event recognition module is used to input the real-time joint angle of the elbow joint and the real-time joint angle of the knee joint into a decision tree fall classification algorithm to identify a fall event, wherein the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the knee joint are used as training sets to train the decision tree fall classification algorithm, the historical data of the joint angle of the elbow joint and the historical data of the joint angle of the elbow joint are calculated based on the historical data of the coordinates of each skeleton key point output by the GhostNet-OpenPose posture estimation network, and the historical data of the coordinates of the skeleton key point are calculated based on the historical data of the aspect ratio of the bounding box of the human body output by the target detection model.

6. A fall event recognition device, characterized in that: include: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the fall event recognition method according to any one of claims 1 to 4 based on instructions stored in the memory.

7. A computer storable medium, characterized in that: Computer program instructions are stored thereon, and when the instructions are executed by a processor, the fall event recognition method as described in any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • All-day unaccompanying ward patient falling detection method based on video monitoring

    CN114495280A

  • Intelligent medicine delivery inspection robot for old-age care institutions

    CN117565076A

  • An eco-friendly bubbling bath bomb

    KR102717513B1

  • Falling behaviour detection method and device

    WO2021227874A1