A Human Fall Detection Device and Method Based on Dual-Light Perception Information Fusion

Abstract

The present invention discloses a human fall detection device and method based on dual-light perception information fusion, which relates to the technical field of human behavior detection. It aims to solve the problems in existing target detection technology, such as visible light images being greatly affected by lighting; infrared images having low contrast, not rich feature information, and high noise; and the large amount of computational complexity of existing human posture recognition models. An adaptive algorithm based on light intensity is proposed to fuse visible light images with infrared thermal imaging information at the feature level, and to screen out candidate human target areas through temperature-width masks, thereby reducing the algorithm's computational complexity, improving the model's recognition accuracy and real-time performance, and effectively realizing the recognition and detection of human postures in nursing scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of human behavior detection, and more specifically to a human fall detection device and method based on dual-light perception information fusion. Background Art

[0002] China is entering an aging society. Due to rapid social development, the number of elderly people living alone or in isolation is increasing at an unprecedented rate. In recent years, there has been a surge in deaths among elderly people living alone due to accidental falls. Studies have shown that timely assistance after a fall can significantly reduce the risk of death and long-term hospitalization.

[0003] Existing fall detection systems use wearable sensors and visual cameras to monitor falls. However, wearable devices suffer from issues like poor battery life and poor comfort, and elderly people tend to forget to wear them. Visual cameras can effectively address these shortcomings. Fall detection systems currently available are typically based on visible light cameras, which pose privacy risks and are unsuitable for dimly lit or overexposed environments. Our system utilizes a dual-light camera, primarily using infrared thermal imaging, to acquire sensor information. Using deep learning methods, we detect abnormal falls and alert caregivers to provide timely assistance. This system operates 24 / 7, eliminates privacy concerns, and offers significant application value for home care. Summary of the Invention

[0004] The present invention can improve the model recognition accuracy and real-time performance, and effectively realize the recognition and detection of human body posture in nursing scenarios.

[0005] In order to achieve the above objectives, the following technical solutions are specifically adopted: a human fall detection device based on dual-light perception information fusion, including a dual-light camera, a terminal processor and an alarm device;

[0006] The dual-light camera is installed on the upper part of the wall in the room at a horizontal angle of -15°, overlooking the monitoring area. At the same time, the dual-light camera is connected to the terminal processor via a network, and the captured video stream is transmitted to the terminal processor. The deep learning model deployed in the terminal processor monitors the received video stream in real time according to the detection algorithm process. After detecting abnormal fall information, an action instruction is sent to the alarm device, and the alarm device issues an alarm after receiving the action instruction.

[0007] In order to achieve the above-mentioned purpose, a detection method of a human fall detection device based on dual-light sensing information fusion is also included, comprising the following steps:

[0008] Step S1: Setting a fixed temperature range [15°C, 40°C], collecting human posture video images from multiple angles and multiple occasions through the dual-light camera, and simultaneously acquiring infrared data and visible light data of the same time series, which include multiple human postures;

[0009] Step S2: The infrared data is subjected to a median filter operator to remove noise, and the infrared image data is subjected to a nonlinear grayscale transformation operator to map the grayscale of the image to a new interval to increase the image contrast;

[0010] Step S3: Extract edge information of the visible light image data through the Scharr operator to obtain the characteristic image of the visible light image data, and then read the average brightness value L of the visible light image data. mean , and normalize it to get the brightness influence factor γ:

[0011]

[0012] The feature image I of the visible light image data is transformed into rgb With infrared image data I r Perform feature-level fusion:

[0013] I=I r +γI rgb ;

[0014] Step S4: Label the fused image data and set three categories: fall, stand, and sit. Label strictly according to posture angles, and set the training set, validation set, and test set according to the ratio of 8:1:1 for the multiple image data.

[0015] Step S5: The present invention adopts the CenterNet network model, which mainly adopts the anchor-free detection model;

[0016] Step S6: Binary encode the output of the model. When the detection result is the fall state, the output value is x=1. When the detection result is the sit or stand state, the output value is x=0. Encode all the detection results for one frame of image and add the output values:

[0017] sum=x1+x2+...+x i

[0018] Based on the following judgment conditions, if sum ≥ 1, the fall information flag is transmitted to the control channel and the timer starts. If the flag information is continuously received within ts, an action command is issued to the alarm device to notify relevant personnel that an abnormal fall behavior has occurred in the monitoring area:

[0019]

[0020] Further: Step S5 includes the following steps:

[0021] Step S51: First, resize the input data into a size of 512×512, use ResNet50 as the feature extraction backbone network, add a temperature-width mask to the feature image with an output size of (16, 16, 2048) to obtain the possible target area;

[0022] Step S52: Then, upsampling is performed using cubic deconvolution to obtain a high-resolution feature image of size (128, 128, 64). The feature image divides the original image into 128×128 regions, where the masked region is the candidate region. Each candidate region has a feature point. If the center point of the human body falls in a region, the feature point in the upper left corner of the region is used as the replacement.

[0023] Step S53: Finally, the process is carried out through three branches, including heat map prediction, center point prediction, and width and height prediction.

[0024] Furthermore: the heat map prediction is a Heatmap heat map prediction, which sets the number of human posture categories c=3, and generates a three-channel heat map through a 3×3+1×1 convolution kernel. The heat map of each channel represents a human posture category, and the heat point on the heat map of each channel represents whether a human posture of that category appears.

[0025] Further: the center point prediction is Reg center point prediction, the number of output channels is 2, representing the offset of each target from the thermal point, thereby obtaining the offset of the current feature point corresponding to the predicted point.

[0026] Furthermore: the width and height prediction is WH width and height prediction, the output channel is 2, representing the width and height prediction of each target, and obtaining the width and height of the prediction box corresponding to the current feature point.

[0027] Further: the temperature width mask is based on the human body temperature width information (T min ,T max ) Set a mask to obtain possible human target areas;

[0028] Temperature range: T i ={T1, T2..., T n}

[0029] Image grayscale value range: h min ~h max

[0030] max||Ti ||-min||T i ||=n

[0031] h max -h min =s

[0032] Gray value area range corresponding to each °C: m = s / n

[0033] h min +n*m=min||T i ||+n

[0034] According to research, under standard atmospheric pressure, the surface temperature of various parts of the human body is distributed between 33.2℃ and 36.4℃. By fitting the relationship between the surface temperature of clothing and the surface temperature of the human body, it is concluded that the temperature of the human body area wearing light clothing is distributed between 26.7℃ and 36.4℃, and the grayscale area is in the range of 120-230. A mask is set according to the grayscale and added to the backbone network output feature image to retain possible human target areas.

[0035] The beneficial effects of the present invention are as follows:

[0036] 1: A feature-level fusion algorithm based on adaptive illumination intensity is proposed. It enhances the edge information of infrared images and improves the robustness of target detection in different lighting environments. It selects candidate human target areas through temperature-width masks, improves the real-time performance of the model, and uses the CenterNet network model to classify and detect data, determine human behavior, and effectively realize the recognition and detection of human posture in nursing scenarios.

[0037] 2: A temperature-width mask mechanism is proposed. The mask range is set according to the temperature-width information of the human body to obtain a rough human target area, reduce the calculation complexity of the model, and facilitate the judgment of human posture.

[0038] 3: It can monitor the monitoring area in real time, and when someone falls, it can promptly issue an audible and visual alarm on the spot, and send a message to the relevant personnel to remind them. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic flow chart of the method of the present invention;

[0040] Figure 2 It is a schematic diagram of the algorithm flow of the present invention;

[0041] Figure 3 Schematic diagram of the fitting function of temperature width and gray value in the present invention;

[0042] Figure 4 This is a schematic diagram of possible human body regions extracted by using the human body temperature width information mask in the present invention;

[0043] Figure 5 This is a schematic diagram of the fusion and comparison of natural light image data and infrared image data in the present invention;

[0044] Figure 6 This is a diagram of the CenterNet network structure;

[0045] Figure 7 It is a schematic diagram of the system structure of the fall detection device. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. It should be understood that the described embodiments are only a portion of the embodiments of the present invention, not all of them. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

[0047] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0048] like Figure 6 As shown, the present invention provides a human fall detection device based on dual-light perception information fusion, including a dual-light camera, a terminal processor and an alarm device;

[0049] The dual-light camera is installed on the upper part of a wall in the room at a horizontal angle of -15°, overlooking the monitoring area. At the same time, the dual-light camera is connected to the terminal processor via a network, and the captured video stream is transmitted to the terminal processor. A trained neural network model is set in the terminal processor, and the received video stream is monitored in real time according to the detection algorithm process. After detecting abnormal fall information, an action instruction is sent to the alarm device, and the alarm device issues an alarm after receiving the action instruction.

[0050] like Figure 1-Figure 5 As shown, the present invention also provides a detection method of a human fall detection device based on dual-light perception information fusion, comprising the following steps:

[0051] Step S1: Setting a fixed temperature range [15°C, 40°C], collecting human posture video images from multiple angles and multiple occasions through the dual-light camera, and simultaneously acquiring infrared data and visible light data of the same time series, which include multiple human postures;

[0052] Step S2: The infrared data is subjected to a median filter operator to remove noise, and the infrared image data is subjected to a nonlinear grayscale transformation operator to map the grayscale of the image to a new interval to increase the image contrast;

[0053] Step S3: Extract edge information of the visible light image data through the Scharr operator to obtain the characteristic image of the visible light image data, and then read the average brightness value L of the visible light image data. mean , and normalize it to get the brightness influence factor γ:

[0054]

[0055] The feature image I of the visible light image data is transformed into rgb With infrared image data I r Perform feature-level fusion:

[0056] I=I r +γI rgb ;

[0057] Step S4: Label the fused image data and set three categories: fall, stand, and sit. Label strictly according to posture angles, and set the training set, validation set, and test set according to the ratio of 8:1:1 for the multiple image data.

[0058] Step S5: The present invention adopts the CenterNet network model, which mainly adopts the anchor-free detection model; first, the input data is resized to 512×512 size, and ResNet50 is used as the feature extraction backbone network. The temperature and width mask is added to the feature image with an output size of (16, 16, 2048) to obtain the possible target area; then, upsampling is performed using cubic deconvolution to obtain a high-resolution feature image with a size of (128, 128, 64). The feature image divides the original image into 128×128 regions, where the mask area is the candidate area. Each candidate area has a feature point. If the center point of the human body falls in an area, it is replaced by the feature point in the upper left corner of the area; finally, it is performed through three branches, including heat map prediction, center point prediction, and width and height prediction.

[0059] Step S6: Binary encode the output of the model. When the detection result is the fall state, the output value is x=1. When the detection result is the sit or stand state, the output value is x=0. Encode all the detection results for one frame of image and add the output values:

[0060] sum=x1+x2+…+xi

[0061] Based on the following judgment conditions, if sum ≥ 1, the fall information flag is transmitted to the control channel and the timer starts. If the flag information is continuously received within ts, an action command is issued to the alarm device to notify relevant personnel that an abnormal fall behavior has occurred in the monitoring area:

[0062]

[0063] The heat map prediction is Heatmap heat map prediction, which sets the number of human posture categories c=3, and generates a three-channel heat map through a 3×3+1×1 convolution kernel. The heat map of each channel represents a human posture category, and the heat point on the heat map of each channel represents whether a human posture of that category appears; the center point prediction is Reg center point prediction, and the number of output channels is 2, which represents the offset of each target from the heat point, thereby obtaining the offset of the current feature point corresponding to the predicted point; the width and height prediction is WH width and height prediction, and the output channel is 2, which represents the width and height prediction of each target, and obtains the width and height of the prediction box corresponding to the current feature point.

[0064] The temperature width mask is based on the human body temperature width information (T min ,T max ) Set a mask to obtain possible human target areas;

[0065] Temperature range: T i ={T1, T2, T n}

[0066] Image grayscale value range: h min ~h max

[0067] max||T i ||-min||T i ||=n

[0068] h max -h min =s

[0069] Gray value area range corresponding to each °C: m = s / n

[0070] h min +n*m=min||T i ||+n

[0071] According to research, under standard atmospheric pressure, the surface temperature of various parts of the human body is distributed between 33.2℃ and 36.4℃. By fitting the relationship between the surface temperature of clothing and the surface temperature of the human body, it is concluded that the temperature of the human body area wearing light clothing is distributed between 26.7℃ and 36.4℃, and the grayscale area is in the range of 120-230. A mask is set according to the grayscale and added to the backbone network output feature image to retain possible human target areas.

Claims

1. A detection method for a human fall detection device based on dual-light sensing information fusion, characterized in that: The human fall detection device based on dual-light sensing information fusion includes a dual-light camera, a terminal processor and an alarm device; The dual-light camera is installed on the upper part of the wall in the room at a horizontal angle of -15 degrees, overlooking the monitoring area. At the same time, the dual-light camera is connected to the terminal processor via a network, and the captured video stream is transmitted to the terminal processor. The deep learning model deployed in the terminal processor monitors the received video stream in real time according to the detection algorithm process. After detecting abnormal fall information, an action instruction is sent to the alarm device, and the alarm device issues an alarm after receiving the action instruction; The detection method of a human fall detection device based on dual-light sensing information fusion includes the following steps: Step S1: Setting a fixed temperature range [15°C, 40°C], collecting human posture video images from multiple angles and multiple occasions through the dual-light camera, and simultaneously acquiring infrared data and visible light data of the same time series, which include multiple human postures; Step S2: The infrared data is subjected to a median filter operator to remove noise, and the infrared image data is subjected to a nonlinear grayscale transformation operator to map the grayscale of the image to a new interval to increase the image contrast; Step S3: Extract edge information of visible light image data through Scharr operator to obtain feature image of visible light image data, and then read the average brightness value of visible light image data. , and normalize it to get the brightness impact factor : ； The feature image of visible light image data is transformed into With infrared image data Perform feature-level fusion: ； Step S4: Label the fused image data and set three categories: fall, stand, and sit. Label them strictly according to the posture angles, and set the training set, validation set, and test set according to the ratio of 8:1:1 for the multiple image data. Step S5: using the CenterNet network model, which uses an anchor-free detection model; Step S6: Binary encode the output of the model. When the detection result is fall state, the output value is ,When the detection result is sit or stand state, the output value is , encode all detection results of a frame image and add the output values: ； According to the following judgment conditions, if sum≥1, the fall information flag is transmitted to the control channel and the timing is started. If flag information is continuously received within seconds, an action instruction will be sent to the alarm device to remind relevant personnel that abnormal falling behavior occurs in the monitoring area: 。 2. The detection method of a human fall detection device based on dual-light sensing information fusion according to claim 1, characterized in that: The step S5 comprises the following steps: Step S51: First, resize the input data into a size of 512×512, use ResNet50 as the feature extraction backbone network, add a temperature-width mask to the feature image with an output size of (16, 16, 2048), and obtain the possible target area; Step S52: Then, upsampling is performed using cubic deconvolution to obtain a high-resolution feature image of size (128, 128, 64). The feature image divides the original image into 128×128 regions, where the mask region is the candidate region. Each candidate region has a feature point. If the center point of the human body falls in a region, the feature point in the upper left corner of the region is used to replace it. Step S53: Finally, the process is carried out through three branches, including heat map prediction, center point prediction, and width and height prediction.

3. The detection method of a human fall detection device based on dual-light sensing information fusion according to claim 2, characterized in that: The heat map prediction is a heatmap prediction, which sets the number of human posture categories c=3, and generates a three-channel heat map through a 3×3+1×1 convolution kernel. The heat map of each channel represents a human posture category, and the heat point on the heat map of each channel represents whether a human posture of that category appears.

4. The detection method of a human fall detection device based on dual-light sensing information fusion according to claim 2, characterized in that: The center point prediction is Reg center point prediction, and the output channel is 2, which represents the offset of each target from the thermal point, thereby obtaining the offset of the current feature point corresponding to the predicted point.

5. The detection method of a human fall detection device based on dual-light sensing information fusion according to claim 2, characterized in that: The width and height prediction is WH width and height prediction, and the output channel is 2, which represents the width and height prediction of each target, and obtains the width and height of the prediction box corresponding to the current feature point.

Images