Fall detection method and system based on infrared sensor and laser sensor

Abstract

The invention relates to the technical field of tumble detection, in particular to a tumble detection method and system based on an infrared sensor and a laser sensor. The system comprises a laser sensor module; an infrared image acquisition module; a data preprocessing module; the detection method comprises the following steps: the tumble detection system is characterized by comprising the following steps: obtaining the distance information between the infrared image of the human body posture and the human body activity; the acquired human body posture infrared image is preprocessed by using a retinexformer algorithm, and a preprocessed data set is labeled; training in a yolov8 model by using the obtained data set to obtain a weight file, and updating model configuration parameters to obtain a trained detection model; and combining the trained detection model with laser attitude evaluation, and distinguishing tumble and tumble attitude through a distance threshold value and a distance change rate.

Description

Technical Field

[0001] The present invention relates to the technical field of fall detection, and particularly relates to a fall detection method and system based on an infrared sensor and a laser sensor. Background Art

[0002] In recent years, the detection of falls in the elderly has become an important research field, which has promoted the development of many detection methods. According to the requirements for wearable devices, these methods can be roughly divided into two categories. The first category is the detection method relying on wearable devices, which mainly obtains human body posture information through physical sensors such as accelerometers and gyroscopes to determine whether a fall occurs; such detection methods have simple hardware, but there are problems such as the need to be worn on the body, device battery life, and device volume. The second category of fall detection methods is non-wearable devices, which mainly monitor through visible light cameras. This method does not rely on wearing but involves privacy protection issues. The existing two mainstream fall detection methods have deficiencies and the detection methods are relatively single. Summary of the Invention

[0003] To solve the above technical problems, this patent conducts research on the infrared characteristics of human falls, designs the hardware of the fall detection system and improves the YOLOv8 fall detection algorithm based on a laser sensor; realizes data enhancement based on the retinexformer algorithm, greatly improving the detection accuracy. Specifically, a fall detection system based on an infrared sensor and a laser sensor is provided, including,

[0004] Laser sensor module: The threshold unit of this sensor is used to monitor in real time whether there is someone in the room. When someone is detected, the sensor sends an interrupt signal to wake up the single-chip microcomputer, and the single-chip microcomputer turns on the load switch to activate the infrared sensor to monitor whether a fall occurs;

[0005] Infrared image acquisition module: It is used to collect human infrared image data in real time and transmit the collected human image data to the server and the embedded platform through WiFi;

[0006] Data preprocessing module: Preprocess the collected human infrared image data based on the Retinexformer network; the data preprocessing module includes a Retinex Transformer module, an encoder-decoder structure, a residual module, an adaptive weight module, an image reconstruction module, and a multi-scale processing module;

[0007] Human pose recognition module: The YOLOv8 detection model is used to analyze human pose data to obtain the pose of the current target. Among them, the CSPDarkNet feature extraction module used as the backbone network, the C3STR module based on swintransformer are adopted to improve the efficiency of collecting context information and capturing global features, and the SPPCSPC module is used to optimize human pose classification. The GAM feature fusion module and the depthwise separable convolution Dwconv module used as the neck network and the prediction module used as the head network are adopted;

[0008] Fall discrimination module: Use the distance threshold and the rate of change of distance to distinguish between falls and fall postures for fall judgment.

[0009] Furthermore, the data processing module is also connected to a storage module. The storage module is connected to the USB-TypeC interface, and the USB-TypeC interface is connected to the computer PC side to upload the infrared image data to the cloud detection module.

[0010] According to another aspect of the present invention, a fall detection method based on an infrared sensor and a laser sensor is also provided, including the following steps:

[0011] Step 1: Obtain the infrared image of the human pose and the distance information of the human activity;

[0012] Step 2: Use the retinexformer algorithm to preprocess the obtained human pose image. The original image is decomposed into two parts according to the illumination component and the reflection component. Subsequently, the human body hub is enhanced and the background noise is suppressed in the reflection component part, and the overall brightness of the illumination component is increased. Then, the preprocessed data set is labeled.

[0013] Step 3: Use the obtained data set to train in the yolov8 model to obtain a weight file, update the model configuration parameters, and obtain a trained detection model;

[0014] Step 4: Add the pose evaluation of the laser to the trained detection model, and distinguish between falls and fall postures by means of a distance threshold and a rate of change of distance.

[0015] Furthermore, the specific steps of processing the human pose infrared image and the human distance data in Step 1 are as follows:

[0016] Step 1.1: Obtain the infrared image of the human pose through the MLX90640 infrared sensor; this sensor generates a 32*24 infrared thermal image by detecting the infrared radiation emitted by the human body, and this image provides the spatial distribution of the human body surface temperature;

[0017] Step 1.2: Use the bilinear interpolation algorithm to increase the resolution of the original image from 32*24 to 160*120, improving the quality of the picture. The specific formula is as follows:

[0018]

[0019] In the formula: I(x,y) is the target interpolation pixel, (x1,y1) and (x2,y2) are the positions of the four adjacent pixels, and Δx and Δy are the intervals in the horizontal and vertical directions;

[0020] Step 1.3: Apply the weighted median filtering algorithm to smooth the image after bilinear interpolation; by performing weighted sorting on the pixel values in the neighborhood and taking the median to remove noise and retain image details, each pixel is updated with the weighted median value of the pixels in its neighborhood. The formula is as follows:

[0021] I filtered (x,y) = median(w1·I(x1,y1), w2·I(x2,y2),...)

[0022] In the formula: w i is the weight, I(x i ,y i ) are the neighborhood pixels, and I filtered (x,y) is the output after weighted median filtering;

[0023] Step 1.4: Use a laser sensor (VL53L1x) to obtain the distance information between the human body and the sensor, and capture the relative position change between the human body and the sensor in real time; the laser sensor periodically obtains the distance data of the human body, that is, the dynamic data of the relative distance between the human body and the sensor changing with time.

[0024] Step 1.5: Smooth the distance data collected by the laser sensor through Kalman filtering. The Kalman filter is a recursive filtering method that can effectively estimate the system state and suppress noise, thereby improving the smoothness of the data. The said smoothing process includes attitude update and observation update.

[0025] Among them, the attitude update formula is:

[0026]

[0027] P k|k-1 = A·P k-1|k-1 ·A T + Q

[0028] Observation update formula:

[0029] H k = P k|k-1 ·HT ·(H·P k|k-1 ·H T +R) -1

[0030]

[0031] P k|k =(I - K k ·H)·P k|k-1

[0032] In the formula: is the estimated value after filtering, and P k|k is the estimated error covariance, K k is the Kalman gain, H is the observation matrix, and Q and R are the process noise and the observation noise respectively.

[0033] Further, in step 2, the Retinexformer network combines the Retinex theory with the transform architecture, extracts features and reconstructs the image using the encoder and the decoder, introduces a short - circuit connection through the residual module to solve the vanishing gradient problem in the deep network, and optimizes the separation process of the reflection and illumination components using the adaptive weights.

[0034] Further, the specific steps for pre - processing the obtained human body pose image in step 2 are as follows:

[0035] Step 2.1: Estimate the illumination component of the image through Gaussian blur; this process can remove the details of the image and only retain the illumination part; Gaussian blur is a commonly used smoothing operation in image processing, and its core idea is to convolve the image with a Gaussian kernel to simulate the illumination component of the image:

[0036]

[0037] where σ controls the intensity of the blur, and (x, y) is the position of the pixel;

[0038] Step 2.2: Use a convolutional layer (Conv) to perform a convolution operation with the Gaussian kernel to smooth the image to obtain the illumination part L(x, y):

[0039] L = Gaussian_Blur(I resized )

[0040] I resized : The resized image;

[0041] Step 2.3: According to the Retinex theory, the input image I(x, y) is expressed as the product of the reflection component R(x, y) and the illumination component L(x, y):

[0042] I(x,y) = R(x,y) × L(x,y)

[0043] The reflected component and the illumination component can be obtained by the following formula:

[0044]

[0045] In actual operation, to avoid division-by-zero errors, a small constant ∈ is usually added to the illumination component L(x,y) to ensure computational stability:

[0046]

[0047] Step 2.4: Calculate the reflected component R(x,y) using L(x,y) obtained from the illumination estimation part and the original image I(x,y), and enhance the reflectance and illumination

[0048] At this stage, the image effect is improved by enhancing or suppressing specific color regions in the image. Emphasis is placed on enhancing the human body region and suppressing environmental factors, while adjusting the brightness of the illumination part.

[0049] Step 2.5: Processing of the human body and environmental regions

[0050] Enhancement of the human body region: By creating a mask M of the human body region P , the brightness of the human body region in the reflected component is increased.

[0051] Suppression of the environmental region: By creating a mask M of the environmental region E , the brightness of the environmental region in the reflected component is decreased.

[0052] Enhancement formula: Enhance the reflected component:

[0053] R enhanced = R × (1 + M P × α P )

[0054] where α P is the enhancement factor.

[0055] Suppress the reflected component:

[0056] R suppressed = R × (1 - M E × α E )

[0057] where α E is the suppression factor, usually set to a small value (such as 0.4).

[0058] Step 2.6: In the HSV color space, create human body and environment masks, and enhance or suppress the reflection component R(x, y).

[0059] Adjust the brightness of the illumination component: Lenhanced = L × β, where β is the illumination brightness enhancement factor.

[0060] Step 2.7: Image fusion; at this stage, fuse the enhanced reflection component and illumination component to obtain the final image.

[0061] Image fusion formula: According to the Retinex theory, the final image Ifused is the product of the reflection component and the illumination component:

[0062] I fused (x, y) = R enhanced (x, y) × L enhanced (x, y)

[0063] Multiply the enhanced reflection component and illumination component pixel by pixel to generate the final image. Limit the pixel values of the image to be within [0, 1] to ensure that the image is not overexposed or underexposed.

[0064] Furthermore, the posture recognition algorithm described in Step 4 distinguishes between falling and falling postures by means of a distance threshold and a distance change rate;

[0065] Step 4.1: Regularly obtain the ranging data D laser (t) of the laser sensor to capture the real-time change of the distance between the human body and the device.

[0066] Step 4.2: Calculate the distance change rate; calculate the distance change rate between the human body and the device at each time step through the continuous distance information D laser (t) provided by the laser sensor. Let Δt be the time interval between two adjacent data acquisitions, then the distance change rate V laser (t) can be calculated by difference:

[0067]

[0068] In the formula: V laser (t) is the distance change rate at time t

[0069] Step 4.3: Based on the distance data and change rate detected by the laser sensor, the direction of falling (front / back / side) can be inferred. The specific judgment rules are as follows:

[0070] Forward / backward fall:

[0071] Backward fall: When the distance D detected by the laser sensor laser(t) Rapid increase indicates that the human body falls backward. When the human body falls, the measured distance by the device will continue to increase, especially when the distance between the head or back and the device increases rapidly.

[0072] Forward fall: If the distance detected by the laser sensor first decreases and then increases, it indicates that the human body has a forward fall.

[0073] Side fall:

[0074] In the case of a side fall, the change in the distance between the human body and the device is not as drastic as that of a forward or backward fall, usually showing a slow change or a change at a medium rate of the distance. At this time, the falling speed is slower, usually a change in distance at a medium rate.

[0075] (V laser (t)) max >V threhold ,D laser (t)↑ Then it is evaluated as a backward fall

[0076] (V laser (t)) min <-V' threhold ,D laser (t)↓↑ Then it is determined as a backward fall

[0077] V laser (t) is within a certain range and the rate remains relatively stable, and D laser (t) is relatively gentle Then it is determined as a side fall

[0078] Step 4.4: What the laser sensor provides is an auxiliary condition, which is to further evaluate the fall posture based on the yolo detection. When the rate of change of the distance detected by the laser sensor is small and the height change is very slow, it may be that the human body lies down or lies prone by itself. At this time, it is necessary to combine the infrared image information for further confirmation of the fall type.

[0079] Furthermore, the pose recognition algorithm uses the YOLOv8 detection model, in which the CSPDarkNet feature extraction module used as the backbone network, the C3STR module based on the swin transformer to improve the efficiency of collecting context information and capturing global features, and the SPPCSPC module to optimize human pose classification, the GAM feature fusion module and the depthwise separable convolution Dwconv module used as the neck network, and the prediction module used as the head network.

[0080] These improvements help to improve the speed and accuracy of the network, thus ensuring effective object detection even on devices with limited resources. Existing fall detection technologies generally only consider a single sensor and lack the fusion of multiple features, resulting in poor product robustness. This patent adopts an infrared sensor to detect and protect family privacy, and adds a laser sensor with a Fov detection perspective to assist in detecting the indoor movement of the human body. Since the device is installed in the corner, it can well detect the change in the height of the human body from the ground and the change in distance during movement. Fall detection is achieved through the imaging of different human postures in the infrared sensor, and fusion fall detection is achieved by combining the height information of the human body from the roof. This patent uses an infrared sensor and a laser sensor, which is more accurate than single-sensor detection. Description of the Drawings

[0081] Figure 1 is a schematic diagram of the system hardware structure of the invention;

[0082] Figure 2 is a flowchart for recognizing fall postures by fusing an infrared sensor and a laser sensor;

[0083] Figure 3 is a system deployment structure diagram;

[0084] Figure 4 is a flowchart of the system operation;

[0085] Figure 5 is a structure diagram of the improved Yolov8 network model;

[0086] Figure 6 is the process of the YOLOv8 algorithm;

[0087] Figure 7 is a schematic diagram of the preprocessing process of the collected infrared images;

[0088] Figure 8 is a diagram of the installation position of the device and the detection perspective;

[0089] Figure 9 is a specific detection perspective diagram of the infrared sensor. Detailed Implementation Manner

[0090] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0091] Reference Figure 1-2 , the present invention provides a fall detection system based on an infrared sensor and a laser sensor, including,

[0092] Laser sensor module: The threshold unit of this sensor monitors in real time whether there is anyone in the room. When someone is detected, the sensor sends an interrupt signal to wake up the single-chip microcomputer, and the single-chip microcomputer turns on the load switch to activate the infrared sensor to monitor whether a fall occurs;

[0093] Infrared image acquisition module: It is used to collect human infrared image data in real time and transmit the collected human image data to the server and the embedded platform via WiFi;

[0094] Data preprocessing module: Preprocess the collected human infrared image data based on the Retinexformer network; the data preprocessing module includes a Retinex Transformer module, an encoder-decoder structure, a residual module, an adaptive weight module, an image reconstruction module, and a multi-scale processing module;

[0095] Human body pose recognition module: Use the YOLOv8 detection model to analyze human body pose data and obtain the pose of the current target. Among them, the CSPDarkNet feature extraction module used as the backbone network, the C3STR module based on swintransformer are used to improve the efficiency of collecting context information and capturing global features, and the SPPCSPC module is used to optimize human body pose classification. The GAM feature fusion module and the depthwise separable convolution Dwconv module are used as the neck network, and the prediction module is used as the head network;

[0096] Fall discrimination module: Use the distance threshold and the rate of change of distance to distinguish between a fall and a fall posture for fall judgment; the hardware deployment module is used to transplant the model trained by the GPU to the NVIDIA Jetson TX2 development board for inference

[0097] The specific design scheme of the hardware of the fall detection system is as Figure 1 shown. The system is designed with the 32-bit single-chip microcomputer STM32F407ZGT6 as the core, which improves the processing ability of the system.

[0098] Use the MLX90640 infrared sensor to obtain the infrared image of the falling human body pose, and use the VL53L1X laser ranging sensor to obtain the distance information of the human body;

[0099] Since the maximum power consumption unit in the fall detection system is the infrared sensor, as Figure 1A load switch of GLF1111 is designed at the front end of the infrared sensor to manage the power supply of the infrared sensor. When the laser sensor detects that there is no one in the scene, the load switch is used to turn off the power of the infrared sensor to save energy consumption; the single-chip microcomputer is in the sleep state and does not perform information interaction, further reducing the system power consumption. When the laser sensor detects that there is someone in the scene, the laser sensor detects through a threshold-type algorithm and simultaneously sends an interrupt signal to wake up the single-chip microcomputer. The single-chip microcomputer is woken up and enters the working state, and the infrared sensor is turned on through the load switch for real-time human fall detection. If the laser sensor does not detect a person within a certain period of time, the system turns off the infrared sensor through the load switch, and the single-chip microcomputer enters the sleep mode again.

[0100] The fall detection system is connected to the Internet cloud platform using the WIFI module ESP8266, and the storage module GD25Qx is used to store the collected information. The monitored data can also be connected to a personal computer (PC) through the USB-typec interface for data storage; the system is powered by a type-c interface and a lithium battery, and at the same time, the LDO buck chip BL8064 is used for buck processing, and BAT54s is added for voltage protection.

[0101] The deployment structure diagram of the system operation is as Figure 3 shown, and the operation flow chart of the system is as Figure 4 shown. The infrared sensor obtains the human infrared image, and the single-chip microcomputer judges in real time whether someone has fallen indoors. If a fall is detected in the cloud, the cloud will send the detection result to the guardian's mobile phone APP in time, so that the elderly can get timely assistance. A laser sensor is used to assist in detecting the human motion posture.

[0102] Based on the above system, the present invention provides a fall detection method based on an infrared sensor and a laser sensor, including the following steps,

[0103] Step 1: Obtain the infrared image of the human posture through MLX90640, and use the bilinear interpolation algorithm and weighted median filtering to improve the image resolution and image quality; obtain the distance information of human activities using the laser sensor, and perform Kalman filtering on the data detected by the laser sensor to improve the data smoothness.

[0104] Step 2: Use the retinexformer algorithm to preprocess the obtained human posture image, decompose the original image into two parts according to the illumination component and the reflection component, then enhance the human hub in the reflection component part, suppress the background noise, and improve the overall brightness of the illumination component. Then annotate the preprocessed data set.

[0105] Step 3: Use the obtained dataset to train in YOLOv8 to obtain a weight file, update the model configuration parameters to get a trained model, and use the trained model for fall detection.

[0106] The specific network structure for improving the YOLOv8 algorithm is as Figure 5 shown. Based on the original YOLOv8 algorithm, the Transformer architecture is optimized. A swin transformer is added to the backbone network to adapt to the characteristics of visual data; SPPCSPC is added to the backbone network to enhance the multi-scale feature expression ability and computational efficiency of the model. In the head structure of the network, conv is replaced with DWconv to reduce the model's computational volume and the number of parameters, while improving the efficiency and speed of the model. Based on the three detection heads of v8, an additional detection head is added, and the GAM-Attention attention mechanism is added after each detection head to improve the small target detection ability and detection accuracy. The improved network includes a comprehensive enhancement strategy. These modules enhance the network's ability to capture and extract features related to fall detection.

[0107] The specific process of applying the improved YOLOv8 algorithm is as Figure 6 shown. The application process is divided into two stages: training and inference. First, the images in the obtained training dataset are input into retinexformer for enhancement. Then, the original images and the enhanced images are sent to the designed YOLO v8 model for training. After training, a model weight file is obtained. To create an offline model that can be used with embedded devices, this paper uses cross-compilation to prune a part of the slicing operator and quantize the model. In the inference stage, the images are directly transmitted to the embedded platform through the WIFI module. Then, the bilinear interpolation algorithm, recursive average filtering, and weighted median filtering are applied to process the collected images. After that, the converted images are sent to the embedded Jetson TX2 processor to obtain the final detection results.

[0108] Step 4: Add the attitude evaluation of the laser to the trained detection model, and distinguish falls and fall postures by means of distance thresholds and distance change rates;

[0109] The installation location and detection area of the fall detection system are as Figure 7 shown, and the specific detection angles of the infrared sensors are as Figure 8 shown. The fall posture recognition algorithm that fuses infrared sensors and laser sensors is as Figure 9 shown.

[0110] The method of using a laser sensor to assist an infrared sensor in fall detection: Combine the human motion trajectory

[0111] The trace and the rate of height change are used to infer the direction of a fall; the direction of the fall can be reflected to some extent by the rate of height change.

[0112] Furthermore, the specific steps of processing the infrared image of the human body posture and the human body distance data in step 1 are as follows:

[0113] Step 1.1: Obtain an infrared image of the human body posture through the MLX90640 infrared sensor; this sensor generates a 32*24 infrared thermal image by detecting the infrared radiation emitted by the human body, and this image provides the spatial distribution of the human body surface temperature;

[0114] Step 1.2: Use the bilinear interpolation algorithm to increase the resolution of the original image from 32*24 to 160*120 to improve the quality of the picture. The specific formula is as follows:

[0115]

[0116] In the formula: I(x,y) is the target interpolated pixel, (x1,y1) and (x2,y2) are the positions of the four adjacent pixels, and Δx and Δy are the intervals in the horizontal and vertical directions;

[0117] Step 1.3: Apply the weighted median filtering algorithm to smooth the image after bilinear interpolation; by performing weighted sorting on the pixel values in the neighborhood and taking the median to remove noise and retain image details, each pixel is updated through the weighted median value of the pixels in its neighborhood. The formula is as follows:

[0118] I filtered (x,y) = median(w1·I(x1,y1), w2·I(x2,y2),...)

[0119] In the formula: w i is the weight, I(x i ,y i ) are the neighborhood pixels, and I filtered (x,y) is the output after weighted median filtering;

[0120] Step 1.4: Use the laser sensor (VL53L1x) to obtain the distance information between the human body and the sensor, and capture the relative position change between the human body and the sensor in real time; the laser sensor periodically obtains the distance data of the human body, that is, the dynamic data of the relative distance between the human body and the sensor changing with time.

[0121] Step 1.5: Smooth the distance data collected by the laser sensor through Kalman filtering. The Kalman filter is a recursive filtering method that can effectively estimate the system state and suppress noise, thereby improving the smoothness of the data. The said smoothing process includes attitude update and observation update.

[0122] The attitude update formula is as follows:

[0123]

[0124] P k|k-1 = A·P k-1|k-1 ·A T + Q

[0125] Observation update formula:

[0126] H k = P k|k-1 ·H T ·(H·P k|k-1 ·H T + R) -1

[0127]

[0128] P k|k = (I - K k ·H)·P k|k-1

[0129] In the formula: is the estimated value after filtering, P k|k is the estimated error covariance, K k is the Kalman gain, H is the observation matrix, and Q and R are the process noise and observation noise respectively.

[0130] Furthermore, in step 2, the Retinexformer network combines the Retinex theory and the transform architecture, and uses the encoder and decoder to

[0131] extract features and reconstruct the infrared image, introduces a short-circuit connection through the residual module to solve the vanishing gradient problem in the deep network, and uses the adaptive weight to optimize the separation process of the reflection and illumination components.

[0132] Furthermore, the specific steps for preprocessing the obtained human body posture image in step 2 are as follows:

[0133] Step 2.1: Estimate the illumination component of the image through Gaussian blur; this process can remove the details of the image and only retain the illumination part; Gaussian blur is a common smoothing operation in image processing, and the core idea is to convolve the image with a Gaussian kernel to simulate the illumination component of the image:

[0134]

[0135] where σ controls the intensity of the blur, and (x, y) is the position of the pixel;

[0136] Step 2.2: Use a convolutional layer (Conv) to perform a convolution operation with a Gaussian kernel to smooth the image and obtain the illumination part L(x, y):

[0137] L = Gaussian_Blur(I resized )

[0138] I resized : The resized image;

[0139] Step 2.3: According to the Retinex theory, the input image I(x, y) is expressed as the product of the reflectance component R(x, y) and the illumination component L(x, y):

[0140] I(x, y) = R(x, y) × L(x, y)

[0141] The reflectance component and the illumination component can be obtained through the following formula:

[0142]

[0143] In actual operation, to avoid division by zero errors, a small constant ∈ is usually added to the illumination component L(x, y) to ensure computational stability:

[0144]

[0145] Step 2.4: Use L(x, y) obtained from the illumination estimation part and the original image I(x, y) to calculate the reflectance component R(x, y) and enhance the reflectance and illumination (Enhancement of Reflectance and Illumination)

[0146] At this stage, the image effect is improved by enhancing or suppressing specific color regions in the image. Emphasize the enhancement of the human body region and the suppression of environmental factors, and at the same time adjust the brightness of the illumination part.

[0147] Step 2.5: Processing of the human body and environmental regions

[0148] Enhancement of the human body region: By creating a mask M P , increase the brightness of the human body region in the reflectance component.

[0149] Suppression of the environmental region: By creating a mask M E , reduce the brightness of the environmental region in the reflectance component.

[0150] Enhancement formula: Enhance the reflectance component:

[0151] R enhanced = R × (1 + M P × α P)

[0152] where α P is the enhancement factor.

[0153] Suppress the reflection component:

[0154] R suppressed = R × (1 - M E × α E )

[0155] where α E is the suppression factor, usually set to a small value (such as 0.4).

[0156] Step 2.6: In the HSV color space, create a human body and environment mask, and enhance or suppress the reflection component R(x, y).

[0157] Adjust the brightness of the illumination component: Lenhanced = L × β where β is the illumination brightness enhancement factor.

[0158] Step 2.7: Image fusion; at this stage, fuse the enhanced reflection component and illumination component to obtain the final image.

[0159] Image fusion formula: According to the Retinex theory, the final image Ifused is the product of the reflection component and the illumination component:

[0160] I fused (x, y) = R enhanced (x, y) × L enhanced (x, y)

[0161] Multiply the enhanced reflection component and illumination component pixel by pixel to generate the final image. Limit the pixel value range of the image to [0, 1] to ensure that the image is not overexposed or underexposed.

[0162] Furthermore, the pose recognition algorithm described in Step 4 distinguishes between a fall and a falling pose by means of a distance threshold and a distance change rate;

[0163] Step 4.1: Regularly obtain the ranging data D laser (t) of the laser sensor to capture the real-time change in the distance between the human body and the device.

[0164] Step 4.2: Calculate the distance change rate; calculate the distance change rate between the human body and the device at each time step through the continuous distance information D laser (t) provided by the laser sensor. Let Δt be the time interval between two adjacent data acquisitions, then the distance change rate V laser (t) can be calculated by finite difference:

[0165]

[0166] Where: V laser (t) is the rate of change of distance at time t

[0167] Step 4.3: Based on the distance data and change rate detected by the laser sensor, the direction of the fall (front / back / side) can be inferred. The specific judgment rules are as follows:

[0168] Front / back fall:

[0169] Falling backwards: When the laser sensor detects the distance D laser (t) increases rapidly, indicating that the person is falling backwards. When a person falls, the distance measured by the device will continue to increase, especially when the distance between the head or back and the device increases rapidly.

[0170] Falling forward: If the distance detected by the laser sensor first decreases and then increases, it means that the person has fallen forward.

[0171] Side fall:

[0172] In the case of a side fall, the change in the distance between the human body and the device is not as dramatic as a front-to-back fall, usually manifested as a slow change in distance or a change at a medium rate. At this time, the speed of the fall is slow, usually with a medium rate of distance change.

[0173] (V laser (t)) max >V threhold , D laser (t)↑ is considered as falling backwards

[0174] (V laser (t)) min <-V' threhold , D laser (t)↓↑ is considered as falling backwards

[0175] V laser (t) is within a certain range and the rate remains relatively stable, and D laser (t) If it is relatively gentle, it is considered a side fall.

[0176] Step 4.4: The laser sensor provides auxiliary conditions to further assess the fall posture based on the yolo detection. When the distance change rate detected by the laser sensor is small and the height changes very slowly, it may be that the person has fallen or lay down on his own. At this time, it is necessary to combine the infrared image information to further confirm the fall type.

[0177] Further, the pose recognition algorithm uses the YOLOv8 detection model, which employs the CSPDarkNet feature extraction module as the backbone network, the C3STR module based on the swin transformer to improve the efficiency of collecting context information and capturing global features, and the SPPCSPC module to optimize human pose classification, and uses the GAM feature fusion module and the depthwise separable convolution Dwconv module as the neck network and the prediction module as the head network; these improvements help to improve the speed and accuracy of the network, thus ensuring effective object detection even on devices with limited resources.

[0178] Fall discrimination algorithm module: uses the method of laser threshold analysis combined with the dynamic data of the relative distance of the human body changing over time to judge the fall pose;

[0179] Hardware deployment module, used to transplant the trained model to an embedded platform for inference.

[0180] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A fall detection system based on infrared sensors and laser sensors, characterized in that: include Laser sensor module: The threshold unit of the sensor is used to monitor in real time whether there is someone in the room. When someone is detected, the sensor sends an interrupt signal to wake up the microcontroller. The microcontroller turns on the load switch and turns on the infrared sensor to monitor whether a fall has occurred. Infrared image acquisition module: used to collect infrared image data of the human body in real time, and transmit the collected human image data to the server and embedded platform via WiFi; Data preprocessing module: Based on the Retinexformer network, the collected human infrared image data is preprocessed to obtain significant human posture data; Human posture recognition module: The YOLOv8 detection model is used to analyze the human posture data and obtain the posture of the current target. The CSPDarkNet feature extraction module is used as the backbone network, the C3STR module based on the swin transformer is used to improve the efficiency of collecting context information and capturing global features, and the SPPCSPC module is used to optimize the human posture classification. The GAM feature fusion module and the deep separable convolution Dwconv module are used as the neck network, and the head network prediction module is used; Fall identification module: Use distance threshold and distance change rate to distinguish between falls and falling postures, and make fall judgments.

2. A fall detection system based on infrared sensor and laser sensor as claimed in claim 1, characterized in that: The data processing module is also connected to a storage module, which is connected to a USB-TypeC interface, and the USB-TypeC interface is connected to a computer PC to upload the infrared image data to the cloud detection module.

3. A fall detection system based on infrared sensor and laser sensor as claimed in claim 1, characterized in that: The data preprocessing module includes a Retinex Transformer module, an encoder-decoder structure, a residual module, an adaptive weight module, an image reconstruction module, and a multi-scale processing module.

4. A fall detection method based on an infrared sensor and a laser sensor, based on the fall detection system according to any one of claims 1 to 3, characterized in that: The following steps are included: Step 1: Obtain infrared images of human body posture and distance information of human activities; Step 2: Use the RetinExformer algorithm to preprocess the acquired human posture infrared image and annotate the preprocessed data set; Step 3: Use the acquired data set to train the yolov8 model, obtain the weight file, update the model configuration parameters, and obtain the trained detection model; Step 4: Combine the trained detection model with the laser posture assessment to distinguish between falls and falling postures by means of distance threshold and distance change rate.

5. A fall detection method based on infrared sensor and laser sensor as claimed in claim 4, characterized in that: Step 1: Process the human body posture infrared image and human body distance data. The specific steps are as follows: Step 1.1: Get a thermal image of the human body posture through the MLX90640 infrared sensor; the sensor generates a raw infrared thermal image with a resolution of 32*24 by detecting the infrared radiation emitted by the human body. The image provides the spatial distribution of the surface temperature of the human body; Step 1.2: Use the bilinear interpolation algorithm to perform bilinear interpolation processing on the original infrared thermal image. The specific formula is: Where: I(x,y) is the target interpolation pixel, (x1,y1) and (x2,y2) are the four adjacent pixel positions, Δx and Δy are the intervals in the horizontal and vertical directions; Step 1.3: Apply the weighted median filtering algorithm to smooth the bilinear interpolated image; by weighted sorting of the pixel values ​​in the neighborhood, taking the median to remove noise and retain image details, each pixel is updated by the weighted median value of the pixels in its neighborhood. The formula is as follows: I filtered (x,y)=median(w1·I(x1,y1),w2·I(x2,y2),...) Where: w i is the weight, I(x i ,y i ) is the neighborhood pixel, I filtered (x,y) is the output after weighted median filtering; Step 1.4: Use a laser sensor to obtain the distance information between the human body and the sensor, and capture the relative position changes between the human body and the sensor in real time; Step 1.5: Perform Draw(t) smoothing processing on the distance data collected by the laser sensor through Kalman filtering; the smoothing processing includes attitude update and observation update; The posture update formula is: P k|k-1 =A·P k-1|k-1 ·A T +Q Observation update formula: H k =P k|k-1 ·H T ·(H·P k|k-1 ·H T +R) -1 P k|k =(I-K k ·H)·P k|k-1 Where: is the estimated value after filtering, P k|k is the estimated error covariance, K k is the Kalman gain, H is the observation matrix, Q and R are the process noise and observation noise respectively.

6. A fall detection method based on infrared sensor and laser sensor as claimed in claim 4, characterized in that: In step 2, the Retinexformer network combines the Retinex theory with the transform architecture, uses the encoder and decoder to extract and reconstruct features of the infrared image, introduces short-circuit connections through the residual module to solve the gradient vanishing problem in the deep network, and uses adaptive weights to optimize the separation process of reflection and illumination components.

7. A fall detection method based on infrared sensor and laser sensor as claimed in claim 6, characterized in that: In step 2, the acquired infrared image of human posture is preprocessed, and the original image is decomposed into two parts according to the illumination component and the reflection component. Then, the human body hub is enhanced in the reflection component, the background noise is suppressed, and the overall brightness of the illumination component is improved. The specific steps are as follows: Step 2.1: Estimate the illumination component of the image through Gaussian blur, remove the details of the image, and retain the illuminated part; Where σ controls the intensity of the blur, (x,y) is the position of the pixel; Step 2.2: Use a convolution layer (Conv) to perform a convolution operation with a Gaussian kernel to smooth the image and obtain the illuminated part L(x,y): L=Gaussian_Blur(I resized ) I resized : resized image; Step 2.3: According to Retinex theory, the input image I(x,y) is represented as the product of the reflection component R(x,y) and the illumination component L(x,y): I(x,y)=R(x,y)×L(x,y) The reflection component and illumination component can be obtained by the following formula: Add a small constant ∈ to the lighting component L(x,y) to ensure computational stability: Step 2.4: Use L(x,y) obtained from the illumination estimation part and the original image I(x,y) to calculate the reflection component R(x,y) and enhance the reflection and illumination; improve the image effect by enhancing or suppressing the specific color areas in the image, focusing on the enhancement of the human body area and the suppression of environmental factors, and adjusting the brightness of the illuminated part; Step 2.5: Treatment of human and environmental areas Human region enhancement: By creating a mask M of the human region P , increase the brightness of the human body area in the reflection component; Ambient region suppression: By creating a mask M of the ambient region E , reduce the brightness of the ambient area in the reflection component; Enhancement formula: Enhance the reflection component: R enhanced =R×(1+M P ×α P ) Among them, α P is the enhancement factor; Suppress the reflection component: R suppressed =R×(1-M E ×α E ) Among them, α E is the suppression factor, usually set to a small value (such as 0.4); Step 2.6: Create human body and environment masks in the HSV color space, and enhance or suppress the reflection component R(x,y); Adjust the brightness of the lighting component: Lenhanced = L × β, where β is the light brightness enhancement factor; Step 2.7: Image fusion; in this stage, the enhanced reflection component and illumination component are fused to obtain the final image; Image fusion formula: According to Retinex theory, the final image Ifused is the product of the reflection component and the illumination component: I fused (x,y)=R enhanced (x,y)×L enhanced (x,y) The enhanced reflection component and illumination component are multiplied pixel by pixel to generate the final image; the pixel value range of the image is limited to [0,1].

8. A fall detection method based on infrared sensor and laser sensor as claimed in claim 4, characterized in that: In step 4, posture assessment is performed through a posture recognition algorithm, and the fall and falling posture are distinguished by means of distance threshold and distance change rate; specifically, as follows: Step 4.1: Periodically obtain the distance data D of the laser sensor laser (t), used to capture the real-time change of the distance between the human body and the device; Step 4.2: Calculate the distance change rate; the continuous distance information D provided by the laser sensor laser (t), calculate the distance change rate between the human body and the device at each time step; let Δt be the time interval between two adjacent data collections, then the distance change rate V laser (t) can be calculated by difference: Where: V laser (t) is the rate of change of distance at time t; Step 4.3: Based on the distance data and change rate detected by the laser sensor, the direction of the fall (front / back / side) can be inferred; the specific judgment rules are as follows: Falling backward: When the distance D detected by the laser sensor is laser (t) Rapid increase, indicating that the person fell backwards; when the person falls, the distance measured by the device will continue to increase, especially when the distance between the head or back and the device increases rapidly; Falling forward: If the distance detected by the laser sensor decreases first and then increases, it means that the person has fallen forward; in the case of a side fall, the change in the distance between the person and the device is not as drastic as a front-to-back fall, usually manifested as a slow change in distance or a change at a medium rate; in this case, the speed of the fall is slow, usually with a change in distance at a medium rate; (V laser (t)) max >V threhold , D laser (t)↑ is assessed as falling backwards; (V laser (t)) min <-V' threhold , D laser (t)↓↑ is considered as falling backwards; V laser (t) is within a certain range and the rate remains relatively stable, and D laser (t) If it is relatively gentle, it is considered a side fall; Step 4.4: When the distance change rate detected by the laser sensor is small and the height changes very slowly, it is necessary to further confirm the fall type in combination with the infrared image information.

9. A fall detection method based on infrared sensor and laser sensor as claimed in claim 8, characterized in that: The posture recognition algorithm adopts the YOLOv8 detection model, in which the CSPDarkNet feature extraction module as the backbone network, the C3STR module based on the swin transformer is used to improve the efficiency of collecting context information and capturing global features, and the SPPCSPC module is used to optimize the human posture classification, and the GAM feature fusion module and the deep separable convolution Dwconv module as the neck network and the head network prediction module are used; Fall identification algorithm module: Use laser threshold analysis combined with dynamic data of human body relative distance changing over time to identify the fall posture.

Images