Fall detection method and system based on infrared and laser sensors

Abstract

The present application relates to the technical field of fall detection, in particular to a fall detection method and system based on infrared sensors and laser sensors; wherein the system comprises a laser sensor module; an infrared image acquisition module; a data preprocessing module; a human posture recognition module and a fall discrimination module; the detection method comprises the following steps, the fall detection system, characterized by comprising the following steps, acquiring the infrared image of the human posture and the distance information of the human activity; the acquired human posture infrared image is preprocessed by using the retinexformer algorithm, and the preprocessed data set is labeled; the acquired data set is trained in the yolov8 model to obtain a weight file, update the model configuration parameters, and obtain a trained detection model; the trained detection model is added to the posture evaluation of the laser, and the fall and the fall posture are distinguished by the distance threshold and the distance change rate.

Description

Technical Field

[0001] This invention relates to the field of fall detection technology, and in particular to a fall detection method and system based on infrared sensors and laser sensors. Background Technology

[0002] In recent years, fall detection among the elderly has become an important research area, driving the development of many detection methods. Based on the need for wearable devices, these methods can be broadly divided into two categories. The first category relies on wearable devices, primarily using physical sensors such as accelerometers and gyroscopes to acquire human posture information to determine if a fall has occurred. This type of method has simple hardware but suffers from limitations such as the need to wear the device, battery life, and size. The second category of fall detection methods uses non-wearable devices, primarily monitoring via visible light cameras. This method does not rely on wearable devices but raises privacy concerns. Both existing mainstream fall detection methods have shortcomings and are relatively limited in their specific detection approaches. Summary of the Invention

[0003] To address the aforementioned technical issues, this patent researches the infrared characteristics of human falls and designs a fall detection system hardware and an improved YOLOv8 fall detection algorithm based on a laser sensor. Data augmentation is implemented using the Retinex Former algorithm, significantly improving detection accuracy. Specifically, it provides a fall detection system based on infrared and laser sensors, including...

[0004] Laser sensor module: The threshold unit of this sensor monitors whether there is a person in the room in real time. When a person is detected, the sensor sends an interrupt signal to wake up the microcontroller. The microcontroller turns on the load switch to activate the infrared sensor to monitor whether a fall has occurred.

[0005] Infrared image acquisition module: used to acquire human infrared image data in real time and transmit the acquired human image data to the server and embedded platform via WiFi;

[0006] Data preprocessing module: preprocesses the acquired human infrared image data based on the Retinex Transformer network; wherein the data preprocessing module includes a Retinex Transformer module, an encoder-decoder structure, a residual module, an adaptive weight module, an image reconstruction module, and a multi-scale processing module;

[0007] Human pose recognition module: The YOLOv8 detection model is used to analyze human pose data and obtain the pose of the current target. The module uses CSPDarkNet feature extraction module as the backbone network, C3STR module based on swintransformer to improve the efficiency of collecting context information and capturing global features, SPPCSPC module to optimize human pose classification, GAM feature fusion module as the neck network, Dwconv module as the depthwise separable convolution, and prediction module as the head network.

[0008] Fall detection module: It distinguishes between falls and fall postures by using distance thresholds and distance change rates to make a fall judgment.

[0009] Furthermore, the data processing module is also connected to a storage module, which is connected to a USB-TypeC interface. The USB-TypeC interface is connected to a PC to upload infrared image data to the cloud detection module.

[0010] According to another aspect of the present invention, a fall detection method based on an infrared sensor and a laser sensor is also provided, comprising the following steps:

[0011] Step 1: Obtain infrared images of human posture and distance information of human movement;

[0012] Step 2: The acquired human pose image is preprocessed using the Retinex Former algorithm. The original image is decomposed into two parts: illumination and reflection components. Then, the human body's rim is enhanced in the reflection component, background noise is suppressed, and the overall brightness of the illumination component is improved. The preprocessed dataset is then labeled.

[0013] Step 3: Use the acquired dataset to train the YOLOv8 model, obtain the weight file, update the model configuration parameters, and obtain the trained detection model;

[0014] Step 4: Add laser posture assessment to the trained detection model, and distinguish between falls and fall postures by using distance thresholds and distance change rates.

[0015] Furthermore, the specific steps for processing the human posture infrared image and human distance data in step 1 are as follows:

[0016] Step 1.1: Acquire an infrared image of the human body posture using the MLX90640 infrared sensor; this sensor generates a 32*24 infrared thermal image by detecting the infrared radiation emitted by the human body, which provides the spatial distribution of the human body surface temperature.

[0017] Step 1.2: Using bilinear interpolation, the original image resolution of 32*24 is increased to 160*120, improving image quality. The specific formula is as follows:

[0018]

[0019] In the formula: I(x,y) is the target interpolated pixel, (x1,y1) and (x2,y2) are the positions of the four neighboring pixels, and Δx and Δy are the horizontal and vertical intervals;

[0020] Step 1.3: Apply the weighted median filtering algorithm to smooth the bilinear interpolated image; remove noise and preserve image details by weighting and sorting the pixel values ​​in the neighborhood and taking the median. Each pixel is updated using the weighted median value of its neighborhood pixels, as shown in the following formula:

[0021] I filtered (x,y)=median(w1·I(x1,y1),w2·I(x2,y2),...)

[0022] In the formula: w i As the weight, I(x) i ,y i ) represents the neighboring pixels, I filtered (x,y) is the output after weighted median filtering;

[0023] Step 1.4: Use a laser sensor (VL53L1x) to acquire distance information between the human body and the sensor, and capture the changes in the relative position of the human body and the sensor in real time; the laser sensor periodically acquires distance data of the human body, that is, dynamic data of the change in the relative distance between the human body and the sensor over time.

[0024] Step 1.5: Perform Draw(t) smoothing on the distance data acquired by the laser sensor using Kalman filtering. The Kalman filter is a recursive filtering method that can effectively estimate the system state and suppress noise, thereby improving the smoothness of the data. The smoothing process includes attitude updates and observation updates.

[0025] The attitude update formula is:

[0026]

[0027] P k|k-1 =A·P k-1|k-1 ·A T +Q

[0028] Observation update formula:

[0029] H k =P k|k-1 ·HT ·(H·P k|k-1 ·H T +R) -1

[0030]

[0031] P k|k =(IK k ·H)·P k|k-1

[0032] In the formula: It is the filtered estimated value, P k|k It is the estimated error covariance, K k H is the Kalman gain, H is the observation matrix, and Q and R are the process noise and observation noise, respectively.

[0033] Furthermore, in step 2, the Retinexformer network combines Retinex theory with the transform architecture, uses an encoder and decoder to extract and reconstruct features from the image, introduces short-circuit connections through residual modules to solve the gradient vanishing problem in deep networks, and uses adaptive weights to optimize the separation process of reflection and illumination components.

[0034] Furthermore, the specific steps for preprocessing the acquired human pose image in step 2 are as follows:

[0035] Step 2.1: Estimate the illumination components of the image using Gaussian blur; this process removes image details, retaining only the illumination components; Gaussian blur is a commonly used smoothing operation in image processing. The core idea is to use a Gaussian kernel to convolve the image to simulate its illumination components.

[0036]

[0037] Where σ controls the intensity of the blur, and (x,y) is the position of the pixel;

[0038] Step 2.2: Perform a convolution operation using a convolutional layer (Conv) and a Gaussian kernel to smooth the image and obtain the illuminated portion L(x,y):

[0039] L = Gaussian_Blur(I resized )

[0040] I resized : Resized image;

[0041] Step 2.3: According to Retinex theory, the input image I(x,y) is represented as the product of the reflection component R(x,y) and the illumination component L(x,y):

[0042] I(x,y)=R(x,y)×L(x,y)

[0043] The reflection component and the illumination component can be obtained using the following formula:

[0044]

[0045] In practice, to avoid division by zero errors, a small constant ∈ is usually added to the illumination component L(x,y) to ensure computational stability.

[0046]

[0047] Step 2.4: Calculate the reflection component R(x,y) using L(x,y) obtained from the illumination estimation part and the original image I(x,y) and enhance reflection and illumination.

[0048] In this stage, image quality is improved by enhancing or suppressing specific color areas. Emphasis is placed on enhancing human figures and suppressing environmental factors, while also adjusting the brightness of illuminated areas.

[0049] Step 2.5: Treatment of human and environmental areas

[0050] Human body region enhancement: By creating a mask M for the human body region P This will increase the brightness of the human body area in the reflective component.

[0051] Environment region suppression: By creating a mask M for the environment region E This reduces the brightness of the ambient area in the reflective component.

[0052] Enhancement formula: Enhance the reflective component:

[0053] R enhanced =R×(1+M) P ×α P )

[0054] Where α P It is an enhancing factor.

[0055] Suppressing reflective components:

[0056] R suppressed =R×(1-M) E ×α E )

[0057] Where α E It is an inhibitory factor, usually set to a small value (such as 0.4).

[0058] Step 2.6: In the HSV color space, create human and environment masks and enhance or suppress the reflection component R(x,y).

[0059] Adjusting the brightness of the light component: Enhanced = L × β, where β is the light brightness enhancement factor.

[0060] Step 2.7: Image Fusion; In this stage, the enhanced reflection component and illumination component are fused to obtain the final image.

[0061] Image fusion formula: According to Retinex theory, the final Ifused image is the product of the reflection component and the illumination component:

[0062] I fused (x,y)=R enhanced (x,y)×L enhanced (x,y)

[0063] The enhanced reflection and illumination components are multiplied pixel by pixel to generate the final image. The pixel values ​​of the image are limited to the range [0,1] to ensure that the image is neither overexposed nor underexposed.

[0064] Furthermore, the posture recognition algorithm described in step 4 distinguishes between falling and falling postures by using distance thresholds and distance change rates;

[0065] Step 4.1: Periodically acquire ranging data D from the laser sensor laser (t) is used to capture real-time changes in the distance between the human body and the device.

[0066] Step 4.2: Calculate the distance change rate; using the continuous distance information D provided by the laser sensor. laser (t), calculate the rate of change of distance between the human body and the device at each time step. Let Δt be the time interval between two adjacent data acquisitions, then the rate of change of distance V laser (t) can be calculated using differences:

[0067]

[0068] In the formula: V laser (t) represents the rate of change of distance at time t.

[0069] Step 4.3: Based on the distance data and rate of change detected by the laser sensor, the direction of the fall (front / back / side) can be inferred. The specific judgment rules are as follows:

[0070] Falling forward / backward:

[0071] Falling backward: When the distance D detected by the laser sensor laser(t) increases rapidly, indicating that the person is falling backward. As the person falls, the distance measured by the device will continue to increase, especially when the distance between the head or back and the device increases rapidly.

[0072] Forward fall: If the distance detected by the laser sensor first decreases and then increases, it indicates that the person has fallen forward.

[0073] Side fall:

[0074] In a side fall, the change in distance between the body and the equipment is less dramatic than in a forward or backward fall, typically manifesting as a slow or moderate rate of change in distance. In this case, the fall is relatively slow, usually involving a moderate rate of distance change.

[0075] (V laser (t)) max >V threhold D laser (t)↑ indicates a backward fall.

[0076] (V laser (t)) min <-V' threhold D laser (t)↓↑ indicates falling backward.

[0077] V laser (t) is within a certain range and the rate remains relatively stable, and D laser (t) If the slope is relatively gentle, it is judged as a side fall.

[0078] Step 4.4: The laser sensor provides auxiliary information to further assess the fall posture based on YOLO detection. When the laser sensor detects a small rate of distance change and a very slow rate of height change, it may indicate that the person has spontaneously fallen to their prone or lying position. In this case, further confirmation of the fall type is needed by combining infrared image information.

[0079] Furthermore, the pose recognition algorithm employs the YOLOv8 detection model, which uses the CSPDarkNet feature extraction module as the backbone network, the C3STR module based on the swin transformer to improve the efficiency of collecting contextual information and capturing global features, and the SPPCSPC module to optimize human pose classification. It also employs the GAM feature fusion module as the neck network, the Dwconv module of depthwise separable convolution, and the prediction module as the head network.

[0080] These improvements help increase network speed and accuracy, ensuring effective target detection even on resource-constrained devices. Existing fall detection technologies generally rely on single sensors without multi-feature fusion, resulting in poor product robustness. This patent utilizes infrared sensor detection to protect home privacy and adds a laser sensor with a FOV detection field of view to assist in detecting indoor human movement. Because the device is installed in a corner, it can effectively detect changes in the height of the human body from the ground and changes in distance during movement. Fall detection is achieved through imaging of the human body in different postures using the infrared sensor, and fused fall detection is achieved by combining the height information of the human body from the ceiling. This patent, using both infrared and laser sensors, is more accurate than single-sensor detection. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of the system hardware structure of the invention;

[0082] Figure 2 This is a flowchart of a fall posture recognition process that integrates infrared and laser sensors;

[0083] Figure 3 This is a system deployment structure diagram;

[0084] Figure 4 It is a flowchart of the system operation;

[0085] Figure 5 This is a diagram of the improved Yolov8 network model structure;

[0086] Figure 6 This is the flow of the YOLOv8 algorithm;

[0087] Figure 7 This is a schematic diagram of the preprocessing process for the acquired infrared images;

[0088] Figure 8 This is a diagram showing the equipment's installation location and the inspection viewpoint;

[0089] Figure 9 This is a diagram showing the specific detection perspective of the infrared sensor. Detailed Implementation

[0090] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0091] refer to Figure 1-2 This invention provides a fall detection system based on infrared and laser sensors, comprising:

[0092] Laser sensor module: The threshold unit of this sensor monitors whether there is a person in the room in real time. When a person is detected, the sensor sends an interrupt signal to wake up the microcontroller. The microcontroller turns on the load switch to activate the infrared sensor to monitor whether a fall has occurred.

[0093] Infrared image acquisition module: used to acquire human infrared image data in real time and transmit the acquired human image data to the server and embedded platform via WiFi;

[0094] Data preprocessing module: preprocesses the acquired human infrared image data based on the Retinex Transformer network; wherein the data preprocessing module includes a Retinex Transformer module, an encoder-decoder structure, a residual module, an adaptive weight module, an image reconstruction module, and a multi-scale processing module;

[0095] Human pose recognition module: The YOLOv8 detection model is used to analyze human pose data and obtain the pose of the current target. The module uses CSPDarkNet feature extraction module as the backbone network, C3STR module based on swintransformer to improve the efficiency of collecting context information and capturing global features, SPPCSPC module to optimize human pose classification, GAM feature fusion module as the neck network, Dwconv module as the depthwise separable convolution, and prediction module as the head network.

[0096] Fall detection module: Uses distance thresholds and distance change rate to distinguish between falls and fall postures for fall detection; Hardware deployment module: Used to port GPU-trained models to the NVIDIA Jetson TX2 development board for inference.

[0097] The specific design scheme of the fall detection system hardware is as follows: Figure 1 As shown, the system is designed with the 32-bit microcontroller STM32F407ZGT6 as its core, which improves the system's processing capabilities.

[0098] The MLX90640 infrared sensor was used to acquire infrared images of the posture of a fallen human body, and the VL53L1X laser rangefinder was used to acquire distance information of the human body.

[0099] Since the most power-consuming unit in a fall detection system is the infrared sensor, such as Figure 1A GLF1111 load switch is designed at the front end of the infrared sensor to manage its power supply. When the laser sensor detects no one in the scene, the load switch shuts off the infrared sensor's power, saving energy; the microcontroller enters sleep mode and does not interact with information, further reducing system power consumption. When the laser sensor detects someone in the scene, it performs detection using a threshold-based algorithm and simultaneously sends an interrupt signal to wake up the microcontroller. The microcontroller is then awakened and enters working mode, turning on the infrared sensor via the load switch for real-time human fall detection. If the laser sensor does not detect anyone within a certain period, the system shuts off the infrared sensor via the load switch, and the microcontroller re-enters sleep mode.

[0100] The fall detection system is connected to an internet cloud platform using the ESP8266 WIFI module, and the collected information is stored using the GD25Qx storage module. The monitored data can also be connected to a personal computer (PC) via a USB-Type-C interface for data storage. The system is powered by a Type-C interface and a lithium battery, and uses an LDO step-down chip BL8064 for voltage reduction and a BAT54s for voltage protection.

[0101] The system deployment structure diagram is as follows: Figure 3 As shown, the system's operation flowchart is as follows: Figure 4 As shown, an infrared sensor acquires infrared images of the human body, and a microcontroller determines in real time whether someone has fallen indoors. If a fall is detected in the cloud, the cloud will promptly send the detection result to the caregiver's mobile app, enabling the elderly to receive timely assistance. A laser sensor is used to assist in detecting the human body's movement posture.

[0102] Based on the above system, the present invention provides a fall detection method based on infrared sensors and laser sensors, comprising the following steps:

[0103] Step 1: Acquire infrared images of human posture using MLX90640, and improve image resolution and quality using bilinear interpolation and weighted median filtering; acquire distance information of human activity using a laser sensor, and perform Kalman filtering on the data detected by the laser sensor to improve data smoothness.

[0104] Step 2: The acquired human pose image is preprocessed using the Retinex Former algorithm. The original image is decomposed into two parts: illumination and reflection components. Then, the human body's rim is enhanced in the reflection component, background noise is suppressed, and the overall brightness of the illumination component is improved. The preprocessed dataset is then labeled.

[0105] Step 3: Use the acquired dataset to train the model in YOLOv8, obtain the weight file, update the model configuration parameters, obtain the trained model, and use the trained model for fall detection.

[0106] The specific network structure for improving the YOLOv8 algorithm is as follows: Figure 5 As shown, the Transformer architecture was optimized based on the original YOLOv8 algorithm. A win transformer was added to the backbone network to adapt to the characteristics of visual data; SPPCSPC was added to the backbone network to improve the model's multi-scale feature representation ability and computational efficiency. In the network head structure, the conv was replaced with DWconv to reduce the model's computational cost and number of parameters, while improving the model's efficiency and speed. An additional detection head was added to the three detection heads of v8, and a GAM-Attention mechanism was added after each detection head to improve the ability to detect small objects and improve detection accuracy. The improved network includes comprehensive enhancement strategies. These modules enhance the network's ability to capture and extract features related to fall detection.

[0107] The specific process of applying the improved YOLOv8 algorithm is as follows: Figure 6 As shown, the application process is divided into two stages: training and inference. First, the images from the obtained training dataset are input into the Retinex Former for augmentation. Then, the original and augmented images are sent to the designed YOLO v8 model for training. After training, the model weight file is obtained. To create an offline model that can be used with embedded devices, this paper uses cross-compilation to prune a portion of the slicing operator and quantizes the model. During the inference stage, the images are directly transmitted to the embedded platform via a WIFI module. Then, bilinear interpolation, recursive average filtering, and weighted median filtering are applied to process the acquired images. Finally, the transformed images are sent to the embedded Jetson TX2 processor to obtain the final detection results.

[0108] Step 4: Add laser posture assessment to the trained detection model, and distinguish between falls and fall postures by using distance thresholds and distance change rates;

[0109] The installation location and detection area of ​​the fall detection system are as follows: Figure 7 As shown, the specific detection angle of the infrared sensor is as follows: Figure 8 As shown, the fall posture recognition algorithm that integrates infrared sensors and laser sensors is as follows: Figure 9 As shown.

[0110] A method for fall detection using laser sensors to assist infrared sensors: combining human motion tracks

[0111] Traces and the rate of change of height can be used to infer the direction of a fall; the direction of a fall can be reflected to some extent by the rate of change of height.

[0112] Furthermore, the specific steps for processing the human posture infrared image and human distance data in step 1 are as follows:

[0113] Step 1.1: Acquire an infrared image of the human body posture using the MLX90640 infrared sensor; this sensor generates a 32*24 infrared thermal image by detecting the infrared radiation emitted by the human body, which provides the spatial distribution of the human body surface temperature.

[0114] Step 1.2: Using bilinear interpolation, the original image resolution of 32*24 is increased to 160*120, improving image quality. The specific formula is as follows:

[0115]

[0116] In the formula: I(x,y) is the target interpolated pixel, (x1,y1) and (x2,y2) are the positions of the four neighboring pixels, and Δx and Δy are the horizontal and vertical intervals;

[0117] Step 1.3: Apply the weighted median filtering algorithm to smooth the bilinear interpolated image; remove noise and preserve image details by weighting and sorting the pixel values ​​in the neighborhood and taking the median. Each pixel is updated using the weighted median value of its neighborhood pixels, as shown in the following formula:

[0118] I filtered (x,y)=median(w1·I(x1,y1),w2·I(x2,y2),...)

[0119] In the formula: w i As the weight, I(x) i ,y i ) represents the neighboring pixels, I filtered (x,y) is the output after weighted median filtering;

[0120] Step 1.4: Use a laser sensor (VL53L1x) to acquire distance information between the human body and the sensor, and capture the changes in the relative position of the human body and the sensor in real time; the laser sensor periodically acquires distance data of the human body, that is, dynamic data of the change in the relative distance between the human body and the sensor over time.

[0121] Step 1.5: Perform Draw(t) smoothing on the distance data acquired by the laser sensor using Kalman filtering. The Kalman filter is a recursive filtering method that can effectively estimate the system state and suppress noise, thereby improving the smoothness of the data. The smoothing process includes attitude updates and observation updates.

[0122] The attitude update formula is:

[0123]

[0124] P k|k-1 =A·P k-1|k-1 ·A T +Q

[0125] Observation update formula:

[0126] H k =P k|k-1 ·H T ·(H·P k|k-1 ·H T +R) -1

[0127]

[0128] P k|k =(IK k ·H)·P k|k-1

[0129] In the formula: It is the filtered estimated value, P k|k It is the estimated error covariance, K k H is the Kalman gain, H is the observation matrix, and Q and R are the process noise and observation noise, respectively.

[0130] Furthermore, in step 2, the Retinexformer network combines Retinex theory with the transform architecture, utilizing encoders and decoders to...

[0131] Infrared images are used for feature extraction and reconstruction. Short-circuit connections are introduced through residual modules to solve the gradient vanishing problem in deep networks. Adaptive weights are used to optimize the separation process of reflection and illumination components.

[0132] Furthermore, the specific steps for preprocessing the acquired human pose image in step 2 are as follows:

[0133] Step 2.1: Estimate the illumination components of the image using Gaussian blur; this process removes image details, retaining only the illumination components; Gaussian blur is a commonly used smoothing operation in image processing. The core idea is to use a Gaussian kernel to convolve the image to simulate its illumination components.

[0134]

[0135] Where σ controls the intensity of the blur, and (x,y) is the position of the pixel;

[0136] Step 2.2: Perform a convolution operation using a convolutional layer (Conv) and a Gaussian kernel to smooth the image and obtain the illuminated portion L(x,y):

[0137] L = Gaussian_Blur(I resized )

[0138] I resized : Resized image;

[0139] Step 2.3: According to Retinex theory, the input image I(x,y) is represented as the product of the reflection component R(x,y) and the illumination component L(x,y):

[0140] I(x,y)=R(x,y)×L(x,y)

[0141] The reflection component and the illumination component can be obtained using the following formula:

[0142]

[0143] In practice, to avoid division by zero errors, a small constant ∈ is usually added to the illumination component L(x,y) to ensure computational stability.

[0144]

[0145] Step 2.4: Calculate the reflection component R(x,y) using L(x,y) obtained from the illumination estimation part and the original image I(x,y) and enhance reflection and illumination.

[0146] In this stage, image quality is improved by enhancing or suppressing specific color areas. Emphasis is placed on enhancing human figures and suppressing environmental factors, while also adjusting the brightness of illuminated areas.

[0147] Step 2.5: Treatment of human and environmental areas

[0148] Human body region enhancement: By creating a mask M for the human body region P This will increase the brightness of the human body area in the reflective component.

[0149] Environment region suppression: By creating a mask M for the environment region E This reduces the brightness of the ambient area in the reflective component.

[0150] Enhancement formula: Enhance the reflective component:

[0151] R enhanced =R×(1+M) P ×α P)

[0152] Where α P It is an enhancing factor.

[0153] Suppressing reflective components:

[0154] R suppressed =R×(1-M) E ×α E )

[0155] Where α E It is an inhibitory factor, usually set to a small value (such as 0.4).

[0156] Step 2.6: In the HSV color space, create human and environment masks and enhance or suppress the reflection component R(x,y).

[0157] Adjusting the brightness of the light component: Enhanced = L × β, where β is the light brightness enhancement factor.

[0158] Step 2.7: Image Fusion; In this stage, the enhanced reflection component and illumination component are fused to obtain the final image.

[0159] Image fusion formula: According to Retinex theory, the final Ifused image is the product of the reflection component and the illumination component:

[0160] I fused (x,y)=R enhanced (x,y)×L enhanced (x,y)

[0161] The enhanced reflection and illumination components are multiplied pixel by pixel to generate the final image. The pixel values ​​of the image are limited to the range [0,1] to ensure that the image is neither overexposed nor underexposed.

[0162] Furthermore, the posture recognition algorithm described in step 4 distinguishes between falling and falling postures by using distance thresholds and distance change rates;

[0163] Step 4.1: Periodically acquire ranging data D from the laser sensor laser (t) is used to capture real-time changes in the distance between the human body and the device.

[0164] Step 4.2: Calculate the distance change rate; using the continuous distance information D provided by the laser sensor. laser (t), calculate the rate of change of distance between the human body and the device at each time step. Let Δt be the time interval between two adjacent data acquisitions, then the rate of change of distance V laser (t) can be calculated using differences:

[0165]

[0166] In the formula: V laser (t) represents the rate of change of distance at time t.

[0167] Step 4.3: Based on the distance data and rate of change detected by the laser sensor, the direction of the fall (front / back / side) can be inferred. The specific judgment rules are as follows:

[0168] Falling forward / backward:

[0169] Falling backward: When the distance D detected by the laser sensor laser (t) increases rapidly, indicating that the person is falling backward. As the person falls, the distance measured by the device will continue to increase, especially when the distance between the head or back and the device increases rapidly.

[0170] Forward fall: If the distance detected by the laser sensor first decreases and then increases, it indicates that the person has fallen forward.

[0171] Side fall:

[0172] In a side fall, the change in distance between the body and the equipment is less dramatic than in a forward or backward fall, typically manifesting as a slow or moderate rate of change in distance. In this case, the fall is relatively slow, usually involving a moderate rate of distance change.

[0173] (V laser (t)) max >V threhold D laser (t)↑ indicates a backward fall.

[0174] (V laser (t)) min <-V' threhold D laser (t)↓↑ indicates falling backward.

[0175] V laser (t) is within a certain range and the rate remains relatively stable, and D laser (t) If the slope is relatively gentle, it is judged as a side fall.

[0176] Step 4.4: The laser sensor provides auxiliary information to further assess the fall posture based on YOLO detection. When the laser sensor detects a small rate of distance change and a very slow rate of height change, it may indicate that the person has spontaneously fallen to their prone or lying position. In this case, further confirmation of the fall type is needed by combining infrared image information.

[0177] Furthermore, the pose recognition algorithm employs a YOLOv8 detection model, which utilizes a CSPDarkNet feature extraction module as the backbone network, a C3STR module based on a swin transformer to improve the efficiency of collecting contextual information and capturing global features, and an SPPCSPC module to optimize human pose classification. It also employs a GAM feature fusion module as the neck network, a depthwise separable convolutional module (Dwconv), and a prediction module as the head network. These improvements help to increase the network's speed and accuracy, thereby ensuring effective target detection even on resource-constrained devices.

[0178] Fall detection algorithm module: It uses laser threshold analysis combined with dynamic data of the relative distance between the human body and the body over time to determine the fall posture;

[0179] The hardware deployment module is used to port the trained model to an embedded platform for inference.

[0180] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fall detection system based on infrared sensors and laser sensors, characterized in that, include Laser sensor module: This sensor monitors the indoor environment in real time by using its threshold unit. When someone is detected, the sensor sends an interrupt signal to wake up the microcontroller. The microcontroller then turns on the load switch to activate the infrared sensor to monitor for falls. Infrared image acquisition module: used to acquire human infrared image data in real time and transmit the acquired human image data to the server and embedded platform via WiFi; Data preprocessing module: Based on the Retinexformer network, the acquired human infrared image data is preprocessed to obtain significant human posture data; Human pose recognition module: The YOLOv8 detection model is used to analyze human pose data and obtain the pose of the current target. The module uses CSPDarkNet feature extraction module as the backbone network, C3STR module based on swin transformer to improve the efficiency of collecting context information and capturing global features, SPPCSPC module to optimize human pose classification, GAM feature fusion module as the neck network, Dwconv module as the depthwise separable convolutional network, and prediction module as the head network. Fall detection module: This module distinguishes between falls and fall postures using distance thresholds and the rate of distance change, thus determining whether a fall has occurred. Details are as follows: Periodically acquire ranging data D from the laser sensor laser (t), used to capture real-time changes in the distance between the human body and the device; Calculate the rate of distance change; use continuous distance information D provided by a laser sensor. laser (t), calculate the rate of change of distance between the human body and the device at each time step; let (t). Given the time interval between two consecutive data acquisitions, the distance change rate is... It can be calculated using differences: ; In the formula: Let be the rate of change of distance at time t; Based on the distance data and rate of change detected by the laser sensor, the direction of the fall is inferred: forward, backward, or sideways. The specific rules are as follows: Falling backward: When the distance D detected by the laser sensor... laser (t) A rapid increase indicates that the person is falling backward; when the person falls, the distance measured by the device will continue to increase, especially when the distance between the head or back and the device increases rapidly; Forward fall: if the distance detected by the laser sensor first decreases and then increases, it indicates that the person has fallen forward; In the case of a side fall, the change in distance between the person and the device is not as drastic as in a forward or backward fall, usually showing a slow or moderate rate of change in distance; at this time, the speed of the fall is relatively slow, usually a moderate rate of distance change; , This is assessed as falling backward; , This is then determined to be a backward fall; Within a certain range and with the rate remaining relatively stable, and If the fall is relatively gentle, it is considered a side fall; When the distance change rate detected by the laser sensor is small and the height change is very slow, it is necessary to combine infrared image information for further confirmation of the fall type.

2. The fall detection system based on infrared and laser sensors as described in claim 1, characterized in that, The data preprocessing module is also connected to a storage module, which is connected to a USB-TypeC interface. The USB-TypeC interface is connected to a PC to upload infrared image data to the cloud detection module.

3. The fall detection system based on infrared sensors and laser sensors as described in claim 1, characterized in that, The data preprocessing module includes a Retinex Transformer module, an encoder-decoder structure, a residual module, an adaptive weighting module, an image reconstruction module, and a multi-scale processing module.

4. A fall detection method based on infrared sensors and laser sensors, based on the fall detection system according to any one of claims 1-3, characterized in that, Includes the following steps, Step 1: Obtain infrared images of human posture and distance information of human movement; Step 2: Preprocess the acquired human pose infrared images using the Retinex Former algorithm, and label the preprocessed dataset. Step 3: Use the acquired dataset to train the YOLOv8 model, obtain the weight file, update the model configuration parameters, and obtain the trained detection model; Step 4: Add laser posture assessment to the trained detection model, and distinguish between falls and fall postures by using distance thresholds and distance change rates.

5. The fall detection method based on an infrared sensor and a laser sensor as described in claim 4, characterized in that, Step 1: Process the infrared image of human posture and the human distance data. The specific steps are as follows: Step 1.1: Acquire a thermal image of the human body posture using the MLX90640 infrared sensor; this sensor generates a raw infrared thermal image with a resolution of 32 * 24 by detecting the infrared radiation emitted by the human body, which provides the spatial distribution of the human body surface temperature; Step 1.2: Perform bilinear interpolation on the original infrared thermal image using a bilinear interpolation algorithm. The specific formula is as follows: ; In the formula: I(x,y) is the target interpolated pixel, (x1,y1) and (x2,y2) are the positions of the four neighboring pixels, and Δx and Δy are the horizontal and vertical intervals; Step 1.3: Apply the weighted median filtering algorithm to smooth the bilinear interpolated image; remove noise and preserve image details by weighting and sorting the pixel values ​​in the neighborhood and taking the median. Each pixel is updated using the weighted median value of its neighborhood pixels, as shown in the following formula: ; In the formula: w i As the weight, I(x) i ,y i ) represents the neighboring pixels, I filtered (x,y) is the output after weighted median filtering; Step 1.4: Use a laser sensor to obtain distance information between the human body and the sensor, and capture the changes in the relative position of the human body and the sensor in real time; Step 1.5: Perform Draw(t) smoothing on the distance data acquired by the laser sensor using Kalman filtering; the smoothing process includes attitude update and observation update; The attitude update formula is: ; ; Observation update formula: ; ; ; In the formula: This is the filtered estimated value. It is the estimated error covariance. H is the Kalman gain, H is the observation matrix, and Q and R are the process noise and observation noise, respectively.

6. The fall detection method based on an infrared sensor and a laser sensor as described in claim 4, characterized in that, In step 2, the Retinexformer network combines Retinex theory with the transformer architecture, using encoders and decoders to extract and reconstruct features from infrared images. Short-circuit connections are introduced through residual modules to solve the gradient vanishing problem in deep networks, and adaptive weights are used to optimize the separation process of reflection and illumination components.

7. The fall detection method based on an infrared sensor and a laser sensor as described in claim 6, characterized in that, In step 2, the acquired infrared image of human posture is preprocessed. The original image is decomposed into two parts: illumination component and reflection component. Then, the human body's rim is enhanced in the reflection component, background noise is suppressed, and the overall brightness of the illumination component is improved. The specific steps are as follows: Step 2.1: Estimate the illumination components of the image using Gaussian blur, remove image details, and retain the illumination components; ; in, The intensity of the blur is controlled, where (x, y) is the pixel position; Step 2.2: Perform a convolution operation with a Gaussian kernel using a convolutional layer (Conv) to smooth the image and obtain the illuminated portion L(x,y): L=Gaussian_Blur(I resized ); I resized : Resized image; Step 2.3: According to Retinex theory, the input image I(x,y) is represented as the product of the reflection component R(x,y) and the illumination component L(x,y): I(x,y)=R(x,y)×L(x,y); The reflection component and the illumination component are obtained using the following formulas: ; Add a small constant to the illumination component L(x,y). Ensure computational stability: ； Step 2.4: Calculate the reflection component R(x,y) using L(x,y) obtained from the illumination estimation part and the original image I(x,y) and enhance reflection and illumination; improve the image effect by enhancing or suppressing specific color areas in the image, focusing on enhancing human body areas and suppressing environmental factors, while adjusting the brightness of the illumination part; Step 2.5: Treatment of human and environmental areas Human body region enhancement: By creating a mask M for the human body region P This will increase the brightness of the human body area in the reflective components; Environment region suppression: By creating a mask M for the environment region E This reduces the brightness of the ambient area in the reflective component; Enhancement formula: Enhance the reflective component: R enhanced =R×(1+M P ×α P ); Where, α P It is an enhancing factor; Suppressing reflective components: R suppressed =R×(1-M E ×α E ); Where, α E It is an inhibitory factor, usually set to a small value (such as 0.4). Step 2.6: In the HSV color space, create human and environment masks and enhance or suppress the reflection component R(x,y); Adjusting the brightness of the light component: Enhanced = L × β, where β is the light brightness enhancement factor; Step 2.7: Image Fusion; In this stage, the enhanced reflection component and illumination component are fused to obtain the final image; Image fusion formula: According to Retinex theory, the final Ifused image is the product of the reflection component and the illumination component. I fused (x,y)=R enhanced (x,y)×L enhanced (x,y); The enhanced reflection component and the illumination component are multiplied pixel by pixel to generate the final image; the pixel values ​​of the image are limited to the range of [0,1].

8. The fall detection method based on an infrared sensor and a laser sensor as described in claim 4, characterized in that, The pose recognition algorithm uses the YOLOv8 detection model, which employs the CSPDarkNet feature extraction module as the backbone network, the C3STR module based on the swin transformer to improve the efficiency of collecting context information and capturing global features, and the SPPCSPC module to optimize human pose classification. It also employs the GAM feature fusion module as the neck network, the Dwconv module of depthwise separable convolution, and the prediction module as the head network. Fall detection algorithm module: It uses laser threshold analysis combined with dynamic data of the relative distance between the human body and the body over time to determine the fall posture.

Images