A weak light real-time human fall detection method based on infrared and thermal imaging

Abstract

The purpose of this invention is to provide a real-time human fall detection method in low-light conditions based on infrared and thermal imaging. This method combines infrared compensation and thermal imaging technologies for fall detection in low-light environments, achieving better results than previous fall detection methods. It overcomes the problem of blurred human body contours and blurring with the environment in dark scenes when captured by infrared night vision. Thermal imaging technology utilizes the principle that all objects in nature radiate heat; when used for nighttime human detection, the captured human body edge data has clear contours and can be well distinguished from the environment. Compared with ordinary fall detection methods, using thermal imaging technology for nighttime fall detection reduces missed and false detections, greatly increasing the protection of personal safety.

Description

Technical Field

[0001] This invention belongs to the technical field of human posture recognition, specifically relating to a real-time human fall detection method in low light based on infrared and thermal imaging. Background Technology

[0002] Intelligent surveillance is a rapidly developing field of computer vision applications, primarily used in various locations such as roads, stations, airports, and schools. Its main tasks typically include pedestrian recognition and abnormal behavior detection. In traditional security surveillance, tasks are mainly completed manually. Due to the heavy workload of staff and the susceptibility to factors such as human attention and fatigue, the effectiveness of surveillance cannot be guaranteed. Intelligent surveillance uses relevant algorithms to automatically identify and retrieve video data, simplifying the workflow and significantly improving work efficiency.

[0003] Meanwhile, human pose recognition is an important research direction in the field of computer vision, with a wide range of applications, such as motion correction and fall detection. Fall detection, in particular, is a human safety technology that automatically determines whether a person is in a fall based on image sequences acquired by monitoring equipment. Its applications are very broad, including home safety protection for the elderly and children.

[0004] Conventional fall detection solutions use color images acquired by surveillance cameras for identification, and infrared illumination to supplement image information in low light conditions. However, surveillance equipment using infrared illumination produces black and white videos in low light, causing the monitored target to blend into the surrounding environment, and the video images become very blurry at greater distances. When this image information is input into a computer for human posture detection, the detection accuracy is severely limited, failing to accurately detect key points on the human body, let alone determine the current posture. If this method is applied to home safety systems, elderly people falling at night may go undetected, potentially leading to unimaginable consequences. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time human fall detection method based on infrared illumination and thermal imaging in low-light conditions, which solves the technical problems of inaccuracy and missed detection in existing human fall detection schemes in low-light conditions.

[0006] The technical solution is shown in the following steps:

[0007] S1: Obtain the dataset;

[0008] S1.1: Under sufficient lighting conditions, use a regular color camera to acquire a color image of the human body a1; use a thermal imaging camera to acquire an infrared thermal image of the human body b1 at the same moment;

[0009] S1.2: Transform the grayscale and brightness of image a1 to generate a black and white image c1, which is used to simulate the image information captured by the infrared supplementary light camera at night.

[0010] S1.3: Denote the images a1, b1, and c1 obtained in steps S1.1 and S1.2 as a single sample x. 1 Repeat steps S1.1 and S1.2 to obtain the original dataset x. 1 x 2 x 3 ...x n n is the number of samples, and the original dataset is denoted as X; each data sample x in X i , by a i b i c i Composition, where i is the sample number; simultaneously, each data sample x in the dataset X is... i All a i b i c i Let them be A, B, and C respectively; that is, dataset A consists of a1, a2, a3...a n Composed of b1, b2, b3...b n Composed of C, which consists of c1, c2, c3...c n Composition, where n is the sample number;

[0011] S2: Training the model;

[0012] S2.1: Input image A into Openpose, where image A is the sum of all a values ​​from the original dataset X. i Data samples, i.e., human body color image data; output A′, where A′ is represented as a′1, a′2, a′3, ..., a′ n ,a′ i This is data containing coordinates of multiple human body key points;

[0013] S2.2: Using the human body keypoint coordinates obtained in step S2.1, mark the human body keypoints in the corresponding images B and C to generate training samples B′ and C′, where B′ consists of b′1, b′2, b′3, ..., b′. n C′ consists of c′1, c′2, c′3……c′ n composition;

[0014] S2.3: Process the images B′ and C′ with keypoint coordinates obtained in step S2.2, and segment the image at each keypoint location to form image sets BK and CK for each keypoint, where BK is represented as bk1, bk2, bk3...bk 18 CK is represented as ck1, ck1, ck3...ck 18 A training dataset with 18 key points was obtained;

[0015] S2.4: Input the image sets of each keypoint in the training samples BK and CK into the pre-set convolutional neural network (CNN), and obtain model M through CNN training. B and M c ;

[0016] S3: Classify the coordinates of key points on the human body to check for fall behavior;

[0017] S3.1: A binocular camera made of an infrared supplementary light camera and a thermal imaging camera acquires low-light image information in real time, namely thermal imaging image and infrared supplementary light head image information, denoted as b and c respectively;

[0018] S3.2: Input b and c into the model M obtained in S2.3 respectively. B and m c In the middle, output the coordinates of the key points of the human skeleton, y b and y c ;

[0019] S3.3: y b and y c Perform mutual compensation to generate the coordinates y of a key point of the human skeleton;

[0020] S3.4: Input the coordinates y of the key points of the human skeleton into the preset Conv-LSTM network for pose classification, and detect fall behavior based on the pose classification results;

[0021] S3.5: If a person is detected to have fallen, the device will issue an alarm; otherwise, it will continue to collect images.

[0022] Preferably, in step S1, the final original dataset X contains each data sample x. i , by a i b i c i Composition, where i is the sample number; simultaneously, each data sample x in the dataset X is... i All a i b i c i Let them be A, B, and C respectively; that is, dataset A consists of a1, a2, a3...an Composed of b1, b2, b3...b n Composed of C, which consists of c1, c2, c3...c n Composition, where n is the sample number.

[0023] Preferably, the process of processing the collected dataset X in S2.1-S2.3 of S2 into training datasets BK and CK.

[0024] Preferably, the model M obtained after training the convolutional neural network in S2.4 is... B and M c .

[0025] Preferably, the image y obtained in steps S3.1 to S3.3 b and y c Mutual compensation is performed to generate the coordinates y of a key point of the human skeleton.

[0026] The beneficial effects of this invention are as follows: The method combining infrared compensation and thermal imaging technology for fall detection in low-light environments achieves better results than previous fall detection methods; it overcomes the problem of blurred human body contours and blending into the environment in relatively dark scenes captured by infrared night vision; thermal imaging technology utilizes the principle that all objects in nature radiate heat for nighttime human body detection, capturing clear human body edge contours that are well distinguishable from the environment; and compared to ordinary fall detection methods, using thermal imaging technology for nighttime fall detection reduces missed and false detections, greatly increasing the protection of personal safety. Attached Figure Description

[0027] Figure 1 This is a flowchart of the present invention.

[0028] Figure 2 This invention relates to a binocular camera.

[0029] Figure 3 This is a flowchart of the model training process of the present invention.

[0030] Figure 4 This is a diagram showing the key points and their numbers on the human body.

[0031] Figure 5 This is a flowchart of the nighttime human fall detection process of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0033] like Figure 1 As shown, the present invention includes the following steps:

[0034] S1: Obtain the dataset;

[0035] S1.1: Under sufficient lighting conditions, use a regular color camera to acquire a color image of the human body a1; use a thermal imaging camera to acquire an infrared thermal image of the human body b1 at the same moment;

[0036] S1.2: Transform the grayscale and brightness of image a1 to generate a black and white image c1, which is used to simulate the image information captured by the infrared supplementary light camera at night.

[0037] S1.3: Denote the images a1, b1, and c1 obtained in steps S1.1 and S1.2 as a single sample x. 1 Repeat steps S1.1 and S1.2 to obtain the original dataset x. 1 x 2 x 3 ...x n n is the number of samples, and the original dataset is denoted as X; each data sample x in X i , by a i b i c i Composition, where i is the sample number; simultaneously, each data sample x in the dataset X is... i All a i b i c i Let them be A, B, and C respectively; that is, dataset A consists of a1, a2, a3...a n Composed of b1, b2, b3...b n Composed of C, which consists of c1, c2, c3...c n Composition, where n is the sample number;

[0038] like Figure 3 As shown, the process of training the model is as follows:

[0039] S2: Training the model;

[0040] S2.1: Input image A into Openpose, where image A is the sum of all a values ​​from the original dataset X. i Data sample, i.e., human body color image data; output A ′ A ′ Represented as a ′ 1, a ′ 2, a ′ 3……a ′ n ,ai ′ This is data containing coordinates of multiple human body key points;

[0041] The OpenPose library used in this invention is an open-source library developed by Carnegie Mellon University (CMU) based on convolutional neural networks and supervised learning, using Caffe (a deep learning framework that combines expressiveness, speed, and cognitive modules) as its framework. It can perform pose estimation for human movements, facial expressions, and finger movements, and is suitable for single and multi-person use, with excellent robustness.

[0042] S2.2: Using the human body keypoint coordinates obtained in step S2.1, mark the human body keypoints in the corresponding images B and C to generate training samples B′ and C′, where B′ consists of b′1, b′2, b′3, ..., b′. n C′ is derived from c′ 1 , c′ 2 , c′ 3 ...c′ n composition;

[0043] S2.3: Process the images B′ and C′ with keypoint coordinates obtained in step S2.2, and segment the image at each keypoint location to form image sets BK and CK for each keypoint, where BK is represented as bk1, bk2, bk3...bk 18 CK is represented as ck1, ck1, ck3...ck 18 18 key points were obtained (such as...) Figure 4 The training dataset (as shown);

[0044] S2.4: Input the image sets of each keypoint in the training samples BK and CK into the pre-set convolutional neural network (CNN), and obtain model M through CNN training. B and M c ;

[0045] A convolutional neural network (CNN) consists of a feature extraction layer, convolutional layers, and a fully connected layer connected in series. The feature extraction network extracts features from any image using convolutional kernels, encoding these features into a feature vector. Finally, the fully connected layer uses the sigmoid function as the activation function, outputting a prediction training model M. B and M c ;

[0046] like Figure 5 As shown, the nighttime fall detection process is as follows:

[0047] S3: Classify the coordinates of key points on the human body to check for fall behavior;

[0048] S3.1: A binocular camera made using an infrared supplementary lighting camera and a thermal imaging camera (e.g.) Figure 2 (As shown) Real-time acquisition of low-light image information, namely thermal imaging image and infrared supplementary light head image information, denoted as b and c respectively;

[0049] S3.2: Input b and c into the model M obtained in S2.3 respectively. B and M c In the middle, output the coordinates of the key points of the human skeleton, y b and y c ;

[0050] S3.3: y b and y c Perform mutual compensation to generate the coordinates y of a key point of the human skeleton;

[0051] S3.4: Input the coordinates y of the key points of the human skeleton into the preset Conv-LSTM (Convolutional Long Short-Term Memory) network for pose classification, and detect fall behavior based on the pose classification results;

[0052] Conv-LSTM (Convolutional Long Short-Term Memory) is a recurrent neural network developed based on LSTM. This network not only has the feature extraction capability of recurrent neural networks, but also adds convolution operations to LSTM, making it better at processing temporal information than LSTM.

[0053] S3.5: If a person is detected to have fallen, the device will issue an alarm; otherwise, image acquisition will continue. An attention mechanism is added to the Conv-LSTM preset in step S3.4.

Claims

1. A method for real-time human fall detection in low light based on infrared and thermal imaging, characterized in that... This includes the following steps: S1: Obtain the dataset; S1.1: Under sufficient lighting conditions, use a regular color camera to acquire a color image of the human body a1; use a thermal imaging camera to acquire an infrared thermal image of the human body b1 at the same moment; S1.2: Transform the grayscale and brightness of image a1 to generate a black and white image c1, which is used to simulate the image information captured by the infrared supplementary light camera at night. S1.3: Denote the images a1, b1, and c1 obtained in steps S1.1 and S1.2 as a sample x. 1 Repeat steps S1.1 and S1.2 to obtain the original dataset x. 1 x 2 x 3 ...x n n is the number of samples, and the original dataset is denoted as X; each data sample x in X i , by a i b i c i Composition, where i is the sample number; simultaneously, each data sample x in the dataset X is... i All a i b i c i Let them be A, B, and C respectively; that is, dataset A consists of a1, a2, a3...a n Composed of b1, b2, b3...b n Composed of C, which consists of c1, c2, c3...c n Composition, where n is the sample number; S2: Training the model; S2.1: Input image A into Openpose, where image A is the sum of all a values ​​from the original dataset X. i Data samples, i.e., human body color image data; output A′, where A′ is represented as a′1, a′2, a′3, ..., a′ n ,a′ i This is data containing coordinates of multiple human body key points; S2.2: Using the human body keypoint coordinates obtained in step S2.1, mark the human body keypoints in the corresponding images B and C, generating training samples B′ and C′, where B′ consists of b′1, b′2, b′3, ..., b′. n C′ consists of c′1, c′2, c′3……c′ n composition; S2.3: Process the images B′ and C′ with keypoint coordinates obtained in step S2.2, and segment the image at each keypoint location to form image sets BK and CK for each keypoint, where BK is represented as bk1, bk2, bk3...bk 18 CK is represented as ck1, ck1, ck3...ck 18 This yielded a training dataset with 18 key points; S2.4: Input the image sets of each keypoint in the training samples BK and CK into the pre-set convolutional neural network (CNN), and obtain model M through CNN training. B and M c ; S3: Classify the coordinates of key points on the human body to check for fall behavior; S3.1: A binocular camera made of an infrared supplementary light camera and a thermal imaging camera acquires low-light image information in real time, namely thermal imaging image and infrared supplementary light head image information, denoted as b and c respectively; S3.2: Input b and c into the model M obtained in S2.3 respectively. B and M c In the middle, output the coordinates of the key points of the human skeleton, y b and y c ; S3.3: y b and y c Perform mutual compensation to generate the coordinates y of a key point of the human skeleton; S3.4: Input the coordinates y of the key points of the human skeleton into the preset Conv-LSTM network for pose classification, and detect fall behavior based on the pose classification results; S3.5: If a person is detected to have fallen, the device will issue an alarm; otherwise, it will continue to collect images.

2. The method for real-time human fall detection in low light based on infrared and thermal imaging according to claim 1, characterized in that: The original dataset X obtained in step S1, and each data sample x in X. i , by a i b i c i Composition, where i is the sample number; simultaneously, each data sample x in the dataset X is... i All a i b i c i Let them be A, B, and C respectively; that is, dataset A consists of a1, a2, a3...a n Composed of b1, b2, b3...b n Composed of C, which consists of c1, c2, c3...c n Composition, where n is the sample number.

3. The method for real-time human fall detection in low light based on infrared and thermal imaging according to claim 1, characterized in that: The process of processing the collected dataset X in S2.1-S2.3 of S2 into training datasets BK and CK.

4. The method for real-time human fall detection in low light based on infrared and thermal imaging according to claim 1, characterized in that: Model M obtained after training the convolutional neural network in S2.4 B and M c .

5. The method for real-time human fall detection in low light based on infrared and thermal imaging according to claim 1, characterized in that: The images y obtained in S3.1 to S3.3 b and y c Mutual compensation is performed to generate the coordinates y of a key point of the human skeleton.

Images