Intelligent personnel falling monitoring method and system based on TOF camera hardware

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides a TOF camera hardware-based personnel falling intelligent monitoring method and system, and the method comprises the following steps: collecting three-dimensional point cloud data of a monitoring region through a ToF area-array camera, carrying out the data preprocessing and background subtraction of the three-dimensional point cloud data, and obtaining the point cloud data of a human body; personnel existence detection and personnel static detection are carried out based on the three-dimensional point cloud data; performing posture recognition, performing normalization conversion on the human body point cloud data to obtain a human body depth map, and performing posture detection on key points of the human body depth map to obtain human body key point data; and predicting the human body key point data based on the trained LSTM model to obtain a person posture state, and when the person posture state is that the person falls down, generating early warning information. The method is wide in application range and suitable for personnel falling detection in public areas and private spaces, the point cloud data only comprises the attitude key points of the personnel, and personnel falling reasons can be known under the condition that privacy is not invaded.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and specifically relates to a method and system for intelligent monitoring of personnel falling based on TOF camera hardware. Background Art

[0002] Currently, the technologies for personnel fall detection mainly include: intelligent wearable devices, video detection technologies, audio detection technologies, WIFI technologies, ultra-wideband radars, and millimeter-wave radars. For some private spaces such as bathrooms and other scenarios, for the technologies used in monitoring devices, issues such as applicability, accuracy, and privacy protection also need to be considered. Video detection technologies have the problem of privacy leakage; WIFI technologies are vulnerable to external signal interference, have poor generalization, and low detection accuracy; ultra-wideband radars and millimeter-wave radar technologies only detect a single falling action, with low accuracy and inability to restore the falling scene. Therefore, it is necessary to provide a method and system for intelligent monitoring of personnel falling based on TOF camera hardware to solve the above problems. Summary of the Invention

[0003] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide a method and system for intelligent monitoring of personnel falling based on TOF camera hardware to solve the problems in the above background art.

[0004] The present invention is implemented as follows. A method for intelligent monitoring of personnel falling based on TOF camera hardware, the method includes the following steps:

[0005] Collect three-dimensional point cloud data of the monitoring area through a ToF area array camera, perform data preprocessing and background subtraction on the three-dimensional point cloud data to obtain human point cloud data;

[0006] Perform personnel presence detection and personnel stillness detection based on the three-dimensional point cloud data;

[0007] Perform pose recognition, perform normalization conversion on the human point cloud data to obtain a human depth map, and perform pose detection on the key points of the human depth map to obtain human key point data;

[0008] Based on the trained LSTM model, predict the human key point data to obtain the personnel pose state. When the personnel pose state is that the personnel is falling, generate a warning message.

[0009] As a further aspect of the present invention: The step of performing data preprocessing and background subtraction on the three-dimensional point cloud data to obtain human point cloud data specifically includes:

[0010] Adopt a statistical outlier filtering algorithm to remove the noise points in the three-dimensional point cloud data;

[0011] Through the spatial change detection of the point cloud dataset, the Octree is used to implement the spatial change detection between the point cloud data of the current scene and the background point cloud data. By recursively comparing the tree structures of the Octree, the spatial changes represented by the differences in the voxels generated by the Octree are identified. Through the double-buffering technology of the Octree, the foreground point data generated by the point cloud data of the current scene compared with the background point cloud data is obtained in real time;

[0012] Through the Euclidean clustering algorithm, the foreground point data is clustered and segmented, and the largest cluster is used as the seed point cloud to obtain the human body point cloud data.

[0013] As a further solution of the present invention: The steps of detecting the presence of personnel and detecting the stillness of personnel based on the three-dimensional point cloud data specifically include:

[0014] After performing background subtraction on the current frame point cloud data and the background point cloud data to obtain the human body point cloud data, the quantity and distribution of the human body point cloud data are statistically analyzed. For the case where it is higher than the first threshold, it is determined that there are personnel present;

[0015] Taking the previous frame point cloud data as the background data, using the current frame point cloud data and the background data for background subtraction operation to obtain the changed part between the current frame and the previous frame. The quantity and distribution of the data points in the changed part are statistically analyzed. For the case where it is higher than the second threshold, it is determined that the personnel are in a non-still state; otherwise, it is determined that the personnel are in a still state, and the duration of the still state of the personnel is accumulated. When the duration exceeds the set time, the long-term still state of the personnel is output.

[0016] As a further solution of the present invention: The steps of detecting the pose of the key points of the human body depth map to obtain the human body key point data specifically include:

[0017] Based on the Mediapipe library, the key points of the human body in the human body depth map are detected to obtain the pose detection result;

[0018] For the key point coordinates of the pose detection result, through the pixel index of the two-dimensional image data projected by the three-dimensional point cloud and the data index in the ordered point cloud data, the conversion from the two-dimensional key point coordinates to the three-dimensional key point coordinates is realized.

[0019] As a further solution of the present invention: When training the LSTM model, the point cloud data of people falling in different scenarios is collected, the key points of the human body point cloud are identified, the human body key points are used as the feature point dataset, the dataset labels are manually marked, and the LSTM model is used to train the dataset to obtain a custom model suitable for human body pose detection.

[0020] Another object of the present invention is to provide a personnel fall intelligent monitoring system based on TOF camera hardware, and the system includes:

[0021] A point cloud data processing module, configured to collect three-dimensional point cloud data of a monitoring area through a ToF area array camera, perform data preprocessing and background subtraction on the three-dimensional point cloud data to obtain human point cloud data;

[0022] A personnel detection module, configured to perform personnel presence detection and personnel stillness detection based on the three-dimensional point cloud data;

[0023] A human body pose detection module, configured to perform pose recognition, perform normalization conversion on the human point cloud data to obtain a human body depth map, and perform pose detection on the key points of the human body depth map to obtain human body key point data;

[0024] An LSTM model prediction module, configured to predict the human body key point data based on the trained LSTM model to obtain a personnel pose state, and generate a warning message when the personnel pose state is a personnel fall.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] The present invention has a wide range of applications, is suitable for personnel fall detection in public areas, and is also applicable to personnel monitoring in private spaces (such as bathrooms, etc.); it has strong anti-interference ability, the ToF device is not affected by light and does not require light conditions; it can achieve high accuracy for special populations (such as those in wheelchairs, using crutches, children, etc.); it can restore the fall scene. After a personnel fall occurs, the historical point cloud data can be viewed back. The point cloud data only contains the pose key points of the personnel, and the cause of the personnel fall can be understood without infringing on privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of a personnel fall intelligent monitoring method based on TOF camera hardware.

[0028] Figure 2 It is a structural schematic diagram of a personnel fall intelligent monitoring system based on TOF camera hardware. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following further describes the present invention in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0030] The following describes in detail the specific implementation of the present invention with reference to specific embodiments.

[0031] As Figure 1As shown in the figure, an embodiment of the present invention provides a method for intelligent monitoring of personnel falling based on the hardware of a TOF camera. The method includes the following steps:

[0032] S100, collect three-dimensional point cloud data of the monitoring area through a ToF area array camera, perform data preprocessing and background subtraction on the three-dimensional point cloud data to obtain human point cloud data;

[0033] S200, perform personnel presence detection and personnel stillness detection based on the three-dimensional point cloud data;

[0034] S300, perform pose recognition, perform normalization conversion on the human point cloud data to obtain a human depth map, and perform pose detection on the key points of the human depth map to obtain human key point data;

[0035] S400, predict the human key point data based on the trained LSTM model to obtain the personnel pose state. When the personnel pose state is a personnel fall, generate a warning message.

[0036] In the embodiment of the present invention, three-dimensional point cloud data of personnel and objects in the monitored scene is collected through a ToF camera, without infringing on the privacy information of the personnel in the monitored area. The three-dimensional point cloud of the personnel in the scene is extracted through a point cloud spatial change detection algorithm, and the personnel pose information is obtained after intelligent analysis of the three-dimensional point cloud of the personnel. Finally, the fall state of the personnel is monitored in real time without violating the principle of privacy. The present invention uses a ToF area array camera to collect three-dimensional point cloud data of the monitoring area, preprocess the point cloud data, extract the human point cloud, detect the human key points, analyze the personnel state, and then output the analysis results.

[0037] In the embodiments of the present invention, the step of performing data preprocessing and background subtraction on the three-dimensional point cloud data to obtain human point cloud data specifically includes: removing the noise points in the three-dimensional point cloud data by using a statistical outlier filtering algorithm, which is a probability method based on the distance distribution of target points to neighboring points in the input data. By comparing the average distance E of the nearest k points in the neighborhood of all points with the threshold Td, it is determined whether the point is a valid point, and finally all target points are determined through iterative calculation. Then background subtraction is performed. For the processing of point cloud data with a fixed position of the ToF camera, background subtraction is a very effective background segmentation method. The input of the algorithm is the background frame before no person enters the scene and the point cloud data frame of the person moving in the scene, and the output is the point cloud data only containing the person. Specifically, through the spatial change detection of the point cloud data set, the Octree is used to realize the spatial change detection between the current scene point cloud data and the background point cloud data. The Octree is a tree structure for managing sparse 3D data. By recursively comparing the tree structure of the Octree, the spatial changes represented by the differences in the voxels generated by the Octree are identified. Through the double-buffer technology of the Octree, the foreground point data generated by the current scene point cloud data compared with the background point cloud data is obtained in real time; through the Euclidean clustering algorithm, the foreground point data is clustered and segmented, and the largest cluster is used as the seed point cloud to obtain the human point cloud data.

[0038] In the embodiments of the present invention, the step of performing human presence detection and human stillness detection based on the three-dimensional point cloud data specifically includes: obtaining human point cloud data by performing background subtraction operation on the current frame point cloud data and the background (no person) point cloud data, and counting the quantity and distribution of the human point cloud data. When it is higher than the first threshold, it is determined that there is a person present; at the same time, in order to prevent the interference of noise data and small dynamic objects, the above-mentioned human point cloud needs to be filtered, and the detection results should also be statistically analyzed for multiple frames within the last 2-3 seconds to improve the accuracy of the output results. When performing human stillness detection, the previous frame point cloud data is used as the background data, and the background subtraction operation is performed using the current frame point cloud data and the background data to obtain the changed part between the current frame and the previous frame. The quantity and distribution of the data points in the changed part are statistically analyzed. When it is higher than the second threshold, it is determined that the person is in a non-still state; otherwise, it is determined that the person is in a still state, and the duration of the human stillness state is accumulated. When the duration exceeds the set time, the long-term stillness state of the person is output.

[0039] In the embodiments of the present invention, the basic logic of human pose recognition based on point cloud data can be transformed into human pose recognition based on depth maps. By combining the pose recognition results of two-dimensional images with depth values, the coordinates of human key points in the world coordinate system can be obtained. First, a depth map transformation is performed. The Z values of the point cloud data containing only the human body are normalized to obtain a human depth map. Compared with the depth map directly converted from the overall point cloud, the depth value range of this depth map is smaller, so the converted grayscale information is more refined, making it more accurate for identifying human poses.

[0040] In the embodiments of the present invention, the step of performing pose detection on the key points of the human depth map to obtain human key point data specifically includes: detecting the human key points in the human depth map based on the Mediapipe library to obtain pose detection results; Mediapipe is an open-source framework developed by Google Research, aiming to help developers easily build, test, and deploy complex multi-modal, multi-task machine learning models. It is particularly good at real-time processing and analysis of multimedia data such as audio and video. In addition, the main advantage of Mediapipe is that it can also perform real-time processing when relying solely on the CPU as the computing resource. For the key point coordinates of the pose detection results, through the pixel indices of the two-dimensional image data projected from the three-dimensional point cloud and the data indices in the ordered point cloud data, the mapping is performed to realize the conversion from two-dimensional key point coordinates to three-dimensional key point coordinates and achieve spatial transformation.

[0041] It should be noted that the Long Short-Term Memory (LSTM) model is a subtype of the Recurrent Neural Network (RNN), mainly used for identifying data sequences, such as patterns appearing in sensor data, stock prices, or natural language. Its purpose is to implement a neural network that can learn what information to store, how long to store it, and what information to discard. This ability is crucial for processing sequences where relevant information spans a large time interval. Different from traditional neural networks, LSTM combines feedback connections, enabling it to process the entire data sequence rather than just a single data point. This makes it very effective in understanding and predicting patterns in time series data such as time series, text, and speech. To build an LSTM model suitable for this project, first, collect the point cloud data of people falling in different scenarios, identify the key points of the human point cloud in it, use the human key points as the feature point data set, manually label the data set labels (no one, someone and normal, someone and falling), and use the LSTM model to train the data set to obtain a custom model suitable for human pose detection; use this model as the human fall detection status detection sub-module of the intelligent analysis service to predict the human key point data labels in real time to obtain the current status of the target person.

[0042] AsFigure 2 As shown in the figure, an intelligent personnel fall monitoring system based on TOF camera hardware is further provided in an embodiment of the present invention. The system includes:

[0043] A point cloud data processing module 100, configured to collect three-dimensional point cloud data of a monitoring area through a ToF area array camera, perform data preprocessing and background subtraction on the three-dimensional point cloud data, and obtain human point cloud data;

[0044] A personnel detection module 200, configured to perform personnel presence detection and personnel stillness detection based on the three-dimensional point cloud data;

[0045] A human body pose detection module 300, configured to perform pose recognition, perform normalization conversion on the human point cloud data to obtain a human body depth map, and perform pose detection on key points of the human body depth map to obtain human body key point data;

[0046] An LSTM model prediction module 400, configured to predict the human body key point data based on the trained LSTM model to obtain a personnel pose state, and generate a warning message when the personnel pose state is a personnel fall.

[0047] The advantages of the embodiment of the present invention compared with the existing personnel fall detection technology are as follows in the table:

[0048]

[0049]

[0050] The above only describes the preferred embodiments of the present invention in detail, and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0051] It should be understood that although the steps in the flowcharts of the embodiments of the present invention are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0052] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0053] After considering the specification and the disclosure of the embodiments, those skilled in the art will readily conceive of other embodiments of the present disclosure. The present application is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

Claims

1. An intelligent monitoring method for personnel falling based on TOF camera hardware, characterized in that, The method includes the following steps: Collect three-dimensional point cloud data of the monitoring area through a ToF area array camera, perform data preprocessing and background subtraction on the three-dimensional point cloud data to obtain human point cloud data; Perform personnel presence detection and personnel stillness detection based on the three-dimensional point cloud data; Perform pose recognition, perform normalization conversion on the human point cloud data to obtain a human depth map, and perform pose detection on the key points of the human depth map to obtain human key point data; Predict the human key point data based on the trained LSTM model to obtain the personnel pose state. When the personnel pose state is a person falling, generate a warning message.

2. The intelligent monitoring method for personnel falling based on the TOF camera hardware according to claim 1, wherein The step of performing data preprocessing and background subtraction on the three-dimensional point cloud data to obtain human point cloud data specifically includes: Use a statistical outlier filtering algorithm to remove noise points in the three-dimensional point cloud data; Through spatial change detection of the point cloud dataset, use Octree to implement spatial change detection between the current scene point cloud data and the background point cloud data. By recursively comparing the tree structure of Octree, identify the spatial changes represented by the differences in the voxels composed of Octree. Through the double-buffering technology of Octree, obtain the foreground point data generated by the current scene point cloud data compared with the background point cloud data in real time; Through the Euclidean clustering algorithm, cluster and segment the foreground point data, and use the largest cluster as the seed point cloud to obtain human point cloud data.

3. The intelligent personnel fall monitoring method based on TOF camera hardware according to claim 1, wherein The step of performing personnel presence detection and personnel stillness detection based on the three-dimensional point cloud data specifically includes: After performing background subtraction operation on the current frame point cloud data and the background point cloud data to obtain human point cloud data, count the quantity and distribution of the human point cloud data. For the situation higher than the first threshold, it is determined that there are personnel present; Use the previous frame point cloud data as the background data, perform background subtraction operation on the current frame point cloud data and the background data to obtain the changed part between the current frame and the previous frame. Statistically analyze the quantity and distribution of the data points in the changed part. For the situation higher than the second threshold, it is determined that the personnel are in a non-still state; otherwise, it is determined that the personnel are in a still state, accumulate the duration of the personnel still state, and when the duration exceeds the set time, output the long-term still state of the personnel.

4. The intelligent personnel fall monitoring method based on the TOF camera hardware according to claim 1, characterized in that, The step of performing pose detection on the key points of the human depth map to obtain human key point data specifically includes: Detect the human key points in the human depth map based on the Mediapipe library to obtain the pose detection result; For the key point coordinates of the pose detection result, map them through the pixel index of the two-dimensional image data projected from the three-dimensional point cloud to the data index in the ordered point cloud data to realize the conversion from two-dimensional key point coordinates to three-dimensional key point coordinates.

5. The intelligent personnel fall monitoring method based on TOF camera hardware according to claim 1, characterized in that, When training the LSTM model, collect the point cloud data of people falling in different scenarios, identify the key points of the human point cloud among them, use the human key points as the feature point dataset, manually label the dataset labels, and use the LSTM model to train the dataset to obtain a custom model suitable for human pose detection.

6. An intelligent monitoring system for personnel fall based on TOF camera hardware, characterized in that, The system includes: A point cloud data processing module, which is used to collect three-dimensional point cloud data of a monitoring area through a ToF area array camera, perform data preprocessing and background subtraction on the three-dimensional point cloud data to obtain human point cloud data; A personnel detection module, which is used to perform personnel presence detection and personnel stillness detection based on the three-dimensional point cloud data; A human body pose detection module, which is used to perform pose recognition, normalize and transform the human point cloud data to obtain a human body depth map, and perform pose detection on the key points of the human body depth map to obtain human body key point data; An LSTM model prediction module, which is used to predict the human body key point data based on the trained LSTM model to obtain the personnel pose state. When the personnel pose state is a person falling, a warning message is generated.

Images