Abnormal behavior detection method based on millimeter wave radar action recognition

Abstract

The invention discloses an abnormal behavior detection method based on millimeter wave radar action recognition. The method comprises the following steps: S1, collecting synchronous millimeter wave radar data; s2, performing radius filtering and other preprocessing operations on the acquired radar data; s3, performing RANSAC (Random Sample Consensus) plane segmentation on the preprocessed data, and distinguishing the ground and other objects on the ground; s4, clustering the point cloud data by using a DBSCAN algorithm, and dividing the point cloud data into a plurality of independent parts to distinguish a human body from a building; s5, analyzing a clustering result, extracting characteristics such as the length-width-height ratio and the point cloud density of each independent part, and analyzing the characteristics to obtain a human body classification result; s6, carrying out dimension conversion from three dimensions to two dimensions, and then carrying out human body key point positioning on the point cloud data by adopting an OpenPose method; and S7, performing abnormal behavior detection by using the space-time diagram convolutional network.

Description

Technical Field

[0001] The present invention relates to the fields of computer monitoring technology and human body posture recognition, and in particular to a method for detecting abnormal human behavior. Background Art

[0002] Abnormal human behavior detection is a technology that uses computer vision, artificial intelligence and machine learning technologies to identify and analyze abnormal states or abnormal activities of human behavior. This technology is widely used in many fields such as security monitoring, intelligent security, medical health, smart home, public safety, etc. Although traditional detection methods such as video surveillance can play a deterrent and recording role to a certain extent, they have limitations in terms of privacy protection, light conditions, and occlusion. For example, video surveillance can easily infringe on people's privacy rights, especially in sensitive areas such as dressing rooms and toilets. In addition, when the light is insufficient or blocked by objects, the effect of video surveillance will be greatly reduced, resulting in monitoring blind spots.

[0003] In order to better monitor and prevent abnormal behavior, the present invention utilizes millimeter wave radar, an emerging non-contact detection technology. Millimeter wave radar has the advantages of strong penetration, high resolution, and good privacy protection, and can efficiently monitor the surrounding environment without infringing personal privacy. It can also detect and identify the position and motion state of objects by transmitting and receiving millimeter wave signals, and accurately capture dynamic changes even in insufficient light or blocked by objects. Since millimeter wave radar does not rely on visible light, it can work stably under all-weather and all-environment conditions. In addition, millimeter wave radar can extract detailed motion features such as speed, direction, and acceleration by processing reflected signals, thereby realizing accurate recognition of human behavior. Unlike video surveillance, millimeter wave radar does not capture clear images, but only generates point cloud data reflecting the motion characteristics of objects, which greatly reduces the risk of privacy infringement. Therefore, millimeter wave radar can not only improve the accuracy of abnormal human behavior detection, but also effectively protect people's privacy.

[0004] The millimeter wave radar device used in the present invention is a 60GHz indoor human body monitoring radar developed by Yingqi Intelligent Company. Summary of the invention

[0005] The purpose of the present invention is to provide an abnormal behavior detection method based on millimeter wave radar action recognition to solve the problems raised in the above background technology.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0007] A method for detecting abnormal behavior based on millimeter wave radar action recognition, characterized in that it comprises the following steps:

[0008] S1. Use the 60GHz indoor human monitoring radar developed by Yingqi Intelligent Company to collect millimeter wave radar data;

[0009] S2, performing pre-processing operations such as radius filtering on the collected radar data;

[0010] S3, performing RANSAC plane segmentation on the preprocessed data to distinguish the ground and other objects on the ground;

[0011] S4. Use the DBSCAN algorithm to cluster the point cloud data, separate different objects, and divide the point cloud data into multiple independent parts;

[0012] S5. Analyze the clustering results, extract the length, width and height ratio and point cloud density of each independent part, and obtain the classification result of whether it is a human body;

[0013] S6, using the OpenPose method to locate key points of the human body in point cloud data;

[0014] S7. Abnormal behavior detection using spatiotemporal graph convolutional networks.

[0015] As a detailed description of the data acquisition method, it is characterized in that: in step S1, the installation location of the radar equipment should be away from areas with strong electromagnetic interference, such as near large transformers, motors and other equipment; the safety of the radar equipment should also be considered to avoid human damage or theft, so it should be installed at a height of about 2.5 meters from the ground, and the downward tilt angle of the radar is set to about 30 degrees, which can make the radar cover as comprehensive as possible; avoid installing the radar in a place blocked by large objects, which may affect the propagation and reflection of the signal; the transmission frequency is selected to be 60GH, which can more accurately detect and distinguish target objects; since the size of the bathroom and student dormitory is not large, the transmission power is set to 5dBm, which is enough to cover a distance of about 10 meters while maintaining low power consumption and electromagnetic radiation; the radar scans 20 times per second; the device itself cannot be used without a power supply and needs to be powered by a Type-C data cable, so a power supply is required at the location where the radar is installed.

[0016] As a detailed description of the data preprocessing method, it is characterized in that: in step S2, there may be a situation where the collected data is missing point cloud, and it needs to be completed. The present invention uses the PointNet++ network to complete the point cloud. The network architecture includes the following parts: Input layer: including three-dimensional coordinate information (x, y, z) containing point cloud data. Sampling and grouping layer: The sampling layer performs layered sampling on the point cloud to reduce the number of points while retaining key points. The grouping layer divides the sampled point cloud into multiple local areas, each of which contains a certain number of neighboring points. Local feature extraction layer: In each local area, the PointNet module is used to extract local features. The PointNet module includes a multi-layer perceptron (MLP) and a maximum pooling layer (Max Pooling). Local feature aggregation layer: The features of all local areas are aggregated into global features to form a global representation of the point cloud. Multi-scale feature fusion layer: Repeat the above process at different scales, and obtain richer point cloud features through multi-scale feature extraction and fusion. Feature decoding and completion layer: The fused global features are input into the feature decoder, and the completed point cloud data is generated through layer-by-layer upsampling and convolution operations. Output layer: outputs the completed point cloud data.

[0017] As a detailed description of the clustering result analysis method, it is characterized in that: in step S5, the features of each part distinguished by the clustering method are extracted, including the length, width and height ratio, the point cloud density, the number of points of each clustering result and the motion feature. The motion feature extraction is analyzed in groups of five frames to distinguish between moving targets (i.e., human bodies) and stationary targets (i.e., buildings).

[0018] As a detailed description of the human body key point positioning method, it is characterized in that: in step S6, before positioning the human body key points, a perspective projection method is used to convert the dimensions from three-dimensional to two-dimensional.

[0019] As a detailed description of the human key point positioning method, it is characterized in that: in step S6, OpenPose is used to estimate the human posture of each frame image to obtain the coordinates of the human key points. The posture estimation results of each frame are stored to form time series posture data. The key point coordinates of each frame are constructed into a spatial graph (nodes are key points, and edges are skeleton connections). The corresponding key points of adjacent frames are connected in the time dimension to form a spatiotemporal graph.

[0020] As a detailed description of the abnormal behavior detection method, it is characterized in that: in step S7, abnormal behaviors are defined as the following: fast movement: detecting the speed change of the joint point, and determining it as fast movement when it exceeds the set threshold; sudden stop: detecting the sharp change of the speed of the joint point from high to low; physical contact: detecting the distance change between multiple joint points, and determining it as physical contact when it approaches a certain threshold; aggressive posture: pre-defining some aggressive posture templates, matching the current posture in real time; defensive posture: pre-defining some defensive posture templates, matching the current posture in real time. The time window is set to ten seconds. When these abnormal behaviors all occur within a time window, it is determined that an abnormal behavior event has occurred.

[0021] As a detailed description of the abnormal behavior detection method, it is characterized in that: in step S7, the spatiotemporal graph convolutional network (ST-GCN) architecture includes the following parts: input layer: input node features (joint point positions) and edges (skeleton connections and time connections) of the spatiotemporal graph; graph convolution layer: perform graph convolution operations on the spatiotemporal graph to extract spatial and temporal features; spatial convolution: perform graph convolution in each frame to extract spatial features of joint points; temporal convolution: perform convolution between joint points of adjacent frames to capture dynamic changes; pooling layer: reduce dimension and aggregate features to reduce computational complexity; fully connected layer: classify the extracted features and calculate the result probability. The activation function selects softmax to convert the model output into a probability distribution. The result is whether an abnormal behavior event occurs, so the number of neurons is set to 2.

[0022] As a detailed description of the feature extraction method, it is characterized in that: the motion feature is to identify the moving target (ie, human body) and the stationary target (ie, building) by analyzing five frames of data.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: the traditional video surveillance system directly captures and records images, which may infringe on people's privacy, especially in sensitive areas such as locker rooms and toilets. The present invention uses millimeter-wave radar for detection, and only generates point cloud data reflecting the motion characteristics of objects, without specific personal images, thereby effectively protecting students' privacy. The performance of video surveillance will be significantly reduced under insufficient light or bad weather conditions (such as night, rainy days, etc.), and monitoring blind spots are prone to occur. The millimeter-wave radar technology of the present invention does not rely on visible light, can work stably under various lighting conditions and weather, and provide all-weather monitoring capabilities. Millimeter-wave radar has good penetration ability and can penetrate non-metallic objects such as clothing and paper to effectively detect obscured targets. This enables millimeter-wave radar to better capture potential abnormal behaviors in complex environments and reduce monitoring blind spots. The millimeter-wave radar system has the characteristics of small size, light weight, and low power consumption, which is convenient for flexible deployment and integration in different places. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of an abnormal behavior detection method provided by an embodiment of the present invention;

[0025] Figure 2 Schematic diagram of a neural network model architecture for abnormal behavior detection provided by an embodiment of the present invention;

[0026] Figure 3 It is one of the offensive postures provided by the embodiment of the present invention;

[0027] Figure 4 It is one of the defensive postures provided by the embodiment of the present invention;

[0028] Figure 5 It is a human body posture image provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the present invention will be described more clearly and completely in conjunction with the accompanying drawings and specific embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0030] Figure 1 FIG. 2 is a flow chart of abnormal human behavior detection provided by this embodiment. Figure 1 As shown, the abnormal behavior detection method provided by the present invention comprises the following steps:

[0031] S1, collect synchronous millimeter wave radar data;

[0032] S2, performing pre-processing operations such as radius filtering on the collected radar data;

[0033] S3, performing RANSAC plane segmentation on the preprocessed data to distinguish the ground and other objects on the ground;

[0034] S4. Use the DBSCAN algorithm to cluster the point cloud data, separate different objects, and divide the point cloud data into multiple independent parts;

[0035] S5. Analyze the clustering results, extract the length, width and height ratio and point cloud density of each independent part, and obtain the classification result of whether it is a human body;

[0036] S6, using the OpenPose method to locate key points of the human body in point cloud data;

[0037] S7. Abnormal behavior detection using spatiotemporal graph convolutional networks.

[0038] In this embodiment, the device used is a 60GHz indoor human monitoring radar developed by Yingqi Intelligent Company for data financial work. The device has a 60G carrier frequency, 3G bandwidth, 4 transmit and 4 receive, and can communicate through WIFI, 4G, etc.; the deployment method is wall-mounted, the height is about 2.5 meters, the downward tilt angle is about 30 degrees, and a power supply is required near the installation location.

[0039] In this embodiment, there may be outliers and missing point clouds for the collected radar data. The radius filtering method is used to remove outliers to enhance the quality of the data; if the number of neighboring points within a specified radius of a point is less than the set threshold, the point is considered to be an outlier and is removed; for the case of missing point clouds, it is necessary to complete it. This embodiment uses the PointNet++ network to complete the point cloud. The network architecture includes the following parts: Input layer: including three-dimensional coordinate information (x, y, z) containing point cloud data. Sampling and grouping layer: The sampling layer performs hierarchical sampling on the point cloud to reduce the number of points while retaining key points. The grouping layer divides the sampled point cloud into multiple local areas, each of which contains a certain number of neighboring points. Local feature extraction layer: In each local area, the PointNet module is used to extract local features. The PointNet module includes a multi-layer perceptron (MLP) and a maximum pooling layer (Max Pooling). Local feature aggregation layer: The features of all local areas are aggregated into global features to form a global representation of the point cloud. Multi-scale feature fusion layer: Repeat the above process at different scales, and obtain richer point cloud features through multi-scale feature extraction and fusion. Feature decoding and completion layer: Input the fused global features into the feature decoder, and generate the completed point cloud data through layer-by-layer upsampling and convolution operations. Output layer: Output the completed point cloud data.

[0040] In this embodiment, the RANSAC algorithm is used to identify and remove the ground plane point cloud, which helps to reduce data noise and highlight the human body and other objects. The distance between the point and the plane is calculated to determine whether the point belongs to the plane. The distance formula is as follows:

[0041]

[0042] After removing the ground, this embodiment uses the DBSCAN clustering method to distinguish the point cloud data into multiple independent parts. DBSCAN can effectively process noise and identify dense areas in the point cloud. For each independent area, its features are extracted, including the aspect ratio feature: the aspect ratio of the human body is different from that of the building, and a threshold can be set to distinguish it. For example, the height of the human body is usually significantly greater than the width and depth, while the aspect ratio of the building is diverse; point cloud density feature: the point cloud of the building is usually dense and regular, while the point cloud density of the human body is low and the distribution is irregular; point count feature: calculate the number of points for each cluster. Buildings usually contain more points, while the number of points in the human body cluster is relatively small; motion feature: by performing motion analysis on five frames of data, moving targets (ie, human bodies) and stationary targets (ie, buildings) can be identified.

[0043] In this embodiment, the OpenPose method of video data processing is used to locate the key points of the human body on the point cloud data. Before using this method to locate the key points, it is necessary to convert the dimension from three-dimensional to two-dimensional. This embodiment uses perspective projection to complete the dimension conversion. Through perspective projection, the depth and distance of three-dimensional objects can be expressed on a two-dimensional plane. Perspective projection projects three-dimensional points onto a two-dimensional plane. The basic formula is as follows:

[0044]

[0045] Among them, (x, y, z) is a point in three-dimensional space, and (x′, y′) is a point on the two-dimensional projection plane.

[0046] After the projection is completed, OpenPose is used to estimate the human posture of each frame image and obtain the coordinates of the key points of the human body. The posture estimation results of each frame are then stored to form time series posture data. The key point coordinates of each frame are constructed into a spatial graph (nodes are key points, and edges are skeleton connections). The corresponding key points of adjacent frames are connected in the time dimension to form a spatiotemporal graph.

[0047] This embodiment uses a spatiotemporal graph convolutional network (ST-GCN) to detect abnormal behavior. Figure 2 As shown in the figure, the network architecture includes the following parts: input layer: input node features (joint point positions) and edges (skeleton connections and time connections) of the spatiotemporal graph; graph convolution layer: perform graph convolution operations on the spatiotemporal graph to extract spatial and temporal features; spatial convolution: perform graph convolution in each frame to extract spatial features of joints; temporal convolution: perform convolution between joints of adjacent frames to capture dynamic changes; pooling layer: reduce the dimension and aggregate the features to reduce the computational complexity; fully connected layer: classify the extracted features and calculate the result probability. The activation function selects softmax to convert the model output into a probability distribution. The result is whether an abnormal behavior event occurs, so the number of neurons is set to 2.

[0048] This embodiment defines abnormal behaviors as follows: Rapid movement: Detect the speed change of joint points, and determine it as rapid movement when it exceeds the set threshold. Sudden stop: Detect the sudden change of the speed of joint points from high to low. Physical contact: Detect the distance change between multiple joint points, and determine it as physical contact when it approaches a certain threshold. Aggressive posture: Predefine some aggressive posture templates to match the current posture, such as Figure 3 Defensive posture: Predefine some defensive posture templates to match the current posture such as Figure 4 shown.

[0049] The pre-trained ST-GCN model is used to segment the video and obtain the feature vector of each window (ten seconds as a window). The features in each window are analyzed, and when these abnormal behaviors appear in a time window, it is determined that abnormal behavior has occurred.

[0050] This embodiment performs a series of operations on the collected data to generate a human body posture image such as Figure 5 shown.

[0051] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting abnormal behavior based on millimeter wave radar action recognition, characterized in that: The steps include: S1. Use the 60GHz indoor human monitoring radar developed by Yingqi Intelligent Company to collect millimeter wave radar data; S2, performing pre-processing operations such as radius filtering and point cloud completion on the collected radar data; S3, performing RANSAC plane segmentation on the preprocessed data to distinguish the ground and other objects on the ground; S4. Use the DBSCAN algorithm to cluster the point cloud data, separate different objects, and divide the point cloud data into multiple independent parts; S5. Analyze the clustering results, extract the length, width and height ratio and point cloud density of each independent part, and obtain the classification result of whether it is a human body; S6, using the OpenPose method to locate key points of the human body in point cloud data; S7. Abnormal behavior detection using spatiotemporal graph convolutional networks.

2. The method for data collection according to claim 1, characterized in that: In step S1, the installation location of the radar equipment should be away from areas with strong electromagnetic interference, such as near large transformers, motors and other equipment; the safety of the radar equipment should also be considered to avoid human damage or theft, so it should be installed at a height of about 2.5 meters from the ground, and the downward tilt angle of the radar is set to about 30 degrees, which can make the radar cover as comprehensive as possible; avoid installing the radar in a place blocked by large objects, which may affect the propagation and reflection of the signal; the transmission frequency is selected as 60GH, which can more accurately detect and distinguish target objects; because the size of the toilet and student dormitory is not large, the transmission power is set to 5dBm, which is enough to cover a distance of about 10 meters while maintaining low power consumption and electromagnetic radiation; the radar scans 20 times per second; the device itself cannot be used without a power supply and needs to be powered by a Type-C data cable, so a power supply is required at the location where the radar is installed.

3. The method for data preprocessing according to claim 1, characterized in that: In step S2, there may be a situation where the collected data is missing a point cloud, and it needs to be completed. The present invention uses the PointNet++ network to complete the point cloud. The network architecture includes the following parts: Input layer: including three-dimensional coordinate information (x, y, z) containing point cloud data. Sampling and grouping layer: The sampling layer performs hierarchical sampling on the point cloud to reduce the number of points while retaining key points. The grouping layer divides the sampled point cloud into multiple local areas, each of which contains a certain number of neighboring points. Local feature extraction layer: In each local area, the PointNet module is used to extract local features. The PointNet module includes a multi-layer perceptron (MLP) and a maximum pooling layer (Max Pooling). Local feature aggregation layer: Aggregate the features of all local areas into global features to form a global representation of the point cloud. Multi-scale feature fusion layer: Repeat the above process at different scales, and obtain richer point cloud features through multi-scale feature extraction and fusion. Feature decoding and completion layer: Input the fused global features into the feature decoder, and generate completed point cloud data through layer-by-layer upsampling and convolution operations. Output layer: outputs the completed point cloud data.

4. The method for clustering result analysis according to claim 1, characterized in that: In step S5, the features of each independent part distinguished by the clustering method are extracted, including the aspect ratio, point cloud density, the number of points in each clustering result, and motion features. The motion features are extracted and analyzed in groups of five frames to distinguish between moving targets (i.e., human bodies) and stationary targets (i.e., buildings).

5. The method for locating key points of a human body according to claim 1, characterized in that: In step S6, before locating the key points of the human body, a perspective projection method is used to convert the dimensions from three dimensions to two dimensions.

6. The method for locating key points of a human body according to claim 1, characterized in that: In step S6, OpenPose is used to estimate the human posture of each frame image and obtain the coordinates of the key points of the human body. The posture estimation results of each frame are stored to form time series posture data. The key point coordinates of each frame are constructed into a spatial graph (nodes are key points and edges are skeleton connections). The corresponding key points of adjacent frames are connected in the time dimension to form a spatiotemporal graph.

7. The method for abnormal behavior detection according to claim 1, characterized in that: In step S7, abnormal behaviors are defined as the following: rapid movement: detecting the speed change of the joint point, and determining it as rapid movement when it exceeds the set threshold; sudden stop: detecting the rapid change of the speed of the joint point from high to low; Physical contact: Detect the distance change between multiple joints and determine it as physical contact when it approaches a certain threshold; Aggressive posture: pre-define some aggressive posture templates, and match the current posture in real time; defensive posture: pre-define some defensive posture templates, and match the current posture in real time. The time window is set to ten seconds. When these abnormal behaviors occur within a time window, it is determined that an abnormal behavior event has occurred.

8. The abnormal behavior detection method according to claim 1, characterized in that: In step S7, the spatiotemporal graph convolutional network (ST-GCN) architecture includes the following parts: input layer: input node features (joint point positions) and edges (skeleton connections and temporal connections) of the spatiotemporal graph; graph convolution layer: perform graph convolution operations on the spatiotemporal graph to extract spatial and temporal features; spatial convolution: perform graph convolution in each frame to extract spatial features of joints; temporal convolution: perform convolution between joints of adjacent frames to capture dynamic changes; pooling layer: reduce the dimension and aggregate the features to reduce the computational complexity; fully connected layer: classify the extracted features and calculate the result probability. The activation function selects softmax to convert the model output into a probability distribution. The result is whether an abnormal behavior event occurs, so the number of neurons is set to 2.

9. The feature extraction method according to claim 2, characterized in that: The motion features are analyzed on five frames of data to identify moving targets (i.e. human body) and stationary targets (i.e. buildings).

Images