Human body abnormal behavior detection method, device, electronic equipment and storage medium

Abstract

The present invention provides a method, device, electronic device and storage medium for detecting abnormal human behavior, which relates to the field of computer technology, wherein the method comprises: identifying a target to be detected in an image data frame, extracting features, and determining key points of the human body; judging the direction deviation based on the key points of the trunk, and determining that the target is in a tilted posture; judging whether the target to be detected is in an abnormal behavior state based on the outlier factors of the key points of the human body in each image data frame. Through the determined key points of the trunk, the direction deviation of the target to be detected relative to the world coordinate system is judged to determine that the target is in a tilted posture. And based on the outlier factors of the key points of the human body in each image data frame, the angle and distance are combined for analysis, and the influence of the angle and distance of the key points of the human body at different positions is comprehensively considered, so as to achieve accurate determination of whether the target to be detected is in an abnormal behavior state, and improve the accuracy of abnormal human behavior detection.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, electronic equipment and storage medium for detecting abnormal human behavior. Background Art

[0002] Through video surveillance, various situations can be recorded, and abnormal behaviors can be identified based on video surveillance, including falling down, squatting, etc.

[0003] Existing methods generally use target detection to detect abnormal behaviors such as people falling and squatting. Specifically, they generally use typical target detection algorithms to detect abnormal behaviors. In existing methods, abnormal behavior detection directly based on algorithms is limited by the scene and angle, which is prone to false detection and leads to low detection accuracy. Summary of the invention

[0004] The present invention provides a method, device, electronic device and storage medium for detecting abnormal human behavior, so as to improve the accuracy of detecting abnormal human behavior.

[0005] The present invention provides a method for detecting abnormal human behavior, comprising the following steps:

[0006] Acquire multiple consecutive image data frames;

[0007] Identify the target to be detected in the image data frame, extract features of the identified target to be detected, and determine the key points of the human body of the target to be detected;

[0008] Based on the torso key points among the human body key points, judging the direction deviation of the target to be detected relative to the world coordinate system, and determining that the target to be detected is in a tilted posture;

[0009] Based on judging the outlier factor of the human body key points in each image data frame, it is determined whether the target to be detected is in an abnormal behavior state, and the outlier factor is determined based on the angle of each human body key point relative to the torso key point, and the distance of each human body key point relative to the torso key point.

[0010] According to a method for detecting abnormal human behavior provided by the present invention, the method determines whether the target to be detected is in an abnormal behavior state based on judging the outlier factor of the key points of the human body in each image data frame, including:

[0011] When the outlier factors of the key points of the human body in each image data frame are greater than a preset determination threshold, determining that the abnormal behavior state of the target to be detected is a falling state;

[0012] The outlier factor is:

[0013] ;

[0014] in, is the outlier factor; is the sum of relative distances, which is determined based on the distances of each human key point relative to the torso key point; is the sum of relative angles, which is determined based on the angles of each human key point relative to the torso key point; as well as is a hyperparameter.

[0015] According to a method for detecting abnormal human behavior provided by the present invention, the process of determining the sum of relative distances includes:

[0016] Normalizing each human body key point in the image data frame, and determining the Euclidean distance of each human body key point after the normalization process relative to the torso key point;

[0017] The relative distance sum is determined based on the sum of the obtained multiple Euclidean distances and the number of human body key points in the image data frame.

[0018] According to a method for detecting abnormal human behavior provided by the present invention, the process of determining the sum of relative angles includes:

[0019] Normalizing each human body key point in the image data frame, and determining the angle of each human body key point after the normalization process relative to the torso key point;

[0020] The relative angle sum is determined based on the sum of the multiple angles obtained and the number of key points of the human body in the image data frame.

[0021] According to a method for detecting abnormal human behavior provided by the present invention, the feature extraction of the identified target to be detected and the determination of the key points of the human body of the target to be detected include:

[0022] Based on a recurrent neural network, feature extraction is performed on the target to be detected to obtain a feature map of the target to be detected;

[0023] Based on the convolutional neural network, the feature map is further extracted to determine the key points of the human body of the target to be detected.

[0024] According to a method for detecting abnormal human behavior provided by the present invention, based on the trunk key points among the key points of the human body, judging the direction deviation of the target to be detected relative to the world coordinate system, and determining that the target to be detected is in a tilted posture, comprises:

[0025] Based on the torso key points among the human body key points, determining the angle between the direction of the target to be detected and the Y-axis direction of the camera coordinate system;

[0026] Based on the angle, adjusting the direction of the target to be detected;

[0027] Based on the parameters of the camera, the adjusted vertical angle of the target to be detected relative to the world coordinate system is determined, and based on the vertical angle, it is determined that the target to be detected is in a tilted posture.

[0028] According to a method for detecting abnormal human behavior provided by the present invention, determining that the target to be detected is in a tilted posture based on the vertical angle includes:

[0029] When it is determined that the vertical angle is greater than a preset tilt angle threshold, it is determined that the target to be detected is in a tilted posture.

[0030] According to a method for detecting abnormal human behavior provided by the present invention, identifying the target to be detected in the image data frame includes:

[0031] Based on a recurrent neural network, feature extraction is performed on the target to be detected to obtain a feature map of the target to be detected;

[0032] Based on the convolutional neural network, further feature extraction is performed on the feature map to determine the detection frame of the target to be detected and the confidence score of the detection frame.

[0033] According to a method for detecting abnormal human behavior provided by the present invention, after determining the detection frame of the target to be detected and the confidence score of the detection frame, the method further includes:

[0034] It is determined that the confidence score of the detection frame of the target to be detected in each image data frame is greater than a preset detection frame score threshold.

[0035] According to a method for detecting abnormal human behavior provided by the present invention, after determining that the target to be detected is in a tilted posture, the method further includes:

[0036] Determine the aspect ratio of the detection frame of the target to be detected, and when it is determined that the aspect ratio is less than a preset aspect ratio threshold, determine that the target to be detected is in a squatting state.

[0037] According to a method for detecting abnormal human behavior provided by the present invention, after determining that the target to be detected is in a tilted posture, the method further includes:

[0038] Based on the multiple continuous image data frames, determining the movement distance of the key points of the human body between two image data frames;

[0039] Determining a time interval between two image data frames based on a sampling frequency of the image data frames;

[0040] Based on the moving distance and the time interval, a moving speed of the target to be detected is determined, and it is determined that the moving speed is less than a preset moving speed threshold.

[0041] The present invention also provides a device for detecting abnormal human behavior, comprising the following modules:

[0042] An image acquisition module, used for acquiring a plurality of continuous image data frames;

[0043] An extraction module is used to identify the target to be detected in the image data frame, extract features of the identified target to be detected, and determine the key points of the human body of the target to be detected;

[0044] A posture judgment module, used to judge the direction deviation of the target to be detected relative to the world coordinate system based on the torso key points among the key points of the human body, and determine that the target to be detected is in a tilted posture;

[0045] The abnormal behavior judgment module is used to determine whether the target to be detected is in an abnormal behavior state based on the outlier factor of the human body key points in each image data frame, and the outlier factor is determined based on the angle of each human body key point relative to the torso key point, and the distance of each human body key point relative to the torso key point.

[0046] The present invention also provides an abnormal behavior detection device, comprising the above-mentioned human abnormal behavior detection device.

[0047] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned methods for detecting abnormal human behavior is implemented.

[0048] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for detecting abnormal human behavior as described above is implemented.

[0049] The human body abnormal behavior detection method, device, electronic device and storage medium provided by the present invention determine the direction deviation of the target to be detected relative to the world coordinate system through the determined torso key points, and determine that the target to be detected is in a tilted posture. And further based on the outlier factor of the human body key points in each image data frame, the angle and distance are analyzed in combination, and the influence of the angle and distance of the human body key points at different positions is comprehensively considered to achieve accurate determination of whether the target to be detected is in an abnormal behavior state, thereby improving the accuracy of human body abnormal behavior detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is a flow chart of the method for detecting abnormal human behavior provided by the present invention.

[0052] Figure 2 It is a schematic diagram of key points of the human body provided by the present invention.

[0053] Figure 3 It is a schematic diagram of the human body key point detection process provided by the present invention.

[0054] Figure 4 It is a flow chart of the method for detecting abnormal human behavior provided by the present invention.

[0055] Figure 5 It is a structural schematic diagram of the human abnormal behavior detection device provided by the present invention.

[0056] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Figure 1 FIG. 1 is a flow chart of the method for detecting abnormal human behavior provided by the present invention. Figure 1 As shown, the method includes the following:

[0059] Step 110, acquiring a plurality of continuous image data frames;

[0060] Step 120, identifying the target to be detected in the image data frame, extracting features of the identified target to be detected, and determining the key points of the human body of the target to be detected;

[0061] Step 130, judging the direction deviation of the target to be detected relative to the world coordinate system based on the torso key points in the human body key points, and determining that the target to be detected is in a tilted posture;

[0062] Step 140, based on judging the outlier factor of the human body key points in each image data frame, determine whether the target to be detected is in an abnormal behavior state, and the outlier factor is determined based on the angle of each human body key point relative to the torso key point, and the distance of each human body key point relative to the torso key point.

[0063] The execution subject of the abnormal human behavior detection method provided by the present invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), or a personal computer (PC), etc., which is not specifically limited by the present invention.

[0064] The following takes the method for detecting abnormal human behavior provided by the present invention executed by a computer as an example to describe the technical solution of the present invention in detail.

[0065] In step 110, a plurality of consecutive image data frames are acquired.

[0066] Access a video stream and extract multiple continuous image data frames from the video stream.

[0067] Optionally, after obtaining a plurality of continuous image data frames, each image data frame may be preprocessed.

[0068] The specific process of preprocessing includes: scaling the image data frame size to a specific size. The specific size is set to 640×640, and then scaling the image data frame size without changing the aspect ratio of the image data frame, scaling the largest side to 640, and filling the small side with grayscale;

[0069] For multiple consecutive image data frames, the selected image data frames may be required to be adjacent or equal detection and reading image data frames. For example, if the camera rate is 30FPS, the number of image data frames is 6, and the interval for reading video frames is 200ms, that is, 6 frames are read per second.

[0070] In step 120, the target to be detected in the image data frame is identified, and features of the identified target to be detected are extracted to determine the key points of the human body of the target to be detected.

[0071] The target to be detected in the image data frame can be identified and the key points of the human body can be extracted. Specifically, based on the deep learning method, the feature representation in the image can be automatically learned and the feature extraction can be performed to realize the detection process of the key points of the human body. The schematic diagram of the extracted key points of the human body can be as follows: Figure 2 The key points of the human body provided by the present invention are shown in the schematic diagram.

[0072] Figure 2 It contains the bounding box of the human target and 11 human key points, namely KP_1, KP_2, KP_3, KP_4, KP_5, KP_6, KP_7, KP_8, KP_9, KP_10 and KP_11.

[0073] It can be understood that by dividing the key points of the human body, the key points of the torso can be obtained as KP_1, KP_2 and KP_6.

[0074] Optionally, after obtaining the key points of the human body from the image frame, a connection relationship between the key points of the human body is established to form a skeleton model of the torso.

[0075] By defining the connection relationship between the key points of the human body, a skeleton model can be constructed. This skeleton model can be represented by a graph data structure, where the key points of the human body are nodes and the connection relationship is an edge.

[0076] After building the skeleton model, the key points of the torso can be extracted according to the following method:

[0077] Relative position: Filter out the key points related to the torso based on relative coordinates. You can set rules such as distance, angle, etc.

[0078] Filtering: Apply threshold filtering to remove possible noise or falsely detected keypoints.

[0079] Skeleton shape analysis: Analyze the skeleton shape to determine the validity of the key points of the torso. For example, verify its reliability by checking the continuity and stability of the spine.

[0080] By establishing the connection relationship between the key points and forming a skeleton model of the torso, the key points of the torso can be effectively extracted.

[0081] In step 130, based on the torso key points among the human body key points, the direction deviation of the target to be detected relative to the world coordinate system is judged to determine that the target to be detected is in a tilted posture.

[0082] It is understandable that the trunk, as the core part of the human body, the positions of its key points can directly reflect the overall posture and inclination of the human body.

[0083] Based on the torso key points in the human body key points, the direction deviation of the target to be detected relative to the world coordinate system is judged. Figure 2 The torso key points KP_1, KP_2 and KP_6 in the image are connected to obtain a straight line in the torso direction. By judging the deviation between the straight line and the vertical method of the real world coordinate system, it can be determined whether the target to be detected is in a tilted posture.

[0084] Based on determining whether the target to be detected is in a tilted posture, a preliminary judgment can be made on whether the target to be detected is in an abnormal behavior state. If it is determined that the target to be detected is in a tilted posture, it can be preliminarily determined that the target to be detected may be in an abnormal behavior state.

[0085] In step 140, based on judging the outlier factor of the human body key points in each image data frame, it is determined whether the target to be detected is in an abnormal behavior state, and the outlier factor is determined based on the angle of each human body key point relative to the torso key point, and the distance of each human body key point relative to the torso key point.

[0086] Judging abnormal behavior status based on the outlier factor of human body key points mainly relies on the analysis of human body key points (such as head, shoulders, elbows, wrists, hips, knees and ankles, etc.), and identifying abnormal behavior by calculating the angle and distance of these key points relative to the torso key points (usually the chest or abdomen).

[0087] The outlier factor refers to whether the performance of certain human key points relative to the torso key points in a specific behavior mode significantly deviates from the normal range. When the angle or distance of the human key points exceeds the preset threshold, the behavior can be considered to be abnormal.

[0088] The angle can be calculated by the vector in the three-dimensional coordinate system. Specifically, the trunk key point can be set as the coordinate origin, and the relative position of other human key points relative to the trunk key point can be calculated. The angle between the human key point and the trunk key point is calculated using the relative position vector. By analyzing the angle data of normal behavior, a reasonable angle range can be set for subsequent abnormality detection.

[0089] The Euclidean distance between each human body key point and the torso key point can be calculated. By analyzing the distance data of normal behavior, a reasonable distance range can be set for subsequent anomaly detection.

[0090] Combine angle and distance for analysis to determine whether the target to be detected is in an abnormal behavior state.

[0091] It can be understood that simultaneous detection is performed based on multiple continuous image data frames, and only when all conditions are met can the person's behavior be determined to be abnormal, thereby avoiding the use of target tracking and other means, and further improving the detection efficiency of abnormal human behavior detection.

[0092] The human body abnormal behavior detection method provided by the present invention determines the direction deviation of the target to be detected relative to the world coordinate system through the determined trunk key points, and determines that the target to be detected is in a tilted posture. And further based on the outlier factor of the human body key points in each image data frame, the angle and distance are analyzed, and the influence of the angle and distance of the human body key points at different positions is comprehensively considered to achieve accurate determination of whether the target to be detected is in an abnormal behavior state, thereby improving the accuracy of human body abnormal behavior detection.

[0093] In one embodiment, determining whether the target to be detected is in an abnormal behavior state based on judging the outlier factor of the key point of the human body in each image data frame includes:

[0094] When the outlier factors of the key points of the human body in each image data frame are greater than a preset determination threshold, determining that the abnormal behavior state of the target to be detected is a falling state;

[0095] The outlier factor is:

[0096] ;

[0097] in, is the outlier factor; is the sum of relative distances, which is determined based on the distances of each human key point relative to the torso key point; is the sum of relative angles, which is determined based on the angles of each human key point relative to the torso key point; as well as is a hyperparameter.

[0098] The process of determining the relative distance sum includes:

[0099] Normalizing each human body key point in the image data frame, and determining the Euclidean distance of each human body key point after the normalization process relative to the torso key point;

[0100] The relative distance sum is determined based on the sum of the obtained multiple Euclidean distances and the number of human body key points in the image data frame.

[0101] The process of determining the relative angle sum includes:

[0102] Normalizing each human body key point in the image data frame, and determining the angle of each human body key point after the normalization process relative to the torso key point;

[0103] The relative angle sum is determined based on the sum of the multiple angles obtained and the number of key points of the human body in the image data frame.

[0104] Specifically, the process of judging the outlier factors of the key points of the human body in each image data frame can be:

[0105] In the camera coordinate system, the human body key points of each person detected in each image data frame are normalized. For example, if 11 human body key points are detected, the 11 points are normalized; if 5 human body key points are detected, the 5 points are normalized.

[0106] The normalization formula is as follows:

[0107] ;

[0108] ;

[0109] in and is the mean of the key points detected. and is the standard deviation of the key points detected on the human body; For detection point Normalized value.

[0110] The normalized Euclidean distance of each human key point relative to the torso key point. Assume that the 11 human key points are KP_1, KP_2, KP_3, KP_4, KP_5, KP_6, KP_7, KP_8, KP_9, KP_10 and KP_11. The torso key points include KP_1, KP_2 and KP_6.

[0111] For example, the torso key point KP_2 is , the Euclidean distance calculation formula relative to the torso key point is as follows:

[0112] ;

[0113] To calculate the angle of the human body key point relative to the torso key point, KP_2 and KP_6 are preferred. If only one of these two points exists, KP_1 is used as the other. , KP_6 is .

[0114] The calculation formula for the angle of the human body key point relative to the torso key point is:

[0115] ;

[0116] The relative distance sum calculation formula is:

[0117] ;

[0118] in, is the sum of relative distances, , , and is the scale factor, is the number of human key points detected in the image data frame. If no corresponding point is detected, the distance of the point is 0;

[0119] The relative angle sum is calculated as:

[0120] ;

[0121] in, is the sum of relative angles, , , and is the scale factor, is the number of human key points detected in the image data frame. If no corresponding point is detected, the angle of the point is 0;

[0122] Calculate the outlier factor, which is expressed as follows:

[0123] ;

[0124] in, is the outlier factor; is the sum of relative distances, which is determined based on the distances of each human key point relative to the torso key point; is the sum of relative angles, which is determined based on the angles of each human key point relative to the torso key point; as well as is a hyperparameter.

[0125] A preset determination threshold may be set to determine whether the calculated outlier factor is greater than the preset determination threshold. If all outlier factors of a plurality of consecutive image data frames are greater than the preset determination threshold, the target to be detected is determined to be in a fallen state.

[0126] The method for detecting abnormal human behavior provided by the present invention comprehensively determines whether a person has abnormal behavior by adopting the method of outlier factors, and the outlier factors comprehensively consider the influence of the angle and distance of key points of the human body at different positions. At the same time, multiple continuous image data frames are detected simultaneously, and only when all conditions are met, the person's behavior is determined to be abnormal, avoiding the use of means such as target tracking, and further improving the detection efficiency of the target.

[0127] In one embodiment, feature extraction is performed on the identified target to be detected to determine the human body key points of the target to be detected, including: based on a recurrent neural network, feature extraction is performed on the target to be detected to obtain a feature map of the target to be detected; based on a convolutional neural network, further feature extraction is performed on the feature map to determine the human body key points of the target to be detected.

[0128] Identifying the target to be detected in the image data frame includes:

[0129] Based on a recurrent neural network, feature extraction is performed on the target to be detected to obtain a feature map of the target to be detected;

[0130] Based on the convolutional neural network, further feature extraction is performed on the feature map to determine the detection frame of the target to be detected and the confidence score of the detection frame.

[0131] After determining the detection frame of the target to be detected and the confidence score of the detection frame, the method further includes:

[0132] It is determined that the confidence score of the detection frame of the target to be detected in each image data frame is greater than a preset detection frame score threshold.

[0133] Specifically, the process of determining the detection frame of the key points of the human body and the target to be detected can be as follows: Figure 3 The human body key point detection process provided by the present invention is shown in the schematic diagram.

[0134] Step 301, input a plurality of continuous image data frames;

[0135] Step 302, sorting the image data frames in chronological order, reading the subsequent pictures in sequence from the first frame to form a picture sequence, and preprocessing the picture sequence;

[0136] Perform image preprocessing and scale the image to a specific size. The specific size is set to 640×640. During the image scaling process, the image’s aspect ratio is not changed. The largest side is scaled to 640, and the smaller side is filled with grayscale. Continuous image data frames require that the selected images must be adjacent or equal detection read image data frame images.

[0137] If the camera rate is 30FPS, the number of consecutive frames is 6, and the interval for reading video frames is 200ms, that is, 6 frames are read per second.

[0138] Step 303, the processed image data frame is passed through the first layer of sequential recurrent neural network RNN, and the network is subjected to Dropout processing to prevent overfitting; the first layer of sequential RNN network may be a long short-term memory network LSTM;

[0139] Step 304, input the obtained feature map into the second-layer sequential RNN network, and perform Dropout processing on the network to prevent overfitting; the second-layer sequential RNN network may be an LSTM network;

[0140] Step 305, input the obtained feature map into the convolutional neural network CNN, and perform BN processing on the features;

[0141] Step 306, input the obtained feature map into the third-layer sequential RNN network, and perform Dropout processing on the network to prevent overfitting; the third-layer sequential RNN network may be an LSTM network;

[0142] Step 307, input the obtained feature map into the CNN network for further feature extraction;

[0143] Step 308, adding the features obtained in the previous step and the features obtained in step 305, performing feature fusion processing, increasing the dominant features of the features, and avoiding the loss of important information;

[0144] Step 309: The feature information obtained in the previous step is processed through 1 1CNN network performs feature extraction;

[0145] Step 310: The feature information obtained in the previous step is processed through 1 1. The CNN network performs feature extraction to obtain the key point information of the human body and the corresponding scores of the key points;

[0146] Step 311, the feature information obtained in step 308 is processed through 1 1CNN network performs feature extraction;

[0147] Step 312: The feature information obtained in the previous step is processed through 1 1. The CNN network performs feature extraction to obtain the detection frame of the target to be detected and the score of the detection frame;

[0148] The program ends.

[0149] After obtaining the detection frame of the target to be detected and the confidence score of the detection frame, it is determined that the confidence score of the detection frame of the target to be detected in each image data frame is greater than the preset detection frame score threshold to ensure that the target detection in each image data frame is accurate and can be used for subsequent judgment processes.

[0150] Optionally, the image data frames may be screened based on the comparison between the confidence score and a preset detection frame score threshold. The image data frames whose confidence scores of the detection frames of the targets to be detected in each image data frame are greater than the preset detection frame score threshold are retained, and the image data frames whose confidence scores of the detection frames of the targets to be detected in each image data frame are less than or equal to the preset detection frame score threshold are deleted.

[0151] In one embodiment, based on the torso key points among the human body key points, judging the direction deviation of the target to be detected relative to the world coordinate system, and determining that the target to be detected is in a tilted posture, includes:

[0152] Based on the torso key points among the human body key points, determining the angle between the direction of the target to be detected and the Y-axis direction of the camera coordinate system;

[0153] Based on the angle, adjusting the direction of the target to be detected;

[0154] Based on the parameters of the camera, the adjusted vertical angle of the target to be detected relative to the world coordinate system is determined, and based on the vertical angle, it is determined that the target to be detected is in a tilted posture.

[0155] Determining that the target to be detected is in an inclined posture based on the vertical angle includes:

[0156] When it is determined that the vertical angle is greater than a preset tilt angle threshold, it is determined that the target to be detected is in a tilted posture.

[0157] Assume that the 11 human body key points are KP_1, KP_2, KP_3, KP_4, KP_5, KP_6, KP_7, KP_8, KP_9, KP_10 and KP_11. The torso key points include KP_1, KP_2 and KP_6.

[0158] According to the information of any two points among the trunk key points KP_1, KP_2 and KP_6, determine the direction of the human body relative to the camera coordinate system;

[0159] Determine the angle direction of the human body relative to the Y axis of the camera coordinate system, that is, if the points KP_1 and KP_2 are determined, the coordinates of KP_1 are ( ), the coordinates of KP_2 are ( ); then the angle relative to the Y axis for:

[0160] ;

[0161] According to the determined , adjust the direction of the target to be detected.

[0162] Furthermore, according to the camera posture, the direction of the human body is adjusted to a direction relative to the world coordinate system;

[0163] Assume that the rotation matrix R represents the rotation from the camera coordinate system to the world coordinate system, and the rotation angle is (the angle between the camera coordinate system and the world coordinate system), the rotation matrix can be expressed as:

[0164] ;

[0165] The midpoint of the camera coordinate system KP_1( ) and KP_2( ) is mapped to the world coordinate system:

[0166] ;

[0167] ;

[0168] Compute the vector between two points in world coordinates:

[0169] ;

[0170] make:

[0171] ;

[0172] Therefore, the adjusted vertical angle of the target to be detected relative to the world coordinate system is:

[0173] ;

[0174] based on , to determine whether the human body is in a tilted posture in the actual coordinate system. Specifically, The tilt angle is compared with the preset threshold value. When the tilt angle is greater than a preset tilt angle threshold, it is determined that the target to be detected is in a tilted posture.

[0175] It can be understood that the preset tilt angle threshold is used to assist in determining whether the target to be detected has fallen, and is to achieve a preliminary determination of whether the target to be detected has fallen.

[0176] In one embodiment, after determining that the target to be detected is in a tilted posture, it also includes: determining the aspect ratio of the detection frame of the target to be detected, and when it is determined that the aspect ratio is less than a preset aspect ratio threshold, determining that the target to be detected is in a squatting state.

[0177] Specifically, calculate the aspect ratio of the detection frame of the target to be detected:

[0178] If the coordinates of the detection box of the target to be detected are (x1, y1, x2, y2, score1), where (x1, y1) is the coordinate of the upper left corner of the detection box, and (x2, y2) is the coordinate of the lower right corner of the detection box, then the aspect ratio of the detection box can be expressed as:

[0179] ;

[0180] A preset aspect ratio threshold is set in advance to determine whether the aspect ratio of the detection frame is less than the preset aspect ratio threshold, and if so, it is determined to be a squatting state.

[0181] In one embodiment, after determining that the target to be detected is in a tilted posture, it also includes: determining the moving distance of the key points of the human body between two image data frames based on the multiple consecutive image data frames; determining the time interval between two image data frames based on the sampling frequency of the image data frames; determining the moving speed of the target to be detected based on the moving distance and the time interval, and determining that the moving speed is less than a preset moving speed threshold.

[0182] Specifically, the moving speed of the target to be detected is determined based on multiple consecutive image data frames. The determination of the moving speed can be set according to the sampling frequency; the coordinates of any point in the torso key point position such as KP_2 and KP_6 are used to calculate the speed of the person. For example, taking KP_2 as an example, the corresponding coordinates of the first frame are KP_2_1 ( ), the coordinates of the sixth frame are KP_2_6( ); Map the coordinates of the two points to the time coordinate system:

[0183] ;

[0184] ;

[0185] The rotation angle is (the angle between the camera coordinate system and the world coordinate system).

[0186] The moving distance of the point between two frames is:

[0187] ;

[0188] ;

[0189] The moving speed of this point is:

[0190] ;

[0191] A preset moving speed threshold is set in advance, and it is determined that the moving speed of the target to be detected is less than the preset moving speed threshold.

[0192] It is understandable that when the target to be detected moves quickly, the body may be in a tilted posture. Based on the judgment of the moving speed, misjudgment that may be caused by the moving speed is eliminated.

[0193] The present invention also provides a flow chart of the method for detecting abnormal human behavior provided by the present invention, as shown in FIG. Figure 4 As shown, the process includes:

[0194] Step 401: access a video stream and collect a plurality of continuous image data frames from the video stream.

[0195] Step 402: After obtaining a plurality of continuous image data frames, each image data frame may be preprocessed.

[0196] The specific process of preprocessing includes: scaling the image data frame size to a specific size. The specific size is set to 640×640, and then scaling the image data frame size without changing the aspect ratio of the image data frame, scaling the largest side to 640, and filling the small side with grayscale;

[0197] For multiple consecutive image data frames, the selected image data frames may be required to be adjacent or equal detection and reading image data frames. For example, if the camera rate is 30FPS, the number of image data frames is 6, and the interval for reading video frames is 200ms, that is, 6 frames are read per second.

[0198] Step 403: input the processed image data frame into a key point detection network.

[0199] Step 404, through the key point detection network, determine whether there is a human target in the camera area, if not, return to step 401, otherwise continue to execute;

[0200] Step 405, obtaining the detection frame and human key points of the target to be detected, the information of the detection frame includes coordinates and scores, specifically (x1, y1, x2, y2, score1); the information of the human key points, a total of 11, the information of each human key point includes coordinates and scores, specifically (kpx, kpy, score);

[0201] Step 406, determining whether the score of the detection frame of the target to be detected of all image data frames is greater than a set threshold, if greater than the set threshold, continue to execute, otherwise return to step 401;

[0202] Step 407, determining whether the confidence scores of any two of the torso key points KP_1, KP_2 and KP_6 of all image data frames are greater than the set threshold, if so, continue to execute, otherwise return to step 401;

[0203] Step 408, determining the direction of the human body relative to the camera coordinate system based on information of any two points among KP_1, KP_2 and KP_6;

[0204] Step 409, according to the camera posture, the direction of the human body is adjusted to the direction relative to the world coordinate system, and it is determined whether the human body is in an inclined posture in the actual coordinate system, that is, whether the vertical angle relative to the world coordinate system is greater than a set threshold. If so, the next step is executed, otherwise, the execution returns to step 401;

[0205] Step 410, calculating the aspect ratio of the detection frames of all the objects to be detected in the image data frame;

[0206] Step 411, determine whether the aspect ratio of the detection frame is less than a set threshold, if so, determine it as a squatting state, otherwise proceed to the next step;

[0207] Step 412, based on the continuous video frames, determine the moving speed of the target to be detected. The moving speed can be determined based on the sampling frequency set in step 402; the moving speed of the target to be detected is calculated by using the coordinates of any point in the trunk key point position, such as KP_2 and KP_6.

[0208] Step 413, determine whether the moving speed is less than a set threshold, if not, return to 401, if yes, continue to execute;

[0209] Step 414, calculating an outlier factor based on the angle of each human body key point relative to the trunk key point and the distance of each human body key point relative to the trunk key point;

[0210] Step 415, determine whether the outlier factor is greater than a set threshold, if yes, continue to execute, otherwise return to step 401;

[0211] Step 416, if all outlier factors of the continuous image data frames meet the conditions, it is determined that the target to be detected is in a falling state;

[0212] Step 417, the process ends.

[0213] The human body abnormal behavior detection device provided by the present invention is described below. The human body abnormal behavior detection device described below and the human body abnormal behavior detection method described above can be referenced to each other.

[0214] like Figure 5 As shown, the device comprises:

[0215] An image acquisition module 510 is used to acquire a plurality of continuous image data frames;

[0216] The extraction module 520 is used to identify the target to be detected in the image data frame, extract features of the identified target to be detected, and determine the key points of the human body of the target to be detected;

[0217] A posture judgment module 530 is used to judge the direction deviation of the target to be detected relative to the world coordinate system based on the torso key point in the human body key point, and determine that the target to be detected is in a tilted posture;

[0218] The abnormal behavior judgment module 540 is used to determine whether the target to be detected is in an abnormal behavior state based on the outlier factor of the human body key points in each image data frame, and the outlier factor is determined based on the angle of each human body key point relative to the torso key point, and the distance of each human body key point relative to the torso key point.

[0219] The human body abnormal behavior detection device provided by the present invention determines the direction deviation of the target to be detected relative to the world coordinate system through the determined torso key points, and determines that the target to be detected is in a tilted posture. And further based on the outlier factor of the human body key points in each image data frame, the angle and distance are analyzed, and the influence of the angle and distance of the human body key points at different positions is comprehensively considered to achieve accurate determination of whether the target to be detected is in an abnormal behavior state, thereby improving the accuracy of human body abnormal behavior detection.

[0220] In one embodiment, the abnormal behavior determination module 540 is specifically used to:

[0221] Based on judging the outlier factors of the key points of the human body in each image data frame, determining whether the target to be detected is in an abnormal behavior state includes:

[0222] When the outlier factors of the key points of the human body in each image data frame are greater than a preset determination threshold, determining that the abnormal behavior state of the target to be detected is a falling state;

[0223] The outlier factor is:

[0224] ;

[0225] in, is the outlier factor; is the sum of relative distances, which is determined based on the distances of each human key point relative to the torso key point; is the sum of relative angles, which is determined based on the angles of each human key point relative to the torso key point; as well as is a hyperparameter.

[0226] In one embodiment, the abnormal behavior determination module 540 is further specifically configured to:

[0227] The process of determining the relative distance sum includes:

[0228] Normalizing each human body key point in the image data frame, and determining the Euclidean distance of each human body key point after the normalization process relative to the torso key point;

[0229] The relative distance sum is determined based on the sum of the obtained multiple Euclidean distances and the number of human body key points in the image data frame.

[0230] In one embodiment, the abnormal behavior determination module 540 is further specifically configured to:

[0231] The process of determining the relative angle sum includes:

[0232] Normalizing each human body key point in the image data frame, and determining the angle of each human body key point after the normalization process relative to the torso key point;

[0233] The relative angle sum is determined based on the sum of the multiple angles obtained and the number of key points of the human body in the image data frame.

[0234] In one embodiment, the extraction module 520 is specifically used to:

[0235] The step of extracting features of the identified target to be detected and determining key points of the human body of the target to be detected includes:

[0236] Based on a recurrent neural network, feature extraction is performed on the target to be detected to obtain a feature map of the target to be detected;

[0237] Based on the convolutional neural network, the feature map is further extracted to determine the key points of the human body of the target to be detected.

[0238] In one embodiment, the posture determination module 530 is specifically used for:

[0239] The step of judging the direction deviation of the target to be detected relative to the world coordinate system based on the trunk key points in the human body key points, and determining that the target to be detected is in a tilted posture, includes:

[0240] Based on the torso key points among the human body key points, determining the angle between the direction of the target to be detected and the Y-axis direction of the camera coordinate system;

[0241] Based on the angle, adjusting the direction of the target to be detected;

[0242] Based on the parameters of the camera, the adjusted vertical angle of the target to be detected relative to the world coordinate system is determined, and based on the vertical angle, it is determined that the target to be detected is in a tilted posture.

[0243] In one embodiment, the posture determination module 530 is further specifically used for:

[0244] The step of determining, based on the vertical angle, that the target to be detected is in an inclined posture comprises:

[0245] When it is determined that the vertical angle is greater than a preset tilt angle threshold, it is determined that the target to be detected is in a tilted posture.

[0246] In one embodiment, the extraction module 520 is further specifically configured to:

[0247] The identifying the target to be detected in the image data frame includes:

[0248] Based on a recurrent neural network, feature extraction is performed on the target to be detected to obtain a feature map of the target to be detected;

[0249] Based on the convolutional neural network, further feature extraction is performed on the feature map to determine the detection frame of the target to be detected and the confidence score of the detection frame.

[0250] In one embodiment, the extraction module 520 is further specifically configured to:

[0251] After determining the detection frame of the target to be detected and the score of the detection frame, the method further includes:

[0252] It is determined that the confidence score of the detection frame of the target to be detected in each image data frame is greater than a preset detection frame score threshold.

[0253] In one embodiment, the extraction module 520 is further specifically configured to:

[0254] After determining that the target to be detected is in an inclined posture, the method further includes:

[0255] Determine the aspect ratio of the detection frame of the target to be detected, and when it is determined that the aspect ratio is less than a preset aspect ratio threshold, determine that the target to be detected is in a squatting state.

[0256] In one embodiment, the posture determination module 530 is further specifically used for:

[0257] After determining that the target to be detected is in an inclined posture, the method further includes:

[0258] Based on the multiple continuous image data frames, determining the movement distance of the key points of the human body between two image data frames;

[0259] Determining a time interval between two image data frames based on a sampling frequency of the image data frames;

[0260] Based on the moving distance and the time interval, a moving speed of the target to be detected is determined, and it is determined that the moving speed is less than a preset moving speed threshold.

[0261] The present invention also provides an abnormal behavior detection device, comprising the human abnormal behavior detection apparatus as described above.

[0262] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communications interface 620 and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the abnormal human behavior detection method, which includes: acquiring a plurality of continuous image data frames;

[0263] Identify the target to be detected in the image data frame, extract features of the identified target to be detected, and determine the key points of the human body of the target to be detected;

[0264] Based on the torso key points among the human body key points, judging the direction deviation of the target to be detected relative to the world coordinate system, and determining that the target to be detected is in a tilted posture;

[0265] Based on judging the outlier factor of the human body key points in each image data frame, it is determined whether the target to be detected is in an abnormal behavior state, and the outlier factor is determined based on the angle of each human body key point relative to the torso key point, and the distance of each human body key point relative to the torso key point.

[0266] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0267] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the human body abnormal behavior detection method provided by the above methods, the method includes: acquiring a plurality of continuous image data frames;

[0268] Identify the target to be detected in the image data frame, extract features of the identified target to be detected, and determine the key points of the human body of the target to be detected;

[0269] Based on the torso key points among the human body key points, judging the direction deviation of the target to be detected relative to the world coordinate system, and determining that the target to be detected is in a tilted posture;

[0270] Based on judging the outlier factor of the human body key points in each image data frame, it is determined whether the target to be detected is in an abnormal behavior state, and the outlier factor is determined based on the angle of each human body key point relative to the torso key point, and the distance of each human body key point relative to the torso key point.

[0271] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the human body abnormal behavior detection method provided by the above methods, the method comprising: acquiring a plurality of continuous image data frames;

[0272] Identify the target to be detected in the image data frame, extract features of the identified target to be detected, and determine the key points of the human body of the target to be detected;

[0273] Based on the torso key points among the human body key points, judging the direction deviation of the target to be detected relative to the world coordinate system, and determining that the target to be detected is in a tilted posture;

[0274] Based on judging the outlier factor of the human body key points in each image data frame, it is determined whether the target to be detected is in an abnormal behavior state, and the outlier factor is determined based on the angle of each human body key point relative to the torso key point, and the distance of each human body key point relative to the torso key point.

[0275] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0276] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0277] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting abnormal human behavior, characterized in that: The method comprises: Acquire multiple consecutive image data frames; Identify the target to be detected in the image data frame, extract features of the identified target to be detected, and determine the key points of the human body of the target to be detected; Based on the torso key points among the human body key points, judging the direction deviation of the target to be detected relative to the world coordinate system, and determining that the target to be detected is in a tilted posture; Determining whether the target to be detected is in an abnormal behavior state based on judging the outlier factor of the human body key points in each image data frame, wherein the outlier factor is determined based on the angle of each human body key point relative to the trunk key point and the distance of each human body key point relative to the trunk key point; The determining whether the target to be detected is in an abnormal behavior state based on judging the outlier factor of the human body key point in each image data frame includes: When the outlier factors of the key points of the human body in each image data frame are greater than a preset determination threshold, determining that the abnormal behavior state of the target to be detected is a falling state; The outlier factor is: ； in, is the outlier factor; is the sum of relative distances, which is determined based on the distances of each human key point relative to the torso key point; is the sum of relative angles, which is determined based on the angles of each human key point relative to the torso key point; as well as is a hyperparameter.

2. The method for detecting abnormal human behavior according to claim 1, characterized in that: The process of determining the relative distance sum includes: Normalizing each human body key point in the image data frame, and determining the Euclidean distance of each human body key point after the normalization process relative to the torso key point; The relative distance sum is determined based on the sum of the obtained multiple Euclidean distances and the number of human body key points in the image data frame.

3. The method for detecting abnormal human behavior according to claim 1, characterized in that: The process of determining the relative angle sum includes: Normalizing each human body key point in the image data frame, and determining the angle of each human body key point after the normalization process relative to the torso key point; The relative angle sum is determined based on the sum of the multiple angles obtained and the number of key points of the human body in the image data frame.

4. The method for detecting abnormal human behavior according to claim 1, characterized in that: The step of extracting features of the identified target to be detected and determining key points of the human body of the target to be detected includes: Based on a recurrent neural network, feature extraction is performed on the target to be detected to obtain a feature map of the target to be detected; Based on the convolutional neural network, the feature map is further extracted to determine the key points of the human body of the target to be detected.

5. The method for detecting abnormal human behavior according to claim 1, characterized in that: The step of judging the direction deviation of the target to be detected relative to the world coordinate system based on the trunk key points in the human body key points, and determining that the target to be detected is in a tilted posture, includes: Based on the torso key points among the human body key points, determining the angle between the direction of the target to be detected and the Y-axis direction of the camera coordinate system; Based on the angle, adjusting the direction of the target to be detected; Based on the parameters of the camera, the adjusted vertical angle of the target to be detected relative to the world coordinate system is determined, and based on the vertical angle, it is determined that the target to be detected is in a tilted posture.

6. The method for detecting abnormal human behavior according to claim 5, characterized in that: The step of determining, based on the vertical angle, that the target to be detected is in an inclined posture comprises: When it is determined that the vertical angle is greater than a preset tilt angle threshold, it is determined that the target to be detected is in a tilted posture.

7. The method for detecting abnormal human behavior according to claim 1, characterized in that: The identifying the target to be detected in the image data frame includes: Based on a recurrent neural network, feature extraction is performed on the target to be detected to obtain a feature map of the target to be detected; Based on the convolutional neural network, further feature extraction is performed on the feature map to determine the detection frame of the target to be detected and the confidence score of the detection frame.

8. The method for detecting abnormal human behavior according to claim 7, characterized in that: After determining the detection frame of the target to be detected and the confidence score of the detection frame, the method further includes: It is determined that the confidence score of the detection frame of the target to be detected in each image data frame is greater than a preset detection frame score threshold.

9. The method for detecting abnormal human behavior according to claim 8, characterized in that: After determining that the target to be detected is in an inclined posture, the method further includes: Determine the aspect ratio of the detection frame of the target to be detected, and when it is determined that the aspect ratio is less than a preset aspect ratio threshold, determine that the target to be detected is in a squatting state.

10. The method for detecting abnormal human behavior according to claim 1, characterized in that: After determining that the target to be detected is in an inclined posture, the method further includes: Based on the multiple continuous image data frames, determining the movement distance of the key points of the human body between two image data frames; Determining a time interval between two image data frames based on a sampling frequency of the image data frames; Based on the moving distance and the time interval, a moving speed of the target to be detected is determined, and it is determined that the moving speed is less than a preset moving speed threshold.

11. A device for detecting abnormal human behavior, characterized in that: include: An image acquisition module, used for acquiring a plurality of continuous image data frames; An extraction module is used to identify the target to be detected in the image data frame, extract features of the identified target to be detected, and determine the key points of the human body of the target to be detected; A posture judgment module, used to judge the direction deviation of the target to be detected relative to the world coordinate system based on the torso key points among the key points of the human body, and determine that the target to be detected is in a tilted posture; An abnormal behavior judgment module is used to determine whether the target to be detected is in an abnormal behavior state based on the outlier factor of the human body key points in each image data frame, wherein the outlier factor is determined based on the angle of each human body key point relative to the trunk key point and the distance of each human body key point relative to the trunk key point; The determining whether the target to be detected is in an abnormal behavior state based on judging the outlier factor of the human body key point in each image data frame includes: When the outlier factors of the key points of the human body in each image data frame are greater than a preset determination threshold, determining that the abnormal behavior state of the target to be detected is a falling state; The outlier factor is: ； in, is the outlier factor; is the sum of relative distances, which is determined based on the distances of each human key point relative to the torso key point; is the sum of relative angles, which is determined based on the angles of each human key point relative to the torso key point; as well as is a hyperparameter.

12. An abnormal behavior detection device, characterized in that: It comprises the device for detecting abnormal human behavior as described in claim 11.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting abnormal human behavior as described in any one of claims 1 to 10 is implemented.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting abnormal human behavior as described in any one of claims 1 to 10 is implemented.

Images