A method for detecting abnormal human behavior in indoor scenes

Abstract

The present invention discloses a method for detecting abnormal human behavior in indoor scenes, and relates to the technical field of abnormal human behavior detection. The method combines the multi-target tracking algorithm FairMOT, the human posture recognition algorithm AlphaPose and the SVM (Support Vector Machines) model, extracts human feature information by using joint point coordinate data, and marks human movements, aiming to effectively identify abnormal movements in indoor environments, avoid interference from environmental factors, and improve detection speed and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal human behavior detection, and in particular to a method for detecting abnormal human behavior in an indoor scene. Background Art

[0002] With the gradual development of science and technology in modern society, surveillance equipment such as cameras has gradually played a vital role in many places, and analyzing abnormal human behavior from video surveillance has become a hot topic of current research.

[0003] The existing methods for detecting abnormal human behavior generally process and classify the human feature information directly extracted from the image in the early stage. For example, by real-time tracking and detecting the motion trajectory of the human body and setting a threshold to determine whether the motion trajectory is a normal motion trajectory or an abnormal motion trajectory, abnormal behavior is detected. Since the set threshold is difficult to distinguish between normal and abnormal motion trajectories, its detection ability is limited and the accuracy is not high; the existing methods for automatically detecting abnormal behavior based on video processing are greatly affected by the video environment during the detection process, and it is necessary to extract multiple features during the detection process, resulting in a slow detection speed; the existing multi-feature human distraction behavior detection algorithm based on convolutional neural networks is only for single-person abnormal detection. When the number of people is too large and the environment is complex, the detection accuracy will also be greatly reduced. In addition, most researchers currently use image information as the direct research object, which will cause the extracted human feature information to be subject to the influence of different appearances, perspectives, occlusions, etc. in the image, which will bring great difficulties to the analysis process. Summary of the invention

[0004] In view of the shortcomings of the above-mentioned prior art, the present invention provides a method for detecting abnormal human behavior in indoor scenarios. The method combines the multi-target tracking algorithm FairMOT, the human posture recognition algorithm AlphaPose and the SVM (Support Vector Machines) model, and extracts human feature information by using joint point coordinate data to mark human movements. The purpose is to effectively identify abnormal movements in indoor environments, avoid interference from environmental factors, and improve detection speed and accuracy.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for detecting abnormal human behavior in an indoor scene, the method comprising the following steps:

[0007] Step 1: Obtain the original surveillance video and track and detect the people appearing in the video;

[0008] Step 2: Use the human posture recognition algorithm AlphaPose to estimate the human posture of the person tracked and detected in step 1, and obtain the horizontal and vertical coordinates of each joint point of the person through the posture estimation result;

[0009] Step 3: Sliding window segmentation is performed on the data obtained in step 2 as a whole with a preset data length f frame as a unit, so as to obtain a plurality of unit video frame sequences of equal length and a plurality of windows, wherein each of the windows corresponds to each of the unit video frame sequences one by one;

[0010] Step 4: Determine whether the number of video frames containing the same person in each window is less than or equal to a preset missing frame number threshold. If so, it is considered that the person has fewer missing frames, and a frame supplement operation needs to be performed on the person with the missing frames, adding the human joint point information of the previous or next frame of the missing video frame of the person; if not, it is considered that the person has more missing frames, and the person appearing in the corresponding window is deleted;

[0011] Step 5: extracting the human body feature information of each character according to the joint point coordinate information of each character in each window, and obtaining the human body feature information sequence of each window;

[0012] Step 6: Use the EM algorithm to perform feature conversion on the human feature information of each person in each window, converting it into the corresponding mean and variance to reduce the amount of data;

[0013] Step 7: Based on the feature conversion results of step 6, use the K-means clustering algorithm to identify the person with abnormal behavior and obtain the ID sequence of the person with abnormal behavior;

[0014] Step 8: According to the method of step 1 to step 5, the human body feature information of each person is extracted from the video with the custom abnormal action, and each human body feature information is marked as the custom abnormal action to obtain a training data set;

[0015] Step 9: Use the training data set to train the SVM model to obtain a trained SVM model;

[0016] Step 10: Use the trained SVM model to classify the human feature information corresponding to each ID in the ID sequence of the abnormal behavior person obtained in step 7, determine the specific abnormal actions of the abnormal behavior person, and mark them accordingly in the original surveillance video.

[0017] Furthermore, according to the method for detecting abnormal human behavior in indoor scenes, in step 1, a multi-target tracking algorithm FairMOT is used to track and detect the people appearing in the video.

[0018] Furthermore, according to the method for detecting abnormal human behavior in indoor scenes, the output information obtained by tracking and detecting the characters appearing in the video using the multi-target tracking algorithm FairMOT includes the ID of the characters appearing in the video, the sequential number of each video frame, and the coordinate information of the upper left corner of the prediction box and the coordinate information of the lower right corner of the prediction box.

[0019] Further, according to the method for detecting abnormal human behavior in indoor scenes, the method for extracting human feature information of each person according to the joint point coordinate information of each person in each window described in step 5 is: first, the coordinates of the center point C of the human body are calculated using the hip joint coordinate information and the shoulder joint coordinate information of the person; then the midpoint of the left upper arm is used as the vertex of angle A2, and the line between the midpoint of the left upper arm and the midpoint of the left forearm and the line between the midpoint of the left upper arm and the center point C of the human body are used as the two sides of angle A2 to calculate the cosine value of angle A2. At the same time, the midpoint of the right upper arm is taken as the vertex of angle A3, the line between the midpoint of the right upper arm and the midpoint of the right forearm, and the line between the midpoint of the right upper arm and the center point C of the human body are taken as the two sides of angle A3, and the cosine value ZA3 of angle A3 is calculated; finally, these two cosine values and As human feature information, it represents the changes in a person's movements over a period of time.

[0020] Further, according to the method for detecting abnormal human behavior in indoor scenes, the method for converting the human feature information of each person in each window into corresponding means and variances using the EM algorithm described in step 6 is: let each human feature information sequence correspond to a Gaussian mixture model, and estimate the mean and variance of each Gaussian model in the Gaussian mixture model by the EM algorithm. It is known that there are a total of K clusters in the Gaussian mixture model, and it is necessary to find K groups of means and variances. The specific process is as follows:

[0021] 1) Introduce hidden variables, which are the probability that each human feature sequence belongs to each cluster;

[0022] 2) Initialize the parameters and latent variables of the Gaussian mixture model, and update the latent variables through the E step of the EM algorithm, which is the proportion of each cluster;

[0023] 3) In the M step of the EM algorithm, the parameter mean and variance of the Gaussian mixture model are updated by using the latent variables obtained;

[0024] 4) Repeat 2) to 3) until the parameters converge, and obtain the mean and variance corresponding to each human feature information sequence.

[0025] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0026] 1. It can make full use of the image information in the current video, obtain human feature information to the greatest extent from a practical perspective, and has stronger universal applicability.

[0027] 2. Expressing human behavior based on the data itself is more realistic and close to human behavioral changes, in line with the human body structure, with fast data processing speed, further improving the efficiency of detection and processing.

[0028] 3. If overlapping people occur during person tracking and detection, the overlapping people will be missing frames and their joint information will be missing. By setting the sliding window and the missing frame threshold, the overlapping people that exceed the missing frame threshold will be discarded. Otherwise, the missing frames will be supplemented with the joint information of the previous or next frame. This will not affect the overall result. At the same time, since the entire video is not processed at one time, the recognition speed can be increased by setting the window size. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the process of detecting abnormal human behavior in indoor scenes according to the present embodiment;

[0030] Figure 2 Schematic diagram of human body posture estimation in this embodiment;

[0031] Figure 3 This is a schematic diagram of human body features in this embodiment;

[0032] Figure 4 Schematic diagram of the sliding window in this embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. The specific implementation methods described here are only used to explain the present invention and are not used to limit the present invention.

[0034] The core idea of ​​the method of the present invention is: first, obtain the video data to be identified. Track and detect the target person appearing in the video, input the result information of the tracking detection into the human posture recognition algorithm AlphaPose, estimate the posture of the detected target person through the human posture recognition algorithm AlphaPose, obtain the information such as the coordinates of the human joint points through the posture estimation, and save the result as a json file. Set a sliding window to read the json file data corresponding to the video frame sequence of a fixed number of frames each time. Store the information such as the coordinates of the human joint points in the json file into a queue. At this time, due to the overlapping phenomenon of characters during tracking and detection, the target overlapping characters may be missing frames, so that the information such as the coordinates of the human joint points is missing. The missing frame situation is judged by setting the missing frame number threshold. Discard the overlapping characters that are greater than the missing frame number threshold, otherwise, perform a frame filling operation. Make the information data such as the coordinates of the human joint points more complete to avoid affecting the subsequent extraction of human feature information.

[0035] The center point of the human body is calculated using the hip joint and the shoulder joint, and the cosine value on the left is calculated using the midpoint of the left upper arm and the lower arm and the center of the human body. The same is true for the right side, and the two are used as human feature information. Then the two cosine values ​​are input into the EM algorithm for feature conversion, so that the coordinates of the human joint points are converted into mean and variance, so that the huge amount of joint point information is converted into the corresponding eigenvalues ​​(mean and variance) extracted according to the joint point coordinates. The clustering algorithm is used to perform cluster analysis on the new eigenvalues ​​to obtain the ID sequence of the person with abnormal behavior. Finally, the results of the cluster analysis are further analyzed by the SVM model to obtain the specific abnormal actions of the person with abnormal behavior.

[0036] Figure 1 is a schematic diagram of a specific flow chart of a method for detecting abnormal human behavior in an indoor scene according to this embodiment. Figure 1 As shown, the method for detecting abnormal human behavior in an indoor scene includes the following steps:

[0037] Step 1: Obtain the original surveillance video and track and detect the people appearing in the video;

[0038] In this embodiment, the method for tracking and detecting the person appearing in the original surveillance video is specifically: using the multi-target tracking algorithm FairMOT to track and detect the person appearing in the video. The specific process is as follows:

[0039] 1) First, the original surveillance video is used in frames to detect people using the target detection network CenterNet in the multi-target tracking algorithm FairMOT to obtain the detection result of each person in the current frame image. The detection result is the position and embedding vector of the prediction box of each person in the current frame image.

[0040] 2) Then, the trajectory of each character is generated through the detection results, and a trajectory pool is set up to put all the trajectories into the trajectory pool. The appearance distance matrix is ​​calculated by the detection results and the trajectory pool. At this time, the Kalman algorithm is used to predict the mean and variance of the trajectory pool. The mean and variance can be used to predict the possible current position of each trajectory in the trajectory pool, and the new cost distance matrix is ​​calculated based on the position of the predicted trajectory, the position of the predicted box in the detection result, and the appearance distance matrix.

[0041] 3) The new cost distance matrix is ​​then used to match the trajectory pool and the detection results through the Hungarian algorithm to obtain the optimal matching result and complete the tracking detection of the person. For the detection results that are not matched, they are initialized as a new trajectory and added to the trajectory pool.

[0042] 4) Repeat 1) to 3) until the entire video segment is processed.

[0043] Finally, the multi-target tracking detection algorithm FairMOT outputs the ID of the person appearing in the video, the sequence number of each video frame, and the coordinates of the upper left and lower right corners of the prediction box, and stores all the information as a .txt file.

[0044] Step 2: Use the human posture recognition algorithm AlphaPose to estimate the human posture of the person tracked and detected in step 1, and obtain the horizontal and vertical coordinates of each joint point of the person through the posture estimation result;

[0045] In this embodiment, a top-down method is adopted. According to the person ID obtained in step 1, the sequence number of each video frame, and the coordinates of the upper left corner and the lower right corner of the prediction frame, the human posture recognition algorithm AlphaPose is used to first detect the person in the prediction frame to obtain the human detection frame, and then the posture of the person is estimated in each human detection frame. Then, the coordinate information of each joint point of the person is obtained through the posture estimation result and stored as a .json file. The json file includes the person ID, the sequence number of each video frame, and the horizontal and vertical coordinates of each joint point of the person. The joint points of the person are as follows: Figure 2 shown.

[0046] Step 3: Segment the data obtained in step 2 in a sliding window manner with a preset data length f frame as a unit to obtain a number of unit video frame sequences of equal length;

[0047] Since the subsequent steps always process data in units of fixed f frames, each time only the json file data corresponding to the fixed length of the unit video frame sequence is processed, so the data needs to be segmented in this step. The specific method is to read the json file and use the sliding window to segment the data in the json file as a whole, such as Figure 4As shown, the width of the sliding window is equal to the preset data length f frame, and in this implementation manner, f=60 is preset.

[0048] Step 4: Determine whether the number of video frames containing the same person in each window is less than or equal to a preset missing frame number threshold. If so, it is considered that the missing frames of the person are relatively small, and a frame filling operation needs to be performed on the missing frame of the person, adding the data of the previous frame or the next frame of the missing video frame of the person; if not, it is considered that the missing frames of the person are relatively large, and the person appearing in the corresponding window is deleted;

[0049] When characters overlap, the character's ID will switch, resulting in missing frames, which in turn leads to missing data on the coordinates of the character's joints. Therefore, it is necessary to pre-set a missing frame threshold to judge the missing frames of each character. If it is less than or equal to the preset missing frame threshold, it is considered that the character has fewer missing frames, and the missing frame operation needs to be performed on the character, specifically, the data of the previous or next frame of the missing video frame of the character is added. Otherwise, it is considered that the character has more missing frames, and the character appearing in the corresponding window is deleted.

[0050] After the processing of this step is completed, the coordinate data of each joint point with the same character ID is stored in the queue format shown in Table 1.

[0051] Table 1 Queue storage table

[0052]

[0053] Step 5: extracting the human body feature information of each character according to the joint point coordinate information of each character in each window, and obtaining the human body feature information sequence of each window;

[0054] In this embodiment, the method for extracting the human body feature information of each person according to the joint point coordinate information of each person in each window is as follows: first, the coordinates of the human body center point C are calculated according to formula (1) using the hip joint coordinate information and the shoulder joint coordinate information of the person; Figure 3 As shown, the midpoint of the left upper arm is taken as the vertex of angle A2, the line between the midpoint of the left upper arm and the midpoint of the left forearm and the line between the midpoint of the left upper arm and the center point C of the human body are taken as the two sides of angle A2, and the cosine value of angle A2 is calculated according to formula (2): At the same time, if Figure 3 As shown, the midpoint of the right upper arm is taken as the vertex of angle A3, the line between the midpoint of the right upper arm and the midpoint of the right forearm and the line between the midpoint of the right upper arm and the center point C of the human body are taken as the two sides of angle A3, and the cosine value of angle A3 is calculated according to formula (3): These two cosine values ​​are used as human feature information to represent the changes in a person's movements over a period of time.

[0055] The formula is as follows:

[0056]

[0057]

[0058]

[0059] Among them, (x0, y0) is the coordinate of the center point C of the human body; Represents the sum of the horizontal coordinates of the shoulder joint point and the hip joint point; Represents the sum of the ordinates of the shoulder joint and the hip joint. (x1, y1) is the coordinate of the midpoint of the left upper arm, (x2, y2) is the coordinate of the midpoint of the left forearm, (x3, y3) is the coordinate of the midpoint of the right upper arm, and (x4, y4) is the coordinate of the midpoint of the right forearm.

[0060] Step 6: Use the EM algorithm to perform feature conversion on the human feature information of each person in each window, converting it into the corresponding mean and variance to reduce the amount of data;

[0061] In this embodiment, this step specifically uses the EM algorithm to perform feature conversion on the human feature information of each character, specifically converting the two cosine values ​​of each character Input into the EM algorithm for iteration, converting the coordinates of the human joint points into mean and variance, thus reducing the huge amount of data of the joint point information and improving the detection speed. The specific process is as follows:

[0062] The Gaussian mixture model is composed of multiple Gaussian models, each of which has its own mean and variance. Each human feature information sequence corresponds to a Gaussian mixture model, and the EM (Expectation Maximization) algorithm is used to estimate the mean and variance of each Gaussian model in the Gaussian mixture model. It is known that there are K clusters in the Gaussian mixture model, so it is necessary to find the mean and variance of K groups. This further reduces the amount of data for the subsequent clustering algorithm. The specific process is as follows:

[0063] 1) Introduce hidden variables, which are the probability that each human feature sequence belongs to each cluster.

[0064] 2) Initialize the parameters and latent variables of the Gaussian mixture model, and update the latent variables through the E step (Expectation-Step, E step) in the EM algorithm, which is the proportion of each cluster.

[0065] 3) In the M-step (Maximization-Step, M-step) of the EM algorithm, the obtained latent variables are used to update the parameter mean and variance of the Gaussian mixture model.

[0066] 4) Repeat 2) to 3) until the parameters converge and obtain the final eigenvalues: mean and variance.

[0067] Step 7: Based on the feature conversion results of step 6, use the K-means clustering algorithm to identify the person with abnormal behavior and obtain the ID sequence of the person with abnormal behavior;

[0068] In this embodiment, the mean and variance obtained in step 6 are clustered separately. Taking the mean as an example, the mean is first used as the initial data set for clustering, and K=2 (normal behavior class, abnormal behavior class) is set. K points are randomly selected from the data set as cluster centers, and the distance between each point in the data set and each cluster center is calculated, and each point in the data set is regressed to the cluster center closest to it. In this way, the categories of all points in the data set after the first round of iteration are obtained, and then the mean of all points in each class is calculated as the new cluster center, and so on, and iterates continuously until the cluster center no longer changes, and the final abnormal behavior class is determined, and then the ID sequence of the person with abnormal behavior is determined.

[0069] Step 8: Extract the human feature information of each person from the video with the custom abnormal action, and mark each human feature information as the custom abnormal action to obtain a training data set;

[0070] In this embodiment, by shooting videos with custom abnormal actions and searching for videos with custom abnormal actions in some real scenes, human feature information corresponding to abnormal actions is extracted, and each human feature information is artificially marked as a custom abnormal action, with a total of N categories marked. The human feature information corresponding to all abnormal actions and their corresponding marking information constitute a training data set.

[0071] Step 9: Use the training data set to train the SVM model to obtain a trained SVM model;

[0072] Using the one-to-one method in the SVM model, N types of training data are combined in pairs to build N×(N-1) / 2 SVM models. Assume that N=4, which are four types of abnormal actions: A, B, C, and D. When training, select the vectors corresponding to (A, B), (A, C), (A, D), (B, C), (B, D), and (C, D) as the training set, and you will get six training results. When testing, test the corresponding vectors on the six results respectively, and then take voting to get the classification results. Finally, get a set of test results, and the SVM model training is complete.

[0073] Step 10: Use the trained SVM model to classify the human feature information corresponding to each ID in the ID sequence of the abnormal behavior person, determine the specific abnormal action of the abnormal behavior person, and feed back the final identified abnormal action to the original surveillance video, and mark it accordingly in the original surveillance video, that is, mark the abnormal action in the original surveillance video.

[0074] In this embodiment, the voting method in the trained SVM model is used to vote on the human feature information A2 and A3 corresponding to each ID in the ID sequence of the abnormal behavior person, and A2 and A3 are classified according to the number of votes, and finally the abnormal actions corresponding to A2 and A3 are obtained.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. Therefore, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for detecting abnormal human behavior in indoor scenes, characterized in that: The method comprises the following steps: Step 1: Obtain the original surveillance video and track and detect the people appearing in the video; Step 2: Use the human posture recognition algorithm AlphaPose to estimate the human posture of the person tracked and detected in step 1, and obtain the horizontal and vertical coordinates of each joint point of the person through the posture estimation result; Step 3: Sliding window segmentation is performed on the data obtained in step 2 as a whole with a preset data length f frame as a unit, so as to obtain a plurality of unit video frame sequences and a plurality of windows of equal length, wherein each of the windows corresponds to each of the unit video frame sequences one by one; Step 4: Determine whether the number of video frames containing the same person in each window is less than or equal to a preset missing frame number threshold. If so, it is considered that the person has fewer missing frames, and a frame supplement operation needs to be performed on the person with the missing frames, adding the human joint point information of the previous or next frame of the missing video frame of the person; if not, it is considered that the person has more missing frames, and the person appearing in the corresponding window is deleted; Step 5: extracting the human body feature information of each character according to the joint point coordinate information of each character in each window, and obtaining the human body feature information sequence of each window; Step 6: Use the EM algorithm to perform feature conversion on the human feature information of each person in each window, converting it into the corresponding mean and variance to reduce the amount of data; Step 7: Based on the feature conversion results of step 6, use the K-means clustering algorithm to identify the person with abnormal behavior and obtain the ID sequence of the person with abnormal behavior; Step 8: According to the method of step 1 to step 5, extract the human feature information of each person from the video with the custom abnormal action, and mark each human feature information as the custom abnormal action to obtain a training data set; Step 9: Use the training data set to train the SVM model to obtain a trained SVM model; Step 10: Use the trained SVM model to classify the human feature information corresponding to each ID in the ID sequence of the abnormal behavior person obtained in step 7, determine the specific abnormal actions of the abnormal behavior person, and mark them accordingly in the original surveillance video; The method for extracting the human body feature information of each person according to the joint point coordinate information of each person in each window described in step 5 is as follows: firstly, the coordinates of the human body center point C are calculated using the coordinate information of the hip joint and the coordinate information of the shoulder joint of the person; and then the midpoint of the left upper arm is used as the angle. The vertex of the left upper arm, the line between the midpoint of the left upper arm and the midpoint of the left forearm, and the line between the midpoint of the left upper arm and the center point C of the human body are used as the angles. The two sides of The cosine value of At the same time, the midpoint of the right upper arm is used as the angle The vertex of the right upper arm, the line between the midpoint of the right upper arm and the midpoint of the right forearm, and the line between the midpoint of the right upper arm and the center point C of the human body are used as angles. The two sides of The cosine value of ; Finally, these two cosine values and As human feature information, it represents the changes in a person's movements over a period of time.

2. The method for detecting abnormal human behavior in indoor scenes according to claim 1, characterized in that: In step 1, a multi-target tracking algorithm FairMOT is used to track and detect the people appearing in the video.

3. The method for detecting abnormal human behavior in indoor scenes according to claim 2, characterized in that: The output information obtained by tracking and detecting the characters appearing in the video using the multi-target tracking algorithm FairMOT includes the ID of the characters appearing in the video, the sequential number of each video frame, and the coordinate information of the upper left corner of the prediction box and the coordinate information of the lower right corner of the prediction box.

4. The method for detecting abnormal human behavior in indoor scenes according to claim 1, characterized in that: The method of using the EM algorithm to convert the human feature information of each person in each window into the corresponding mean and variance in step 6 is as follows: let each human feature information sequence correspond to a Gaussian mixture model, and estimate the mean and variance of each Gaussian model in the Gaussian mixture model by the EM algorithm. It is known that there are K clusters in the Gaussian mixture model, and it is necessary to find K groups of means and variances. The specific process is as follows: 1) Introduce hidden variables, which are the probability that each human feature sequence belongs to each cluster; 2) Initialize the parameters and latent variables of the Gaussian mixture model, and update the latent variables through the E step of the EM algorithm, which is the proportion of each cluster; 3) In the M step of the EM algorithm, the parameter mean and variance of the Gaussian mixture model are updated by using the latent variables obtained; 4) Repeat 2) to 3) and iterate until the parameters converge to obtain the mean and variance corresponding to each human feature information sequence.

Images