A method, device, electronic device and medium for identifying abnormal behavior of personnel
Through the probability grammar dictionary, the prior probability distribution of the time and space changes of people in the monitoring scenario is described, the human body posture vector is extracted and syntax analysis is performed, and the problem of limited reliance on labeled data and detection categories in the existing technology is solved, and a variety of abnormal behaviors is realized. Identification and accurate detection reduces costs and manual participation.
Patent Information
- Application Number
- CN202410198747.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-02-22
AI Technical Summary
Existing indoor monitoring systems rely on a large amount of manual participation and labeling data, resulting in limited detection categories, high cost and poor generalization capabilities, making it difficult to identify multiple types of abnormal behaviors.
The probability grammar dictionary is used to describe the prior probability distribution of the temporal and spatial changes of people through unlabeled historical monitoring data, extract the human posture vector and perform grammatical analysis, identify behavior abnormalities, and reduce dependence on the annotated data.
The identification of various types of behavior abnormalities is realized, the recognition scope is expanded, the algorithm cost is reduced, and the detection accuracy and efficiency is improved.
Smart Images

Figure CN117877124B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, electronic device and medium for identifying abnormal human behavior. Background Art
[0002] To ensure the safety of production and life, more and more occasions are equipped with monitoring systems to monitor people's behavior. For example, the conference room is an important place for people to carry out daily work, and people pay more and more attention to the safety of the conference room. In some important meeting occasions, supervisors often need to know the activities of all participants in a timely manner. Therefore, the indoor monitoring system is used to monitor the participants to determine whether there are abnormal behaviors, such as physical conflicts, people falling, illegal recording, filming, illegal intrusion of the venue, etc., so as to make the conference room safer, more efficient and more confidential.
[0003] However, the existing indoor monitoring system still requires a lot of manual participation to identify various abnormal behaviors, and due to limited monitoring manpower, it is impossible to monitor multiple conference rooms at the same time, resulting in high monitoring costs and low efficiency.
[0004] Alternatively, there are also existing technologies that use machine learning algorithms trained with a large amount of labeled data to detect abnormal behavior of personnel. However, such algorithms have limited detection categories and can only detect abnormal behavior categories pre-defined in the training data. Once other types of abnormal behaviors that are not labeled during the training process appear, the algorithm will not be able to accurately detect them; this will cause the R&D cost of the algorithm to continue to increase as the types of abnormal behaviors concerned increase, and the hardware configuration requirements will also increase as the complexity of the system increases, resulting in high system deployment costs. At the same time, machine learning algorithms require a large amount of labeled data of different types of abnormal behaviors, that is, there is a problem of labeled data dependence, but in reality, abnormal data is relatively scarce, and it is difficult to obtain enough data to train multiple types of abnormal behavior recognition models, resulting in poor generalization of the algorithm and high training costs. Summary of the invention
[0005] In view of this, the purpose of the present application is to provide a method, device, electronic device and medium for identifying abnormal human behavior, so as to solve the problems existing in the prior art of reliance on labeled data, limited detection categories and high detection costs.
[0006] The present invention provides a method for identifying abnormal behavior of a person, the method comprising:
[0007] Extracting multiple frames of surveillance images within a target time window from the surveillance video of the target surveillance scene;
[0008] Extracting a human body posture vector of each person in the target monitoring scene within the target time window from the multiple frames of monitoring images;
[0009] For each person, a pre-constructed probabilistic grammar dictionary is used to perform grammatical analysis on the person's human posture vector to determine the amount of information corresponding to the person's current performance behavior; wherein the probabilistic grammar dictionary uses the grammatical structure of a spatiotemporal AND / OR graph to describe the prior probability distribution related to the spatiotemporal changes of the person in the target monitoring scene through unlabeled historical monitoring data;
[0010] Based on the amount of information corresponding to the person's current behavior and a preset information threshold, it is determined that the person is currently behaving abnormally.
[0011] The present application also provides a device for identifying abnormal behavior of a person, the device comprising:
[0012] A monitoring image acquisition module is used to extract multiple frames of monitoring images within a target time window from the monitoring video of the target monitoring scene;
[0013] A human posture vector extraction module is used to extract the human posture vector of each person in the target monitoring scene within the target time window from multiple frames of monitoring images;
[0014] An information volume calculation module is used to perform grammatical analysis on the human posture vector of each person using a pre-built probabilistic grammar dictionary to determine the information volume corresponding to the current behavior of the person; wherein the probabilistic grammar dictionary uses the grammatical structure of a spatiotemporal AND / OR graph to describe the prior probability distribution related to the spatiotemporal changes of the person in the target monitoring scene through unlabeled historical monitoring data;
[0015] The identification module is used to determine whether the person is currently behaving abnormally based on the amount of information corresponding to the person's current behavior and a preset information threshold.
[0016] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of a method for identifying abnormal personnel behavior as described above are performed.
[0017] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for identifying abnormal behavior of a person as described above are executed.
[0018] The embodiments of the present application provide a method, device, electronic device and medium for identifying abnormal human behavior. By describing the prior probability distribution related to the spatiotemporal changes of people in the target monitoring scene through a probabilistic grammar dictionary, various types of abnormal behaviors can be identified without relying on labeled data, thereby expanding the recognition range, eliminating the dependence on labeled data and reducing the algorithm cost.
[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 A flow chart of a method for identifying abnormal human behavior provided by an embodiment of the present application is shown;
[0022] Figure 2 A schematic diagram showing the structure of a space-time AND / OR graph provided in an embodiment of the present application is shown;
[0023] Figure 3 A schematic diagram showing an AUC detection result provided in an embodiment of the present application is shown;
[0024] Figure 4 A schematic diagram of the structure of a device for identifying abnormal human behavior provided in an embodiment of the present application is shown;
[0025] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0026] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.
[0027] According to research, in order to ensure the safety of production and life, more and more occasions are equipped with monitoring systems to monitor people's behavior. For example, the conference room is an important place for people to carry out daily work, and people pay more and more attention to the safety of the conference room. In some important conference occasions, supervisors often need to know the activities of all participants in a timely manner. Therefore, the indoor monitoring system is used to monitor the participants to determine whether there are abnormal behaviors, such as physical conflicts, people falling, illegal recording, filming, illegal intrusion of the venue, etc., so as to make the conference room safer, more efficient and more confidential.
[0028] However, the existing indoor monitoring system still requires a lot of manual participation to identify various abnormal behaviors, and due to limited monitoring manpower, it is impossible to monitor multiple conference rooms at the same time, resulting in high monitoring costs and low efficiency.
[0029] Alternatively, there are also existing technologies that use machine learning algorithms trained with a large amount of labeled data to detect abnormal behavior of personnel. However, such algorithms have limited detection categories and can only detect abnormal behavior categories pre-defined in the training data. Once other types of abnormal behaviors that are not labeled during the training process appear, the algorithm will not be able to accurately detect them; this will cause the R&D cost of the algorithm to continue to increase as the types of abnormal behaviors concerned increase, and the hardware configuration requirements will also increase as the complexity of the system increases, resulting in high system deployment costs. At the same time, machine learning algorithms require a large amount of labeled data of different types of abnormal behaviors, that is, there is a problem of labeled data dependence, but in reality, abnormal data is relatively scarce, and it is difficult to obtain enough data to train multiple types of abnormal behavior recognition models, resulting in poor generalization of the algorithm and high training costs.
[0030] Based on this, the embodiments of the present application provide a method, device, electronic device and medium for identifying abnormal human behavior to solve the problems existing in the prior art of reliance on labeled data, limited detection categories and high detection costs.
[0031] See also Figure 1 , Figure 1 This is a flow chart of a method for identifying abnormal human behavior provided by an embodiment of the present application. Figure 1 As shown in , the identification method provided by the embodiment of the present application includes:
[0032] S101. Extracting multiple surveillance image frames within a target time window from a surveillance video of a target surveillance scene.
[0033] Here, the target monitoring scene can be the monitoring scene of a conference room in a company, and the monitoring video is the video captured by the monitoring camera in the conference room. The target time window can be understood as a window that moves forward in the time domain with a specific step size. For example, the length of the target time window is set to n frames. When the length of the target time window is n frames, n frames of scene monitoring images should be extracted from the monitoring video.
[0034] In this step, the surveillance video captured by the surveillance camera in the target surveillance scene can be firstly obtained, and then multiple frames of surveillance images within the target time window can be extracted from the surveillance video.
[0035] S102: extracting a human body posture vector of each person in the target monitoring scene within the target time window from the multiple frames of monitoring images.
[0036] Here, the human body posture vector can be a vector representing the human body posture formed by the coordinates of the human body key points, rotation angle and other features of each person in the monitoring image. Specifically, the generally used 3D human body model contains 24 human body key points. Considering that the lower body of the human body will be inaccurately estimated due to occlusion in the conference room scene, only 14 key points of the upper body are taken according to the characteristics of the target monitoring scene in the embodiment of the present application. Therefore, each person contains 14 3D (rotation angle along the x-axis, rotation angle along the y-axis, rotation angle along the z-axis, in radians) posture data in each frame of the monitoring image. Exemplarily, these key points can be expressed as:
[0037] J i =(θ xi ,θ yi ,θ zi ),i=1,2,…,24
[0038] Among them, i represents the index of the key point.
[0039] For the above step S102, in the specific implementation, for each frame of the scene monitoring image, first extract the posture vector of each person in the scene monitoring image, and then flatten the key point angles of each person in the continuous n-frame scene monitoring image to obtain the human posture vector of the person in the target time window. Specifically, flatten the three-dimensional rotation angles of the key points of each person in the continuous n-frame scene monitoring image to obtain the human posture vector of n*14*3, which is recorded as V=[v1,v2,…,v n ], where v i It is a 42*3-dimensional column vector, representing the posture vector of the human body in the i-th frame.
[0040] In a possible implementation, step S102 may include:
[0041] S1021. Input multiple frames of surveillance images into a pre-trained human body detection model, detect each person in each frame of surveillance image, and mark a human body detection frame on each frame of surveillance image.
[0042] For the above step S1021, in the specific implementation, multiple frames of monitoring images are input into a pre-trained human body detection model for human body detection, and the detection results of the human body position and size of each person in each frame of monitoring image are obtained, and the human body detection frame corresponding to each person is marked in each frame of scene monitoring image to record the detection results. Here, the human body detection model can be a model such as MTCNN, SSD, YOLO, etc., and this application does not make specific limitations on this.
[0043] S1022: Input the multiple surveillance images into a pre-trained human tracking model to obtain the motion trajectory of each person in the multiple surveillance images.
[0044] For the above step S1022, during the specific implementation, the human body tracking model performs target tracking according to the human body detection frame corresponding to each person in each frame of the scene monitoring image. Specifically, each frame of the monitoring image marked with the human body detection frame is input into the pre-trained human body tracking model to determine the motion trajectory of each person in multiple frames of scene monitoring images. The human body tracking model here can be DeepSort, Kalman filter, particle filter, etc., and this application does not make specific limitations on this. Assuming that there are a total of x people active in the target monitoring scene, then if the tracking is successful, x human body target IDs and their motion trajectories in n frames of scene monitoring images will be obtained.
[0045] S1023, inputting the motion trajectory of each person in the multiple frames of monitoring images into a pre-trained human posture detection model to obtain posture features of multiple human body key points of each person in each frame of monitoring images.
[0046] For the above step S1023, in the specific implementation, the posture data of the human image corresponding to the human tracking frame in each frame of the monitoring image is extracted. Specifically, the human image corresponding to the human detection frame of each person is input into a pre-trained human posture detection model to obtain the posture features of multiple key points of each person in each frame of the scene monitoring image. Here, the human posture detection model can be OpenPose, AlphaPose, MetaPose, etc., which is not specifically limited in this application.
[0047] S1024. Flatten the posture features of multiple human body key points of each person in each frame of the monitoring image to obtain a human body posture vector of each person in the target time window.
[0048] Regarding the above step S1024, in a specific implementation, the posture features of each key point in each frame of monitoring image are spliced and flattened to form a human body posture vector of each person.
[0049] S103: For each person, use a pre-built probabilistic grammar dictionary to perform grammatical analysis on the person's human posture vector to determine the amount of information corresponding to the person's current performance behavior.
[0050] Among them, the probabilistic grammar dictionary uses the grammatical structure of Spatial-Temporal And-Or Graph (ST-AOG) to describe the prior probability distribution related to the spatiotemporal changes of people in the target monitoring scene through unlabeled historical monitoring data.
[0051] According to the monitoring needs of different indoor scenes, different types of prior probability distributions can be defined, and then the probability distribution is learned and modeled based on the training data to obtain the ST-AOG containing nodes, edges and probabilities, completing the construction of the entire probabilistic grammar dictionary.
[0052] More specifically, the probabilistic grammar dictionary can be constructed in the following manner:
[0053] Step 1: using multiple frames of historical monitoring images in the historical monitoring video of the target monitoring scene as the historical monitoring data, and extracting multiple sample human body posture vectors from the multiple frames of historical monitoring images.
[0054] In the specific implementation, the training data is the historical monitoring data of normal meeting records. For example, the historical monitoring video with a duration of about 30 minutes is sampled at 5 frames per second to obtain nearly 9,000 historical monitoring images, which are recorded as D1.
[0055] Before starting model training, the training data needs to be structured, that is, multiple sample human posture vectors are extracted. The specific method can refer to the description in step S102, and the same effect can be achieved.
[0056] Step 2: Define the original structure of the spatiotemporal AND / OR graph based on the characteristics of the target monitoring scene.
[0057] Here, ST-AOG is a directed acyclic graph, and the original structure of the space-time AND-OR graph includes nodes and edges representing the association relationship between nodes; there are multiple ways to define the ST-AOG of the conference room, and an example of the original structure of the space-time AND-OR graph will be described in detail below.
[0058] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of a time-space AND / OR graph provided in an embodiment of the present application. Figure 2As shown, each time t corresponds to a 3-layer AND / OR subgraph structure, and a total ST-AOG is formed through the connection of root nodes between different times.
[0059] The types of nodes include: scene nodes representing the target monitoring scene at different times, area nodes into which the monitoring image of the target monitoring scene is divided, and personnel nodes where there are people in each monitoring area. "Monitoring scene t" represents the scene node of the target monitoring scene at the current moment, and is an "AND node"; its child nodes "area 1", "area 2", etc. are area nodes of all monitoring areas divided from the monitoring image, for example, area 1 is the desktop, area 2 is the upper right side of the desktop, etc. The area node is an OR node, and the connection relationship with the child nodes is "OR". If area 1 is connected to two child nodes "person 1" and "person 2", it means that visually "person 1" and "person 2" are both in area 1. The "person" node is a leaf node. A person may visually exist in multiple areas at the same time, so the personnel node of a person will be connected to multiple area nodes.
[0060] In this way, by connecting the monitoring scene at the current moment with the monitoring scene at the previous moment, we construct a ST-AOG that can describe the spatiotemporal changes of the scene. Each node in the ST-AOG has rich attribute information. Exemplarily, in the embodiment of the present application, the attribute information includes the area and shape of the region in the region node, and the position of the human body, the position of the key points of the human body, the quaternion of the 3D joint point posture, and other information in the personnel node. The attributes of each node can be flattened into a 1-dimensional vector to constitute the attribute vector of the node.
[0061] Step 3: Define the probability type of the prior probability distribution described by the probability grammar dictionary.
[0062] Dozens of prior probabilities can be defined in the probability grammar dictionary, including but not limited to: the probability of node attribute occurrence; the probability of node attribute jump; the probability of node co-occurrence; the probability of connection between personnel nodes and regional nodes; the probability of a node appearing with a certain area as a condition, etc. Different types of prior probability distributions can be defined according to the monitoring needs of different indoor scenes.
[0063] Step 4: Count multiple frames of sample monitoring images and multiple sample human posture vectors to obtain a probability model for each prior probability distribution.
[0064] In this step, a probability model of each defined prior probability distribution is constructed by learning and modeling the probability distribution of multiple frames of sample monitoring images and multiple sample human posture vectors.
[0065] Step 5: Use the probability parameters of the probability model of each prior probability distribution as the node attributes of the personnel node and add them to the spatiotemporal AND / OR graph to obtain the probability grammar dictionary.
[0066] In this step, the probability model can be characterized by a number of probability parameters, and the probability parameters are used as node attributes of the personnel nodes and added to the personnel nodes in the spatiotemporal AND / OR graph to obtain a probability grammar dictionary including nodes, edges and probabilities.
[0067] In one example, the probability type of the defined prior probability distribution includes at least one of the following: the prior probability distribution of sample human body posture vector values, the prior probability distribution of sample human body posture vector jumps, and the prior probability distribution of the appearance of a human body in each monitoring area in the monitoring image.
[0068] First, for the prior probability distribution P1(w i ), that is, the probability distribution of attribute values: the human posture vectors w of all human bodies in the historical surveillance videos are i By performing value statistics, we can obtain a prior probability distribution P1(w i ); a Gaussian mixture probability model can be used to fit this high-dimensional probability distribution, and the model parameters of the successfully fitted Gaussian mixture probability model are recorded. For example, a Gaussian mixture probability model with a probability parameter of 5 can be used to fit this high-dimensional probability distribution.
[0069] Second, for the prior probability distribution P2(w i |w j ), that is, the attribute jump probability: the human body posture vector w of all human bodies in the historical surveillance video i Perform statistics on the values of the previous and next frames, that is, take the value w at the current moment j In the case of i The probability of the human body posture vector value change is obtained by the prior probability distribution P2(w i |w j ); The prior probability distribution can be discretized to obtain an attribute change probability matrix M as a model parameter.
[0070] Second, for the prior probability distribution of the presence of a human body in each surveillance area in the surveillance image, that is, the connection probability between the person node and the area node: the appearance positions of all human bodies in the historical surveillance video are counted, denoted as P3(A i ), that is, area A i The prior probability distribution of the human body appearing in .
[0071] In this way, the historical monitoring data used in the process of building the probabilistic grammar dictionary does not need to be pre-labeled, which can eliminate the dependence on labeled data and reduce algorithm costs. On the other hand, compared with the machine learning algorithm trained with labeled data, the prior probability distribution is described by the probabilistic grammar dictionary, which can identify other types of unlabeled abnormal behaviors when applied, thereby expanding the recognition range.
[0072] In specific implementation, step S103 may include:
[0073] For each person, the probability model of each prior probability distribution is restored from the probability parameters included in the node attributes of the person node corresponding to the person in the probability grammar dictionary; the human body posture vector of the person is used as the sampling point of each prior probability distribution and substituted into the corresponding probability model to obtain each probability related to the current performance of the person; based on each probability related to the current performance of the person, the amount of information corresponding to the current performance of the person is determined.
[0074] Here, the purpose of parsing the human body posture vector of the person by the probabilistic grammar dictionary is to obtain the spatiotemporal parsing features of the current performance behavior. The specific process of grammar parsing is divided into two main parts: spatial feature extraction and temporal feature extraction.
[0075] For spatial features, a probabilistic grammar model is used to calculate the probability value and information content of each joint point. The information value probability and information content of these joint points can be expressed as:
[0076] P(J i ),i=1,2,…,24
[0077] Info(J i )=-log2P(J i ),i=1,2,…,24
[0078] The extraction of time series features focuses on analyzing the changes of joint points over time. By setting the sampling detection S (unit: second) and the number of sampling points T, a series of sequence features that change over time can be extracted by sliding the time window forward from the current time scale t. This can be expressed by the following formula:
[0079] TJ t,S,T ={J i (tn·S)|n=0,1,…,T-1}
[0080] For the temporal feature information, the probability grammar model is used to calculate the state transition probability of a joint point at two consecutive moments. Assume that the state of the joint point at time t is The state at time t+1 is The state transition probability can be expressed by the conditional probability density function, that is, In a given The probability distribution under the condition is expressed as
[0081] In one example, for the newly input continuous n-frame image, the above attribute expansion method is used to obtain a n*14*3 vector, which is recorded as V=[v1,v2,…,v n ] Consider V as the sampling point of probability distribution P1 and substitute it into Gaussian mixture model to obtain P1(v1), P1(v2), ..., P1(v n )n probability values; v i 、v j As sampling points of probability distribution P2, we can obtain P2(v2|v1), P2(v3|v2), P2(v4|v3), ..., P1(v n |v n-1 ); v i 、v j As the sampling point of probability distribution P3, determine P3(A i ); then, the information volume I1 and I2 are calculated by the following formula:
[0082]
[0083] or
[0084]
[0085] It should be noted that the formula for determining the amount of information corresponding to the person's current behavior based on each probability related to the person's current behavior can adopt other formulas in the prior art, as long as I is inversely proportional to the probability.
[0086] S104: Determine that the person is currently behaving abnormally based on the amount of information corresponding to the person's current behavior and a preset information amount threshold.
[0087] In a possible implementation manner, the preset information volume threshold is determined in the following manner:
[0088] Determining a first preset information amount threshold value of the information amount according to the prior probability distribution of the sample human body posture vector value and the prior probability distribution of the sample human body posture vector jump; and / or,
[0089] A second preset information amount threshold value of the information amount is determined according to a priori probability distribution of the presence of a human body in each monitoring area in the monitoring image.
[0090] Exemplarily, the first preset information volume threshold I thr1 The calculation formula can be expressed as:
[0091]
[0092] or,
[0093] The second preset information volume threshold I thr2 The calculation formula can be expressed as:
[0094]
[0095] or,
[0096] Where n i The time window length of the i-th sample.
[0097] In this step, if I1≥I thr1 Or I2 ≥ I thr2 , it is considered that the person has abnormal behavior in the frame at time t=n.
[0098] Furthermore, after step S104, the identification method further includes:
[0099] S105 , concatenating the body posture vector of the person in the target time window and the node information corresponding to the current behavior of the person to obtain a behavior feature vector of the person.
[0100] In this step, the human posture vectors corresponding to the multi-frame surveillance images within the time window of the person and the node information corresponding to the current behavior of the person are flattened according to the order and time sequence of the human key points to obtain the corresponding behavior feature vectors of the person in the multi-frame surveillance images, so as to form the spatiotemporal feature description of the person in the target time window.
[0101] For example, m human key points are selected:
[0102] J=[0,1,2,3,4,5,6,9,13,14,16,17,18,19,20,21,22,23]
[0103] set up is the Euler angle of the i-th joint point in the t-th frame is the information amount of the i-th joint point in the t-th frame, n is the number of frames considered, then the feature vector of the t-th frame can be expressed as:
[0104]
[0105] S106: Input the behavior feature vector of the person into a pre-trained behavior classification model to determine the abnormal type of the current abnormal behavior of the person.
[0106] In this embodiment, random forest is selected as the classifier, and the number of forest trees n_estimators=50 and the maximum depth of the tree max_depth=8 are selected.
[0107] The flattened eigenvectors F1, F2, ... F m Input the trained classifiers in sequence to obtain the classification labels (i.e. the abnormal types of abnormal behaviors) L1, L2, ...L m .
[0108] The behavior classification model is trained in the following way:
[0109] Step 1: extracting multiple frames of abnormal historical monitoring images with abnormal behaviors from the historical monitoring video of the target monitoring scene.
[0110] Step 2: annotate the multiple frames of abnormal history monitoring images with abnormal type labels, and extract multiple abnormal human posture vectors from the multiple frames of abnormal history monitoring images.
[0111] For steps 1 and 2 above, first construct a dataset for training the behavior classification model. Specifically, combine the temporal and spatial attributes and information content of a small number of labeled samples to form a feature vector and assign corresponding labels to establish a dataset. Split the entire dataset into a training set, a validation set, and a test set to ensure that data points in the same trajectory are not distributed in these three sets at the same time. Here, each trajectory point is regarded as an independent sample in the dataset. This training set will be used to train the classifier so that it can classify new input samples as abnormal.
[0112] Step 3: After constructing the probabilistic grammar dictionary, an abnormal behavior feature vector of a person exhibiting abnormal behavior is generated based on the probabilistic grammar dictionary and the abnormal human posture vector.
[0113] In this step, after constructing the probabilistic grammar dictionary, the probabilistic grammar dictionary is used to generate abnormal human posture vectors based on the training set data in the same manner as the behavior feature vectors generated in the application phase.
[0114] Step 4: Train the behavior classification model based on the abnormal behavior feature vector and the corresponding abnormal type label.
[0115] Here, as a classifier, the nonlinear model random forest is preferred, and decision tree can also be used. According to the selected nonlinear classifier, the relevant parameters are initialized and configured. After that, the classifier is iteratively trained using the divided training set and its corresponding labels combined with the selected machine learning algorithm. The validation set is used to test the performance of the trained model. The evaluation criteria include but are not limited to accuracy and F1 score. After the training is completed, the model is persisted and saved for subsequent use in abnormal behavior recognition.
[0116] In an experiment, for the detection results of abnormal behavior, the embodiment of the present application uses the area under the curve (AUC) for measurement at the region level (region-based). Figure 3 , Figure 3 A schematic diagram of an AUC detection result provided by an embodiment of the present application. Region-based evaluation (Region-based AUC): This evaluation index mainly focuses on the model's ability to detect abnormal areas. The calculation method is to use the ratio of the number of detected abnormal areas to the total number of abnormal areas as the y-axis (called RBDR, i.e., abnormal area detection rate), and the false positive rate as the x-axis, draw the ROC curve, and calculate the area under the curve, which is the Region-based AUC. This indicator can reflect the performance of the model in terms of spatial accuracy. The larger its value, the stronger the model's ability to locate abnormal areas. Experiments show that the recognition method provided by the embodiment of the present application can accurately identify abnormal behavior of personnel.
[0117] For the detection results of abnormal behavior types, this application uses accuracy and F1-score to measure. In the experiment, the accuracy is 82.83% and the F1-score is 79.20%.
[0118] In summary, the method for identifying abnormal personnel behavior provided in the embodiment of the present application has the following beneficial effects: it can realize the detection of various unlabeled abnormal behaviors; it can realize the classification of abnormal behaviors based on small samples, reduce the demand for data volume, and reduce the cost of manual labeling; it has low requirements for hardware configuration and low deployment cost; it has high accuracy in identifying abnormal behaviors and detecting specific abnormal types.
[0119] Based on the same inventive concept, the present application also provides a device for identifying abnormal behavior of a person. Figure 4 As shown in , the identification device 400 provided in the embodiment of the present application includes:
[0120] A monitoring image acquisition module 410 is used to extract multiple frames of monitoring images within a target time window from the monitoring video of the target monitoring scene;
[0121] A human posture vector extraction module 420 is used to extract the human posture vector of each person in the target monitoring scene within the target time window from multiple frames of monitoring images;
[0122] The information volume calculation module 430 is used to perform grammatical analysis on the human posture vector of each person using a pre-built probabilistic grammatical dictionary to determine the information volume corresponding to the current performance behavior of the person; wherein the probabilistic grammatical dictionary uses the grammatical structure of a spatiotemporal AND / OR graph to describe the prior probability distribution related to the spatiotemporal changes of the person in the target monitoring scene through unlabeled historical monitoring data;
[0123] The identification module 440 is used to determine that the person is currently behaving abnormally according to the amount of information corresponding to the person's current behavior and a preset information threshold.
[0124] Further, when the human body posture vector extraction module 420 is used to extract the human body posture vector of each person in the target monitoring scene within the target time window from multiple frames of monitoring images, the human body posture vector extraction module 420 is used to:
[0125] Input multiple surveillance images into a pre-trained human detection model, detect each person in each surveillance image, and mark a human detection frame on each surveillance image;
[0126] Input multiple frames of surveillance images into a pre-trained human tracking model to obtain the motion trajectory of each person in the multiple frames of surveillance images;
[0127] The motion trajectory of each person in multiple frames of surveillance images is input into a pre-trained human posture detection model to obtain the posture features of multiple human key points of each person in each frame of surveillance images;
[0128] According to the posture features of multiple human body key points of each person in each frame of monitoring image, the human body posture vector of each person in the target time window is flattened.
[0129] Furthermore, the recognition device 400 further includes: a construction module (not shown in the figure); the construction module is used to construct the probabilistic grammar dictionary in the following manner:
[0130] Using multiple frames of historical monitoring images in the historical monitoring video of the target monitoring scene as the historical monitoring data, and extracting multiple sample human body posture vectors from the multiple frames of historical monitoring images;
[0131] The original structure of the spatiotemporal AND / OR graph is defined based on the characteristics of the target monitoring scene; wherein the original structure of the spatiotemporal AND / OR graph includes nodes and edges representing associations between nodes; the types of nodes include: scene nodes representing the target monitoring scene at different times, area nodes into which the monitoring images of the target monitoring scene are divided, and personnel nodes where personnel exist in each monitoring area;
[0132] Defining the probability type of the prior probability distribution described by the probability grammar dictionary;
[0133] Perform statistics on multiple frames of sample monitoring images and multiple sample human posture vectors to obtain a probability model for each prior probability distribution;
[0134] The probability parameters of the probability model of each prior probability distribution are used as node attributes of the personnel node and added to the spatiotemporal AND / OR graph to obtain the probability grammar dictionary.
[0135] Further, the probability type of the prior probability distribution includes at least one of the following: the prior probability distribution of the sample human posture vector value, the prior probability distribution of the sample human posture vector jump, and the prior probability distribution of the human body appearing in each monitoring area in the monitoring image; the recognition module 440 is also used to determine the preset information amount threshold in the following way:
[0136] Determining a first preset information amount threshold value of the information amount according to the prior probability distribution of the sample human body posture vector value and the prior probability distribution of the sample human body posture vector jump; and / or,
[0137] A second preset information amount threshold value of the information amount is determined according to a priori probability distribution of the presence of a human body in each monitoring area in the monitoring image.
[0138] Furthermore, when the information volume calculation module 430 is used to perform grammatical analysis on the human body posture vector of each person using a pre-built probabilistic grammar dictionary to determine the amount of information corresponding to the current performance behavior of the person, the information volume calculation module 430 is used to:
[0139] For each person, restore the probability model of each prior probability distribution from the probability parameters included in the node attributes of the person node corresponding to the person in the probability grammar dictionary;
[0140] The person's human posture vector is used as a sampling point of each prior probability distribution and substituted into the corresponding probability model to obtain each probability related to the person's current performance behavior;
[0141] Based on each probability associated with the person's current behavior, the amount of information corresponding to the person's current behavior is determined.
[0142] Furthermore, the identification device 400 further includes: a classification module (not shown in the figure); the classification module is used to:
[0143] Splicing the human body posture vector of the person in the target time window and the node information corresponding to the person's current performance behavior to obtain the person's behavior feature vector; wherein the person's behavior feature vector is used to describe the spatiotemporal characteristics of the person in the target time window;
[0144] The behavior feature vector of the person is input into a pre-trained behavior classification model to determine the abnormal type of the current abnormal behavior of the person.
[0145] Furthermore, the identification device 400 further includes: a training module (not shown in the figure); the training module is used to train the behavior classification model in the following manner:
[0146] Extracting multiple frames of abnormal historical monitoring images with abnormal behaviors from the historical monitoring video of the target monitoring scene;
[0147] Annotating the multiple frames of abnormal history monitoring images with abnormal type labels, and extracting multiple abnormal human posture vectors from the multiple frames of abnormal history monitoring images;
[0148] After constructing the probabilistic grammar dictionary, generating an abnormal behavior feature vector of a person who exhibits abnormal behavior based on the probabilistic grammar dictionary and the abnormal human posture vector;
[0149] The behavior classification model is obtained by training based on the abnormal behavior feature vector and the corresponding abnormal type label.
[0150] Since the principle of solving the problem by the identification device in the embodiment of the present application is similar to the above-mentioned identification method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0151] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in , the electronic device 500 includes a processor 510 , a memory 520 and a bus 530 .
[0152] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1The steps of a method for identifying abnormal behavior of personnel in the method embodiment shown are specifically implemented in accordance with the method embodiment and will not be described in detail here.
[0153] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of a method for identifying abnormal behavior of personnel in the method embodiment shown are specifically implemented in accordance with the method embodiment and will not be described in detail here.
[0154] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0155] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0156] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0158] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0159] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for identifying abnormal behavior of personnel, characterized in that: The identification method comprises: Extracting multiple frames of surveillance images within a target time window from a surveillance video of a target surveillance scene; the target surveillance scene is a surveillance scene of a conference room; Extracting a human body posture vector of each person in the target monitoring scene within the target time window from the multiple frames of monitoring images; For each person, a pre-built probabilistic grammar dictionary is used to perform grammatical analysis on the person's human posture vector to determine the amount of information corresponding to the person's current performance behavior; wherein the probabilistic grammar dictionary uses the grammatical structure of a spatiotemporal AND / OR graph to describe the prior probability distribution related to the spatiotemporal changes of the person in the target monitoring scene through unlabeled historical monitoring data; According to the information amount corresponding to the current behavior of the person and the preset information amount threshold, it is determined that the current behavior of the person is abnormal; The probability types of the prior probability distribution include: the prior probability distribution of the sample human body posture vector values, the prior probability distribution of the sample human body posture vector jumps, and the prior probability distribution of the human body appearing in each monitoring area in the monitoring image; the prior probability distribution of the sample human body posture vector values and the prior probability distribution of the sample human body posture vector jumps are used to determine the first preset information amount threshold value of the information amount; the prior probability distribution of the human body appearing in each monitoring area in the monitoring image is used to determine the second preset information amount threshold value of the information amount; The determining that the person currently has abnormal behavior according to the amount of information corresponding to the person's current behavior and a preset information amount threshold value includes: If the first amount of information is greater than or equal to a first preset amount of information threshold, or the second amount of information is greater than or equal to a second preset amount of information threshold, it is determined that the person is currently behaving abnormally.
2. The identification method according to claim 1, characterized in that: Extracting the human body posture vector of each person in the target monitoring scene within the target time window from the multiple frames of monitoring images, including: Input multiple surveillance images into a pre-trained human detection model, detect each person in each surveillance image, and mark a human detection frame on each surveillance image; Input multiple frames of surveillance images into a pre-trained human tracking model to obtain the motion trajectory of each person in the multiple frames of surveillance images; The motion trajectory of each person in multiple frames of surveillance images is input into a pre-trained human posture detection model to obtain the posture features of multiple human key points of each person in each frame of surveillance images; According to the posture features of multiple human body key points of each person in each frame of monitoring image, the human body posture vector of each person in the target time window is flattened.
3. The identification method according to claim 1, characterized in that: The probabilistic grammar dictionary is constructed in the following way: Using multiple frames of historical monitoring images in the historical monitoring video of the target monitoring scene as the historical monitoring data, and extracting multiple sample human body posture vectors from the multiple frames of historical monitoring images; The original structure of the spatiotemporal AND / OR graph is defined based on the characteristics of the target monitoring scene; wherein the original structure of the spatiotemporal AND / OR graph includes nodes and edges representing associations between nodes; the types of nodes include: scene nodes representing the target monitoring scene at different times, area nodes into which the monitoring images of the target monitoring scene are divided, and personnel nodes where personnel exist in each monitoring area; Defining the probability type of the prior probability distribution described by the probability grammar dictionary; Perform statistics on multiple frames of sample monitoring images and multiple sample human posture vectors to obtain a probability model for each prior probability distribution; The probability parameters of the probability model of each prior probability distribution are used as node attributes of the personnel node and added to the spatiotemporal AND / OR graph to obtain the probability grammar dictionary.
4. The identification method according to claim 1, characterized in that: For each person, a pre-built probabilistic grammar dictionary is used to perform grammatical analysis on the person's human posture vector to determine the amount of information corresponding to the person's current performance behavior, including: For each person, restore the probability model of each prior probability distribution from the probability parameters included in the node attributes of the person node corresponding to the person in the probability grammar dictionary; The person's human posture vector is used as a sampling point of each prior probability distribution and substituted into the corresponding probability model to obtain each probability related to the person's current performance behavior; Based on each probability associated with the person's current behavior, the amount of information corresponding to the person's current behavior is determined.
5. The identification method according to claim 1, characterized in that: After determining that the person is currently behaving abnormally based on the amount of information corresponding to the person's current behavior and a preset information amount threshold, the identification method further includes: Splicing the human body posture vector of the person in the target time window and the node information corresponding to the person's current performance behavior to obtain the person's behavior feature vector; wherein the person's behavior feature vector is used to describe the spatiotemporal characteristics of the person in the target time window; The behavior feature vector of the person is input into a pre-trained behavior classification model to determine the abnormal type of the current abnormal behavior of the person.
6. The identification method according to claim 5, characterized in that: The behavior classification model is trained in the following way: Extracting multiple frames of abnormal historical monitoring images with abnormal behaviors from the historical monitoring video of the target monitoring scene; Annotating the multiple frames of abnormal history monitoring images with abnormal type labels, and extracting multiple abnormal human posture vectors from the multiple frames of abnormal history monitoring images; After constructing the probabilistic grammar dictionary, generating an abnormal behavior feature vector of a person who exhibits abnormal behavior based on the probabilistic grammar dictionary and the abnormal human posture vector; The behavior classification model is obtained by training based on the abnormal behavior feature vector and the corresponding abnormal type label.
7. A device for identifying abnormal behavior of a person, characterized in that: The identification device comprises: A monitoring image acquisition module, used to extract multiple frames of monitoring images within a target time window from a monitoring video of a target monitoring scene; the target monitoring scene is a monitoring scene of a conference room; A human posture vector extraction module is used to extract the human posture vector of each person in the target monitoring scene within the target time window from multiple frames of monitoring images; An information volume calculation module is used to perform grammatical analysis on the human posture vector of each person using a pre-built probabilistic grammar dictionary to determine the information volume corresponding to the current behavior of the person; wherein the probabilistic grammar dictionary uses the grammatical structure of a spatiotemporal AND / OR graph to describe the prior probability distribution related to the spatiotemporal changes of the person in the target monitoring scene through unlabeled historical monitoring data; An identification module is used to determine that the person's current behavior is abnormal based on the amount of information corresponding to the person's current behavior and a preset information threshold; The probability types of the prior probability distribution include: the prior probability distribution of the sample human body posture vector values, the prior probability distribution of the sample human body posture vector jumps, and the prior probability distribution of the human body appearing in each monitoring area in the monitoring image; the prior probability distribution of the sample human body posture vector values and the prior probability distribution of the sample human body posture vector jumps are used to determine the first preset information amount threshold value of the information amount; the prior probability distribution of the human body appearing in each monitoring area in the monitoring image is used to determine the second preset information amount threshold value of the information amount; When the identification module is used to determine that the person is currently behaving abnormally based on the amount of information corresponding to the person's current behavior and a preset information threshold, the identification module is used to: If the first amount of information is greater than or equal to a first preset amount of information threshold, or the second amount of information is greater than or equal to a second preset amount of information threshold, it is determined that the person is currently behaving abnormally.
8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of a method for identifying abnormal personnel behavior as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for identifying abnormal human behavior as described in any one of claims 1 to 6 are executed.
