A human abnormal behavior detection method based on physical law

Abstract

The application discloses a human abnormal behavior detection method based on physical law, and solves the problem that visual features are easily affected by observation noise during detection. The application comprises the following steps: positioning a human body region and a joint part through a visual representation module, and obtaining coarse-grained visual features and fine-grained visual features respectively; processing the input coarse-grained and fine-grained visual features through a coarse-grained part and a fine-grained part of a physical intuition module respectively, outputting simulated force borne by the human body and force borne by each joint of the human body, calculating the acceleration of each joint, and updating the human body posture; and combining the visual representation module and the physical intuition module to detect human abnormal behavior. The method simultaneously considers visual clues and physical constraints, so that robust representation of human behavior which is not affected by observation noise can be obtained, and human abnormal behavior detection can be more accurate and reasonable.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and relates to human abnormal behavior detection, in particular to a human abnormal behavior detection method based on physical laws. BACKGROUND

[0002] Human abnormal video detection is a task of locating actions that do not match the regular patterns from videos. Due to the imbalance of surveillance videos, almost all available training datasets have no abnormal annotations, because it is costly to manually select a small subset with anomalies, and it is difficult to understand human behavior from limited observations in real scenarios. Therefore, the difficulty of this task is mainly caused by the unpredictability of abnormal semantics and the lack of training data.

[0003] Many weakly supervised or unsupervised methods have been used to distinguish abnormal events from a large number of events. Existing methods train an encoder-decoder based model for reconstruction. The reconstruction error of normal behavior is small, and the reconstruction error of abnormal behavior is high. Such methods include autoencoders, hidden regression models and adversarial learning models. Other methods determine abnormalities according to the coherence of human motion. These methods can predict future poses based on historical poses in normal behaviors after training, and determine abnormalities by high prediction errors in abnormal behaviors. Typical methods of this type include skeleton prediction, LSTM (Long Short-Term Memory Network) or GAN (Generative Adversarial Network) based prediction and spatio-temporal dual-stream models. Most of the above methods rely on frame-level, region-level or skeleton-level features, however, in crowded scenes with mutual occlusion, incomplete observations can lead to noisy features, hindering accurate reconstruction and prediction.

[0004] In order to solve the problem of lack of training data annotation, some existing methods propose a clustering-based strategy to distinguish actions according to the similarity of spatial and temporal features. They map videos to a high-dimensional feature space where abnormal videos are far away from normal videos. Some methods propose meta-learning based methods that fine-tune the model using some unlabeled samples in the target environment. Other weakly supervised methods include attention-based methods that make the model focus on key areas while reducing the influence of the background. In addition, transfer learning based methods, multiple instance learning based methods, self-supervised methods, pseudo-label generation and distribution analysis methods have also been proposed. Although these methods have achieved good results, the use of only visual cues without physical constraints can lead to predictions that are not physically reasonable under observation noise, because the predicted poses can be physically unreasonable.

[0005] Existing methods solve the anomaly detection problem in two ways. The first method focuses on multiple visual features. They first locate the target and then parse the target action. These actions are represented by multiple complementary cues, including skeletal trajectories, appearance, motion patterns, and moving directions. Even though different visual cues are complementary, they are sensitive to observation noise that affects visual features. The second method utilizes the regularity of action over time. However, the regularity is acquired from limited training data. Even though some methods separate this regularity over time into an intrinsic attribute part and a time-varying part, these methods are only validated on ideal training data without occlusion. In addition, existing methods that utilize physical laws in video prediction are only applied to simple rigid objects. However, the human body is a complex combination of countless rigid and elastic objects. Therefore, it is necessary to simplify the modeling of self-driving and external forces before applying physical laws to the parsing of human behavior. SUMMARY

[0006] Based on the above problems, the present application provides a human abnormal behavior detection method based on physical laws, which provides a hierarchical physical intuition (PI) module and a visual representation (VR) module, which are combined to extract features, that is, both visual cues and physical constraints are considered, so that a robust representation that is not affected by observation noise can be obtained, and human abnormal behavior detection can be more accurate and reasonable.

[0007] The human abnormal behavior detection method based on physical laws provided by the present application comprises the following steps:

[0008] Step 1: Locate the human region and joint parts through the visual representation module, and obtain coarse-grained visual feature vector representation and fine-grained visual feature vector representation respectively. The coarse-grained visual feature vector representation includes the appearance feature vector of the located pedestrian and surrounding objects, and the relative position of the pedestrian and the surrounding objects; the fine-grained visual feature vector representation includes the joint position of the pedestrian.

[0009] Step 2: constructing the input data of the physical intuition module. The present application uses the physical intuition module to model the human body as a large number of connected particles and analyzes the force on them. The physical intuition module is divided into a coarse-grained part and a fine-grained part; the coarse-grained part regards each person as a particle, inputs the appearance features of the pedestrian, the appearance features I1 of the objects in the pedestrian's neighborhood, and the relative position I2 of the pedestrian and the objects in the neighborhood; the fine-grained part regards each joint as a particle, inputs the joint position I3 of the human body and the joint semantic connection information I4, which is obtained according to whether there is a bone connected between the joints. The physical intuition module simulates the force on the human body through the coarse-grained force output by the coarse-grained part and the fine-grained force output by the fine-grained part.

[0010] Step 3: the coarse-grained part and the fine-grained part of the physical intuition module process the input data respectively. The coarse-grained part processes the input matrices I1 and I2 to output the coarse-grained force of the pedestrian, which is the resultant force of the self-driving force and the external force on the pedestrian; the fine-grained part processes the input matrices I3 and I4 to output the fine-grained force of the pedestrian, which contains the force on all joints; the physical intuition module simulates the force on the pedestrian using the coarse-grained force and the fine-grained force, calculates the acceleration of each joint of the pedestrian according to the output of the coarse-grained part and the fine-grained part; the physical intuition module updates the position of each joint and the posture of the human body after obtaining the acceleration of each joint of the pedestrian.

[0011] Step 4: optimizing the weight parameters of the physical intuition module network. The position of the human body joint obtained by the physical intuition module is used to approximate the position of the human body joint obtained by the visual representation module, and the network parameters of the physical intuition module are optimized by minimizing the approximation error.

[0012] Step 5: combining the visual representation module and the physical intuition module to detect abnormal behavior of the human body.

[0013] The visual representation module and the physical intuition module first infer the behavior of the pedestrian in the training video to obtain the normal behavior distribution, and the behavior of the pedestrian is represented as the position of each joint of the human body; then the acceleration of each joint of the human body is obtained from the real-time collected image, and then the abnormal score of the human body is calculated according to the probability distribution of the normal behavior distribution it obeys, and the abnormal behavior of the human body is detected according to the score.

[0014] Compared with the prior art, the present application has the advantages and positive effects that:

[0015] (1) The method of the present application proposes to use the distribution of the forces acting on the whole body and the local part of the pedestrian for anomaly detection, and is realized through hierarchical PI modules (coarse-grained whole body force and fine-grained local force). The module carries out physical simulation in three-dimensional space, and uses motion parameters conforming to physical laws to fit the dynamic information of the observed target. Since the parameters of the PI module conform to the physical law, they are not affected by observation noise such as occlusion and have robustness to incomplete observation. The PI module of the present application can correct the estimation results of the visual VR module.

[0016] (2) The method of the present application proposes a constraint optimization target for PI module design, which facilitates the PI module to obtain human body dynamics parameters conforming to physical laws through gradient back propagation, and combines the PI module with the VR module. Compared with the traditional method using only visual cues, the method of the present application can generate physically reasonable and discriminative representations of human motion.

[0017] (3) The method of the present application constructs a behavior distribution space according to the force information on the human body judged by the PI module, and proposes a new anomaly score. Through experimental detection, the method of the present application can better detect abnormal behavior, has good detection performance, and has wide applicability. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The architecture diagram of the VR module combining visual features and the PI module containing physical constraints of the present application is described.

[0019] Figure 2 The network architecture diagram of the PI module in the present application containing coarse-grained and fine-grained analysis. DETAILED DESCRIPTION

[0020] In order to facilitate those skilled in the art to understand and implement the present application, the present application is further described in detail and in depth below in combination with the drawings. Based on the following embodiments, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0021] The human body abnormal behavior detection method based on physical laws of the embodiment of the present application processes the collected video images in the following five steps.

[0022] Step 1: The input video image is positioned by a visual representation module (referred to as VR module) to locate the human body region and joint parts, and coarse-grained visual feature vector representation and fine-grained visual feature vector representation are obtained respectively. The coarse-grained representation represents the whole body feature, and the fine-grained representation represents the local joint feature.

[0023] The VR module includes a detector for locating a human body and a pose estimator for locating 14 body joints. The detector is capable of locating 80 common objects such as pedestrians, vehicles, numbers, etc., and extracting features of the located objects to identify the object categories; the pose estimator further determines the positions of the 14 joints based on the located pedestrians.

[0024] For batch input of T-frame video images, the detector estimates the relative positions of the human body and surrounding objects and extracts features of the objects as coarse-grained input to estimate coarse-grained forces. The coarse-grained visual feature vector representation is obtained by the detector and includes appearance information of the located pedestrians and neighboring objects and position information of the pedestrians and neighboring objects. In an embodiment of the present application, the detector divides the image into 52x52 rectangular grids, determines the upper, lower, left and right boundary positions of the most likely object centered on each grid and a 32-dimensional appearance feature vector of the object. After determining the boundary box and feature vector of the pedestrian, the detector determines the most likely object box and appearance features on the 8 nearest neighboring rectangular grids around the pedestrian. The pose estimator locates the 14 joint positions of the pedestrian within the pedestrian boundary box and outputs a fine-grained visual feature vector representation. The fine-grained visual feature vector representation includes the joint position information of the pedestrian.

[0025] In an embodiment of the present application, a 70-dimensional feature vector is formed by random sampling around the body joint area to represent the appearance of a person, and the appearance similarity can be represented by the cosine similarity between the feature vectors. The trajectory estimation between different persons is achieved by the optimal transport method. That is, the M t of the person at time t and the M t+1 of the person at time t+1 are determined by the optimal transport method. t The matching between the persons is achieved by minimizing the overall matching error of all pairs between the two time frames. The overall matching error refers to that all possible pedestrian matching relationships between the two time frames form a M t+1 matrix, each row of the matrix represents the inverse of the appearance similarity between a person in the previous frame and all persons in the next frame, and the smaller the value is, the higher the similarity is. All elements in the matrix are positive, and the sum of the elements in the matrix is the overall error.

[0026] The VR model is represented as θ h,1 ,...,θ h,T , where θ h,t ,t=1,...,T represents the joint position information of the h person at time t. θ h,1 ,...,θ h,T may exist estimation errors caused by visual noise, and the PI module uses the pose sequence conforming to the physical law to correct θ h,1 ,...,θ h,TA more reasonable pose sequence is obtained by approximation. h represents the pedestrian identification number.

[0027] Step 2: Construct the input data for the Physical Intuition Module (PI module).

[0028] This invention utilizes a physical intuition module to model the human body as a large number of interconnected point masses and performs force analysis on them. The PI module is divided into coarse-grained and fine-grained components, such as... Figure 1 As shown, the forces acting on the object are simulated by the sum of coarse-grained and fine-grained forces. The PI module estimates the pedestrian dynamic parameters in an iterative manner that conform to both visual observation characteristics and physical laws.

[0029] The physical intuition module constructs physical dynamic parameter simulations in three-dimensional space and approximates visual observation of dynamic information: the human body is modeled as connected mass points, and the mass of the i-th joint is denoted as m. i To simplify the calculation, assume V represents the number of joints.

[0030] In the coarse-grained module of the PI module, when predicting the state of a pedestrian at time t, the input is the appearance of the h-th person at the previous t-1 time, the appearance information I1 of the neighboring objects, and the position information I2 of the neighboring objects. The output is the coarse-grained force of the h-th person at time t. yes Figure 1 The main force F in s and external force F e The resultant force. The superscript 'c' indicates a coarse-grained network. Active forces, or self-driving forces, are determined by posture; for example, a person must squat before jumping. External forces include not only the interaction forces between adjacent objects but also the supporting force and friction. External forces are also posture-dependent because the supporting force comes into play when the body lands. Gravity is posture-independent; for any person with V joints, gravity is Vmg, where g represents gravitational acceleration. The coarse-grained part only simulates the interaction forces between the pedestrian and surrounding objects, determining the resultant force at the center of the body.

[0031] like Figure 2 As shown, in this embodiment of the invention, I1 and I2 are obtained based on the coarse-grained visual feature vector representation.

[0032] I1 encoding is used to encode the appearance features of all pedestrians and the appearance features of objects within their 8 neighborhoods, centered on each pedestrian. Let M represent the total number of pedestrians, O represent the dimension of each object's appearance feature vector, and Q = 8 neighborhoods for each pedestrian's center point. Since the force and acceleration at time t (time T in this embodiment) to be predicted are determined by the appearance and relative position at the previous T-1 times, the input matrix can be obtained. Represents the set of real numbers. The superscript indicates the dimension of the matrix.

[0033] I2 encoding determines the relative positions of all pedestrians and their neighboring objects over the first T-1 time steps. For each pedestrian, the encoding includes the positional relationships between objects within Q+1 regions of the pedestrian's center point and its 8 neighboring areas.

[0034] The fine-grained module of the PI module primarily analyzes posture-related forces, treating each human joint as a point mass, such as... Figure 2 The grid diagram in the lower left corner shows the connections between joints. Black dots indicate that the joints in the corresponding rows and columns are connected because there is a force between them. When predicting the state of the h-th pedestrian at time t, the input to the fine-grained module is the joint coordinate information and semantic connection information of the h-th pedestrian at time t-1 before. The output is the fine-grained force of the h-th person at time t. The fine-grained force includes the force of each joint. This represents the force acting on the i-th joint. The superscript f indicates a fine-grained network. Forces acting on joints It is the main force F s Support force S and frictional force F r The combined force. Joint semantic connectivity information is obtained based on whether there is a bone connection between joints.

[0035] like Figure 2 As shown, in this embodiment of the invention, I3 is obtained based on fine-grained visual feature vector representation. The I3 encoding represents the coordinates of all pedestrian joints at time T-1 before the start of the process. C in C represents the coordinate system dimension. in =3. I4 encoding of the semantic connectivity information of all pedestrians at time T-1 before the key. For each pedestrian at each time point, there is a V×V matrix. The elements in this matrix represent whether there is a semantic connection between the key points in the corresponding row and column. The values ​​are 0 or 1, where 0 indicates no semantic connection and 1 indicates a semantic connection. A semantic connection exists when key points are connected by bones; otherwise, there is no semantic connection.

[0036] In this embodiment of the invention, the forces acting on M pedestrians are predicted. The input data I1, I2, I3 and I4 are the data of the M pedestrians collected by the visual representation module. In practice, I1, I2, I3 and I4 can be collected according to the number of pedestrians to be predicted.

[0037] Step 3: Combine the visual representation module with the physical intuition module for anomaly detection.

[0038] The visual representation module extracts the appearance features and relative positions of pedestrians and neighboring objects at various time points from the input video images. This information is then input into the coarse-grained network of the physical intuition module to estimate the forces exerted on the pedestrian by surrounding objects. The visual representation module also obtains the pedestrian's joint coordinates at various time points, which, along with joint semantic connectivity information, are input into the fine-grained network to simulate the forces acting on the joints related to posture. The physical intuition module combines the outputs of the coarse-grained and fine-grained components to calculate the resultant force and acceleration acting on each joint.

[0039] like Figure 2 As shown, the coarse-grained part treats each person as a point mass, and the appearance and spatial relationships of each pedestrian and objects in their surrounding environment are used to estimate the coarse-grained forces. Figure 2 The human center coordinates in the diagram refer to the center coordinates of each pedestrian. The coarse-grained part consists of two branches. The first branch is composed of 1×1 convolutional layers, used to process the appearance features of pedestrians and surrounding objects, modeling the coupling relationship between the appearance features of nearby objects. The second branch determines the interaction forces through the relative positional relationships of nearby objects. In the second branch, a 1×3 spatial convolution is first used to model the spatial interaction between nearby objects, then a weighted sum is taken with the input matrix I2, and finally a 3×1 temporal convolution is used to extract motion characteristics. The coarse-grained force of the pedestrian is obtained by summing the two branches. The coarse-grained force is the resultant force of the pedestrian's active force and the external forces acting on them. Let the final output tensor of the coarse-grained part be... C represents the predicted forces acting on M pedestrians. in =3 indicates the force acting in the three directions of the coordinate axes.

[0040] The coarse-grained portion represents the analysis of the interaction forces between different pedestrians as a whole. (This is from an embodiment of the invention.) Figure 2 In the first branch, F1 = f1 represents the relationship between the relative positions of different pedestrians and the interaction force, and its input is the coordinate position of the center point of different pedestrians; in the second branch, F2 = f2 represents the relationship between the change of the position of different pedestrians over time and the interaction force. For example, if the position changes quickly, the interaction force will change quickly.

[0041] In the fine-grained estimation stage, each human joint is treated as a point mass. The joint coordinates and semantic connectivity information of the pedestrian are input into the fine-grained network. When inputting into the fine-grained network, the matrix form of I3 is first changed, and I3 is equivalently transformed into... Human joint position information is converted into relative distances between joint points, stored, and then input into the network. The fine-grained part includes weights... The real-time related layers and weights are The real-time weighted layer. The real-time correlation layer obtains the relative positions between connected nodes, and the real-time weighted layer extracts time clues.

[0042] like Figure 2 As shown, in the T-1 channels of the real-time correlation layer, since joint forces can only be applied by adjacent joints, only the node weights corresponding to elements with a value of 1 in matrix I4 are active. The real-time weighted layer is a 1×1 convolutional layer that maps the T-1 input channels to 2 output channels, outputting a tensor. This represents the predicted forces on the joints of M individuals, where each element represents the force exerted by the row-indexed joint on the column-indexed joint. The two channels represent the forces along the tangential and normal directions of the skeleton, respectively. Only element values ​​corresponding to an I2 value of 1 are valid, because joint forces can only be exerted by adjacent joints.

[0043] Pedestrian acceleration is divided into coarse-grained whole-body acceleration and fine-grained local joint acceleration, the sum of which is the actual acceleration of each joint. This invention calculates joint acceleration based on the resultant force output by the coarse-grained and fine-grained components, such as... Figure 2 As shown, the acceleration of V joints for each pedestrian is calculated. Let the coarse-grained force of the h-th person at time t be... The fine-grained force of the h-th person at time t is Let F h,t,i for and The resultant force of the two forces, then the acceleration a of the i-th joint of the h-th person at time t. h,t,i According to F h,t,i -mg=ma h,t,i The calculations yielded a value where i = 1, 2, ..., V, m is the joint mass, and g is the acceleration due to gravity. The position of the joint can be determined based on its acceleration.

[0044] Fine-grained forces consist of two components: one along the tangential direction and the other along the normal direction of the skeleton. These two components are summed and then added to the coarse-grained forces. All parts of the same pedestrian experience the same coarse-grained force.

[0045] The coarse-grained component explains the surrounding environment and external forces that cause the anomalies, while the fine-grained component explains the abnormal human movements and behaviors. Since the coarse-grained forces are related to the relative positions of the pedestrian and surrounding objects, and the fine-grained forces are correlated with posture, the PI module alternates between the two steps. In the first step, the coarse-grained forces determined by relative position and the fine-grained forces determined by posture are combined to determine the acceleration. In the second step, the human posture and relative position are updated using the acceleration. The new posture is used to update the input of the PI module, and new forces are determined based on the updated information. Let X be the position of the V joints of the h-th person obtained by the PI module at time t. h,t ={X h,t,1 ,...,X h,t,V}

[0046] Step 4: Optimize the network parameters of the PI module so that the network parameters conform to both the physical rules introduced in Steps 2 and 3 and the human visual dynamic features extracted from the observation information in Step 1.

[0047] The parameters in the PI module remain constant at different times. By reducing the difference between the human joint positions simulated by the PI module and those observed by the VR module, the PI module can provide network weight parameters that conform to both visual characteristics and physical laws. Even under conditions of incomplete visual features, such as occlusion or blurring, complete human motion parameters can be recovered.

[0048] The parameters in the PI module include learnable weight parameters for the convolutional layers in the coarse-grained module and learnable weight parameters for the correlation and fully connected layers in the fine-grained module. These parameters represent the relationship between the pedestrian's relative position to surrounding objects and the pedestrian's posture and the forces acting on the body, respectively. Dynamically updating the parameters in the PI module makes the pedestrian motion patterns simulated by the PI module more consistent with visual observation information. Suppose that the PI module can obtain the trajectory of pedestrian h at T time points. By using Approximate θ h,1 ,...,θ h,T Minimize the approximation error to optimize the PI module parameters W h ={W coarse,h W fine,h}, where W h These are the PI module network parameters for pedestrian h, W coarse,h W fine,h These are the network parameters for the coarse-grained and fine-grained parts, respectively. W fine,h Includes weight parameters W for the real-time correlation layer and the real-time weighted layer. fine1,h W fine2,h .

[0049] This invention optimizes the parameters of the PI module to maximize the consistency between simulated human posture and visual motion features. Based on a differentiable physical model, physical parameters are automatically calculated to maximize the consistency between PI model predictions and VR model predictions. The objective can be simplified as follows:

[0050]

[0051] Among them, X h,t ={X h,t,1 ,...,X h,t,V} represents the position of the h-th person at time t in the PI model simulation. For X h,t The average value of the coordinates of all relevant nodes; Y h,t-1 Let X represent the appearance feature vector of the h-th person at time t-1; to simulate the force-induced motion of joints, the physical quantities satisfy the following relationship: X h,t,i =X h,t-1,i +v h,t-1,i dt+0.5a h,t,i dt 2 The velocity v of the i-th joint of the h-th person at time (t-1) is... h,t-1,i =(X h,t-1,i -X h,t-2,i ) / (dt), where dt is the time interval between adjacent video frames. In this embodiment of the invention, dt = 0.04.

[0052] In each person's motion trajectory, this invention aims to construct a pedestrian motion sequence in three-dimensional space that conforms to Newton's second law using a PI module. At each moment in each pedestrian trajectory, the PI module uses a sliding window of length T to determine the force state and acceleration of all joints of the pedestrian based on the past T-1 observations. Furthermore, it calculates the simulation error by projecting the simulated posture in three-dimensional space onto two-dimensional space and matching it with the posture observed from RGB images.

[0053] The three-dimensional spatial pose obtained at time t is projected into two-dimensional space as follows:

[0054] x h,t =f proj (X h,t ) = sΠ(RX h,t )+t trans

[0055] Where, x h,t Indicates according to X h,t Projection yields the position of the h-th person in two-dimensional space at time t, f. proj Here, Π represents the projection function; s, R, and t represent the projection operation. transThese represent the scaling, rotation, and translation matrices of the 3D spatial projection, respectively, which are estimated by the VR module.

[0056] The optimization process of the PI module parameters is represented by the loss function L as follows:

[0057]

[0058] Where, θ h,t ,t=1,...,T represents the keypoint position of the h-th person at time t, obtained by the VR model from the input video image, and is the inferred three-dimensional coordinates of the pedestrian keypoint. f proj (θ h,t ) is based on θ h,t The two-dimensional spatial position is obtained through projection. After adaptive adjustment to minimize the objective function L, the joint position information x, which conforms to physical laws, is obtained. h,t The above optimization process, on the one hand, makes x h,t It should closely resemble visually observable features while also conforming to physical laws.

[0059] The optimization process for parameters in the PI module aims to ensure that the pedestrian motion parameters represented by the PI module are consistent with the dynamic features observed visually. The optimization process is modeled as changing W... h , The process of minimizing the loss function L above, W h The parameters represent the simulated pedestrian motion pattern h learned by the PI module. λ1 and λ2 are the adjustment weights of the two constraint terms; constraint terms L1 and L2 are expressed as follows:

[0060]

[0061]

[0062] Here, C(i,j) = 1 indicates a semantic connection between joints i and j. The loss is calculated in two-dimensional space because 3D information is partially missing in RGB images. L1 represents the invariance of skeleton size; for two joints i and j satisfying C(i,j) = 1, a semantic connection exists, such as the left elbow and left wrist being connected. L2 is based on statistical information about the angles between adjacent bones conforming to the physiological characteristics of normal pedestrians. γ i This represents the angle between bones connected to the i-th joint, obtained by the VR module for positioning. and σ γi This represents the mean and variance of the angle that best reflects the physiological characteristics of a pedestrian. An angle that more closely matches physiological characteristics helps reduce loss; therefore, using constrained optimization can make the predicted pose more consistent with physical laws.

[0063] Step 5: Combine the visual representation module with the physical intuition module to detect abnormal human behavior.

[0064] The distribution of human behavior is described based on the similarity between coarse-grained and fine-grained human accelerations estimated by the PI module, and the anomaly detection score S is calculated.

[0065] First, the VR module and the PI module infer pedestrian motion information from the training video to obtain the distribution of normal behavior. Let the behavior of pedestrian h at time t be represented as a vector. Where a h,t,i The acceleration at joint i is represented by the PI module. A h,t The distribution is a Gaussian mixture distribution with K clusters.

[0066] Then, for a real-time calculated acceleration A at each joint of the h-th person's body... h ′ ,t Its abnormal score S h,t It is determined by the inverse probability of the probability distribution it follows, as follows:

[0067]

[0068] μ k ,σ k These represent the mean and variance of the Gaussian distribution of the k-th cluster. The calculated anomaly score is used to determine whether abnormal behavior exists.

[0069] Even when anomalous behavior is unrelated to fine-grained actions, the coarse-grained branch of the PI module can still detect anomalies through the interaction forces between nearby objects and the central pedestrian. For example, in a specific scenario, walking in the wrong direction can cause the coarse-grained module to output a large outlier value S.

[0070] To better illustrate the effectiveness of this invention, it is compared with other existing methods on four datasets: CUHK Avenue, ShanghaiTech, UR Fall, and Street Scene. Quantitative evaluation is performed using the area under the ROC curve (AUC) obtained by continuously changing the anomaly detection threshold; a higher AUC value indicates better performance. Unlike other datasets, the Street Scene dataset uses trajectory-based detection rate (TBDR) to evaluate the proportion of correctly detected trajectories and region-based detection rate (TBDR) to evaluate the correctly detected regions.

[0071] Table 1. Comparison of AUC between the method of this invention and other methods.

[0072]

[0073] The numbers in Table 1 represent AUC values, and N / A indicates that this method was not tested on a specific dataset in this literature. The method of this invention is compared with four existing methods. The four existing methods are as follows:

[0074] (1) (Lu et al. 2020) refers to the method in reference 1 (Lu, Y.; Yu, F.; Reddy, MKK; and Wang, Y. 2020. Few-Shot Scene-Adaptive Anomaly Detection. In European Conference on Computer Vision, 125–141. Springer);

[0075] (2) (Liu et al. 2018) refers to the method in reference 2 (Liu, W.; Luo, W.; Lian, D.; and Gao, S. 2018. Future frame prediction for anomaly detection – a new baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6536–6545. IEEE.).

[0076] (3) (Ionescu et al. 2019b) refers to the method in reference 3 (Ionescu, RT; Khan, FS; Georgescu, M.-I.; and Shao, L. 2019b. Object-centric auto-encoders and dummyanomalies for abnormal event detection in video. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 7842–7851. IEEE.).

[0077] (4) (Ramachandra and Jones 2020) refers to the method in reference 4 (Ramachandra, B.; and Jones, M. 2020. Street Scene: A new dataset and evaluation protocol for videoanomaly detection. In Proceedings of the IEEE / CVF Winter Conference on Ap-plications of Computer Vision, 2569–2578. IEEE.).

[0078] The methods described above are all methods for detecting abnormal behavior. As can be seen from the comparison results in Table 1, the method of this invention exhibits better performance on various datasets compared to other methods. This demonstrates that the method of this invention can effectively detect abnormal human behavior and has better detection performance.

Claims

1. A method for detecting abnormal human behavior based on physical laws, characterized in that, Includes the following steps: Step 1: For the input video image, the pedestrian area and joints are located through the visual representation module, and coarse-grained visual feature vector representation and fine-grained visual feature vector representation are obtained respectively; the coarse-grained visual feature vector includes the appearance features of the located pedestrian and surrounding objects, as well as the relative position of the pedestrian and surrounding objects; the fine-grained visual feature vector representation includes the joint positions of the human body. Step 2: Construct the input data for the physics intuition module; The physical intuition module is divided into a coarse-grained part and a fine-grained part. The coarse-grained part treats each person as a point mass and inputs the appearance features of the pedestrian, the appearance features of objects in the pedestrian's neighborhood I1, and the relative position of the pedestrian and the objects in the neighborhood I2. The fine-grained part treats each joint as a point mass and inputs the joint position of the human body I3 and the semantic connection information of the joint I4. The semantic connection information of the joint is obtained based on whether there are bones connecting the joints. Step 3: The coarse-grained and fine-grained parts of the physics intuition module process the input data respectively; The coarse-grained part processes the input matrices I1 and I2 to output the coarse-grained force of the pedestrian, which is the resultant force of the pedestrian's self-driving force and the external forces acting on it; the fine-grained part processes the input matrices I3 and I4 to output the fine-grained force of the pedestrian, which includes the forces acting on all relevant nodes. The Physics Intuition module simulates the forces acting on a pedestrian using coarse-grained and fine-grained forces, and calculates the acceleration of each joint of the pedestrian based on the outputs of the coarse-grained and fine-grained components; after obtaining the acceleration of each joint of the pedestrian, the Physics Intuition module updates the position of each joint and the human posture. The coarse-grained part of the network consists of two branches. The first branch is composed of 1×1 convolutional layers, used to process the appearance features of pedestrians and their neighboring objects. In the second branch, 1×3 spatial convolution is first used to model the spatial interaction between neighboring objects, then weighted and summed with the input matrix I2, and finally the motion characteristics are extracted by 3×1 temporal convolution. The fine-grained part of the network includes a real-time correlation layer and a real-time weighted layer. In the real-time correlation layer, only the weights of nodes with an element value of 1 in matrix I4 are set to the active state. The output of the real-time correlation layer is then input into the real-time weighted layer for processing. The real-time weighted layer is a 1×1 convolutional layer that maps all input channels to two output channels. The two output channels output the forces along the tangential direction of the human joints of each pedestrian and the forces along the normal direction, respectively. Step 4: Optimize the network parameters of the physical intuition module so that the human posture predicted by the physical intuition module conforms to both human physical constraints and visual observation. The human joint positions obtained by the physical intuition module are used to approximate the human joint positions obtained by the visual representation module. The network parameters of the physical intuition module are optimized by minimizing the approximation error. Step 5: Combine the visual representation module with the physical intuition module to detect abnormal human behavior; The visual representation module and the physical intuition module first infer the behavior of pedestrians in the training video to obtain the normal behavior distribution, and the pedestrian behavior is represented by the position of each joint of the human body; then, for the real-time captured images, the acceleration of each joint of the human body is obtained, and the abnormal human body score is calculated according to the probability distribution of the normal behavior distribution it follows, and abnormal human behavior is detected based on the score.

2. The method according to claim 1, characterized in that, The visual representation module includes: a detector for identifying object types and locating human bodies, and a pose estimator for locating human body joint positions; the detector extracts appearance feature vectors of pedestrians and surrounding objects from the input video image, calculates the relative position of pedestrians and objects in the neighborhood, and generates a coarse-grained visual feature vector representation. The pose estimator locates the joint points of each pedestrian's body.

3. The method according to claim 1, characterized in that, In step 2, assuming the prediction is for the behavior of M pedestrians at time t, the visual representation module obtains the appearance features I1 of the pedestrians and their neighboring objects at the previous (t-1) time steps, and the relative positions I2 of all pedestrians and their neighboring objects at the previous (t-1) time steps. Assuming the dimension of the appearance feature vector is O and the number of pedestrian neighbors is Q, then... The visual representation module obtains the human body joint positions I3 of the pedestrian at the previous (t-1) time step. Where V represents the number of human joints identified, and C in Represents coordinate system dimensions; semantic connection information of key points. Each pedestrian corresponds to a V×V matrix at each time step. The elements in the matrix represent whether the key points in the corresponding row and column have a semantic connection. The values ​​are 0 or 1, where 0 indicates no semantic connection and 1 indicates a semantic connection. A semantic connection is indicated when there is a skeleton connecting the key points; otherwise, there is no semantic connection.

4. The method according to claim 1, characterized in that, In step 3, the physical intuition module, wherein: Let the final output tensor of the coarse-grained part be Z. coarse Record the forces acting on each pedestrian in the three directions of the coordinate axes; The human joint positions stored in input matrix I3 are converted into relative distances between human joints, and then input together with matrix I4 into the fine-grained part of the network. First, they are processed by the real-time correlation layer. Each time step contained in matrix I3 corresponds to an input channel, which is then input into one processing channel of the real-time correlation layer. The output of the real-time correlation layer is then input into the real-time weighted layer for further processing. Let the fine-grained part output tensor Z... fine Record the forces at the joints of each pedestrian's body along the tangential and normal directions of the skeleton; For each pedestrian, the resultant force of the coarse-grained force output by the coarse-grained component and the fine-grained force output by the fine-grained component is calculated. Then, the acceleration of each joint of the human body is calculated to further update the position of each joint of the pedestrian and the posture of the human body.

5. The method according to claim 1 or 4, characterized in that, In step 4, when optimizing the network parameters of the physical intuition module, the objective function is as follows: Where M represents the number of pedestrians, and T represents time T; X h,t and X h,t-1 These represent the positions of the h-th person's key points at time t and time t-1, as predicted by the physical intuition module. and respectively for X h,t and X h,t-1 Calculate the average of the coordinates of all joints; Y h,t-1 W represents the appearance characteristics of the h-th person at time t-1; coarse,h W fine,h These are the network parameters for the coarse-grained and fine-grained parts of the Physical Intuition module, respectively; X h,t ={X h,t,1 ,...,X h,t,V }, X h,t,i (i = 1, 2, ..., V) represents the position of the i-th joint of the h-th person at time t, and V is the number of human joints identified. The physical intuition module then obtains the acceleration 'a' of the h-th person at the i-th joint at time t. h,t,i Then, further obtain X h,t,i =X h,t-1,i +v h,t-1,i dt+0.5a h,t, i dt 2 Wherein the velocity v of the joint at the previous moment h,t-1,i =(X h,t-1,i -X h,t-2,i ) / (dt), where dt is the time interval between two adjacent frames; W h These are the physical intuition module network parameters for pedestrian h; During the optimization process, a sliding window of length T is used to acquire video frame images. The visual representation module observes the positions of human joints at the past T-1 times. The physical intuition module predicts the force and acceleration of the human joints at time T, simulates the positions of human joints, and then projects the acquired human joint positions into a two-dimensional space to match the human joint positions observed by the visual representation module from the image at time T, and calculates the approximation error. Let X be the value obtained by the h-th person at time t. h,t Projected into two-dimensional space, it is represented as f proj (X h,t ), f proj The projection function is θ; the three-dimensional coordinates of the human joints observed by the visual representation module from the video frame image at time t are θ. h,t Then the loss function for parameter optimization of the physical intuition module is expressed as: Where λ1 and λ2 are the adjustment weights of the two constraint terms; the constraint terms L1 and L2 are as follows: Constraint L1 represents the invariance of skeleton dimensions; for two joints i and j with semantic connection C(i,j) = 1, a semantic connection exists. L2 is based on statistical information about the angle between adjacent bones, γ. i This represents the angle between the two bones connected to the i-th joint. as well as This represents the mean and variance of the angle obtained from the training set.

6. The method according to claim 1, characterized in that, In step 5, the normal behavior distribution is assumed to be a Gaussian mixture distribution with K clusters, and the acceleration of each joint of the h-th person's body, calculated from the real-time acquired images, is A′. h,t Then the pedestrian's abnormal score S h,t The calculation is as follows: Where, μ k ,σ k N(A′) represents the mean and variance of the Gaussian distribution of the k-th cluster. h,t |μ k ,σ k ) represents A′ h,t It follows the probability distribution of the k-th cluster.

7. The method according to claim 1, characterized in that, In step 5, when detecting abnormal human behavior, if the physical intuition module detects abnormal interaction forces between a pedestrian and objects in the vicinity, it will determine that the pedestrian is exhibiting abnormal behavior.

Images