A human abnormal behavior detection method combining appearance texture and motion skeleton

Abstract

The application discloses a human abnormal behavior detection method combining appearance texture and motion skeleton, and belongs to the field of computer vision; first, an original video is divided at equal intervals, and human appearance key regions in each frame and skeleton key points of each human body are extracted respectively; global motion trajectories of all human bodies are calculated; then, based on STGAT, trajectories of skeleton key points of each human body in a future frame are predicted; for each human body, predicted skeleton key points are converted into dense flow heat maps as guidance information by using DFE, and the guidance information is input into CGAN to generate human appearance key regions corresponding to skeleton key point postures; in addition, pixel-by-pixel analysis is used to eliminate background deviation of the appearance key regions to improve accuracy; finally, skeleton key point prediction values and appearance key region generation values of each human body are calculated with corresponding label values to obtain error, and the error is weighted and summed to obtain an abnormal score; the application effectively reduces the false alarm rate and realizes rapid and effective video anomaly detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image, video analysis and processing in computer vision, and is a human abnormal behavior detection method combining appearance texture and motion skeleton. BACKGROUND

[0002] Abnormal detection is defined as the process of detecting data samples that deviate significantly from the normal pattern in a specific domain. With the rapid popularization of monitoring video equipment, human behavior anomaly detection plays an important role in social life and public safety. Unlike fully supervised learning tasks such as target detection and action recognition, the abnormal behavior detection task faces problems such as low coverage, high time cost and unclear definition in the process of using artificial labeling, which is caused by the complexity, uncertainty and imbalance of abnormal data itself. Therefore, the existing mainstream data and literature define this task as a one-class classification problem using only normal data for training, aiming to further expand the gap between a small amount of abnormal behavior and the vast majority of normal events.

[0003] In the past few years, a large number of research results have adopted deep learning-based detection methods, which have brought significant progress to video analysis-related tasks. Deep learning-based video anomaly detection methods usually show significantly better performance than traditional methods.

[0004] Deep learning-based video anomaly detection methods generate accurate descriptions of normal rule behaviors through two common modes: reconstruction and prediction. Reconstruction-based methods reconstruct the input by learning embedding features to generate high-error data corresponding to abnormal behaviors. However, reconstruction-based methods are easily affected by irrelevant background noise, and even fail to effectively detect anomalies due to model overfitting. Prediction-based methods use historical video frames to predict future frames and calculate prediction errors to generate anomaly scores. Prediction-based methods fail to fully learn prior knowledge, which is beneficial for detecting anomalies but difficult to detect normal events, resulting in an increase in false alarm rate.

[0005] Recently, Li et al. explored a simple and parallel way to combine reconstruction and prediction, which only achieved limited improvement in accuracy. Wang et al. provided a novel progressive relationship to handle the combination of the above two modes, but only used appearance pixels as a single feature.

[0006] In video anomaly detection, deep learning methods based on pre-extracted features have attracted much attention due to their better accuracy. Existing multi-dimensional pre-extracted features can be roughly divided into two categories: appearance texture and motion description, which can be used alone or in combination. Ano-Graph proposed by Pourreza et al. regards the appearance key regions extracted from off-the-shelf object detectors as nodes of a spatio-temporal topological graph, and uses edges to describe the interaction between human bodies. Compared with methods that directly generate complete pixels, Ano-Graph reduces background noise to some extent and relieves the computational burden. However, this method cannot effectively separate the depth coupling between foreground and background regions, nor can it make corresponding adjustments according to complex situations such as strong light, oblique shooting, extreme weather, etc.

[0007] Yu et al. combined appearance with gradient images and motion cues represented by optical flow to accurately locate abnormal activity regions in video events. Gradient images roughly describe the target motion trend in a limited direction and within a local region through a hand-designed descriptor; while dense optical flow refines the motion representation, but inevitably brings more estimation noise.

[0008] More difficultly, the classic methods of generating these two features require the video shooting angle to remain stationary in consecutive frames, which greatly hinders their application in more scenarios. Morais et al. proposed a model named MPED-RNN based on RNN, which decomposes human skeletons into two sub-parts, global motion trajectory and local body pose, as motion description features. Luo et al. explored the motion rules of human joints in abnormal behaviors without pixel-level redundancy, while Zeng et al. characterized the interaction between individuals at different levels through a hierarchical topological graph. Similar to other single-modal-based methods, Pang et al. have proved that skeleton-based methods have certain limitations in detecting complex and diverse anomalies. Liu et al. proposed a method named HF 2 -VAD, which is computationally expensive for extracting full-size optical flow operations and less efficient than skeleton-based methods. SUMMARY

[0009] The present application aims at the deficiencies of existing human abnormal behavior detection algorithms in videos, and proposes a human abnormal behavior detection method combining appearance texture and motion skeleton, which improves the overall detection effect.

[0010] The human abnormal behavior detection method combining appearance texture and motion skeleton, comprising the following steps:

[0011] Step one, for the original video data containing the human behavior to be tested, it is divided into several segments at equal intervals;

[0012] Step two, for all independent video frames in each segment, use the target detection algorithm to extract all human appearance key regions in each frame;

[0013] The coordinate set of N human appearance key regions in all frame images within the current segment T is represented as S = (x1, x2, y1, y2), x1, x2, y1, y2 respectively correspond to the positions of the four vertices of the human coordinate frame. The coordinate frame of the nth human in the tth frame image is represented as t∈T, n represents the total number of all humans in each frame image, x and y represent the horizontal and vertical coordinates of the coordinate frame including the human body;

[0014] Step three, for each segment, use a multi-target tracking algorithm to calculate the global motion trajectory of all humans in the segment;

[0015] Step 301, for the segment T time, calculate all human region coordinate frames as prior information;

[0016] Step 302, calculate the intersection over union IOU of all human coordinate frames between adjacent frames, and obtain the corresponding cost matrix M;

[0017] Let two adjacent frames t1, t2 contain n1, n2 humans respectively, and the matrix M has n1 rows and n2 columns. Each row represents all n2 human coordinate frames in the current frame t2, each column represents all n1 human coordinate frames in the previous frame t1, and each element is the IOU value, which represents the mutual correlation degree of any two human coordinate frames in adjacent frames.

[0018] Step 303, use the Sinkhorn optimal transport algorithm to iterate the standardized cost matrix M, further expand the distance between elements with high and low correlation degrees, and obtain the approximate solution of the optimal transport distance

[0019] Step 304, use the Hungarian algorithm to calculate the bipartite graph matching of the approximate solution , that is, complete the pairwise matching of each human in the front and back frames, obtain the position of each person in the adjacent front and back frames, count the motion trajectory of each human body in all frames within T time, and obtain the global motion trajectory of N human bodies, denoted as

[0020] where represents the geometric center point of the ith human body in the trajectory coordinates of the nth human body in the tth frame;

[0021] Step four, use the pose detection algorithm to extract 2D skeleton key points in all human key regions;

[0022] The N human region coordinate frames within T time Input pose detection algorithm, extract the corresponding human posture respectively, each human body has 17 key points;

[0023] T time N human skeleton key point coordinate set The jth key point of the nth human body in the tth frame;

[0024] Step five, using the global motion trajectory of all human bodies in each segment, combining the skeleton key points of each human body, based on the multi-scale attention mechanism of space-time graph attention network, predicting the trajectory of each human skeleton key point in the future frame in the segment;

[0025] The space-time graph attention network STGAT consists of three sub-modules: trajectory graph generator TGO, space-time self-attention network STTN and skeleton transfer network STN.

[0026] For the segment T time, TGO converts the global trajectory of all human bodies in all frames into space-time graph structure data respectively;

[0027] The nth human global trajectory corresponding to the tth frame picture Respectively converted into space-time graph structure data {G1, G2,.., Gn} t |t∈T}; The conversion formula is:

[0028] G t =(V(G t ),E(G t ))

[0029] V(G t )={V i (G t )|i∈{1,...,n}}

[0030]

[0031]

[0032] Where V(G t ) represents the set of single-frame graph nodes, n represents the total number of nodes, corresponding to the coordinates of each human body; The degree of association between all graph nodes is described by the adjacency matrix A, is the element of the i-th row and j-th column of the adjacency matrix, which represents the degree of association between the i-th and j-th graph nodes; All associated elements represent the edges of a single-frame graph, that is, the set E(G t );

[0033] The STN is responsible for effectively encoding the spatio-temporal graph structure data, and improving the spatio-temporal consistency of trajectory prediction through the series connection between the spatial self-attention network and the temporal self-attention network.

[0034] Based on the graph theory, in the structured graph, the process of information transmission from the graph node j to the graph node i is defined as m j→i For any one structured graph, each graph node i is associated with its feature h i and its neighborhood set Nb(i);

[0035] Based on the original self-attention function ( vjbelongs to the original function), the graph convolution operation of the self-attention mechanism of the graph node i is:

[0036]

[0037] h′ i = f out (Att(i))+Att(i)

[0038] Where f out is an output function, and h′ i is the update of the graph convolution function of the self-attention mechanism to the graph node i.

[0039] For each human body in each frame of image in the segment, the STN is responsible for normalizing the skeleton key points of the human body in the previous frame, and migrating the result to the global trajectory represented by the spatio-temporal graph, and jointly predicting the trajectory of the skeleton key points of the human body in the future frame.

[0040] The regularization layer is used to migrate the skeleton key points, and the multi-layer perception is used to predict the trajectory of the skeleton key points, and the migration and prediction process is uniformly represented as:

[0041]

[0042] Where x represents the skeleton key points of the current human body, y represents the global trajectory of the current human body, and a(y) represents a scaling function, and β(y) represents a bias function. Where μ(x), σ(x) represent the mean and standard deviation of the sample respectively; the sample is the skeleton key points of the current human body.

[0043] Step six, for each human body, the predicted skeleton key points in the future frame are converted into dense flow heat maps as guidance information by using the dense flow estimator DFE, and input into the conditional generative adversarial network CGAN to generate the appearance key regions of the human body corresponding to the skeleton key point posture; the two together constitute the dense flow conditional generative adversarial network DFCGAN.

[0044] DFE is specifically: first, the skeleton key points are converted into 17-channel posture heat map as structure guidance information, let p s and p t respectively represent the skeleton key points of the source image x s and the target image x t .

[0045] The dense flow estimator F takes p s , p t , x s and x t as input and generates a dense flow field w, which is expressed as:

[0046] w=F(p s , p t , x s )

[0047] CGAN generates real texture images The formula is:

[0048]

[0049] G is a conditional generative adversarial network, and the joint loss function LD is generated by the loss function Adversarial loss function L adv , perceptual loss function L perc and style loss function L style , which is trained end-to-end to directly learn the mapping relationship between the latent appearance distribution and the motion information extracted from the image, so as to directly generate a large number of texture samples.

[0050]

[0051] Where λ a , λ p and λ s are the respective loss weights.

[0052] In the training of DFCGAN, a pixel-by-pixel analysis method based on background elimination BE is adopted, which generates pixel-by-pixel texture by making DFCGAN reconstruct the key areas in the current frame and the predicted future frame at the same time, so as to balance the overall brightness and saturation, and reduce the sharpness noise in the background.

[0053] Specifically, at t and t+1, there are skeleton key points p obs , p obs+1 , (predicted) and appearance key regions x obs , x obs+1 ; the BE process is specifically represented as:

[0054]

[0055] where p sou = p obs , and x sou = x obs for prediction; p sou = p obs , x tar = x obs+1 and x sou = x obs for reconstruction.

[0056] Step seven, for each human body in each frame of image, use two anomaly value judgment modules to take the skeleton key point prediction value and the appearance key region generated value as input, and calculate the error with the corresponding label value, get two types of anomaly scores, and perform weighted summation to get the final anomaly score;

[0057] 1) Calculate the generation error between the generated value and the label value;

[0058] The label value is defined as the real skeleton key point coordinates and appearance key region at the current time, and the mean square error (MSE) loss function is selected to calculate the difference between the framework output value and the label value, and the anomaly scores L1 of the skeleton key points and the anomaly scores L2 of the appearance are obtained respectively, and the calculation formula is:

[0059]

[0060]

[0061]

[0062] where f s and f r represent the skeleton key point score output by the anomaly value judgment module and the appearance key region score output by the DFCGAN respectively, and represent the real labels of the skeleton key points and the appearance key regions.

[0063] The weighted summation formula is

[0064]

[0065] where μ1, σ1, μ2, σ2 are the mean and standard deviation of L1 and L2 respectively, and ω1 and ω2 are the weights of L1 and L2.

[0066] Step eight, perform anomaly behavior detection on the final anomaly score in a binary classification manner, that is, distinguish the time length containing anomalies and normal in a specific segment in the data set, so as to obtain the human body with abnormal behavior.

[0067] The characteristics and benefits of the present application are as follows:

[0068] (1) A human abnormal behavior detection method combining appearance texture and motion skeleton, using a Spatio-Temporal Graph Attention Transformer (STGAT) based on a multi-scale attention mechanism, encodes the spatio-temporal features of the global historical trajectory into a graph structure, and integrates the corresponding local skeleton to more accurately jointly predict the trajectory of the skeleton. This method follows the solution paradigm of a class of classification problems, ensuring that the model learns well in the normal mode, thereby sensitively detecting human abnormal behavior.

[0069] (2) A human abnormal behavior detection method combining appearance texture and motion skeleton, a joint detection framework is proposed, which combines human skeleton joints and appearance key regions in a progressive manner through multi-modal fusion, thereby improving the detection accuracy of human abnormal behavior in video. Specifically, a Dense Flow Conditional Generative Adversarial Network (DFCGAN) based on dense flow is designed, which reorganizes the source image and the target skeleton predicted by STGAT, uses the predicted skeleton as guidance information to generate human appearance key regions, so that the posture skeleton containing abnormal behavior deteriorates the quality of the corresponding pixel generation, and further expands the gap between normal and abnormal samples.

[0070] (3) A human abnormal behavior detection method combining appearance texture and motion skeleton, in the pixel generation process, a pixel-by-pixel analysis method is used, without adding new network structures, by simultaneously reconstructing the current label value and predicting the future appearance key region, the foreground consistency enhancement and background deviation elimination are realized. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is a flow chart of the human abnormal behavior detection method combining appearance texture and motion skeleton of the present application;

[0072] Figure 2 is a whole working principle diagram of the human abnormal behavior detection algorithm proposed in the embodiment of the present application;

[0073] Figure 3 is a global architecture diagram of the human abnormal behavior detection joint framework proposed in the present application;

[0074] Figure 4 is a specific structure diagram of the spatio-temporal self-attention network based on graph attention mechanism designed in the present application;

[0075] Figure 5It is a specific structure diagram of the generative adversarial network based on dense flow conditions designed by the application.

[0076] Figure 6 It is a part of the effect diagram of the qualitative research results of the detection method described in the application.

[0077] Figure 7 It is a detection accuracy diagram of the detection method proposed by the application on multiple data sets. DETAILED DESCRIPTION

[0078] In order to facilitate those skilled in the art to understand and implement the application, the application will be further described in detail and in depth in combination with the drawings.

[0079] The application proposes a human abnormal behavior detection method combining appearance texture and motion skeleton, which trains positive samples through a detection framework to maximize the rules of normal behavior, establishes a paradigm to distinguish normal and abnormal events related to human behavior, and combines multiple features and multiple modeling methods to improve the accuracy of video abnormal behavior detection, uses a pixel generation method to input historical frames and skeleton guidance information to predict future frames and reconstruct current frames. Before that, the future skeleton is predicted according to the historical skeleton trajectory, which helps to gradually propagate the error of skeleton prediction, and finally deteriorates the pixel-level abnormal generation error to further improve the detection performance.

[0080] Firstly, the application extracts the multi-modal features of the fixed segment video input in the public human abnormal behavior data set, including motion skeleton and appearance texture; then, it uses a spatio-temporal self-attention network to predict the future skeleton according to the historical skeleton trajectory, and simultaneously uses it as guidance information, uses a dense flow condition generative adversarial network to input the target skeleton and source image as input to jointly generate the target appearance key area, obtains the generated values of the two types of abnormal criteria of skeleton and texture, and further improves the overall method accuracy by reconstructing the key area of the source image and predicting the target key area to enhance the foreground consistency and eliminate background noise; finally, according to the scene crowd density, environment light background, specific behavior and camera shooting angle and other factors, the two types of abnormal scores are weighted and fused to obtain the final abnormal score, which effectively improves the detection accuracy of the framework and reduces the false alarm rate, realizes fast and effective video abnormal behavior detection.

[0081] The human abnormal behavior detection method combining appearance texture and motion skeleton, as shown in Figure 1 includes the following steps:

[0082] Step one, for the original video data containing the human behavior to be detected, it is divided into several segments at equal intervals;

[0083] Step two, for each independent video frame in each segment, use the target detection algorithm to extract all human appearance key regions in each frame;

[0084] The coordinate set of N human appearance key regions in all frame images within the T time of the current segment is represented as B = (x1, x2, y1, y2), x1, x2, y1, y2 respectively correspond to the positions of the four vertices of the human coordinate frame. The coordinate frame of the nth human in the tth frame image is represented as t∈T, n represents the total number of all humans in each frame image, x and y respectively represent the horizontal and vertical coordinates of the coordinate frame including the human body;

[0085] Step three, for each segment, use a multi-target tracking algorithm to calculate the global motion trajectory of each human body within the segment;

[0086] Step 301, for the T time of the segment, calculate all human region coordinate frames as prior information;

[0087] Step 302, calculate the intersection over union IOU of all human coordinate frames between adjacent frames to obtain the corresponding cost matrix M;

[0088] Intersection over union Let two adjacent frames t1, t2 contain n1, n2 human bodies respectively, and the matrix M has n1 rows and n2 columns. Each row represents all n2 human body coordinate frames of the current frame t2, each column represents all n1 human body coordinate frames of the previous frame t1, and each element is the IOU value, which represents the mutual correlation degree of any two human body coordinate frames in adjacent frames.

[0089] Step 303, use the Sinkhorn optimal transport algorithm to iterate the standardized cost matrix M to further expand the distance between elements with high and low correlation degrees, and obtain the approximate solution of the optimal transport distance

[0090] Step 304, use the Hungarian algorithm to calculate the bipartite graph matching of the approximate solution , that is, to complete the pairwise matching of each human body in adjacent frames, to obtain the position of each human in adjacent frames, to count the motion trajectory of each human body in all frames within the T time, and to obtain the global motion trajectory of N human bodies, denoted as

[0091] wherein represents the geometric center point of the ith human body in the trajectory coordinates of the nth human body in the tth frame;

[0092] Step four, use a pose detection algorithm to extract 2D skeleton key points in all human key regions;

[0093] The N person body area coordinate frame in T time The posture detection algorithm is input, and the respective corresponding human body postures are extracted, and each human body has 17 key points;

[0094] The set of N person body skeleton key point coordinates in T time The jth key point of the ith person body in the nth person body in the tth frame

[0095] Step five, using the global motion trajectory of all human bodies in each segment, combining the skeleton key points of each human body, based on the multi-scale attention mechanism of the space-time graph attention network, predicting the trajectory of the skeleton key points of each human body in the segment in the future frame;

[0096] The space-time graph attention network STGAT is composed of three sub-modules: the trajectory graph generator TGO, the space-time self-attention network STTN, and the skeleton transfer network STN.

[0097] For the segment T time, TGO converts the global trajectory of all human bodies in all frames into space-time graph structure data respectively;

[0098] The global trajectory of the nth person body corresponding to the tth frame picture Respectively converted into space-time graph structure data {G1, G2,.., Gn} t |t∈T}; The conversion formula is:

[0099] G t = (V(G t ), E(G t ))

[0100] V(G t ) = {V i (G t )|i∈{1,...,n}}

[0101]

[0102]

[0103] Where V(G t ) represents the set of single-frame graph nodes, n represents the total number of nodes, corresponding to the coordinates of each human body, and the degree of association between all graph nodes is described using an adjacency matrix A, is the element in the ith row and jth column of the adjacency matrix, representing the degree of association between the ith and jth graph nodes; all associated elements represent the edges of a single-frame graph, i.e., the set E(G t );

[0104] The STN is responsible for effectively encoding the spatio-temporal graph structure data, and improving the spatio-temporal consistency of trajectory prediction through the series connection between the spatial self-attention network and the temporal self-attention network.

[0105] Based on the graph theory, in the structured graph, the process of information transmission from the graph node j to the graph node i is defined as m j→i For any one structured graph, each graph node i is associated with its feature h i and its neighborhood set Nb(i);

[0106] Based on the original self-attention function ( v j belongs to the original function), the graph convolution operation of the self-attention mechanism of the graph node i is:

[0107]

[0108] h′ i =f out (Att(i))+Att(i)

[0109] Where f out is an output function, and h′ i is the update of the graph convolution function of the self-attention mechanism to the graph node i.

[0110] For each human body in each frame of image in the segment, the STN is responsible for normalizing the skeleton key points of the human body in the previous frame, and migrating the result to the global trajectory represented by the spatio-temporal graph, and jointly predicting the trajectory of the skeleton key points of the human body in the future frame.

[0111] The regularization layer is used to migrate the skeleton key points, and the multi-layer perception is used to predict the trajectory of the skeleton key points. The migration and prediction process is uniformly represented as:

[0112]

[0113] Where x represents the skeleton key points of the current human body, y represents the global trajectory of the current human body, and a(y) represents a scaling function, and b(y) represents a bias function. Where m(x), s(x) represent the mean and standard deviation of the sample respectively; the sample is the skeleton key points of the current human body.

[0114] Step six, for each human body, the predicted skeleton key points in the future frame are converted into dense flow heat maps as guidance information by using the dense flow estimator DFE, and input into the conditional generative adversarial network CGAN to generate the appearance key regions of the human body corresponding to the skeleton key point posture; the two together constitute the dense flow conditional generative adversarial network DFCGAN.

[0115] The dense flow condition generation adversarial network DFCGAN is composed of two parts: a dense flow estimator (DFE) and a conditional generation adversarial network (CGAN). First, the trajectory of the historical skeleton key points of the current human body is input into the spatio-temporal graph graph attention network to generate a skeleton joint prediction value, and then the generated skeleton is converted into a dense flow heat map as guidance information, and a high-quality appearance key region corresponding to the skeleton posture is output using the conditional generation adversarial network.

[0116] In addition, by using the pixel-by-pixel analysis method, the label value is reconstructed and the key region is predicted, thereby improving the foreground consistency in a single frame and eliminating certain background noise.

[0117] The DFE describes the relative motion between a pair of source skeletons and target skeletons, converts the sparse skeleton into a dense flow pixel field, generates a dense flow heat map as guidance information, and inputs the CGAN to generate a target key region corresponding to the posture. Specifically, first, the skeleton key points are converted into 17-channel posture heat maps as structural guidance information, and p s and p t represent the skeleton key points of the source image x s and the target image x t , respectively.

[0118] The dense flow estimator F takes p s , p t , x s , and x t as inputs and generates a dense flow field w, which is represented as:

[0119] w=F(p s , p t , x s )

[0120] The conditional generation adversarial network G takes x s , p s , p t , and w as inputs and generates a real texture image The formula is:

[0121]

[0122] The CGAN explicitly calculates the spatial deformation by grid sampling the dense flow to align the appearance features of the encoder with the target feature space. The joint loss function LD is generated by the loss function the adversarial loss function L adv , the perceptual loss function L per c, and the style loss function L style to train the conditional generation adversarial network G end-to-end, directly learn the mapping relationship between the latent appearance distribution and the motion information to the image, and thereby directly generate a large number of texture samples.

[0123]

[0124] wherein λ a , λ p and λ s are respective loss weights.

[0125] In the training of the DFCGAN, a pixel-by-pixel analysis method based on background elimination BE is adopted, by making the DFCGAN simultaneously reconstruct the key regions in the current frame and the predicted future frame to generate pixel-by-pixel texture, to balance the overall brightness and saturation, and to reduce the sharpness noise in the background.

[0126] Specifically, at time t and t+1, there are skeleton key points p obs , p obs+1 , (predicted) and appearance key regions x obs , x obs+1 ; the BE process is specifically represented as:

[0127]

[0128] wherein p sou =p obs , and x sou =x obs are used for prediction; p sou =p obs , x tar =x obs+1 and x sou =x obs are reconstructed.

[0129] Step seven, for each human body in each frame of image, using two outlier judgment modules, the skeleton key point prediction value and the appearance key region generated value are taken as input, and the error is calculated with the corresponding label value, two types of abnormal scores are obtained, and the final abnormal score is obtained by weighted summation;

[0130] 1) Calculate the generation error between the generated value and the label value;

[0131] The label value is defined as the real skeleton key point coordinates and appearance key region at the current time, the mean square error MSE loss function is selected to calculate the difference between the framework output value and the label value, to obtain the abnormal score L1 of the skeleton key point and the abnormal score L2 of the appearance, the calculation formula is:

[0132]

[0133]

[0134]

[0135] where fs and f r respectively represent the skeleton key point score output by the anomaly value judgment module and the appearance key region score output by the DFCGAN, and represent the real labels of the skeleton key points and the appearance key regions.

[0136] The weighted sum formula is

[0137]

[0138] where μ1, σ1, μ2, σ2 are the mean and standard deviation of L1 and L2 respectively, and ω1 and ω2 are the weights of L1 and L2 respectively.

[0139] Step eight, the final anomaly score is detected in a binary classification manner, that is, the lengths of time containing anomalies and normal in a specific segment in the data set are distinguished, so as to obtain the human body with abnormal behavior.

[0140] Considering the complexity, unknowability and uneven amount of positive and negative samples of abnormal behavior, the present application detects abnormal behavior in a binary classification manner, that is, the lengths of time containing anomalies and normal in a specific segment in the data set are distinguished, rather than identifying the type of specific behavior. The evaluation standard for the classical binary classification problem is usually carried out by the area under the receiver operating characteristic curve (AUROC), and the reasonable range of AUROC is between 0 and 1, and the higher the output value represents the better performance of the classifier.

[0141] The detection framework proposed in the present application reports the AUROC performance on four large public abnormal behavior data sets, and achieves good results.

[0142] Embodiment:

[0143] As shown in Figure 2 , the following specific steps are included:

[0144] Step one, after the original video is divided into small segments at equal intervals, first, the appearance key region in each frame is extracted using a target detection algorithm, that is, the human body position information is detected; then the motion trajectory of each human body in multiple video segments is calculated using a multi-target tracking algorithm, and finally the skeleton joints in the key region of the human body are extracted using a skeleton key point positioning algorithm. The final multiple pre-extracted features include the trajectories of the appearance key region and the human body skeleton key points.

[0145] 1) Position the coordinates of the human appearance region;

[0146] The YOLOv5 object detection algorithm is used to extract human body region coordinates (x1, x2, y1, y2) from video frames, resulting in a set B containing the coordinates of all human body regions within a fixed frame. tnxy , where t represents time, n represents the number of people, and x and y represent the horizontal and vertical axes, respectively.

[0147] 2) Extract the key points of the 2D skeleton within the appearance area;

[0148] Using the HR-Net pose detection algorithm, based on B obtained in 1), tnxy As input, 17 key points are extracted to obtain a set P containing the coordinates of all human skeleton key points within a fixed frame. tnxy Compared to appearance region features based on RGB information, the skeleton can more concisely and clearly describe the motion characteristics of the human body.

[0149] 3) Use Sinkhorn's multi-target tracking algorithm to extract human motion trajectories;

[0150] To obtain the human motion trajectory within multiple frames, this embodiment uses the human region coordinates B tnxy As prior information, a cost matrix M is constructed by calculating the Inter-Coordinate U (IoU) of adjacent frame coordinate regions to describe the correlation between frames preceding and following the same target. M is then iteratively normalized using the Sinkhorn optimal transmission algorithm. The core of Sinkhorn is minimizing the Wasserstein distance. Iterate repeatedly until the minimum is reached. Until the preset threshold is met, the optimal approximate solution of M is obtained.

[0151]

[0152]

[0153] Where α and β are constant parameters for solving the problem, and λ is a hyperparameter that controls the convergence rate.

[0154] Finally, the classic matching algorithm, the Hungarian algorithm, is used to find... Bipartite graph matching is used to extract the global motion trajectory of multiple targets.

[0155] Step 2: A spatiotemporal graph attention network based on a multi-scale attention mechanism predicts the future skeleton; a dense flow conditional generative adversarial network generates key regions of the target's appearance, forming the core components of the joint framework.

[0156] First, the trajectory of the historical skeleton key point is input into the spatio-temporal graph attention network to generate a skeleton joint prediction value, and then the generated skeleton is converted into a dense flow heat map as guidance information, and a conditional generative adversarial network is used to construct a high-quality appearance key region corresponding to the skeleton posture.

[0157] In addition, by reconstructing the label value and predicting the key region analysis method based on pixels, the foreground consistency in a single frame is improved and certain background noise is eliminated.

[0158] 1) using a spatio-temporal self-attention network to predict a future skeleton;

[0159] The application designs a spatio-temporal self-attention network (STGAT) based on a multi-scale attention mechanism, which predicts a future skeleton joint through a graphical historical skeleton key point trajectory.

[0160] As shown in Figure 3 and Figure 4 , the specific structure of the time self-attention module of the STGAT sub-module is shown.

[0161] The self-attention module first takes the query matrix key matrix and value matrix as inputs.

[0162] For the i-th node (person), it is represented as:

[0163]

[0164]

[0165]

[0166] Where f Q , f K and f V are the query function, key function and value function corresponding to the i-th node.

[0167] The attention features of each node are calculated respectively:

[0168]

[0169]

[0170] Where head j = Att j (Q i , K i , V i ), f Ois a full connection layer, responsible for fusing the kth head and Att j indexed jth head of the feature.

[0171] Figure 3 Also show the specific structure of the spatial self-attention module, which is used to represent the spatial interaction between the human body.

[0172] The application proposes a novel graph convolution operation based on self-attention mechanism, which can transmit information on undirected graph structure. For a feature vector h in the spatial feature set i , its corresponding query vector is represented as q i = f Q (h i ), the key vector is k i = f K (h i ), and the value vector is v i = f V (h i ). The information transmission from node j to i is defined as:

[0173]

[0174] The attention function can be recorded as:

[0175]

[0176] Similar to the traditional graph attention convolution, but based on the strong attention mechanism brought by Transformer, it has better feature extraction ability. For any graph G = (V, E), where V = {1, 2,..., n} is the node set, is the edge set. Assume that each node i is associated with an embedding feature h i and a neighborhood set Nb(i). The graph convolution operation of the self-attention mechanism of node i can be written as:

[0177]

[0178] h′ i = f out (Att(i))+Att(i)

[0179] where f out is the output function, and the full connection layer h′ i is the updated embedding of the graph convolution function of the self-attention mechanism for node i.

[0180] 2) Generate target appearance key areas using dense flow condition generation adversarial network;

[0181] This invention designs a Dense Flow Conditional Generative Adversarial Network (DFCGAN) that generates high-quality key regions of human appearance by deeply decoupling and reconstructing the texture information of the source image and the predicted skeleton provided by STGAT. Figure 5 As shown, DFCGAN consists of two parts: Dense Flow Estimator (DFE) and Conditional Generative Adversarial Network (CGAN).

[0182] First, the 17 key points of the skeleton were converted into a pose heatmap as a guide for the skeleton structure, such as... Figure 6 As shown in (e). Let p s and p t Representing the source image x s and target image x t Guided by the skeleton structure, the dense flow estimator F will p s p t x s and x t As input, a dense flow field w is generated, represented as:

[0183] w = F(p) s p t x s )

[0184] F is designed as a fully convolutional network, and a sampling correctness loss function L is proposed. c This makes the source image feature v s The dense flow field w is aligned with the grid sampling to find a reasonable position. The dense flow field w describes the source image features v. s The degree of distortion establishes a mapping relationship between source and target features. The sampling correctness loss function, based on the similarity between features generated by specific layers of the pre-trained VGG19 network and label value features, can constrain the output of F to ensure the correct position of w. The corresponding v is generated through grid sampling. t Sampling accuracy loss function L c Calculate v s，w and v t The relative cosine similarity between them is denoted as:

[0185]

[0186] in Indicates v s，w and v t The cosine similarity between them, with the coordinate set Ω representing the N feature maps of VGG19. and This indicates that v is located at coordinates l = (x, y). s，w and v t Features This is the normalization term.

[0187] CGAN explicitly computes spatial warping by grid sampling dense flow to align the encoder's appearance features with the target feature space region.

[0188] The conditional generative adversarial network G takes x s , p s , p t and w as input and produces a realistic texture image denoted as:

[0189]

[0190] G is trained jointly by the l1 loss function, the generative adversarial loss function, the perceptual loss function and the style loss function. The l1 loss function is denoted as:

[0191]

[0192] The generative adversarial loss function is denoted as:

[0193]

[0194] where D is the discriminator of G. The perceptual loss function computes the l1 distance between the activation maps of a pre-trained network. It can be denoted as:

[0195]

[0196] where φ i is the activation map of the i-th layer of the pre-trained network. The style loss function computes the statistical error between the activation maps as:

[0197]

[0198] where is the Gram matrix constructed from the activation maps φ j .

[0199] The overall method is trained end-to-end by optimizing the joint loss function L D in DFE and CGAN simultaneously, directly generating a large number of samples by directly learning the latent appearance distribution and the mapping relationship from motion information to images.

[0200]

[0201] 3) Use the pixel-by-pixel analysis method to enhance foreground consistency and eliminate background bias

[0202] To make DFCGAN perform well in different scenarios, DFCGAN generates pixel-wise texture of human appearance by reconstructing key regions in the current frame and predicting key regions in the future frame, including balancing the brightness and saturation of the foreground in the key regions to enhance consistency between the source and target images, and smoothing the sharpness of the image background to reduce noise.

[0203] Finally, in the stage of calculating the anomaly score, the pixel-wise analysis method pays more attention to the hue difference of the foreground region and outputs a pair of highly similar background regions to offset the deviation existing in each other without adding additional models or network structures. As shown in Figure 6 (d) and (e) with Figure 6 (j) and (1) show that after pixel-wise analysis, the brightness of the foreground region remains consistent, highlighting the hue difference, and the noisy background region is significantly blurred and tends to be similar.

[0204] Step three, use two anomaly value judgment modules to input the skeleton prediction value and appearance generation value as inputs, and calculate the error with the corresponding label value, and divide it into anomaly score 1 and anomaly score 2. The final anomaly score is obtained by weighted fusion of anomaly score 1 and 2. According to the evaluation standard of area under the receiver operating characteristic curve (AUROC), the accuracy of the overall detection framework is tested, and a full comparison is made with other latest methods.

[0205] 1) Calculate the generation error between the generated value and the label value;

[0206] After step two, the motion skeleton prediction value f s and the appearance region generation value f r For training samples in the normal mode in a one-class classification problem, the mean square error (MSE) loss function is selected to calculate the difference between the framework output value and the label value, and the anomaly scores L1 and L2 are obtained respectively;

[0207] 2) Weighted fusion of anomaly scores;

[0208] In view of the abnormal behavior in complex scenes, variable light, different crowd density and various interactive actions and other factors, two types of anomaly scores L1 and L2 need to be added with different weights to obtain the optimal anomaly score L3, and the weighted sum formula is

[0209]

[0210] Where μ1, σ1, μ2, σ2 are the mean and standard deviation of the skeleton trajectory error and appearance region error of all training samples, and ω1 and ω2 are the weights of the two anomaly scores.

[0211] 3) Calculate the detection framework accuracy and compare it with other latest methods;

[0212] In the experimental test phase, the application runs in an experimental environment of NVIDIA GTX3080 graphics card, Ubuntu 18.04 system and Pytorch deep learning framework.

[0213] The area under the receiver operating characteristic curve (AUROC) is used as the test standard of the overall algorithm.

[0214] The application is tested on four large public abnormal behavior data sets: CUHK Avenue, ShanghaiTech, Corridor and ADOC, and in order to further prove that the application has better precision and robustness for human-related abnormal behavior, according to the strategy of other mainstream methods, some irrelevant data is removed to form the human behavior related HR-Avenue, HR-ShanghaiTech and HR-Corridor data sets. As shown in the following table, the method of the application leads other latest mainstream methods on multiple data sets.

[0215]

[0216] As shown in Figure 6 , it is a qualitative research effect diagram of the human abnormal behavior detection method combined with appearance texture and motion skeleton running in multiple different scenes of ShanghaiTech data set. As can be seen from Figure 6 (a) and (b), for abnormal behavior, STGAT predicts skeleton joints according to normal mode, which is significantly different from the historical skeleton; for normal behavior, the similarity between the predicted skeleton and the historical skeleton is obviously improved. As can be seen from Figure 6 (j) and (l), DFCGAN reconstructs the appearance key area of the source image well, which helps to enhance the foreground consistency and eliminate background noise, and also predicts the target image with high quality, which helps to amplify the skeleton level abnormal error at the pixel level. As can be seen from Figure 6 (m), the method of the application outputs a relatively chaotic and fuzzy difference map when detecting abnormal behavior, and outputs a very clear and clean difference map when detecting normal behavior.

[0217] As shown in Figure 7 , it is a precision test diagram of the human abnormal behavior detection method combined with appearance texture and motion skeleton in multiple different scenes of CUHK Avenue, ShanghaiTech and Corridor data sets. The method proposed by the application shows high precision and better generalizability across multiple data sets.

Claims

1. A method for human abnormal behavior detection combining appearance texture and motion skeleton, characterized in that, The specific steps are as follows: step one, for the original video data containing the human behavior to be tested, the original video data is equally divided into several segments; Step two, for all independent video frames in each segment, use the target detection algorithm to extract all human appearance key regions in each frame; Step three, for each segment, use a multi-target tracking algorithm to calculate the global motion trajectory of all human bodies in the segment; Step four, use a posture detection algorithm to extract 2D skeleton key points in all human key regions; Step five, using the global motion trajectory of all human bodies in each segment, combining the skeleton key points of each human body, based on the multi-scale attention mechanism of the spatio-temporal graph attention network STGAT, the trajectory of the skeleton key points of each human body in the segment in the future frame is predicted; Step six, for each human body, the predicted skeleton key points in the future frame are converted into dense flow heat maps as guidance information by using a dense flow estimator DFE, which are input into a conditional generative adversarial network CGAN to generate human appearance key regions corresponding to the posture of the skeleton key points; The DFE and the CGAN together constitute a dense flow conditional generative adversarial network DFCGAN; The DFE is specifically: First, the skeleton key points are converted into 17-channel pose heat maps as structure guidance information, and the pose heat maps are used to guide the generation of the target image and respectively represent the skeleton key points of the source image and the target image respectively The dense flow estimator DFE takes as input , , and and generates a dense flow field denoted as: CGAN produces realistic texture images The formula is: a conditional generative adversarial network, by a loss function , an adversarial loss function , a perceptual loss function , and a style loss function to generate a joint loss function end-to-end training, directly learning the mapping relationship of the latent appearance distribution and the motion information to the image, thereby directly generating a large number of texture samples; In the training of the DFCGAN, a pixel-by-pixel analysis method based on background elimination BE is used, which balances the overall brightness and saturation and reduces the sharpness noise in the background by making the DFCGAN reconstruct the key regions in the current frame and the predicted future frame to generate pixel-by-pixel textures; Specifically, at and moments, there are skeleton key points , , and appearance key regions , ; the BE process is specifically represented as: wherein , and for predicting; , and reconstructing; Step seven, for each human body in each frame of image, use two outlier judgment modules to input the skeleton key point prediction value and the appearance key region generation value as input, and calculate the error with the corresponding label value to obtain two types of outlier scores, and then perform weighted summation to obtain the final outlier score; Step eight, the final outlier score is detected for abnormal behavior in a binary classification manner, that is, the time length containing abnormalities and normality in a specific segment in the data set is distinguished, so as to obtain the human body with abnormal behavior.

2. The method of claim 1, wherein the method further comprises: In the second step, in the T time of the current segment, all frame images The coordinate set of the key area of the personal body appearance is represented as , , respectively corresponding to the positions of the four vertices of the body coordinate frame; represents the first frame image the coordinate frame of the personal body, , represents the total number of all personal bodies in each frame image, and respectively represents the horizontal and vertical coordinates of the coordinate frame including the personal body.

3. The method of claim 1, wherein the method further comprises: The step three is specifically: Step 301, for the segment T time, all human body region coordinate frame as prior information; At step 302, a two-by-two intersection-over-union (IOU) of all human coordinate frames between adjacent frames is calculated to obtain a corresponding cost matrix ; Let two adjacent frames Within, respectively contain Personal body, matrix Common Line Column; each row represents all Personal body coordinate frame of the current frame Each column represents all Personal body coordinate frame of the previous frame Each element, namely IOU value, represents the mutual correlation degree of any two personal body coordinate frames in the adjacent frames before and after; Step 303, using Sinkhorn optimal transport algorithm to iterate the standardized cost matrix , further expanding the distance between elements with high and low correlation, and obtaining an optimal transport distance approximation solution ; Step 304, calculating the approximate solution by using the Hungarian algorithm The bipartite graph matching of the two frames, that is, completing the pairwise matching of each human body in the front and rear frames, obtaining the position of each person in the adjacent front and rear frames, counting the motion trajectory of each human body in all frames in T time, obtaining the global motion trajectory of the human body, denoted as . ; wherein , represents the intra-frame the trajectory coordinates of the individual body. The geometric center point of the individual body.

4. The method of claim 1, wherein the method further comprises: In the fourth step, the T time is divided into Personal body area coordinate frame The input posture detection algorithm extracts the respective corresponding human body postures, and each human body has 17 key points. T time A set of personal body skeleton key point coordinates ; For the Intra-frame The The Key point.

5. The method of claim 1, wherein the method further comprises: In step five, the spatio-temporal graph attention network STGAT is composed of three sub-modules: a trajectory graph generator TGO, a spatio-temporal self-attention network STTN, and a skeleton transfer network STN; For the T time of the segment, the TGO converts the global trajectory of all human bodies in all frames into spatio-temporal graph structure data respectively; The first The frame picture corresponds to The global trajectory of the individual body Respectively converted into spatio-temporal graph structure data The conversion formula is: wherein represents a set of single-frame graph nodes, represents the total number of nodes, corresponding to the coordinates of each human body; represents a set of single-frame graph edges, using an adjacency matrix to represent the degree of association between all graph nodes described, is the element in the th row and the th column of the adjacency matrix, representing the degree of association between the th and the th graph nodes; The STTN is responsible for effectively encoding the spatio-temporal graph structure data, and improving the spatio-temporal consistency of trajectory prediction through the series connection between the spatial self-attention network and the temporal self-attention network; Based on graph theory, in a structured graph, the process of information passing from a graph node to a graph node is defined as ; for any one structured graph, each graph node is associated with its features and its neighborhood set ; Based on the original self-attention function , Belonging to the original function, the graph node The graph convolution operation of the self-attention mechanism is: wherein is an output function, the features output by the fully connected layer is an update of the graph node by the graph convolution function of the self-attention mechanism ; For each human body in each frame of image in the segment, the STN is responsible for normalizing the skeleton key points of the human body in the previous frame, and transferring the results to the global trajectory represented by the spatio-temporal graph, and jointly predicting the trajectory of the skeleton key points of the human body in the future frame; The regularization layer is used to transfer the skeleton key points, and the multi-layer perceptron is responsible for predicting the trajectory of the skeleton key points, and the transfer and prediction process is uniformly represented as: wherein represent global trajectories of the current human body, represent global trajectories of the current human body, denotes a scaling function, denotes a bias function; wherein , denote the mean and standard deviation of the sample, respectively; the sample being the skeleton joints of the current human body.

6. The method of claim 1, wherein the method further comprises: In step seven, the generation error between the generated value and the label value is calculated, which is specifically: The label value is defined as the real skeleton key point coordinate and the appearance key region at the current moment, and a mean square error (MSE) loss function is selected to calculate the difference between the framework output value and the label value, to obtain an abnormal score of the skeleton key point and an abnormal score of the appearance, respectively and the appearance , and the calculation formula is: wherein and skeleton keypoint scores outputted by the outlier judging module and appearance key region scores outputted by the DFCGAN respectively, and represent the real labels of the skeleton keypoint and the appearance key region; The weighted summation formula is where , , , are the mean and standard deviation of and , and are the weights of and .

Images