Abnormal event detection method and device, equipment and medium

Abstract

The invention discloses an abnormal event detection method and device, equipment and a medium, and relates to the technical field of deep learning, and the method comprises the steps: determining a plurality of skeleton sequences corresponding to each human skeleton track of all pedestrians in a to-be-detected video, and constructing a space-time skeleton diagram based on the plurality of skeleton sequences; determining a global feature component through the feature vector and the length and width of the detection bounding box, and determining a local feature component based on the global feature component; obtaining a reconstruction sequence and a prediction sequence of the local feature component by using the trained space-time diagram convolution auto-encoder network, and determining a target reconstruction anomaly score and a target prediction anomaly score of the video frame at the target moment based on the reconstruction sequence and the prediction sequence; and determining a target abnormal score based on the target reconstruction abnormal score and the target prediction abnormal score, and detecting whether an abnormal event exists in the to-be-detected video according to the target abnormal score. Therefore, the abnormal event in the video can be accurately detected.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to an abnormal event detection method, device, equipment and medium. Background Art

[0002] In recent years, deep learning-based video anomaly detection technology has developed rapidly. However, most advanced semi-supervised anomaly detection methods based on deep learning networks directly use pixel-level underlying surface features (spatiotemporal gradients or pixel intensities) and motion features (optical flow) as event representations to simultaneously detect both surface and motion anomalies. While this approach avoids the time-consuming task of designing effective handcrafted features tailored to specific scene events and improves the method's adaptability, the underlying pixel-level visual features are extracted from every pixel in the video image. Using these as input for anomaly detection inevitably introduces background noise, increasing the computational burden of building a normal event model to distinguish between noise and valid signals. Although some studies have filtered out spatiotemporal feature cubes composed entirely of background and exhibiting minimal motion to minimize computational overhead, this still introduces some redundant background information present in the foreground feature cubes. Furthermore, most methods use pixel features corresponding to the entire video image as input without any preprocessing.

[0003] It can be seen that how to detect abnormal events without adding an additional computational burden of background change modeling is a problem that those skilled in the art need to solve. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method, apparatus, device, and medium for detecting abnormal events, which can detect abnormal events based on human skeleton features without adding the computational burden of background change modeling. The specific scheme is as follows:

[0005] In a first aspect, the present invention provides a method for detecting an abnormal event, comprising:

[0006] Determine a number of skeleton sequences corresponding to the skeleton trajectories of all pedestrians in the video to be detected, and construct a spatiotemporal skeleton graph based on the skeleton sequences;

[0007] The global feature component is determined by the feature vector and the length and width of the detection bounding box, and the local feature component is determined based on the global feature component; the feature vector is a vector determined based on the coordinate vectors of each node in the spatiotemporal skeleton graph; the node is the joint point of the human skeleton; the global feature component is a feature component that represents the position change of the entire human skeleton in the video to be detected at different times; the local feature component is a feature component that represents the position change of the joint point in the video to be detected at different times;

[0008] The trained spatiotemporal graph convolutional autoencoder network is used to obtain the reconstructed sequence and predicted sequence of local feature components, and the target reconstruction anomaly score and target prediction anomaly score of the video frame at the target time are determined based on the reconstructed sequence and the predicted sequence respectively; the trained spatiotemporal graph convolutional autoencoder network is a single encoder-dual decoder architecture network, and the decoder includes a spatiotemporal graph deconvolution network unit that performs temporal deconvolution using 2D deconvolution;

[0009] The target anomaly score is determined based on the target reconstruction anomaly score and the target prediction anomaly score, and whether there is an abnormal event in the video to be detected is detected according to the target anomaly score.

[0010] Optionally, determining a plurality of skeleton sequences corresponding to the skeleton trajectories of all pedestrians in the video to be detected, and constructing a spatiotemporal skeleton graph based on the plurality of skeleton sequences, including:

[0011] Use the preset posture estimation and trajectory tracking algorithm to extract the human skeleton trajectories of all pedestrians in the video to be detected;

[0012] Using a sliding window, each human skeleton trajectory is divided into several skeleton sequences of fixed time length;

[0013] Determine the joint points of the human skeleton in the skeleton sequence as nodes of the spatiotemporal skeleton graph;

[0014] The natural connections between different joints in the spatial dimension that conform to the human body structure are determined as the spatial edges of the spatiotemporal skeleton graph, and the connections between the same joints in consecutive frames in the temporal dimension are determined as the temporal edges of the spatiotemporal skeleton graph;

[0015] A spatiotemporal skeleton graph is constructed based on the nodes, spatial edges, and temporal edges of the spatiotemporal skeleton graph.

[0016] Optionally, determining a global feature component by using the feature vector and the length and width of the detection bounding box to determine a local feature component based on the global feature component includes:

[0017] Determine the center point coordinates of the detection bounding box and the length and width of the detection bounding box according to the feature vector;

[0018] Determine the global feature component based on the center point coordinates, the length and width of the detection bounding box;

[0019] Determine a first difference between the abscissa of each joint point and the abscissa of the center point of the detection bounding box;

[0020] determining the target horizontal coordinate according to a ratio of the first difference value to a width of the detection bounding box;

[0021] determining a second difference between the longitudinal coordinate of each joint point and the longitudinal coordinate of the center point of the detection bounding box;

[0022] determining the target vertical coordinate based on a ratio of the second difference value to the length of the detection bounding box;

[0023] The local feature components are determined by the target abscissa and the target ordinate.

[0024] Optionally, before using the trained spatiotemporal graph convolutional autoencoder network to obtain the reconstructed sequence and predicted sequence of the local feature components, the following steps are also included:

[0025] The encoder is constructed based on a target number of layers of spatiotemporal convolutional network units. In the spatiotemporal convolutional network units, the local feature components of the input are subjected to spatial graph convolution using a nonlinear activation function, and temporal convolution using a 2D convolution operation. After the temporal convolution, batch normalization and activation operations based on a linear rectification function are performed to obtain the corresponding hidden layer features.

[0026] A decoder is constructed based on a target number of layers of spatiotemporal graph deconvolutional network units. In the spatiotemporal graph deconvolutional network units, the hidden layer features are spatially convolved using a nonlinear activation function, and temporally convolved using a 2D deconvolution operation. After the temporal convolution, batch normalization and activation operations based on a linear rectification function are performed to obtain the corresponding initial reconstructed sequence and initial predicted sequence.

[0027] Construct an initial spatiotemporal graph convolutional autoencoder network through the encoder and decoder;

[0028] Construct the loss function by the mean square error produced by the decoder;

[0029] The initial spatiotemporal graph convolutional autoencoder network is trained using the loss function to obtain a trained spatiotemporal graph convolutional autoencoder network.

[0030] Optionally, determining a target reconstruction anomaly score and a target prediction anomaly score of a video frame at a target moment based on the reconstruction sequence and the prediction sequence, respectively, includes:

[0031] Determine the reconstruction error corresponding to the local feature component at the target moment based on the reconstruction sequence;

[0032] Determine the prediction error corresponding to the local feature component at the target moment according to the prediction sequence;

[0033] Determine the initial reconstruction anomaly score at the target moment through the reconstruction error;

[0034] Determine the initial prediction anomaly score at the target moment through the prediction error;

[0035] The target reconstruction anomaly score is determined based on the initial reconstruction anomaly scores of each human skeleton at the target moment using the maximum pooling operation;

[0036] The maximum pooling operation is used to determine the target reconstruction anomaly score based on the initial predicted anomaly scores corresponding to each human skeleton at the target moment.

[0037] Optionally, determining a target anomaly score based on the target reconstruction anomaly score and the target prediction anomaly score includes:

[0038] The sum between the target reconstructed anomaly score and the target predicted anomaly score is determined as the target anomaly score.

[0039] Optionally, detecting whether there are any abnormal events in the video to be detected based on the target anomaly score includes:

[0040] Determine the relationship between the target anomaly score and the preset threshold score;

[0041] If the target anomaly score is greater than the preset threshold score, it is determined that there is an abnormal event in the video to be detected;

[0042] If the target anomaly score is less than or equal to the preset threshold score, it is determined that there is no abnormal event in the video to be detected.

[0043] In a second aspect, the present invention provides an abnormal event detection device, comprising:

[0044] A spatiotemporal skeleton graph construction module is used to determine a number of skeleton sequences corresponding to the skeleton trajectories of all pedestrians in the video to be detected, and to construct a spatiotemporal skeleton graph based on the skeleton sequences;

[0045] A local feature component determination module is configured to determine a global feature component using a feature vector and the length and width of a detection bounding box, thereby determining a local feature component based on the global feature component; the feature vector is a vector determined based on the coordinate vectors of each node in the spatiotemporal skeleton graph; the nodes are joints of the human skeleton; the global feature component is a feature component that characterizes the positional changes of the entire human skeleton in the video to be detected at different times; and the local feature component is a feature component that characterizes the positional changes of the joints in the video to be detected at different times.

[0046] An anomaly score determination module is used to obtain a reconstructed sequence and a predicted sequence of local feature components using a trained spatiotemporal graph convolutional autoencoder network, and determine a target reconstructed anomaly score and a target predicted anomaly score for a video frame at a target moment based on the reconstructed sequence and the predicted sequence, respectively. The trained spatiotemporal graph convolutional autoencoder network is a single encoder-dual decoder architecture network, and the decoder includes a spatiotemporal graph deconvolution network unit that performs temporal deconvolution using 2D deconvolution.

[0047] The abnormal event detection module is used to determine the target abnormality score based on the target reconstruction abnormality score and the target prediction abnormality score, and detect whether there is an abnormal event in the video to be detected according to the target abnormality score.

[0048] In a third aspect, the present invention provides an electronic device, comprising:

[0049] memory for storing computer programs;

[0050] A processor is used to execute a computer program to implement the aforementioned abnormal event detection method.

[0051] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the aforementioned abnormal event detection method when executed by a processor.

[0052] In the present invention, several skeleton sequences corresponding to the human skeleton trajectories of all pedestrians in the video to be detected are determined, and a spatiotemporal skeleton graph is constructed based on the several skeleton sequences; a global feature component is determined by a feature vector and the length and width of a detection bounding box, so as to determine a local feature component based on the global feature component; a feature vector is a vector determined based on the coordinate vector of each node in the spatiotemporal skeleton graph; a node is a joint point of the human skeleton; a global feature component is a feature component that characterizes the position change of the entire human skeleton in the video to be detected at different moments; a local feature component is a feature component that characterizes the position change of the joint point in the video to be detected at different moments; a trained spatiotemporal graph convolutional autoencoder network is used to obtain a reconstructed sequence and a predicted sequence of the local feature components, and a target reconstruction anomaly score and a target prediction anomaly score of a video frame at a target moment are determined based on the reconstructed sequence and the predicted sequence respectively; the trained spatiotemporal graph convolutional autoencoder network is a network with a single encoder-dual decoder architecture, and the decoder includes a spatiotemporal graph deconvolution network unit that performs temporal deconvolution using 2D deconvolution; a target anomaly score is determined based on the target reconstruction anomaly score and the target prediction anomaly score, and whether there is an abnormal event in the video to be detected is detected according to the target anomaly score.

[0053] Beneficial effects: This application characterizes the extracted human skeleton features through a spatiotemporal graph and decomposes them into global motion components and local action components. The spatiotemporal graph convolutional autoencoder network then models the normal pattern of human motion based on the local components. At the same time, the spatiotemporal graph convolutional autoencoder network can simultaneously reconstruct the input and predict the corresponding future, and fuse the reconstruction and prediction errors as an anomaly score to detect abnormal events. In this way, the present invention can only focus on the human behavior movement in the video foreground, without including background redundant information, so it does not add the computational burden of modeling background changes, and can accurately and quickly detect abnormal events in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0055] Figure 1 A flow chart of an abnormal event detection method provided by an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of a human body space-time skeleton diagram provided by an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of a decomposition of a human body spatial skeleton diagram provided by an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of an encoder provided by an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of a decoder provided by an embodiment of the present invention;

[0060] Figure 6 A schematic structural diagram of an abnormal event detection device provided by an embodiment of the present invention;

[0061] Figure 7 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] The terms "including" and "having," as used in the present description and accompanying drawings, and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements and may include steps or elements that are not listed.

[0064] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0065] Most advanced semi-supervised abnormal event detection methods based on deep learning networks directly use pixel-level underlying surface features (spatiotemporal gradients or pixel intensity) and motion features (optical flow) as event representations to simultaneously detect both surface and motion abnormal events. While this approach avoids the time-consuming task of designing effective manual features for specific scene events and improves the method's adaptability to specific scenarios, the underlying pixel-level visual features are extracted from each pixel in the video image. Using these as input for anomaly detection inevitably introduces video background noise information, thereby increasing the computational burden of establishing a normal event model to distinguish between noise and valid signals. Although some studies have filtered out spatiotemporal feature cubes composed entirely of background and with minimal motion response to minimize computational overhead, this still introduces some redundant background information present in the foreground feature cubes. To address the aforementioned technical problems, the present invention discloses an abnormal event detection method, apparatus, device, and medium that can detect abnormal events based on human skeleton features without the additional computational burden of modeling background changes.

[0066] See also Figure 1 As shown, an embodiment of the present invention provides a method for detecting abnormal events, including:

[0067] Step S11: determining a plurality of skeleton sequences corresponding to the human skeleton trajectories of all pedestrians in the video to be detected, and constructing a spatiotemporal skeleton graph based on the plurality of skeleton sequences.

[0068] In the embodiment of the present invention, considering that people are the target subjects in key video surveillance areas such as open-air squares, commercial centers, airports and railway stations, and the vast majority of abnormal events are caused by human activities, the present invention describes human behavior based on skeleton features and further characterizes video events, and on this basis, conducts abnormal event detection related to people. Compared with pixel-level features, human skeleton features are more compact and have a stronger structure, and only focus on the movement of human behavior in the foreground of the video, and do not contain redundant background information. At the same time, skeleton features are a kind of high-level semantic features. In crowded scenes, traditional high-level semantic features such as human movement speed or trajectory will not be accurately extracted due to occlusion, while skeleton features can be accurately extracted through human pose estimation algorithms that are robust to occlusion, such as AlphaPose and OpenPose.

[0069] In an embodiment of the present invention, the skeleton trajectory of each pedestrian in the video is first extracted using a posture estimation and trajectory tracking algorithm to describe its behavior, and a sliding window strategy is used to divide each trajectory into a skeleton sequence of fixed size. Then a spatiotemporal skeleton graph is constructed to represent the skeleton sequence. Specifically, the human skeleton trajectories of all pedestrians in the video to be detected are extracted using a preset posture estimation and trajectory tracking algorithm; each human skeleton trajectory is divided into a number of skeleton sequences of fixed time length using a sliding window; the joints of the human skeleton in the skeleton sequence are determined as nodes of the spatiotemporal skeleton graph; the natural connections between different joints in the spatial dimension that conform to the human body structure are determined as spatial edges of the spatiotemporal skeleton graph, and the connections between the same joints in consecutive frames in the temporal dimension are determined as temporal edges of the spatiotemporal skeleton graph; a spatiotemporal skeleton graph is constructed based on the nodes, spatial edges and temporal edges of the spatiotemporal skeleton graph. When the spatiotemporal graph is used to represent the human skeleton features, the skeleton joints are used as nodes of the graph, the natural connections between different joints in the spatial dimension that conform to the human body structure are used as spatial edges of the graph, and the connections between the same joints in consecutive frames in the temporal dimension are used as temporal edges of the graph, such as Figure 2 As shown. The spatiotemporal skeleton graph constructed by the skeleton sequence of T frames containing V joint points is represented as , where Z represents the node set of the graph and E represents the edge set of the graph. Contains all the relevant nodes in the skeleton, and the edge set E contains spatial edges and temporal edges, where k is the number of joint points. The feature vector set of the spatial skeleton graph node set at time t is composed of the coordinate vector of each node, expressed as ; Where C() represents the coordinates of each joint point; is the set of real numbers.

[0070] It should be noted that in the process of extracting the skeleton trajectory of each pedestrian in the video using the pose estimation and trajectory tracking algorithm, the present invention can first use a model such as YOLOv8 (You Only Look Once version 8, a target detection algorithm) to locate the pedestrian bounding box in the video frame, assign a unique ID (Identity) to each pedestrian, and then use DeepSORT (Deep Simple Online and Realtime Tracking, a multi-target tracking algorithm based on deep learning) or ByteTrack algorithm to associate the pedestrian IDs in consecutive frames to generate a spatiotemporal trajectory. Use OpenPose (an open source real-time multi-person pose estimation library) or HRNet (High-Resolution Network, a deep neural network architecture for visual tasks (such as pose estimation, semantic segmentation, target detection, etc.)) to extract pedestrian skeleton key points (such as 18 or 25 joint points). The ZS (Zhang-Suen) refinement algorithm or single-pixel processing is then used to eliminate skeleton noise and improve trajectory coherence. Missing keypoints are repaired using interpolation or time series prediction (such as Kalman filtering). Finally, the skeleton data is organized into a C×T×V format: C (channel): joint coordinates (x, y), velocity, acceleration, and other features. T (time step): sequence of consecutive video frames. V (node): number of skeleton joints (e.g., 18). The present invention can also use multi-view fusion or occlusion-robust pose estimation models to handle occlusion. Dynamic background interference can be addressed by combining background subtraction with camera motion compensation (Ego-motion) to separate pedestrian motion. This allows for precise skeleton feature extraction.

[0071] Step S12, determine the global feature component through the feature vector and the length and width of the detection bounding box, and determine the local feature component based on the global feature component; the feature vector is a vector determined based on the coordinate vector of each node in the spatiotemporal skeleton diagram; the node is the joint point of the human skeleton; the global feature component is the feature component that characterizes the position change of the entire human skeleton in the video to be detected at different times; the local feature component is the feature component that characterizes the position change of the joint point in the video to be detected at different times.

[0072] In the embodiment of the present invention, generally speaking, human behavior includes two parts: global rigid motion and local non-rigid changes in posture. Therefore, for the spatial skeleton image feature vector set at time t , using a decomposition model to decompose it into global feature components and local feature components: the global component is defined as the two-dimensional absolute coordinates of the center point of the skeleton detection bounding box in the video image. This coordinate reflects the position change of the entire skeleton in the image at different times, and thus can provide the rigid motion information of the entire skeleton; while the local component is defined as the two-dimensional relative coordinates of each joint point (graph node) in the original skeleton feature when considering the center point of the bounding box. This coordinate reflects the position change of the joint point in the image at different times (excluding the change caused by the overall displacement of the skeleton), and thus can provide the non-rigid deformation information inside the skeleton. The decomposition process is expressed as follows, Figure 3 As shown:

[0073] ;

[0074] in, Represents the global feature component; Represents the local feature component; is the horizontal coordinate of the node of the spatial skeleton graph at time t; is the vertical coordinate of the node of the spatial skeleton graph at time t.

[0075] However, in a two-dimensional image, due to the lack of depth, using only xy coordinates cannot provide the true position of the target. Considering that the size of the detection bounding box contains the depth information of the target position to some extent (the smaller the bounding box size, the farther the target depth position), the present invention introduces the length of the bounding box into the two-dimensional coordinates of the center point of the skeleton bounding box. and width Composed of global features, expressed as , and is calculated based on the coordinates of all relevant nodes using the following formula (1), while the local features are calculated by removing the global component from the original joint point coordinates in the following formula (2):

[0076] ; (1)

[0077] ; (2)

[0078] That is, the present invention determines the center point coordinates of the detection bounding box and the length and width of the detection bounding box based on the feature vector; determines the global feature component based on the center point coordinates, the length and width of the detection bounding box; determines the first difference between the horizontal coordinate of each joint point and the horizontal coordinate of the center point of the detection bounding box; determines the target horizontal coordinate based on the ratio of the first difference to the width of the detection bounding box; determines the second difference between the vertical coordinate of each joint point and the vertical coordinate of the center point of the detection bounding box; determines the target vertical coordinate based on the ratio of the second difference to the length of the detection bounding box; and determines the local feature component through the target horizontal coordinate and the target vertical coordinate.

[0079] Step S13: Use the trained spatiotemporal graph convolutional autoencoder network to obtain the reconstructed sequence and predicted sequence of the local feature components, and determine the target reconstruction anomaly score and the target prediction anomaly score of the video frame at the target moment based on the reconstructed sequence and the predicted sequence, respectively; the trained spatiotemporal graph convolutional autoencoder network is a network with a single encoder-dual decoder architecture, and the decoder includes a spatiotemporal graph deconvolution network unit that uses 2D deconvolution to perform temporal deconvolution.

[0080] In an embodiment of the present invention, an encoder is first constructed based on a spatiotemporal convolutional network unit of a target number of layers; wherein, in the spatiotemporal convolutional network unit, spatial graph convolution is performed on the local feature components of the input using a nonlinear activation function, and temporal convolution is performed using a 2D convolution operation, and batch normalization and activation operations based on a linear rectification function are performed after the temporal convolution to obtain corresponding hidden layer features; a decoder is constructed based on a spatiotemporal graph deconvolutional network unit of a target number of layers; wherein, in the spatiotemporal graph deconvolutional network unit, spatial graph convolution is performed on the hidden layer features using a nonlinear activation function, and temporal convolution is performed using a 2D deconvolution operation, and batch normalization and activation operations based on a linear rectification function are performed after the temporal convolution to obtain corresponding initial reconstruction sequences and initial prediction sequences; an initial spatiotemporal graph convolutional autoencoder network is constructed through the encoder and the decoder; a loss function is constructed through the mean square error generated by the decoder; and the initial spatiotemporal graph convolutional autoencoder network is trained using the loss function to obtain a trained spatiotemporal graph convolutional autoencoder network. Specifically, the present invention constructs a prediction branch in the decoder part of ST-GCAE (Spatial-temporal Graph Convolutional Autoencoder) to predict future human skeleton sequences. Therefore, unlike the traditional autoencoder architecture, ST-GCAE adopts a single encoder-dual decoder architecture. Among them, the single encoder is used to map the local components of the input skeleton sequence to hidden layer features. The dual decoder includes a reconstruction decoder and a prediction decoder, which share the same hidden layer features but perform different tasks: reconstruct the input past moment sequence and predict the corresponding future moment sequence. More specifically, the encoder of ST-GCAE consists of 9 layers of spatial-temporal graph convolutional network units (Spatial-temporal GraphConvolutional Network, ST-GCN), such as Figure 4 As shown in Figure 2. In each ST-GCN unit, the implementation of spatial graph convolution is not as simple as 2D or 3D convolution. It is implemented using graph convolution, and the specific formula is as follows:

[0081] ;

[0082] in, express Neighborhood, To return one item, Represents the training weight, changing the feature dimension from C to ; is a non-linear activation function.

[0083] Therefore, the spatial graph convolution takes an input tensor of shape C×T×V Convert to shape ×T×V output tensor. For the following time convolution, the traditional 2D convolution operation is used on the input time channel to learn the temporal dynamics, resulting in a shape of Tensor. Unlike traditional ST-GCN units, this paper abandons the ResNet mechanism to avoid introducing noise from lower layers into subsequent network layers. After the temporal convolution, batch normalization and ReLU (Rectified Linear Unit) activation operations are performed.

[0084] Corresponding to the encoder, the decoder of ST-GCAE consists of 9 layers of spatial-temporal graph deconvolutional network (ST-GDCN), such as Figure 5 The difference between the ST-GCN unit and the ST-GDCN unit is that the temporal convolution layer in the former is replaced by a temporal deconvolution layer. More specifically, the temporal deconvolution layer is implemented through a 2D deconvolution operation.

[0085] In the embodiments of the present invention, the basic architecture of the ST-GCAE is an autoencoder network. In the ST-GCAE, the encoder maps the input sequence of local components into hidden layer features, while the dual decoder attempts to simultaneously reconstruct the input from the hidden layer features and predict the corresponding future sequence. To achieve this goal, the present invention incorporates the mean squared error generated by the reconstruction and prediction decoders into the loss function used to train the network. The loss function is defined as follows:

[0086] ;

[0087] Where L is the loss function; N represents the input batch size; and They represent the local component sequence of the j-th input and its reconstruction sequence respectively. and Respectively The corresponding predicted sequence of future moments and its true value, Represents the starting time of the predicted local sequence, T and P represent the time series length of the reconstructed and predicted sequences respectively. The last term is the regularization term, which is used to avoid overfitting, where is the regularization term coefficient; W represents all the weight parameters of the autoencoder network.

[0088] Therefore, the present invention uses the trained ST-GCAE to obtain the local component sequence corresponding to the test skeleton sequence A refactored version of and the predicted version . Then the present application can determine the target reconstruction anomaly score and target prediction anomaly score of the video frame at the target moment based on the reconstruction sequence and the prediction sequence respectively. That is, determine the reconstruction error corresponding to the local feature component at the target moment based on the reconstruction sequence; determine the prediction error corresponding to the local feature component at the target moment based on the prediction sequence; determine the initial reconstruction anomaly score at the target moment through the reconstruction error; determine the initial prediction anomaly score at the target moment through the prediction error; determine the target reconstruction anomaly score based on the initial reconstruction anomaly score corresponding to each human skeleton at the target moment using the maximum pooling operation; determine the target reconstruction anomaly score based on the initial prediction anomaly score corresponding to each human skeleton at the target moment using the maximum pooling operation.

[0089] Specifically, for Skeleton features at time t , calculate its reconstruction error and prediction error :

[0090] ;

[0091] By calculating all the The reconstruction and prediction errors in the skeleton sequence are obtained The reconstruction anomaly score of and predicted anomaly score :

[0092] ;

[0093] in, and Respectively indicate that they contain instances The reconstructed sequence set and the predicted sequence set, and Respectively and The reconstructed and predicted sequences in and Represents the sequence and middle The reconstruction error and prediction error of and The number of sequences in the .

[0094] After calculating the reconstruction and predicted anomaly scores of each skeleton instance in the video frame at time t, the Max Pooling method is used to obtain the target reconstruction anomaly score of the video frame at time t. and target prediction anomaly score :

[0095] ;

[0096] in, Represents the set of skeleton instances in the video frame at time t.

[0097] Step S14: determining a target anomaly score based on the target reconstruction anomaly score and the target prediction anomaly score, and detecting whether there is an abnormal event in the video to be detected according to the target anomaly score.

[0098] In the embodiment of the present invention, the sum of the target reconstruction anomaly score and the target prediction anomaly score is determined as the target anomaly score. That is, the reconstruction and prediction anomaly scores of the video frame at time t are fused to obtain the final target anomaly score. :

[0099] ;

[0100] The target anomaly score can then be used to detect whether an abnormal event exists in the video being tested. Specifically, the relationship between the target anomaly score and a preset threshold score is determined. If the target anomaly score is greater than the preset threshold score, the video being tested is deemed to contain an abnormal event. If the target anomaly score is less than or equal to the preset threshold score, the video being tested is deemed to contain no abnormal event. In this way, based on the reconstruction error obtained by calculating the difference between the local component sequence of the skeleton features being tested and its reconstructed output, the corresponding predicted version obtained based on the input of the corresponding historical moment sequence is compared with the sequence being tested to obtain the prediction error. The two are then combined to form the final anomaly score, achieving complementary advantages.

[0101] Beneficial effects: This application characterizes the extracted human skeleton features through a spatiotemporal graph and decomposes them into global motion components and local action components. The spatiotemporal graph convolutional autoencoder network then models the normal pattern of human motion based on the local components. At the same time, the spatiotemporal graph convolutional autoencoder network can simultaneously reconstruct the input and predict the corresponding future, and fuse the reconstruction and prediction errors as an anomaly score to detect abnormal events. In this way, the present invention can only focus on the human behavior movement in the video foreground, without including background redundant information, so it does not add the computational burden of modeling background changes, and can accurately and quickly detect abnormal events in the video.

[0102] See also Figure 6 As shown, an embodiment of the present invention provides an abnormal event detection device, including:

[0103] A spatiotemporal skeleton graph construction module 11 is used to determine a plurality of skeleton sequences corresponding to the skeleton trajectories of all pedestrians in the video to be detected, and to construct a spatiotemporal skeleton graph based on the plurality of skeleton sequences;

[0104] The local feature component determination module 12 is configured to determine the global feature component using the feature vector and the length and width of the detection bounding box, so as to determine the local feature component based on the global feature component; the feature vector is a vector determined based on the coordinate vectors of each node in the spatiotemporal skeleton graph; the node is a joint point of the human skeleton; the global feature component is a feature component that represents the position change of the entire human skeleton in the video to be detected at different times; the local feature component is a feature component that represents the position change of the joint point in the video to be detected at different times;

[0105] Anomaly score determination module 13 is used to obtain a reconstructed sequence and a predicted sequence of local feature components using a trained spatiotemporal graph convolutional autoencoder network, and determine a target reconstructed anomaly score and a target predicted anomaly score of a video frame at a target moment based on the reconstructed sequence and the predicted sequence, respectively; the trained spatiotemporal graph convolutional autoencoder network is a network with a single encoder-dual decoder architecture, and the decoder includes a spatiotemporal graph deconvolution network unit that performs temporal deconvolution using 2D deconvolution;

[0106] The abnormal event detection module 14 is configured to determine a target abnormality score based on the target reconstruction abnormality score and the target prediction abnormality score, and detect whether an abnormal event exists in the video to be detected according to the target abnormality score.

[0107] Since the embodiments of the device part correspond to the above embodiments, the embodiments of the device part please refer to the description of the embodiments of the method part, which will not be repeated here.

[0108] Beneficial effects: This application characterizes the extracted human skeleton features through a spatiotemporal graph and decomposes them into global motion components and local action components. The spatiotemporal graph convolutional autoencoder network then models the normal pattern of human motion based on the local components. At the same time, the spatiotemporal graph convolutional autoencoder network can simultaneously reconstruct the input and predict the corresponding future, and fuse the reconstruction and prediction errors as an anomaly score to detect abnormal events. In this way, the present invention can only focus on the human behavior movement in the video foreground, without including background redundant information, so it does not add the computational burden of modeling background changes, and can accurately and quickly detect abnormal events in the video.

[0109] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 7This is a structural diagram of an electronic device according to an exemplary embodiment. The content in the diagram should not be considered as any limitation on the scope of use of this application. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the abnormal event detection method disclosed in any of the aforementioned embodiments. In addition, the electronic device in this embodiment may specifically be an electronic computer.

[0110] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0111] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0112] The operating system 221 is used to manage and control the hardware devices on the electronic device and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the abnormal event detection method performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program capable of implementing other specific tasks.

[0113] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned abnormal event detection method is implemented. The specific steps of this method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0114] Furthermore, the present application also discloses a computer program product, including a computer program / instructions; wherein, when executed by a processor, the computer program / instructions implement the aforementioned abnormal event detection method. The specific steps of this method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0116] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0118] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0119] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting abnormal events, characterized in that: include: Determine a plurality of skeleton sequences corresponding to the human skeleton trajectories of all pedestrians in the video to be detected, and construct a spatiotemporal skeleton graph based on the plurality of skeleton sequences; A global feature component is determined by a feature vector and the length and width of a detection bounding box, and a local feature component is determined based on the global feature component; the feature vector is a vector determined based on the coordinate vectors of each node in the spatiotemporal skeleton graph; the nodes are joints of the human skeleton; the global feature component is a feature component that characterizes the position change of the entire human skeleton in the video to be detected at different times; the local feature component is a feature component that characterizes the position change of the joints in the video to be detected at different times; A trained spatiotemporal graph convolutional autoencoder network is used to obtain a reconstructed sequence and a predicted sequence of the local feature components, and a target reconstructed anomaly score and a target predicted anomaly score of a video frame at a target moment are determined based on the reconstructed sequence and the predicted sequence, respectively. The trained spatiotemporal graph convolutional autoencoder network is a network with a single encoder-dual decoder architecture, and the decoder includes a spatiotemporal graph deconvolution network unit that performs temporal deconvolution using 2D deconvolution. A target anomaly score is determined based on the target reconstruction anomaly score and the target prediction anomaly score, and whether an abnormal event exists in the video to be detected is detected according to the target anomaly score.

2. The abnormal event detection method according to claim 1, characterized in that: The step of determining a plurality of skeleton sequences corresponding to the human skeleton trajectories of all pedestrians in the video to be detected, and constructing a spatiotemporal skeleton graph based on the plurality of skeleton sequences, includes: Use the preset posture estimation and trajectory tracking algorithm to extract the human skeleton trajectories of all pedestrians in the video to be detected; Using a sliding window to divide each of the human skeleton trajectories into a plurality of skeleton sequences of fixed time lengths; Determining the joint points of the human skeleton in the skeleton sequence as nodes of the spatiotemporal skeleton graph; Determine the natural connections between different joints in the spatial dimension that conform to the human body structure as the spatial edges of the spatiotemporal skeleton graph, and determine the connections between the same joint in consecutive frames in the temporal dimension as the temporal edges of the spatiotemporal skeleton graph; The spatiotemporal skeleton graph is constructed based on the nodes, the spatial edges and the temporal edges of the spatiotemporal skeleton graph.

3. The abnormal event detection method according to claim 1, characterized in that: The determining of the global feature component by using the feature vector and the length and width of the detection bounding box, and determining the local feature component based on the global feature component, includes: Determine the center point coordinates of the detection bounding box and the length and width of the detection bounding box according to the feature vector; Determine the global feature component based on the center point coordinates, the length and width of the detection bounding box; Determine a first difference between the abscissa of each joint point and the abscissa of the center point of the detection bounding box; determining a target horizontal coordinate according to a ratio of the first difference value to a width of the detection bounding box; determining a second difference between the longitudinal coordinate of each joint point and the longitudinal coordinate of the center point of the detection bounding box; determining the target vertical coordinate according to a ratio of the second difference value to the length of the detection bounding box; The local characteristic component is determined by the target abscissa and the target ordinate.

4. The abnormal event detection method according to claim 1, characterized in that: Before using the trained spatiotemporal graph convolutional autoencoder network to obtain the reconstructed sequence and predicted sequence of the local feature components, the method further includes: Constructing an encoder based on a target number of layers of spatiotemporal convolutional network units; wherein the spatiotemporal convolutional network units perform spatial graph convolution on the input local feature components using a nonlinear activation function, perform temporal convolution using a 2D convolution operation, and perform batch normalization and activation operations based on a linear rectification function after the temporal convolution to obtain corresponding hidden layer features; Constructing a decoder based on the spatiotemporal graph deconvolution network unit of the target number of layers; wherein, in the spatiotemporal graph deconvolution network unit, the hidden layer features are subjected to spatial graph convolution using a nonlinear activation function, temporal convolution using a 2D deconvolution operation, and batch normalization and activation operations based on a linear rectification function are performed after the temporal convolution to obtain corresponding initial reconstructed sequences and initial predicted sequences; Constructing an initial spatiotemporal graph convolutional autoencoder network through the encoder and the decoder; constructing a loss function using the mean square error generated by the decoder; The initial spatiotemporal graph convolutional autoencoder network is trained using the loss function to obtain a trained spatiotemporal graph convolutional autoencoder network.

5. The abnormal event detection method according to claim 1, characterized in that: The determining a target reconstruction anomaly score and a target prediction anomaly score of a video frame at a target moment based on the reconstruction sequence and the prediction sequence respectively includes: Determining a reconstruction error corresponding to the local feature component at the target time based on the reconstruction sequence; Determining a prediction error corresponding to the local feature component at the target moment according to the prediction sequence; Determining an initial reconstruction anomaly score at the target moment using the reconstruction error; Determining an initial prediction anomaly score at the target moment using the prediction error; Determine the target reconstruction anomaly score based on the initial reconstruction anomaly scores corresponding to each human skeleton at the target moment using a maximum pooling operation; The target reconstruction anomaly score is determined based on the initial predicted anomaly scores corresponding to each human skeleton at the target moment using a maximum pooling operation.

6. The abnormal event detection method according to claim 1, characterized in that: The determining a target anomaly score based on the target reconstruction anomaly score and the target prediction anomaly score includes: The sum of the target reconstructed anomaly score and the target predicted anomaly score is determined as the target anomaly score.

7. The abnormal event detection method according to claim 1, characterized in that: The detecting whether there is an abnormal event in the video to be detected according to the target abnormality score includes: Determine the magnitude relationship between the target anomaly score and a preset threshold score; If the target anomaly score is greater than the preset threshold score, it is determined that the abnormal event exists in the video to be detected; If the target anomaly score is less than or equal to the preset threshold score, it is determined that the abnormal event does not exist in the video to be detected.

8. An abnormal event detection device, characterized in that: include: A spatiotemporal skeleton graph construction module is used to determine a plurality of skeleton sequences corresponding to the skeleton trajectories of all pedestrians in the video to be detected, and to construct a spatiotemporal skeleton graph based on the plurality of skeleton sequences; A local feature component determination module is configured to determine a global feature component using a feature vector and the length and width of a detection bounding box, so as to determine a local feature component based on the global feature component; the feature vector is a vector determined based on the coordinate vectors of each node in the spatiotemporal skeleton graph; the nodes are joints of the human skeleton; the global feature component is a feature component that characterizes positional changes of the entire human skeleton in the video to be detected at different moments; and the local feature component is a feature component that characterizes positional changes of the joints in the video to be detected at different moments. An anomaly score determination module is used to obtain a reconstructed sequence and a predicted sequence of the local feature components using a trained spatiotemporal graph convolutional autoencoder network, and determine a target reconstructed anomaly score and a target predicted anomaly score of the video frame at the target moment based on the reconstructed sequence and the predicted sequence, respectively; the trained spatiotemporal graph convolutional autoencoder network is a network with a single encoder-dual decoder architecture, and the decoder includes a spatiotemporal graph deconvolution network unit that performs temporal deconvolution using 2D deconvolution; The abnormal event detection module is used to determine a target abnormality score based on the target reconstruction abnormality score and the target prediction abnormality score, and detect whether there is an abnormal event in the video to be detected according to the target abnormality score.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the abnormal event detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the abnormal event detection method according to any one of claims 1 to 7.

Images