A video anomaly online detection method and device based on a skeletal trajectory diffusion model

Abstract

The application discloses a video anomaly online detection method and device based on a skeleton trajectory diffusion model, and comprises the following steps: acquiring a continuous video frame sequence containing a current video frame of a video to be detected; extracting a real skeleton posture sequence from the continuous video frame sequence by using a skeleton posture network; tracking skeleton information in different video frames by using an optical flow neural network to obtain a real spatiotemporal skeleton sequence of a target; extracting global features and local features from the real spatiotemporal skeleton sequence, and encoding the global features and the local features based on a spatiotemporal autoencoder to obtain first conditional features and second conditional features; generating a predicted spatiotemporal skeleton frame at a current moment by inverse diffusion based on random noise, the first conditional features and the second conditional features by using a skeleton trajectory diffusion model; and determining whether the current video frame is an abnormal video frame by comparing the real spatiotemporal skeleton frame at the current moment with the predicted spatiotemporal skeleton frame, so that fast and high-precision detection can be realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of video image signal processing and machine vision, and particularly relates to a video anomaly online detection method and device based on a skeleton trajectory diffusion model. BACKGROUND

[0002] Video anomaly detection technology is of great significance to promote the automation and intelligence of the video monitoring field. The current mainstream video anomaly detection technology can be divided into vision-based reconstruction and skeleton-based reconstruction. The model using the visual codec structure reconstructs the video frame. The basic assumption of this method is that normal video frames can be reconstructed, and abnormal video frames cannot be reconstructed. However, this reconstruction-based method can also better reconstruct abnormal video frames. In addition, using all visual information contains a large amount of redundant information, which is not conducive to improving the detection accuracy and detection speed.

[0003] Patent application No. CN107222659A discloses a video anomaly detection method. First, according to the characteristics of video jitter local consistency, combined with the block idea, the global motion estimation is completed by using the gray projection algorithm. Then, according to the longest path algorithm, the frame sequence that can represent the video jitter is selected as the effective frame. The jitter parameters such as jitter rate, jitter frequency and jitter amplitude of the video jitter are calculated according to the global motion parameters of the effective frame, and the influence factor of the jitter parameters is set. Finally, the jitter degree of the video is calculated by the weighted average method.

[0004] With the development of deep learning, people often use deep learning to detect target video anomalies. For example, patent application No. CN113435432A discloses a video anomaly detection model training method, which includes: obtaining a training video frame sequence, corresponding sequence feature information and target feature information of a training video frame sequence corresponding to a training screening video frame; inputting the training video frame sequence into an initial video anomaly detection model to obtain the predicted feature information corresponding to the training screening video frame; generating a time sequence positive training sample of a time sequence discriminant model corresponding to the initial video anomaly detection model based on the target feature information and the sequence feature information, and generating a time sequence negative training sample based on the sequence feature information and the predicted feature information; based on the target feature information, the predicted feature information, the time sequence positive training sample and the time sequence negative training sample, the initial video anomaly detection model and the time sequence discriminant model are trained in an adversarial manner until the convergence condition is met, and the target video anomaly detection model is obtained.

[0005] Although the above two technical solutions can detect video anomalies, the detection speed needs to be improved. SUMMARY

[0006] In view of the above, the purpose of the present application is to provide a video anomaly online detection method and device based on a skeletal trajectory diffusion model to improve the accuracy and speed of video anomaly detection.

[0007] To achieve the above-mentioned purpose of the present application, the embodiment of the present application provides a video anomaly online detection method based on a skeletal trajectory diffusion model, comprising the following steps:

[0008] Obtaining a continuous video frame sequence containing a current video frame of a video to be detected;

[0009] Extracting a real skeletal pose sequence from the continuous video frame sequence by using a skeletal pose network, and tracking skeletal information in different video frames by using an optical flow neural network to obtain a real spatio-temporal skeletal sequence of a target;

[0010] Extracting global features and local features from the real spatio-temporal skeletal sequence, and encoding the global features and the local features based on a spatio-temporal autoencoder to obtain first conditional features and second conditional features;

[0011] Generating a predicted spatio-temporal skeletal frame at a current time by inverse diffusion based on random noise, the first conditional features and the second conditional features by using a skeletal trajectory diffusion model;

[0012] Determining whether the current video frame is an abnormal video frame by comparing the real spatio-temporal skeletal frame at the current time with the predicted spatio-temporal skeletal frame.

[0013] Preferably, the global features and the local features extracted from the real spatio-temporal skeletal sequence comprise:

[0014] Extracting a motion trend of a human skeleton in the entire video frame from the real spatio-temporal skeletal sequence as the global features;

[0015] Extracting a motion change of a human pose between video frames from the real spatio-temporal skeletal sequence as the local features.

[0016] Preferably, the encoder and the decoder in the spatio-temporal autoencoder adopt a Unet model, the Unet model is used to learn the global features in a hidden space to obtain the first conditional features, and the Unet model is used to learn the local features in the hidden space to obtain the second conditional features.

[0017] Preferably, the skeletal trajectory diffusion model is used to generate the predicted spatio-temporal skeletal sequence by inverse diffusion based on the random noise, the first conditional features and the second conditional features, comprising:

[0018] Taking the random noise as initial noise, and taking a splicing result of the first conditional features and the second conditional features as a guiding body condition of each time step, performing multi-step denoising to generate the predicted spatio-temporal skeletal sequence.

[0019] Preferably, the predicted spatio-temporal skeleton sequence is generated by inverse diffusion based on the random noise, the first conditional feature and the second conditional feature using the skeletal trajectory diffusion model, comprising:

[0020] a multi-step denoising is performed with the random noise as the initial noise and the first conditional feature as the guide body condition of each time step to generate a first predicted candidate spatio-temporal skeleton sequence;

[0021] a multi-step denoising is performed with the random noise as the initial noise and the second conditional feature as the guide body condition of each time step to generate a second predicted candidate spatio-temporal skeleton sequence;

[0022] the first predicted candidate spatio-temporal skeleton sequence and the second predicted candidate spatio-temporal skeleton sequence are merged to obtain the predicted spatio-temporal skeleton sequence.

[0023] Preferably, whether the current video frame is an abnormal video frame is determined by comparing the real spatio-temporal skeleton frame at the current time and the predicted spatio-temporal skeleton frame, comprising:

[0024] an abnormal score of the real spatio-temporal skeleton frame at the current time and the predicted spatio-temporal skeleton frame at the current time is calculated, and when the abnormal score is higher than a preset threshold, the current video frame is determined as an abnormal video frame.

[0025] Preferably, the abnormal score of the real spatio-temporal skeleton frame and the predicted spatio-temporal skeleton frame is calculated by L1 norm.

[0026] To achieve the above-mentioned purposes, the embodiment further provides a video anomaly online detection device based on a skeletal trajectory diffusion model, comprising:

[0027] a data acquisition module for acquiring a continuous video frame sequence containing a current video frame of a video to be detected;

[0028] a skeleton extraction module for extracting a real skeleton pose sequence from the continuous video frame sequence using a skeleton pose network, and tracking skeleton information in different video frames using an optical flow neural network to obtain a real spatio-temporal skeleton sequence of a target;

[0029] a feature encoding module for extracting global features and local features from the real spatio-temporal skeleton sequence, and encoding the global features and the local features based on a spatio-temporal auto-encoder to obtain a first conditional feature and a second conditional feature;

[0030] a predicted spatio-temporal skeleton frame module for generating a predicted spatio-temporal skeleton frame at the current time based on the random noise, the first conditional feature and the second conditional feature by inverse diffusion using the skeletal trajectory diffusion model;

[0031] an abnormality confirmation module for determining whether the current video frame is an abnormal video frame by comparing the real spatio-temporal skeleton frame at the current time and the predicted spatio-temporal skeleton frame.

[0032] To achieve the above-mentioned object of the application, the embodiment further provides a computing device, comprising a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the above-mentioned video anomaly online detection method based on the diffusion model of the skeleton trajectory.

[0033] To achieve the above-mentioned object of the application, the embodiment further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the above-mentioned video anomaly online detection method based on the diffusion model of the skeleton trajectory.

[0034] Compared with the prior art, the application has the beneficial effects at least including:

[0035] Based on the spatiotemporal skeleton sequence extracted from the continuous video frame sequence, a guide condition containing global features and local features is constructed, a diffusion model is used to generate a predicted spatiotemporal skeleton frame based on the guide condition, and then whether the current video frame is an abnormal video frame is determined based on the real spatiotemporal skeleton frame and the predicted spatiotemporal skeleton frame at the current moment, which can greatly improve the detection speed and also improve the detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0037] Figure 1 is a flowchart of the video anomaly online detection method based on the diffusion model of the skeleton trajectory provided by the embodiment;

[0038] Figure 2 is a training schematic diagram of the diffusion model of the skeleton trajectory provided by the embodiment;

[0039] Figure 3 is a structural schematic diagram of the video anomaly online detection device based on the diffusion model of the skeleton trajectory provided by the embodiment. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0041] As Figure 1As shown, the embodiment provides a video anomaly online detection method based on a skeletal trajectory diffusion model, including the following steps:

[0042] S1, obtaining a continuous video frame sequence containing a current video frame of a video to be detected.

[0043] In the embodiment, the obtained continuous video frame sequence contains the current video frame and also contains historical video frames other than the current video frame, and these video frame sequences are subsequently used to extract skeletal information to construct a guide vector.

[0044] S2, extracting a real skeletal pose sequence from the continuous video frame sequence by using a skeletal pose network, and tracking skeletal information in different video frames by using an optical flow neural network to obtain a real spatio-temporal skeletal sequence of the target.

[0045] In the embodiment, the skeletal pose network can adopt an Alpha Pose model, and the optical flow neural network can adopt a FlowNet.

[0046] S3, extracting global features and local features from the real spatio-temporal skeletal sequence, and encoding the global features and the local features based on a spatio-temporal autoencoder to obtain first conditional features and second conditional features.

[0047] In the embodiment, the motion trend of the human skeleton in the entire video frame is extracted from the real spatio-temporal skeletal sequence as the global features, and the motion change of the human pose between video frames is extracted from the real spatio-temporal skeletal sequence as the local features, and then the spatio-temporal autoencoder is used to encode the global features and the local features to obtain the first conditional features and the second conditional features.

[0048] In the embodiment, the encoder and the decoder in the spatio-temporal autoencoder adopt an Unet model, the Unet model is used to learn the global features in the hidden space to obtain the first conditional features, and the Unet model is used to learn the local features in the hidden space to obtain the second conditional features.

[0049] S4, using a skeletal trajectory diffusion model to generate a predicted spatio-temporal skeletal frame at the current time based on random noise, the first conditional features and the second conditional features through inverse diffusion.

[0050] In the embodiment, the structure of the skeletal trajectory diffusion model is as shown in Figure 2 Before being applied, the skeletal trajectory diffusion model is trained by using a public video anomaly detection data set.

[0051] In the skeletal trajectory diffusion model, according to Markov chain derivation rules, x t represents the image of the t-th diffusion step, so the forward diffusion equation can be expressed as:

[0052]

[0053] where q(x t |x t-1 ) is a normal distribution, and is the mean of the Gaussian function, and β t is the variance of the Gaussian function.

[0054] In the inverse diffusion process, the model needs to learn and represent the conditional probability distribution p θ (x t-1 |x t )

[0055] p θ (x t-1 |x t ) = N(x t-1 ; μ θ (x t , t), ∑ θ (x t , t))

[0056] where the mean and variance also depend on the noise level, and the model needs to represent these means and variances through learning, and the way to choose is to keep the variance fixed, only learn the mean of this conditional probability distribution, x t is the spatio-temporal skeleton sequence sampled from the dataset, and t represents the t-th step in the diffusion process.

[0057] In one way, the skeleton trajectory diffusion model generates a predicted spatio-temporal skeleton sequence through inverse diffusion based on random noise, a first conditional feature, and a second conditional feature, including: taking the random noise as the initial noise, and taking the splicing result of the first conditional feature and the second conditional feature as the guide body condition at each time step, performing multi-step denoising to generate the predicted spatio-temporal skeleton sequence.

[0058] In another way, the skeleton trajectory diffusion model generates a predicted spatio-temporal skeleton sequence through inverse diffusion based on random noise, a first conditional feature, and a second conditional feature, including: taking the random noise as the initial noise, and taking the first conditional feature as the guide body condition at each time step, performing multi-step denoising to generate a first predicted candidate spatio-temporal skeleton sequence; taking the random noise as the initial noise, and taking the second conditional feature as the guide body condition at each time step, performing multi-step denoising to generate a second predicted candidate spatio-temporal skeleton sequence; and merging and fusing the first predicted candidate spatio-temporal skeleton sequence and the second predicted candidate spatio-temporal skeleton sequence to obtain the predicted spatio-temporal skeleton sequence.

[0059] S5, determining whether the current video frame is an abnormal video frame by comparing the real spatio-temporal skeleton frame at the current moment and the predicted spatio-temporal skeleton frame.

[0060] In the embodiment, when the abnormality is determined, the abnormality score of the real space-time skeletal frame at the current moment and the predicted space-time skeletal frame at the current moment is calculated, wherein the abnormality score can be calculated by using L1 norm, that is, the L1 norm of the real space-time skeletal frame and the predicted space-time skeletal frame is calculated as the abnormality score, and when the abnormality score is higher than a preset threshold, the current video frame is determined as an abnormal video frame.

[0061] As shown in Figure 3 The embodiment also provides a video abnormality online detection device based on a skeletal trajectory diffusion model, which comprises a data acquisition module 31, a skeletal extraction module 32, a feature coding module 33, a predicted space-time skeletal frame module 34 and an abnormality confirmation module 35. The data acquisition module 31 is used to acquire a continuous video frame sequence of a video to be detected, which comprises a current video frame. The skeletal extraction module 32 is used to extract a real skeletal pose sequence from the continuous video frame sequence by using a skeletal pose network, and track the skeletal information in different video frames by using an optical flow neural network to obtain a real space-time skeletal sequence of a target. The feature coding module 33 is used to extract global features and local features from the real space-time skeletal sequence, and code the global features and the local features based on a space-time autoencoder to obtain first conditional features and second conditional features. The predicted space-time skeletal frame module 34 is used to generate a predicted space-time skeletal frame at the current moment by inverse diffusion based on random noise, the first conditional features and the second conditional features by using a skeletal trajectory diffusion model. The abnormality confirmation module 35 is used to determine whether the current video frame is an abnormal video frame by comparing the real space-time skeletal frame at the current moment and the predicted space-time skeletal frame.

[0062] It should be noted that the video abnormality online detection device based on the skeletal trajectory diffusion model provided in the above embodiment should be illustrated by the division of the above functional modules when performing video abnormality online detection, and the above functions can be completed by different functional modules according to needs, that is, the internal structure of the terminal or the server is divided into different functional modules to complete all or part of the above described functions. In addition, the video abnormality online detection device based on the skeletal trajectory diffusion model provided in the above embodiment and the video abnormality online detection construction method based on the skeletal trajectory diffusion model belong to the same concept, and the specific implementation process is shown in the video abnormality online detection method based on the skeletal trajectory diffusion model, which will not be described here.

[0063] Based on the same inventive concept, the embodiment also provides a computing device comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the above video abnormality online detection method based on the skeletal trajectory diffusion model, and specifically include the following steps:

[0064] S1, acquiring a continuous video frame sequence of a video to be detected, which comprises a current video frame;

[0065] S2, extracting a real skeletal pose sequence from the continuous video frame sequence by using a skeletal pose network, tracking skeletal information in different video frames by using an optical flow neural network, and obtaining a real spatiotemporal skeletal sequence of the target;

[0066] S3, extracting global features and local features from the real spatiotemporal skeletal sequence, and encoding the global features and the local features based on a spatiotemporal autoencoder to obtain first conditional features and second conditional features;

[0067] S4, generating a predicted spatiotemporal skeletal frame at a current time by inverse diffusion based on random noise, the first conditional features, and the second conditional features by using a skeletal trajectory diffusion model;

[0068] S5, determining whether the current video frame is an abnormal video frame by comparing the real spatiotemporal skeletal frame at the current time and the predicted spatiotemporal skeletal frame.

[0069] The computing device provided in the embodiment, in addition to including a processor and a memory, also includes internal buses, network interfaces, memories, and other hardware required by businesses. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs, so as to implement the video anomaly online detection method based on the skeletal trajectory diffusion model described in S1-S5. Of course, in addition to the software implementation mode, the present application does not exclude other implementation modes, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to the logical units, but can also be hardware or a logic device.

[0070] Based on the same inventive concept, the embodiment also provides a computer readable storage medium having a program stored thereon, which, when executed by a processor, implements the video anomaly online detection method based on the skeletal trajectory diffusion model, and specifically includes the following steps:

[0071] S1, obtaining a continuous video frame sequence of a to-be-detected video including a current video frame;

[0072] S2, extracting a real skeletal pose sequence from the continuous video frame sequence by using a skeletal pose network, tracking skeletal information in different video frames by using an optical flow neural network, and obtaining a real spatiotemporal skeletal sequence of the target;

[0073] S3, extracting global features and local features from the real spatiotemporal skeletal sequence, and encoding the global features and the local features based on a spatiotemporal autoencoder to obtain first conditional features and second conditional features;

[0074] S4, generating a predicted spatiotemporal skeletal frame at a current time by inverse diffusion based on random noise, the first conditional features, and the second conditional features by using a skeletal trajectory diffusion model;

[0075] S5, determining whether the current video frame is an abnormal video frame by comparing the real space-time skeleton frame and the predicted space-time skeleton frame at the current moment.

[0076] In the embodiments, the computer readable medium includes permanent and non-permanent, removable and non-removable media, which can realize information storage by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0077] The above detailed description of the specific embodiments of the present application has described the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not used to limit the present application. Any modification, supplement and equivalent replacement made within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. A method for online video anomaly detection based on skeletal trajectory diffusion model, characterized in that, The method comprises the following steps: obtaining a continuous video frame sequence containing a current video frame of a video to be detected; extracting a real skeleton posture sequence from the continuous video frame sequence by using a skeleton posture network and tracking skeleton information in different video frames by using an optical flow neural network, so as to obtain a real spatiotemporal skeleton sequence of a target; extracting global features and local features from the real spatiotemporal skeleton sequence and encoding the global features and the local features based on a spatiotemporal autoencoder to obtain first conditional features and second conditional features, wherein the extracting the global features and the local features from the real spatiotemporal skeleton sequence comprises: extracting a motion trend of a human skeleton in an entire video frame from the real spatiotemporal skeleton sequence as the global features; and extracting a motion change of a human posture between video frames from the real spatiotemporal skeleton sequence as the local features; generating a predicted spatiotemporal skeleton frame at a current time by reverse diffusion based on random noise, the first conditional features and the second conditional features by using a skeleton trajectory diffusion model, specifically comprising: taking the random noise as initial noise and taking a splicing result of the first conditional features and the second conditional features as a guide body condition at each time step to perform multi-step denoising and generate a predicted spatiotemporal skeleton sequence; or specifically comprising: taking the random noise as initial noise and taking the first conditional features as a guide body condition at each time step to perform multi-step denoising and generate a first predicted candidate spatiotemporal skeleton sequence; taking the random noise as initial noise and taking the second conditional features as a guide body condition at each time step to perform multi-step denoising and generate a second predicted candidate spatiotemporal skeleton sequence; and merging and fusing the first predicted candidate spatiotemporal skeleton sequence and the second predicted candidate spatiotemporal skeleton sequence to obtain the predicted spatiotemporal skeleton sequence; determining whether the current video frame is an abnormal video frame by comparing the real spatiotemporal skeleton frame at the current time with the predicted spatiotemporal skeleton frame.

2. The method of claim 1, wherein the diffusion model is based on a skeletal trajectory. The encoder and the decoder in the spatiotemporal autoencoder adopt an Unet model, the Unet model is used to learn the global features in a hidden space to obtain the first conditional features, and the Unet model is used to learn the local features in the hidden space to obtain the second conditional features. 3.The method of claim 1, wherein, The determination that the current video frame is an abnormal video frame by comparing the real spatiotemporal skeleton frame at the current time with the predicted spatiotemporal skeleton frame comprises: calculating an abnormal score of the real spatiotemporal skeleton frame at the current time and the predicted spatiotemporal skeleton frame at the current time, and determining that the current video frame is an abnormal video frame when the abnormal score is higher than a preset threshold.

4. The method of claim 3, wherein the diffusion model is based on a skeletal trajectory. The abnormal score of the real spatiotemporal skeleton frame and the predicted spatiotemporal skeleton frame is calculated by using an L1 norm.

5. An apparatus for video anomaly online detection based on a skeletal trajectory diffusion model, characterized in that, The method comprises the following steps: a data acquisition module configured to obtain a continuous video frame sequence containing a current video frame of a video to be detected; a skeleton extraction module configured to extract a real skeleton posture sequence from the continuous video frame sequence by using a skeleton posture network and tracking skeleton information in different video frames by using an optical flow neural network, so as to obtain a real spatiotemporal skeleton sequence of a target; a feature encoding module, configured to extract global features and local features from the real spatio-temporal skeleton sequence, and encode the global features and the local features based on a spatio-temporal auto-encoder to obtain first conditional features and second conditional features, wherein the extracting the global features and the local features from the real spatio-temporal skeleton sequence comprises: extracting motion trends of human skeletons in an entire video frame from the real spatio-temporal skeleton sequence as the global features; and extracting motion change conditions of human poses between video frames from the real spatio-temporal skeleton sequence as the local features; a predicted spatio-temporal skeleton frame module, configured to generate a predicted spatio-temporal skeleton frame at a current time by inverse diffusion based on random noise, the first conditional features and the second conditional features using a skeleton trajectory diffusion model, and specifically comprising: performing multi-step denoising by taking the random noise as initial noise and taking a splicing result of the first conditional features and the second conditional features as a guide body condition at each time step to generate a predicted spatio-temporal skeleton sequence; or specifically comprising: performing multi-step denoising by taking the random noise as initial noise and taking the first conditional features as a guide body condition at each time step to generate a first predicted candidate spatio-temporal skeleton sequence; performing multi-step denoising by taking the random noise as initial noise and taking the second conditional features as a guide body condition at each time step to generate a second predicted candidate spatio-temporal skeleton sequence; and merging and fusing the first predicted candidate spatio-temporal skeleton sequence and the second predicted candidate spatio-temporal skeleton sequence to obtain the predicted spatio-temporal skeleton sequence; an abnormality confirmation module, configured to determine whether a current video frame is an abnormal video frame by comparing a real spatio-temporal skeleton frame at the current time and the predicted spatio-temporal skeleton frame.

6. A computing device comprising a memory and one or more processors, the memory having stored therein executable code, wherein, The one or more processors execute the executable code to implement the video anomaly online detection method based on the skeleton trajectory diffusion model in any one of claims 1-4.

7. A computer readable storage medium characterized in that, A program is stored thereon, and the program is executed by a processor to implement the video anomaly online detection method based on the skeleton trajectory diffusion model in any one of claims 1-4.

Images