Video anomaly detection method and device, electronic equipment and storage medium

Abstract

The invention provides a video anomaly detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-detected video, and carrying out the cutting of the to-be-detected video, and obtaining a plurality of video segments; wherein each video segment comprises a to-be-detected video frame and a plurality of continuous to-be-processed video frames; respectively inputting the to-be-processed video frames in the video segments into a video anomaly detection model to obtain predicted video frames corresponding to the video segments; wherein the video anomaly detection model is obtained based on training of a video training sample; and performing anomaly detection based on the predicted video frame and the to-be-detected video frame. The video anomaly detection model provided by the invention predicts the predicted video frame of the next frame based on a certain number of continuous to-be-processed video frames, and further realizes accurate detection of the abnormal event in the video based on the similarity between the predicted video frame and the to-be-detected video frame.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a video anomaly detection method, device, electronic equipment and storage medium. Background Art

[0002] Video Anomaly Detection (VAD) aims to detect abnormal frames in a video that correspond to unexpected events. When only normal videos are available during the training phase, this task is called Unsupervised Video Anomaly Detection (UVAD), which is a more challenging task because it must learn to recognize anomalies without ever seeing anomalies. In recent years, video anomaly detection has received widespread attention from academia and industry, and has been widely used in intelligent surveillance systems and other fields.

[0003] Compared to video classification or action recognition, UVAD is different from classifying the entire video as normal or abnormal. VAD must detect frames corresponding to abnormal events from the video, while most of the content of the video is normal and anomalies appear in multiple modes, which leads to poor accuracy of anomaly detection. Summary of the invention

[0004] The present invention provides a video anomaly detection method, device, electronic equipment and storage medium, which are used to solve the defects in the prior art and improve the accuracy of video anomaly detection.

[0005] The present invention provides a video anomaly detection method, comprising: Acquire a video to be detected, and perform segmentation processing on the video to be detected to obtain multiple video segments; wherein each of the video segments includes a video frame to be detected and multiple consecutive video frames to be processed; Inputting the to-be-processed video frames in each of the video segments into a video anomaly detection model respectively to obtain predicted video frames corresponding to each of the video segments; wherein the video anomaly detection model is trained based on video training samples; Anomaly detection is performed based on the predicted video frame and the video frame to be detected.

[0006] According to a video anomaly detection method provided by the present invention, the video anomaly detection model includes a three-dimensional convolutional network, and the three-dimensional convolutional network includes a basic three-dimensional convolutional module, three down-sampling modules and three up-sampling modules connected in sequence, the down-sampling modules and the up-sampling modules correspond one to one, and the down-sampling modules are jump-connected with the corresponding up-sampling modules.

[0007] According to a video anomaly detection method provided by the present invention, the downsampling module includes a first three-dimensional convolution unit, a deformable three-dimensional convolution unit and a second three-dimensional convolution unit connected in sequence; The deformable three-dimensional convolution unit includes a deformable three-dimensional convolution layer and a separable three-dimensional convolution layer; wherein the separable three-dimensional convolution layer generates an offset for each sampling point of the convolution kernel of the deformable three-dimensional convolution layer so that the convolution kernel is deformed based on the offset.

[0008] According to a video anomaly detection method provided by the present invention, the separable three-dimensional convolution layer includes a one-dimensional convolution and a two-dimensional convolution.

[0009] According to a video anomaly detection method provided by the present invention, the training process of the video anomaly detection model includes: Acquire multiple video training samples and sample labels of each of the video training samples; wherein the video training samples include multiple consecutive training video frames; Constructing a generative adversarial network model; wherein the generative adversarial network model includes a generator and a discriminator, the input of the generator is the video training sample, and the input of the discriminator is the sample label and the simulated video frame generated by the generator; The video training samples and the sample labels are input into the generative adversarial network model, and the generator and the discriminator are trained alternately until the training is completed, thereby obtaining the video anomaly detection model.

[0010] According to a video anomaly detection method provided by the present invention, the video training sample and the sample label are input into the generative adversarial network model, the generator and the discriminator are alternately trained until the training is completed, and the video anomaly detection model is obtained, including: In each round of training, the discriminator is fixed, and the video training samples are input into the generator to obtain the corresponding plurality of simulated video frames; The generator is fixed, and the simulated video frame and the sample label are input into the discriminator for training; wherein the sample label includes a plurality of labeled video frames corresponding to the simulated video frame; Calculate a first loss value of the generator based on the simulated video frame and the sample label, and calculate a second loss value of the discriminator based on an output result of the discriminator; Determine whether the training of the generator and the discriminator is completed based on the first loss value and the second loss value respectively; If the training is completed, the video anomaly detection model is obtained based on the trained generator.

[0011] According to a video anomaly detection method provided by the present invention, the anomaly detection based on the predicted video frame and the to-be-detected video frame comprises: Calculating the similarity between the predicted video frame and the to-be-detected video frame of each of the video segments; Whether the video frame to be detected is abnormal is determined based on the similarity.

[0012] The present invention also provides a video anomaly detection device, comprising: A first acquisition module is configured to acquire a video to be detected, and to perform segmentation processing on the video to be detected to obtain a plurality of video segments; wherein each of the video segments includes a video frame to be detected and a plurality of consecutive video frames to be processed; An input module is configured to input the to-be-processed video frames in each of the video segments into a video anomaly detection model to obtain predicted video frames corresponding to each of the video segments; wherein the video anomaly detection model is trained based on video training samples; The anomaly detection module is configured to perform anomaly detection based on the predicted video frame and the video frame to be detected.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned video anomaly detection methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video anomaly detection method as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video anomaly detection method as described above is implemented.

[0016] The video anomaly detection method, device, electronic device and storage medium provided by the present invention are based on semi-supervised learning technology, input a certain number of continuous video frames to be processed into a pre-trained video anomaly detection model, predict the predicted video frame of the next frame, and then perform anomaly detection based on the similarity between the predicted video frame and the video frame to be detected, thereby realizing accurate detection of abnormal events in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the video anomaly detection method provided by the present invention.

[0019] Figure 2 Schematic diagram of a video anomaly detection model in an embodiment of the present invention.

[0020] Figure 3 It is a schematic diagram of the deformable three-dimensional convolution unit provided by the present invention.

[0021] Figure 4 Schematic diagram of training a video anomaly detection model in an embodiment of the present invention Figure 5 It is a structural schematic diagram of the video anomaly detection device provided by the present invention.

[0022] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Combine the following Figure 1-Figure 6 The present invention describes a video anomaly detection method, device, electronic device and storage medium.

[0025] Figure 1 FIG. 1 is a flow chart of a method for detecting video anomalies according to an exemplary embodiment. Figure 1 As shown, in an exemplary embodiment, the video anomaly detection method includes steps 110 to 130, which are described in detail as follows.

[0026] Step 110, obtaining a video to be detected, and cutting the video to be detected to obtain multiple video segments; wherein each of the video segments includes a video frame to be detected and multiple consecutive video frames to be processed.

[0027] In the embodiment of the present invention, the video to be detected is divided into continuous and overlapping video segments, and the video segment includes a video frame to be detected and a video frame to be processed. In the time dimension, the video frame to be detected is the last frame of the video segment. For example, for a video segment consisting of 10 video frames, the first 1 to 9 frames are used as video frames to be processed, and the 10th frame is used as the video frame to be detected. The non-overlapping video frames of each video segment and the adjacent previous video segment can be used as the video frame to be detected.

[0028] Step 120, respectively input the to-be-processed video frames in each of the video segments into a video anomaly detection model to obtain predicted video frames corresponding to each of the video segments; wherein the video anomaly detection model is trained based on video training samples.

[0029] In the embodiment of the present invention, a video anomaly detection model is pre-trained, and all the to-be-processed video frames of each video segment are stacked in the time dimension to form a multidimensional tensor. For example, if there are m video frames to be processed, the composed multidimensional tensor C×T×H×W is used as the network input, where C, T, H and W are the number of channels, number of frames, frame height and frame width of the video segment, respectively, which can be expressed as { }, and input it into the video anomaly detection model for processing. The video anomaly detection model can generate a predicted video segment corresponding to the video frame to be processed based on the input video frame to be processed. . The number of frames in the predicted video segment is the same as the number of frames in the input video frame to be processed. In the time dimension, the first video frame of the predicted video segment corresponds to the second video frame of the video segment. For example, in the aforementioned video segment composed of 10 video frames, frames 1 to 9 are input into the video anomaly detection model. The video anomaly detection model generates video frames from frames 2 to 10 based on these 9 video frames to be processed. The generated video frames from frames 2 to 10 constitute the predicted video segment, wherein the last frame in the predicted video segment is the predicted video frame, that is, the generated video frame 10 is the predicted video frame.

[0030] Step 130: Perform anomaly detection based on the predicted video frame and the to-be-detected video frame.

[0031] In the embodiment of the present invention, whether the video frame to be detected is abnormal is detected based on the predicted video frame of each video segment.

[0032] In an embodiment of the present invention, based on semi-supervised learning technology, a certain number of consecutive video frames to be processed are input into a pre-trained video anomaly detection model to predict the predicted video frame of the next frame, and then anomaly detection is performed based on the similarity between the predicted video frame and the video frame to be detected, thereby achieving accurate detection of abnormal events in the video.

[0033] In an exemplary embodiment of the present invention, the video anomaly detection model includes a three-dimensional convolutional network, which includes a basic three-dimensional convolutional module, three downsampling modules and three upsampling modules connected in sequence, the downsampling modules and the upsampling modules correspond one to one, and the downsampling modules are jump-connected with the corresponding upsampling modules.

[0034] In an embodiment of the present invention, a video anomaly detection model is composed of a three-dimensional convolutional network (Unet-3D). The three-dimensional convolutional network of the present invention is based on the commonly used Unet-3D network, and the number of channels is reduced in each downsampling stage, and each downsampling module is improved.

[0035] like Figure 2 As shown, the basic 3D convolution module is composed of two ordinary 3D convolution layers (3D conv) to extract visual features in the video frame to be processed. After extracting the visual features, the downsampling module (Deform-down) provided by the present invention is used to extract relevant action features in each downsampling stage. For example, in the three downsampling modules, the input feature map size is changed from the original 8×8×128×128 to 4×16×128×128, 2×32×64×64 and 1×64×32×32 respectively.

[0036] In each upsampling stage that is symmetrical to the downsampling stage, skip connections are used to fuse the upsampled feature map with the corresponding feature map before downsampling to enhance the expressiveness of the feature map, thereby generating the next video frame of the input video training sample.

[0037] Specifically, Figure 2 As shown in the figure, the upsampling module (Concat-up) includes a deconvolution layer and two ordinary 3D convolution layers. The input of each upsampling module will first be processed by an ordinary deconvolution layer (Deconv), and then added with the feature map transmitted by the skip connection module to enhance the expressiveness of the feature map. Subsequently, the fused feature map will pass through two ordinary 3D convolution layers (3Dconv). After being processed by three upsampling modules, the final output size of the generator is , corresponding to the input of the generator.

[0038] In an exemplary embodiment of the present invention, the downsampling module includes a first three-dimensional convolution unit, a deformable three-dimensional convolution unit, and a second three-dimensional convolution unit connected in sequence; The deformable three-dimensional convolution unit includes a deformable three-dimensional convolution layer and a separable three-dimensional convolution layer; wherein the separable three-dimensional convolution layer generates an offset for each sampling point of the convolution kernel of the deformable three-dimensional convolution layer so that the convolution kernel is deformed based on the offset.

[0039] In the embodiment of the present invention, Figure 2 As shown, the downsampling module (Deform-down) includes a first three-dimensional convolution unit (3Dconv), a deformable three-dimensional convolution unit (CD3D) and a second three-dimensional convolution unit (3D Maxpooling) which are connected in sequence.

[0040] like Figure 2 and Figure 3 As shown, the deformable 3D convolution unit includes a deformable 3D convolution layer (Deform 3D conv) and a separable 3D convolution layer (Separable 3D conv). The separable 3D convolution layer predicts the offset of each sampling point of the deformable 3D convolution layer so that the deformable 3D convolution layer is deformed based on the offset.

[0041] In an exemplary embodiment of the present invention, the separable three-dimensional convolution layer includes a one-dimensional convolution and a two-dimensional convolution.

[0042] In the embodiment of the present invention, the separable three-dimensional convolution layer uses a set of two-dimensional convolutions of size 1×3×3 and one-dimensional convolutions of size 2×1×1, so as to achieve accurate motion feature extraction and spatiotemporal feature fusion.

[0043] The separable 3D convolution layer generates an additional offset for each point of the convolution kernel of the deformable 3D convolution layer to guide the deformation of the convolution kernel. The convolution calculation formula is as follows: ; ; in, express The output characteristics at Represents the convolution kernel A point on , is with The corresponding offset, A conventional convolution kernel for convolution on the feature map is defined. Since the offset is usually a fraction, it is calculated by bilinear interpolation. Based on the offset, the convolution kernel is no longer arbitrarily offset in the three directions, but is fixed in the channel direction and can only be offset in the horizontal and numerical directions. While inheriting the excellent spatiotemporal information extraction capability of 3D convolution, the model's ability to distinguish between normal and abnormal events is enhanced.

[0044] One-dimensional convolution and two-dimensional convolution are composed of The convolutions are separated into two dimensions, where the two-dimensional convolution extracts features from each video frame and the one-dimensional convolution fuses the features of adjacent video frames.

[0045] To avoid confusing the order of frames, the Deformable 3D Convolutional Unit does not generate offsets along the temporal dimension between video frames, so there are a total of offset, where , yes The offset of the third dimension is 0, that is, the offset in the time dimension is 0, that is: .

[0046] In the embodiment of the present invention, the deformable 3D convolution unit extracts the image features of each frame, inherits the excellent spatiotemporal information extraction capability of the 3D convolution, and adjusts the scale or receptive field to adapt to irregular geometric changes, thereby accurately focusing on the key points in the video frame. The deformable 3D convolution unit affects the frame prediction errors of normal segments and abnormal segments in opposite ways, making it easier to distinguish abnormal frames from normal frames, thereby making it easier to detect abnormal events.

[0047] In an exemplary embodiment of the present invention, the training process of the video anomaly detection model includes the following steps, which are described in detail as follows.

[0048] Acquire multiple video training samples and sample labels of each of the video training samples; wherein the video training samples include multiple consecutive training video frames.

[0049] Construct a generative adversarial network model; wherein the generative adversarial network model includes a generator and a discriminator, the input of the generator is the video training sample, and the input of the discriminator is the sample label and the simulated video frame generated by the generator.

[0050] The video training samples and the sample labels are input into the generative adversarial network model, and the generator and the discriminator are trained alternately until the training is completed, thereby obtaining the video anomaly detection model.

[0051] In an embodiment of the present invention, the video training sample includes multiple consecutive training video frames, and the corresponding sample labels are composed of video frames delayed by one frame from each training video frame, that is, the first 1 to 9 video frames constitute the video training sample, and the 2 to 10 video frames constitute the sample labels.

[0052] like Figure 3As shown in the figure, a generative adversarial network model (GAN) is constructed. The generative adversarial network model includes a generator Generate and a discriminator Discriminator. The discriminator uses the Pix2Pix network. The generative adversarial network model as a whole adopts an alternating training method. The alternating training method is the key to model training. In each round of iteration, the discriminator is first fixed and the generator is trained so that the generator can generate more realistic images based on the input video training samples to deceive the discriminator. Then, the generator is fixed and the discriminator is trained so that it can more accurately identify the difference between the real image and the generated image. Through this alternating optimization method, the generator and the discriminator can promote each other and make progress together, continuously improving the overall performance of the model.

[0053] In an exemplary embodiment of the present invention, the video training samples and the sample labels are input into the generative adversarial network model, the generator and the discriminator are trained alternately until the training is completed, and the video anomaly detection model is obtained, which includes the following steps, which are described in detail as follows.

[0054] In each round of training, the discriminator is fixed, and the video training samples are input into the generator to obtain the corresponding multiple simulated video frames.

[0055] The generator is fixed, and the simulated video frames and the sample labels are input into the discriminator for training; wherein the sample labels include a plurality of labeled video frames corresponding to the simulated video frames.

[0056] A first loss value of the generator is calculated based on the simulated video frame and the sample label, and a second loss value of the discriminator is calculated based on an output result of the discriminator.

[0057] It is determined whether the training of the generator and the discriminator is completed based on the first loss value and the second loss value respectively.

[0058] If the training is completed, the video anomaly detection model is obtained based on the trained generator.

[0059] In the embodiment of the present invention, the generative adversarial network model as a whole adopts an alternating training method. In each round of iterative training, the discriminator is first fixed, and the video training sample is input into the generator for training, so that it can generate a simulated video frame of the next frame of the training video frame in the video training sample. Then, the generator is fixed, and the simulated video frame and sample label are input into the discriminator for training, so that the discriminator can more accurately identify the difference between the simulated video frame and the sample label. The generator attempts to generate frames that are indistinguishable from real frames, while the discriminator attempts to distinguish them. Through this alternating optimization method, the generator and the discriminator can promote each other, make progress together, and continuously improve the overall performance of the model.

[0060] In the embodiment of the present invention, during the training process, all training video frames of the video training sample are used to construct the loss to better utilize the training data.

[0061] Specifically, during the p-th round of iterative training, the video training sample input by the generator is represented as , k represents the number of training video frames in the video training sample, and the sample label corresponding to the video training sample is expressed as { , the video training sample is input into the generator for processing, and the output of the generator is expressed as .

[0062] For the generator, two loss functions are defined as follows: ; ; in, Represents the output of the generator With sample labels Loss function at the pixel level. Represents the output of the generator With sample labels The loss function at the gradient level is and Represent the gradients in the x and y directions respectively.

[0063] Therefore, the overall loss function of the generator is It can be defined as: ; Furthermore, the loss function of the discriminator is It can be defined as: ; in, , and are the corresponding weights respectively.

[0064] In an exemplary embodiment of the present invention, the abnormality detection based on the predicted video frame and the to-be-detected video frame includes the following steps, which are described in detail as follows.

[0065] The similarity between the predicted video frame and the to-be-detected video frame of each video segment is calculated.

[0066] Whether the video frame to be detected is abnormal is determined based on the similarity.

[0067] In the embodiment of the present invention, Peak Signal-to-Noise Ratio (PSNR) is used as an evaluation index to measure the similarity between the predicted video frame and the video frame to be detected.

[0068] Peak signal-to-noise ratio (PSNR) represents the ratio of the maximum possible signal power to the destructive noise power that affects its representation accuracy. It is an engineering term used to quantify image quality, especially the degree of distortion after image compression or processing. The higher the PSNR value, the smaller the image distortion, that is, the better the image quality. It provides a quantitative way to compare the effects of different image processing algorithms or compression methods. The formula for calculating the peak signal-to-noise ratio is as follows: ; ; Where MSE stands for mean square error. As the peak signal-to-noise ratio decreases, the possibility that the video frame to be detected contains an abnormal event gradually increases.

[0069] The PSNR values ​​of the video frames to be detected in the entire video to be detected are normalized to obtain a score, and this is used to determine which video frames to be detected have abnormalities. The normalized calculation formula is as follows: ; Normalization can convert the PSNR value into a uniform scoring range so that the degree of abnormality between different video frames to be detected can be more intuitively compared. By setting a detection threshold or according to the distribution of PSNR values, the video frames to be detected that are below the detection threshold or in the low segment are determined as abnormal video frames. In this way, automatic detection and identification of video abnormal events can be achieved.

[0070] The video anomaly detection device provided by the present invention is described below. The video anomaly detection device described below and the video anomaly detection method described above can be referred to each other. It should be noted that the device provided in the following embodiment and the method provided in the above embodiment belong to the same concept, and the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, which will not be repeated here.

[0071] In an exemplary embodiment of the present invention, see Figure 5 , Figure 5 A video anomaly detection device according to an exemplary embodiment includes: The first acquisition module 510 is configured to acquire a video to be detected, and perform segmentation processing on the video to be detected to obtain a plurality of video segments; wherein each of the video segments includes a video frame to be detected and a plurality of consecutive video frames to be processed; The input module 520 is configured to input the to-be-processed video frames in each of the video segments into a video anomaly detection model to obtain a predicted video frame corresponding to each of the video segments; wherein the video anomaly detection model is trained based on a video training sample; The anomaly detection module 530 is configured to perform anomaly detection based on the predicted video frame and the to-be-detected video frame.

[0072] In an exemplary embodiment of the present invention, the video anomaly detection model includes a three-dimensional convolutional network, which includes a basic three-dimensional convolutional module, three down-sampling modules and three up-sampling modules connected in sequence, the down-sampling modules and the up-sampling modules correspond one to one, and the down-sampling modules are jump-connected with the corresponding up-sampling modules.

[0073] In an exemplary embodiment of the present invention, the downsampling module includes a first three-dimensional convolution unit, a deformable three-dimensional convolution unit, and a second three-dimensional convolution unit connected in sequence; The deformable three-dimensional convolution unit includes a deformable three-dimensional convolution layer and a separable three-dimensional convolution layer; wherein the separable three-dimensional convolution layer generates an offset for each sampling point of the convolution kernel of the deformable three-dimensional convolution layer so that the convolution kernel is deformed based on the offset.

[0074] In an exemplary embodiment of the present invention, the separable three-dimensional convolutional layer includes a one-dimensional convolution and a two-dimensional convolution.

[0075] In an exemplary embodiment of the present invention, a video anomaly detection device includes: A second acquisition module is configured to acquire a plurality of video training samples and a sample label of each of the video training samples; wherein the video training samples include a plurality of continuous training video frames; A construction module configured to construct a generative adversarial network model; wherein the generative adversarial network model includes a generator and a discriminator, the input of the generator is the video training sample, and the input of the discriminator is the sample label and the simulated video frame generated by the generator; The training module is configured to input the video training samples and the sample labels into the generative adversarial network model, and the generator and the discriminator are trained alternately until the training is completed to obtain the video anomaly detection model.

[0076] In an exemplary embodiment of the present invention, the training module includes: A first fixed submodule is configured to fix the discriminator in each round of training, input the video training sample into the generator, and obtain the corresponding plurality of simulated video frames; A second fixed submodule is configured to fix the generator and input the simulated video frame and the sample label into the discriminator for training; wherein the sample label includes a plurality of labeled video frames corresponding to the simulated video frame; A first calculation submodule is configured to calculate a first loss value of the generator based on the simulated video frame and the sample label, and calculate a second loss value of the discriminator based on an output result of the discriminator; A first determination submodule, configured to determine whether the training of the generator and the discriminator is completed based on the first loss value and the second loss value respectively; The video anomaly detection model submodule is configured to obtain the video anomaly detection model based on the trained generator if the training is completed.

[0077] In an exemplary embodiment of the present invention, the anomaly detection module 530 includes: A second calculation submodule is configured to calculate the similarity between the predicted video frame and the to-be-detected video frame of each of the video segments; The second determination submodule is configured to determine whether the to-be-detected video frame is abnormal based on the similarity.

[0078] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the video anomaly detection method, which includes: obtaining a video to be detected, cutting and processing the video to be detected to obtain a plurality of video segments; wherein each of the video segments includes a video frame to be detected and a plurality of continuous video frames to be processed; Inputting the to-be-processed video frames in each of the video segments into a video anomaly detection model respectively to obtain predicted video frames corresponding to each of the video segments; wherein the video anomaly detection model is trained based on video training samples; Anomaly detection is performed based on the predicted video frame and the video frame to be detected.

[0079] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0080] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the video anomaly detection method provided by the above methods, the method includes: obtaining a video to be detected, cutting and processing the video to be detected to obtain a plurality of video segments; wherein each of the video segments includes a video frame to be detected and a plurality of continuous video frames to be processed; Inputting the to-be-processed video frames in each of the video segments into a video anomaly detection model respectively to obtain predicted video frames corresponding to each of the video segments; wherein the video anomaly detection model is trained based on video training samples; Anomaly detection is performed based on the predicted video frame and the video frame to be detected.

[0081] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the video anomaly detection method provided by the above methods, the method comprising: obtaining a video to be detected, cutting and processing the video to be detected to obtain a plurality of video segments; wherein each of the video segments comprises a video frame to be detected and a plurality of continuous video frames to be processed; Inputting the to-be-processed video frames in each of the video segments into a video anomaly detection model respectively to obtain predicted video frames corresponding to each of the video segments; wherein the video anomaly detection model is trained based on video training samples; Anomaly detection is performed based on the predicted video frame and the video frame to be detected.

[0082] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video anomaly detection method, characterized in that: include: Acquire a video to be detected, and perform segmentation processing on the video to be detected to obtain multiple video segments; wherein each of the video segments includes a video frame to be detected and multiple consecutive video frames to be processed; Inputting the to-be-processed video frames in each of the video segments into a video anomaly detection model respectively to obtain predicted video frames corresponding to each of the video segments; wherein the video anomaly detection model is trained based on video training samples; Anomaly detection is performed based on the predicted video frame and the video frame to be detected.

2. The video anomaly detection method according to claim 1, characterized in that: The video anomaly detection model includes a three-dimensional convolutional network, which includes a basic three-dimensional convolutional module, three down-sampling modules and three up-sampling modules connected in sequence, the down-sampling modules correspond to the up-sampling modules one by one, and the down-sampling modules are jump-connected with the corresponding up-sampling modules.

3. The video anomaly detection method according to claim 2, characterized in that: The downsampling module includes a first three-dimensional convolution unit, a deformable three-dimensional convolution unit and a second three-dimensional convolution unit connected in sequence; The deformable three-dimensional convolution unit includes a deformable three-dimensional convolution layer and a separable three-dimensional convolution layer; wherein the separable three-dimensional convolution layer generates an offset for each sampling point of the convolution kernel of the deformable three-dimensional convolution layer so that the convolution kernel is deformed based on the offset.

4. The video anomaly detection method according to claim 3, characterized in that: The separable three-dimensional convolution layer includes a one-dimensional convolution and a two-dimensional convolution.

5. The video anomaly detection method according to claim 1, characterized in that: The training process of the video anomaly detection model includes: Acquire multiple video training samples and sample labels of each of the video training samples; wherein the video training samples include multiple consecutive training video frames; Constructing a generative adversarial network model; wherein the generative adversarial network model includes a generator and a discriminator, the input of the generator is the video training sample, and the input of the discriminator is the sample label and the simulated video frame generated by the generator; The video training samples and the sample labels are input into the generative adversarial network model, and the generator and the discriminator are trained alternately until the training is completed, thereby obtaining the video anomaly detection model.

6. The video anomaly detection method according to claim 5, characterized in that: The video training sample and the sample label are input into the generative adversarial network model, and the generator and the discriminator are trained alternately until the training is completed to obtain the video anomaly detection model, including: In each round of training, the discriminator is fixed, and the video training samples are input into the generator to obtain the corresponding plurality of simulated video frames; The generator is fixed, and the simulated video frame and the sample label are input into the discriminator for training; wherein the sample label includes a plurality of labeled video frames corresponding to the simulated video frame; Calculate a first loss value of the generator based on the simulated video frame and the sample label, and calculate a second loss value of the discriminator based on an output result of the discriminator; Determine whether the training of the generator and the discriminator is completed based on the first loss value and the second loss value respectively; If the training is completed, the video anomaly detection model is obtained based on the trained generator.

7. The video anomaly detection method according to any one of claims 1 to 6, characterized in that: The performing anomaly detection based on the predicted video frame and the to-be-detected video frame includes: Calculating the similarity between the predicted video frame and the to-be-detected video frame of each of the video segments; Whether the video frame to be detected is abnormal is determined based on the similarity.

8. A video anomaly detection device, characterized in that: include: A first acquisition module is configured to acquire a video to be detected, and to perform segmentation processing on the video to be detected to obtain a plurality of video segments; wherein each of the video segments includes a video frame to be detected and a plurality of consecutive video frames to be processed; An input module is configured to input the to-be-processed video frames in each of the video segments into a video anomaly detection model to obtain predicted video frames corresponding to each of the video segments; wherein the video anomaly detection model is trained based on video training samples; The anomaly detection module is configured to perform anomaly detection based on the predicted video frame and the video frame to be detected.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the video anomaly detection method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video anomaly detection method according to any one of claims 1 to 7 is implemented.

Images