Video anomaly detection method and system based on double-flow structure

Abstract

The application discloses a video anomaly detection method based on a double-flow structure, comprising: acquiring a frame flow through sampling; generating an optical flow vector; extracting spatial and time sequence features of the frame flow by using a 3D convolution kernel; dividing the frame flow into segments, performing convolution on each segment by using the convolution kernel, and obtaining features by 3D convolution processing; compressing the features through a pooling layer to obtain a tensor of a certain size, inputting the tensor as a feature sequence into a one-way long short-term memory network for analysis, and obtaining a final feature vector output; obtaining a first anomaly detection result through classification; processing the optical flow vector by using a convolutional neural network, expressing time sequence information in the optical flow vector in the form of spatial features, and obtaining a second anomaly detection result through classification; and fusing the first and second anomaly detection results to obtain a final video anomaly detection result. The spatial and time sequence features in the frame flow and the time sequence information in the optical flow are extracted and analyzed. The detection precision is improved, and the training speed is optimized.

Description

Technical Field

[0001] This invention belongs to the field of video anomaly detection technology, specifically relating to a video anomaly detection method and system based on a dual-stream structure. Background Technology

[0002] Video anomaly detection is not only an important and active research topic in the field of computer vision, but also a very challenging task. The task of video anomaly detection is to identify events in a video that are clearly inconsistent with normal events, such as fires, traffic accidents, or stampedes. However, due to the large variety of anomalies and their low frequency of occurrence, it is impossible to collect all and enough anomalies to train the model. Furthermore, labeling massive amounts of video frames consumes a significant amount of manpower and time.

[0003] There are currently three commonly used modes for video anomaly detection:

[0004] The first approach is based on manual inspection. The city's video surveillance system only records and transmits video, while professionals review the footage and monitor for anomalies. This approach is time-consuming, labor-intensive, and has low detection accuracy.

[0005] The second approach is an anomaly detection scheme based on object tracking. This involves using YOLO or other object detection models to track and detect the main subject in the video, outputting whether or not an anomaly event is present. This scheme, based on relevant deep learning models, significantly reduces manpower and detection time. However, its application is limited, typically only detecting specific anomalies. Therefore, different models need to be trained for different anomalies, resulting in a large training load and making it difficult to guarantee the performance of each model.

[0006] The third approach is a deep learning-based anomaly detection scheme that does not involve object detection. By optimizing the dataset and model structure, and applying transfer learning, the same model can be applied to different anomaly detection scenarios (such as fires, traffic accidents, etc.). Current deep learning-based detection methods can effectively detect anomalies, but there is still room for improvement in detection accuracy.

[0007] Publication No. CN 112926396 B discloses an action recognition method based on dual-stream convolutional attention. First, the video is preprocessed to obtain frame image sequences and optical flow image sequences, and appearance feature representations and action feature representations are extracted from the video, respectively. Then, a convolutional attention module is constructed to obtain attention feature representations of the frame images and optical flow images, and the two attention representations are fused using a dual-stream fusion module. Next, an action recognition model utilizing the convolutional attention mechanism and dual-stream fusion method is trained. This method constructs a convolutional attention module, taking appearance feature representations and action feature representations as inputs, and outputting channel attention tensors and spatiotemporal attention tensors. The feature representations are weighted to obtain appearance attention feature representations and motion attention features. However, this method is relatively complex, does not extract and analyze temporal information in the optical flow, and its accuracy is not very high. Summary of the Invention

[0008] To address the aforementioned technical problems, the purpose of this invention is to provide a video anomaly detection method and system based on a dual-stream structure, which enables the extraction and analysis of spatial and temporal features in the frame stream and temporal information in the optical flow, thereby improving detection accuracy and optimizing training speed.

[0009] The technical solution of this invention is:

[0010] A video anomaly detection method based on a dual-stream structure includes the following steps:

[0011] S01: Sample the input video to obtain the frame stream;

[0012] S02: Generate optical flow vectors based on the acquired frame stream;

[0013] S03: Utilizing 3D convolutional kernels to extract spatial and temporal features from the frame stream, specifically including:

[0014] Each frame in the frame stream is taken as a segment, and each segment is convolved with a convolution kernel of a certain size and stride to obtain features through 3D convolution. The features are then compressed into a tensor of a certain size through a pooling layer. Finally, the tensor is used as a feature sequence and input into a unidirectional long short-term memory network for analysis to obtain the final feature vector output. Then, classification is performed to obtain the first anomaly detection result.

[0015] S04: The optical flow vector is processed using a convolutional neural network, and the temporal information in the optical flow vector is expressed in the form of spatial features. The classification results are then used to obtain the second anomaly detection result.

[0016] S05: The first anomaly detection result and the second anomaly detection result are fused to obtain the final video anomaly detection result.

[0017] In a preferred embodiment, the sampling method in step S01 includes:

[0018] S11: For the target number of frames K, the sampling rate S = N / K is calculated from the total number of frames N;

[0019] S12: Sample the input video for the first time by extracting 1 frame every S frames, until the end of the video, for a total of M frames.

[0020] S13: Calculate the deletion rate T = M / (MK) based on the difference between the current number of sampled frames M and the target number of frames K.

[0021] S14: Finally, the first sampling result is used as input, and the samples are deleted according to the method of deleting 1 frame per T frames. The deleting sample result is used as the final sampling result and frame stream.

[0022] In a preferred embodiment, the method for generating optical flow in step S02 includes:

[0023] By minimizing the change in pixel grayscale values ​​between two adjacent frames, a cost function based on the optical flow vector of pixel grayscale values ​​is obtained:

[0024] E(u,v) = ∫∫ [(I x u+I y v+I t ) 2 +α(∣∣▽u∣∣ 2 +∣∣▽v∣∣ 2 )]dxdy

[0025] in, u and v These are the two components of the optical flow vector. I x 、I y These are the gradients in the x and y directions of the two images, respectively. I t It is the time derivative, and α is a smoothness parameter. ▽ u and ▽ v represents respectively u and v Gradient in space;

[0026] Taking the derivative of the cost function, we obtain the information about u and v The system of partial differential equations:

[0027] I x (I x u+Iy v+I t )+αΔu=0

[0028] I y (I x u+I y v+I t )+αΔv=0

[0029] Solving the system of equations yields u and v The value of is the value of the corresponding optical flow vector.

[0030] In a preferred embodiment, the method for processing the optical flow vector using a convolutional neural network in step S04 includes:

[0031] S41: Adjust the optical flow size to 32×32, and process it with 16 3×3 convolutional kernels with a stride of 1 in the first convolutional layer;

[0032] S42: Next, the feature maps are fed into multiple densely connected blocks, and each convolutional layer needs to connect to all preceding layers as input:

[0033] x t = H 1 ([x 0 , x 1 … x t-1 ])

[0034] Where x represents the corresponding convolutional layer, and H represents a non-linear transformation operation, including BN, convolution, activation, and pooling;

[0035] S43: The output of the last densely connected block is classified by a classifier to obtain the result of the second anomaly detection.

[0036] In a preferred embodiment, the method for fusing the first anomaly detection result and the second anomaly detection result in step S05 includes:

[0037] For each of the two streams, a certain number of the top-ranked prediction results are taken, and the average of these results is calculated and sorted to obtain the final video anomaly detection results.

[0038] The present invention also discloses a computer storage medium storing a computer program thereon, which, when executed, implements the above-described video anomaly detection method based on a dual-stream structure.

[0039] This invention also discloses a video anomaly detection system based on a dual-stream structure, comprising:

[0040] The sampling module samples the input video to obtain a frame stream;

[0041] The optical flow generation module generates optical flow vectors based on the acquired frame stream;

[0042] The frame stream analysis and processing module uses 3D convolution kernels to extract spatial and temporal features from the frame stream, specifically including:

[0043] Each frame in the frame stream is taken as a segment, and each segment is convolved with a convolution kernel of a certain size and stride to obtain features through 3D convolution. The features are then compressed into a tensor of a certain size through a pooling layer. Finally, the tensor is used as a feature sequence and input into a unidirectional long short-term memory network for analysis to obtain the final feature vector output. Then, classification is performed to obtain the first anomaly detection result.

[0044] The optical flow analysis and processing module uses a convolutional neural network to process the optical flow vector, expresses the temporal information in the optical flow vector in the form of spatial features, and classifies it to obtain the second anomaly detection result;

[0045] The dual-stream fusion module fuses the first and second anomaly detection results to obtain the final video anomaly detection result.

[0046] In a preferred embodiment, the sampling method of the sampling module includes:

[0047] S11: For the target number of frames K, the sampling rate S = N / K is calculated from the total number of frames N;

[0048] S12: Sample the input video for the first time by extracting 1 frame every S frames, until the end of the video, for a total of M frames.

[0049] S13: Calculate the deletion rate T = M / (MK) based on the difference between the current number of sampled frames M and the target number of frames K.

[0050] S14: Finally, the first sampling result is used as input, and the samples are deleted according to the method of deleting 1 frame per T frames. The deleting sample result is used as the final sampling result and frame stream.

[0051] In a preferred embodiment, the optical flow generation method of the optical flow generation module includes:

[0052] By minimizing the change in pixel grayscale values ​​between two adjacent frames, a cost function based on the optical flow vector of pixel grayscale values ​​is obtained:

[0053] E(u,v) = ∫∫ [(I x u+I y v+I t ) 2 +α(∣∣▽u∣∣ 2 +∣∣▽v∣∣ 2 )]dxdy

[0054] in, u and v These are the two components of the optical flow vector. I x 、I y These are the gradients in the x and y directions of the two images, respectively. I t It is the time derivative, and α is a smoothness parameter. ▽ u and ▽ v represents respectively u and v Gradient in space;

[0055] Taking the derivative of the cost function, we obtain the information about u and v The system of partial differential equations:

[0056] I x (I x u+I y v+I t )+αΔu=0

[0057] I y (I x u+I y v+I t )+αΔv=0

[0058] Solving the system of equations yields u and v The value of is the value of the corresponding optical flow vector.

[0059] In a preferred embodiment, the method for processing optical flow vectors using a convolutional neural network in the optical flow analysis and processing module includes:

[0060] S41: Adjust the optical flow size to 32×32, and process it with 16 3×3 convolutional kernels with a stride of 1 in the first convolutional layer;

[0061] S42: Next, the feature maps are fed into multiple densely connected blocks, and each convolutional layer needs to connect to all preceding layers as input:

[0062] x t = H 1 ([x 0 , x 1 … x t-1 ])

[0063] Where x represents the corresponding convolutional layer, and H represents a non-linear transformation operation, including BN, convolution, activation, and pooling;

[0064] S43: The output of the last densely connected block is classified by a classifier to obtain the result of the second anomaly detection.

[0065] Compared with the prior art, the beneficial effects of the present invention are:

[0066] 1. It realizes the extraction and analysis of spatial and temporal features in frame streams, as well as temporal information in optical flow (expressed in the form of spatial features), which improves the ability of the same model to detect different abnormal events, improves the detection accuracy, optimizes the training speed, saves a lot of human resources, and reduces the fusion cost.

[0067] 2. This method optimizes video preprocessing techniques, achieving more even video sampling results. While retaining the number of sampled frames, it also allows frames from the later or end parts of the video to be sampled, enhancing the information integrity of the sampled frame stream and improving feature extraction capabilities.

[0068] 3. The model has strong versatility and robustness, and the trained model can be deployed in different anomaly detection tasks. Attached Figure Description

[0069] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0070] Picture 1 This is a flowchart of the video anomaly detection method based on a dual-stream structure in this embodiment;

[0071] Picture 2 This is a block diagram illustrating the principle of the video anomaly detection system based on a dual-stream structure in this embodiment.

[0072] Picture 3This is the sampling result under direct sampling in this embodiment;

[0073] Picture 4 This is the optimized sampling result for this embodiment;

[0074] Picture 5 This is the optical flow output result of this embodiment;

[0075] Picture 6 This is the frame stream processing flow for this embodiment;

[0076] Picture 7 This is the optical flow processing flow for this embodiment. Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0078] Invention principle:

[0079] The video is preprocessed to obtain frame streams and corresponding optical flow. Then, a deep learning model is used to analyze and process these two streams separately. Finally, a fusion layer fuses the outputs of the two streams to obtain the final detection result. This method utilizes convolutional neural networks and recurrent neural networks, classic technologies in natural language processing and computer vision, to extract and analyze spatial and temporal features from frame streams, as well as temporal information (expressed as spatial features) from optical flow. This improves detection accuracy, optimizes training speed, and saves significant human resources. Example

[0080] like Picture 1 As shown, a video anomaly detection method based on a dual-stream structure includes the following steps:

[0081] S01: Sample the input video to obtain the frame stream;

[0082] S02: Generate optical flow vectors based on the acquired frame stream;

[0083] S03: Utilizing 3D convolutional kernels to extract spatial and temporal features from the frame stream, specifically including:

[0084] Each frame in the frame stream is taken as a segment, and each segment is convolved with a convolution kernel of a certain size and stride to obtain features through 3D convolution. The features are then compressed into a tensor of a certain size through a pooling layer. Finally, the tensor is used as a feature sequence and input into a unidirectional long short-term memory network for analysis to obtain the final feature vector output. Then, classification is performed to obtain the first anomaly detection result.

[0085] S04: The optical flow vector is processed using a convolutional neural network, and the temporal information in the optical flow vector is expressed in the form of spatial features. The classification results are then used to obtain the second anomaly detection result.

[0086] S05: The first anomaly detection result and the second anomaly detection result are fused to obtain the final video anomaly detection result.

[0087] In a preferred embodiment, the sampling method in step S01 includes:

[0088] S11: For the target number of frames K, the sampling rate S = N / K is calculated from the total number of frames N;

[0089] S12: Sample the input video for the first time by extracting 1 frame every S frames, until the end of the video, for a total of M frames.

[0090] S13: Calculate the deletion rate T = M / (MK) based on the difference between the current number of sampled frames M and the target number of frames K.

[0091] S14: Finally, the first sampling result is used as input, and the samples are deleted according to the method of deleting 1 frame per T frames. The deleting sample result is used as the final sampling result and frame stream.

[0092] In a preferred embodiment, the method for generating optical flow in step S02 includes:

[0093] By minimizing the change in pixel grayscale values ​​between two adjacent frames, a cost function based on the optical flow vector of pixel grayscale values ​​is obtained:

[0094] E(u,v) = ∫∫ [(I x u+I y v+I t ) 2 +α(∣∣▽u∣∣ 2 +∣∣▽v∣∣ 2 )]dxdy

[0095] in, u and v These are the two components of the optical flow vector. I x、I y These are the gradients in the x and y directions of the two images, respectively. I t It is the time derivative, and α is a smoothness parameter. ▽ u and ▽ v represents respectively u and v Gradient in space;

[0096] Taking the derivative of the cost function, we obtain the information about u and v The system of partial differential equations:

[0097] I x (I x u+I y v+I t )+αΔu=0

[0098] I y (I x u+I y v+I t )+αΔv=0

[0099] Solving the system of equations yields u and v The value of is the value of the corresponding optical flow vector.

[0100] In a preferred embodiment, the method for processing the optical flow vector using a convolutional neural network in step S04 includes:

[0101] S41: Adjust the optical flow size to 32×32, and process it with 16 3×3 convolutional kernels with a stride of 1 in the first convolutional layer;

[0102] S42: Next, the feature map is fed into three densely connected blocks, and each convolutional layer needs to be connected to all preceding layers as input:

[0103] x t = H 1 ([x 0 , x 1 … x t-1 ])

[0104] Where x represents the corresponding convolutional layer, and H represents a non-linear transformation operation, including BN, convolution, activation, and pooling;

[0105] S43: The output of the last densely connected block is classified by a classifier to obtain the result of the second anomaly detection.

[0106] In a preferred embodiment, the method for fusing the first anomaly detection result and the second anomaly detection result in step S05 includes:

[0107] For each of the two streams, a certain number of the top-ranked prediction results are taken, and the average of these results is calculated and sorted to obtain the final video anomaly detection results.

[0108] In another embodiment, a computer storage medium stores a computer program that, when executed, implements the above-described video anomaly detection method based on a dual-stream structure.

[0109] In another embodiment, such as Picture 2 As shown, a video anomaly detection system based on a dual-stream structure includes:

[0110] Sampling module 10 samples the input video to obtain a frame stream;

[0111] Optical flow generation module 20 generates optical flow vectors based on the acquired frame stream;

[0112] Frame stream analysis and processing module 30 uses 3D convolution kernels to extract spatial and temporal features from the frame stream, specifically including:

[0113] Each frame in the frame stream is taken as a segment, and each segment is convolved with a convolution kernel of a certain size and stride to obtain features through 3D convolution. The features are then compressed into a tensor of a certain size through a pooling layer. Finally, the tensor is used as a feature sequence and input into a unidirectional long short-term memory network for analysis to obtain the final feature vector output. Then, classification is performed to obtain the first anomaly detection result.

[0114] The optical flow analysis and processing module 40 uses a convolutional neural network to process the optical flow vector, expresses the temporal information in the optical flow vector in the form of spatial features, and classifies it to obtain the second anomaly detection result;

[0115] The dual-stream fusion module 50 fuses the first anomaly detection result and the second anomaly detection result to obtain the final video anomaly detection result.

[0116] The following describes in detail the workflow of a video anomaly detection system based on a preferred embodiment, including the following steps:

[0117] Step 1: Sample the input video.

[0118] The traditional direct sampling method is optimized to ensure that the frame stream information in the later part of the video can be sampled. First, for the target number of frames K, the sampling rate S (S=N / K) is calculated from the total number of frames N. Then, the input video is sampled for the first time by extracting 1 frame every S frames until the end of the video, for a total of M frames (M>=K). Based on the difference between the current number of sampled frames M and the target number of frames K, the deletion rate T (T=M / (MK)) is calculated. Finally, the first sampling result is used as input, and 1 frame is deleted every T frames. The deleted sampling result is used as the final sampling result and frame stream.

[0119] Traditional sampling methods are as follows: Picture 3 As shown, for a video of 47 frames (each square represents 1 frame), assuming the target sampling frame count is 16 and the sampling rate should be 2, the sampling result is: Picture 3 All black frames in the video. Direct sampling may ignore information from later parts of the video, leading to a decrease in detection accuracy.

[0120] The optimized sampling method is as follows: Picture 4 As shown, the sampling rate and target number of sampled frames remain unchanged, and gray frames represent frames deleted according to T after the first sampling. The optimized sampling result, while retaining the number of sampled frames, allows frames from the later or end parts of the video to be sampled, enhancing the information integrity of the sampled frame stream and improving feature extraction capabilities.

[0121] Step 2: Generate optical flow. For two adjacent frames in the video, based on minimizing the change in pixel grayscale values ​​between the two frames, a cost function for the optical flow vector based on pixel grayscale values ​​is obtained:

[0122] E(u,v) = ∫∫[(I x u+I y v+I t ) 2 +α(∣∣▽u∣∣ 2 +∣∣▽v∣∣ 2 )]dxdy

[0123] Where u and v are two components of the optical flow vector, I x I y I t These are the gradients and temporal derivatives of the two images in the x and y directions, respectively, and α is a smoothness parameter. ▽ u and ▽ v represents the gradients of u and v in space, respectively. Taking the derivative of this function yields a system of partial differential equations in u and v:

[0124] I x (I x u+I y v+I t ) + αΔu = 0

[0125] I y (I x u+I y v+I t ) + αΔv = 0

[0126] Finally, solving the system of equations yields the values ​​of u and v, which are the corresponding values ​​of the optical flow vector, such as... Picture 5 As shown.

[0127] Step 3: Process the frame stream using 3D convolution and long short-term memory networks, such as... Picture 6 As shown, this method utilizes 3D convolutional kernels to extract spatial and temporal features from the frame stream. Each 64 frames in the stream are treated as a clip, and each clip is convolved using a 3×3×3 kernel with a stride of 1. The 3D convolution process yields 6×30×30×32 features. These features are then compressed using a 2×2×2 pooling layer to obtain a 3×15×15×32 tensor. Finally, this tensor is used as a feature sequence for further analysis by a unidirectional long short-term memory network to obtain the final feature vector output. Finally, classification is performed using fully connected layers and softmax to obtain the video anomaly detection results.

[0128] Step 4: Process the optical flow using the DenseNet convolutional neural network, such as... Picture 7 As shown. Based on the characteristic that temporal information in optical flow is presented in the form of spatial features, a convolutional network with lower computational requirements is used to process the optical flow. First, the optical flow size is adjusted to 32×32. In the first convolutional layer, 16 convolutional kernels of 3×3 with a stride of 1 are used to process it. Then, the feature map is fed into 3 densely connected blocks, and each convolutional layer needs to connect to all previous layers as input:

[0129] x t = H1([x0, x1… x t-1 ])

[0130] Here, x represents the corresponding convolutional layer, and H represents a non-linear transformation operation, including batch normalization (BN), convolution, activation, and pooling. In this scheme, the feature map sizes for each block are 32×32, 16×16, and 8×8, respectively. Finally, a fully connected layer and a softmax classifier are used to classify the output of the last densely connected block to obtain the anomaly detection result.

[0131] Step 5: Fuse the outputs of the two streams. Since the feature maps of frame streams and optical flow are difficult to fuse, and considering computational limitations, an averaging method is used to directly fuse the results of the two streams. The top 5 predictions from each stream are taken, the average prediction value is calculated from these results, and the results are sorted to obtain the final anomaly detection result.

[0132] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A video anomaly detection method based on a dual-stream structure, characterized in that, Includes the following steps: S01: Sample the input video to obtain the frame stream; S02: Generate optical flow vectors based on the acquired frame stream; S03: Utilizing 3D convolutional kernels to extract spatial and temporal features from the frame stream, specifically including: Each frame in the frame stream is taken as a segment, and each segment is convolved with a convolution kernel of a certain size and stride to obtain features through 3D convolution. The features are then compressed into a tensor of a certain size through a pooling layer. Finally, the tensor is used as a feature sequence and input into a unidirectional long short-term memory network for analysis to obtain the final feature vector output. Then, classification is performed to obtain the first anomaly detection result. S04: The optical flow vector is processed using a convolutional neural network to express the temporal information in the optical flow vector in the form of spatial features, and the second anomaly detection result is obtained by classification; the method of processing the optical flow vector using a convolutional neural network includes: S41: Adjust the optical flow size to 32×32, and process it with 16 3×3 convolutional kernels with a stride of 1 in the first convolutional layer; S42: Next, the feature maps are fed into multiple densely connected blocks, and each convolutional layer needs to connect to all preceding layers as input: x t = H 1 ([x 0 , x 1 … x t-1 ]) Where x represents the corresponding convolutional layer, and H represents a non-linear transformation operation, including BN, convolution, activation, and pooling; S43: The output of the last densely connected block is classified by a classifier to obtain the result of the second anomaly detection; S05: The first anomaly detection result and the second anomaly detection result are fused to obtain the final video anomaly detection result.

2. The video anomaly detection method based on a dual-stream structure according to claim 1, characterized in that, The sampling method in step S01 includes: S11: For the target number of frames K, the sampling rate S = N / K is calculated from the total number of frames N; S12: Sample the input video for the first time by extracting 1 frame every S frames, until the end of the video, for a total of M frames. S13: Calculate the deletion rate T = M / (MK) based on the difference between the current number of sampled frames M and the target number of frames K. S14: Finally, the first sampling result is used as input, and the samples are deleted according to the method of deleting 1 frame per T frames. The deleting sample result is used as the final sampling result and frame stream.

3. The video anomaly detection method based on a dual-stream structure according to claim 1, characterized in that, The method for generating optical flow in step S02 includes: By minimizing the change in pixel grayscale values ​​between two adjacent frames, a cost function based on the optical flow vector of pixel grayscale values ​​is obtained: E(u,v) = [(I x u+I y v+I t ) 2 +α(∣∣ u∣∣ 2 +∣∣ v∣∣ 2 )]dxdy in, u and v These are the two components of the optical flow vector. I x 、I y These are the gradients in the x and y directions of the two images, respectively. I t It is the time derivative, and α is a smoothness parameter. u and v represents respectively u and v Gradient in space; Taking the derivative of the cost function, we obtain the following about u and v The system of partial differential equations: I x (I x u+I y v+I t )+αΔu=0 I y (I x u+I y v+I t )+αΔv=0 Solving the system of equations yields u and v The value of is the value of the corresponding optical flow vector.

4. The video anomaly detection method based on a dual-stream structure according to claim 1, characterized in that, The method for fusing the first anomaly detection result and the second anomaly detection result in step S05 includes: For each of the two streams, a certain number of the top-ranked prediction results are taken, and the average of these results is calculated and sorted to obtain the final video anomaly detection results.

5. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the video anomaly detection method based on a dual-stream structure as described in any one of claims 1-4.

6. A video anomaly detection system based on a dual-stream structure, characterized in that, include: The sampling module samples the input video to obtain a frame stream; The optical flow generation module generates optical flow vectors based on the acquired frame stream; The frame stream analysis and processing module uses 3D convolution kernels to extract spatial and temporal features from the frame stream, specifically including: Each frame in the frame stream is taken as a segment, and each segment is convolved with a convolution kernel of a certain size and stride to obtain features through 3D convolution. The features are then compressed into a tensor of a certain size through a pooling layer. Finally, the tensor is used as a feature sequence and input into a unidirectional long short-term memory network for analysis to obtain the final feature vector output. Then, classification is performed to obtain the first anomaly detection result. The optical flow analysis and processing module uses a convolutional neural network to process the optical flow vector, expressing the temporal information in the optical flow vector in the form of spatial features, and classifying it to obtain the second anomaly detection result; the method of processing the optical flow vector using a convolutional neural network in step S04 includes: S41: Adjust the optical flow size to 32×32, and process it with 16 3×3 convolutional kernels with a stride of 1 in the first convolutional layer; S42: Next, the feature maps are fed into multiple densely connected blocks, and each convolutional layer needs to connect to all preceding layers as input: x t = H 1 ([x 0 , x 1 … x t-1 ]) Where x represents the corresponding convolutional layer, and H represents a non-linear transformation operation, including BN, convolution, activation, and pooling; S43: The output of the last densely connected block is classified by a classifier to obtain the result of the second anomaly detection; The dual-stream fusion module fuses the first and second anomaly detection results to obtain the final video anomaly detection result.

7. The video anomaly detection system based on a dual-stream structure according to claim 6, characterized in that, The sampling method of the sampling module includes: S11: For the target number of frames K, the sampling rate S = N / K is calculated from the total number of frames N; S12: Sample the input video for the first time by extracting 1 frame every S frames, until the end of the video, for a total of M frames. S13: Calculate the deletion rate T = M / (MK) based on the difference between the current number of sampled frames M and the target number of frames K. S14: Finally, the first sampling result is used as input, and the samples are deleted according to the method of deleting 1 frame per T frames. The deleting sample result is used as the final sampling result and frame stream.

8. The video anomaly detection system based on a dual-stream structure according to claim 6, characterized in that, The optical flow generation method of the optical flow generation module includes: By minimizing the change in pixel grayscale values ​​between two adjacent frames, a cost function based on the optical flow vector of pixel grayscale values ​​is obtained: E(u,v) = [(I x u+I y v+I t ) 2 +α(∣∣ u∣∣ 2 +∣∣ v∣∣ 2 )]dxdy in, u and v These are the two components of the optical flow vector. I x 、I y These are the gradients in the x and y directions of the two images, respectively. I t It is the time derivative, and α is a smoothness parameter. u and v represents respectively u and v Gradient in space; Taking the derivative of the cost function, we obtain the following about u and v The system of partial differential equations: I x (I x u+I y v+I t )+αΔu=0 I y (I x u+I y v+I t )+αΔv=0 Solving the system of equations yields u and v The value of is the value of the corresponding optical flow vector.

Images