A weakly supervised video anomaly detection method, device, equipment and storage medium

Abstract

The application relates to the field of video detection, and discloses a weakly supervised video anomaly detection method, device and equipment and a storage medium. The method comprises the following steps: extracting video features; constructing an anomaly judgment model comprising a multi-scale time feature fusion module, a multi-scale space feature fusion module and a video anomaly score module; training the anomaly judgment model by using the extracted video features; performing feature extraction on a to-be-detected video and inputting the to-be-detected video into the trained anomaly judgment model to perform anomaly judgment. In this way, the convergence degree of the model and the learning range of the model on the video are increased by fusing the multi-scale time and space features of the video, the understanding ability of the model on the video features is improved, sufficient video information can be learned, the model is more robust, the model can better utilize the existing features without changing the number of training data, the video can be automatically judged for anomaly, and the efficiency of video anomaly detection is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of video detection, and in particular to a method, apparatus, device, and storage medium for detecting anomalies in weakly supervised video. Background Technology

[0002] Anomalies in videos are broadly defined as unusual behaviors or events, i.e., behaviors or events that occur infrequently. Video anomaly detection mainly involves determining whether selected video segments contain anomalies and locating the time period in which the anomaly occurs.

[0003] Because normal and abnormal behaviors in reality are complex and diverse, they cannot be exhaustively listed; the boundary between normal and abnormal behaviors is fuzzy, meaning that the definition of abnormality is uncertain in different scenarios, making it difficult for anomaly detection systems to generalize. Furthermore, due to the massive amount of video data, obtaining accurate frame-level and pixel-level anomaly annotations is extremely expensive. There is also an imbalance between positive and negative samples, meaning that the number of abnormal samples is far less than the number of normal samples, making it difficult to obtain a sufficient number of anomalies.

[0004] Therefore, how to effectively detect video anomalies is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a weakly supervised video anomaly detection method, apparatus, device, and storage medium, which can automatically identify anomalies in videos through an anomaly detection model, greatly improving the efficiency of video anomaly detection. The specific solution is as follows:

[0006] A weakly supervised video anomaly detection method includes:

[0007] Extract video features;

[0008] Construct an anomaly detection model that includes a multi-scale temporal feature fusion module, a multi-scale spatial feature fusion module, and a video anomaly scoring module;

[0009] The anomaly detection model is trained using the extracted video features;

[0010] Feature extraction is performed on the video to be tested, and the feature is input into the trained anomaly detection model for anomaly detection.

[0011] Preferably, in the weakly supervised video anomaly detection method provided in the embodiments of the present invention, extracting video features includes:

[0012] The video is segmented to obtain multiple video clips;

[0013] The video features of the video segment are extracted using a pre-trained I3D model.

[0014] Preferably, in the weakly supervised video anomaly detection method provided in the embodiments of the present invention, the training process of the multi-scale temporal feature fusion module includes:

[0015] The extracted video features are subjected to at least two average pooling operations to obtain the temporal features of the video at different scales.

[0016] Preferably, in the weakly supervised video anomaly detection method provided in the embodiments of the present invention, the training process of the multi-scale spatial feature fusion module includes:

[0017] Perform at least two spatial convolutions at different scales on each of the obtained time features to generate at least two feature data obtained by convolution at different scales;

[0018] The generated feature data are spliced ​​together to obtain video feature data that integrates multiple scales and temporal and spatial dimensions.

[0019] Preferably, in the weakly supervised video anomaly detection method provided in the embodiments of the present invention, the training process for the video anomaly scoring module includes:

[0020] Each dimension of the video feature data at different scales is regressed using three fully connected layers to obtain the video segment score;

[0021] Within each scale, the score with the highest score is selected as the score for that scale.

[0022] The average of the maximum scores at different scales yields the video's anomaly score.

[0023] Preferably, in the weakly supervised video anomaly detection method provided in the embodiments of the present invention, while training the anomaly judgment model, the method further includes:

[0024] The sum of sparse loss, smoothing loss, cross-entropy loss, and spatiotemporal feature loss based on KL divergence is used as the loss function of the anomaly detection model.

[0025] Preferably, in the weakly supervised video anomaly detection method provided in the embodiments of the present invention, feature extraction of the video to be tested and input into the trained anomaly judgment model for anomaly judgment include:

[0026] The video to be tested is segmented to obtain multiple video segments.

[0027] Features of the video segment to be tested are extracted using a pre-trained I3D model;

[0028] The features of the video segment to be tested are input into the trained anomaly detection model, and the multi-scale temporal and spatial features are fused to obtain the anomaly score of the video segment to be tested.

[0029] This invention also provides a weakly supervised video anomaly detection device, comprising:

[0030] The feature extraction unit is used to extract video features;

[0031] The model building unit is used to build an anomaly detection model that includes a multi-scale temporal feature fusion module, a multi-scale spatial feature fusion module, and a video anomaly scoring module.

[0032] The model training unit is used to train the anomaly detection model using the extracted video features;

[0033] The model inference unit is used to extract features from the video under test and input them into the trained anomaly detection model for anomaly detection.

[0034] This invention also provides a weakly supervised video anomaly detection device, including a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the weakly supervised video anomaly detection method described above in this invention.

[0035] This invention also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the weakly supervised video anomaly detection method described above in this invention.

[0036] As can be seen from the above technical solution, the weakly supervised video anomaly detection method provided by the present invention includes: extracting video features; constructing an anomaly judgment model including a multi-scale temporal feature fusion module, a multi-scale spatial feature fusion module, and a video anomaly scoring module; training the anomaly judgment model using the extracted video features; and extracting features from the video to be tested and inputting them into the trained anomaly judgment model for anomaly judgment.

[0037] The weakly supervised video anomaly detection method provided by this invention increases the model's learning range and convergence by fusing multi-scale temporal and spatial features of the video, thereby improving the model's understanding of video features. It can learn enough video information, making the model more robust. Furthermore, without changing the amount of training data, the model can better utilize existing features and automatically judge anomalies in the video, greatly improving the efficiency of video anomaly detection.

[0038] Furthermore, the present invention also provides a corresponding device, equipment, and computer-readable storage medium for weakly supervised video anomaly detection methods, further making the above methods more practical. The device, equipment, and computer-readable storage medium have corresponding advantages. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 A flowchart of a weakly supervised video anomaly detection method provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the architecture of the weakly supervised video anomaly detection method provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram illustrating the training of a multi-scale temporal feature fusion module according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram illustrating the training of a multi-scale spatial feature fusion module according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram illustrating the training of the video anomaly scoring module according to an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of the structure of the weakly supervised video anomaly detection device provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] This invention provides a weakly supervised video anomaly detection method, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0048] S101. Extract video features;

[0049] In specific implementation, step S101 extracts video features, which may include: segmenting the video to obtain multiple video segments; and using a pre-trained I3D model to extract video features from the video segments.

[0050] Specifically, the above steps can be understood as preprocessing the video, namely: first, converting the video into video frames, then cropping the video frames, and using a pre-trained I3D model to extract video features from the cropped portions. The cropped portions can be understood as video segments divided from a video. In this invention, each segment is a multiple of 4. For example, every 16 consecutive frames can be considered a video segment, and a 1*2048 dimension feature vector can be extracted from every 16 consecutive frames.

[0051] For a complete video input dimension of T*W (T is the number of frames, H is the video frame height, and W is the video frame width), the video feature dimension obtained after video preprocessing steps is T / 16*2048.

[0052] S102. Construct an anomaly detection model that includes a multi-scale temporal feature fusion module, a multi-scale spatial feature fusion module, and a video anomaly scoring module;

[0053] It should be noted that this invention introduces a multi-scale spatiotemporal feature fusion learning mechanism. By utilizing the temporal and spatial fusion of video features at multiple scales in the feature extraction part, the robustness of the model can be effectively improved.

[0054] S103. Train the anomaly detection model using the extracted video features;

[0055] It should be noted that the model training in this invention employs a multiple instance learning method, specifically utilizing multiple instance learning to achieve weakly supervised detection. In machine learning, multiple instance learning is a method evolved from supervised learning algorithms, defining a "bag" as a collection of multiple instances, and it has wide applications. Instead of receiving a set of individually labeled instances, the learner receives a set of labeled bags, each containing multiple instances. In the simple case of multiple instance binary classification, a bag can be labeled as negative if all instances in it are normal. Conversely, a bag is labeled as positive if at least one instance in it is anomalous.

[0056] In this invention, a complete video is considered a packet. The video is segmented, and the resulting video segments are multiple instances within the packet. A packet containing instances of unusual behavior or events is a positive packet; a packet without instances of unusual behavior or events is a negative packet. The feature dimension of a video packet in this invention can be 4n*2048.

[0057] S104. Extract features from the video to be tested and input them into the trained anomaly detection model for anomaly detection.

[0058] In the weakly supervised video anomaly detection method provided in the embodiments of the present invention, by fusing multi-scale temporal and spatial features of the video, the learning range and convergence degree of the model are increased, the model's ability to understand video features is improved, and it can learn enough video information, making the model more robust. Furthermore, without changing the amount of training data, the model can better utilize existing features and automatically judge anomalies in the video, greatly improving the efficiency of video anomaly detection.

[0059] In specific implementation, the weakly supervised video anomaly detection method provided in the embodiments of the present invention may include the following during the training of the multi-scale temporal feature fusion module: performing at least two average pooling operations on the extracted video features to obtain the temporal features of the video at different scales.

[0060] like Figure 3 As shown, a video packet input with 4n*2048 dimension feature data is the first scale. Two average pooling operations yield two different scales. Scale 2 uses 2n*2048 dimension feature data, and scale 3 uses n*2048 dimension feature data. These two average pooling operations produce the temporal features of the video at different scales.

[0061] In specific implementation, the weakly supervised video anomaly detection method provided in this embodiment of the invention, during the training of the multi-scale spatial feature fusion module, may specifically include: first, performing at least two spatial convolutions at different scales on each obtained temporal feature to generate at least two feature data obtained from convolutions at different scales; then, concatenating the generated feature data to obtain video feature data that integrates multi-scale temporal and spatial dimensions. This multi-scale spatiotemporal fusion of video features allows the model to understand more temporal and spatial semantic information during the score generation process.

[0062] like Figure 4 As shown, two spatial convolutions at different scales are performed on the video segment features at three scales. Taking scale 3 as an example, two n*1024 feature data points obtained from convolutions at different scales are obtained. The two n*1024 feature data points are concatenated to obtain feature data with a temporal and spatial dimension of n*2048 that incorporates the video.

[0063] In specific implementation, the weakly supervised video anomaly detection method provided in the embodiments of the present invention may include the following steps during the training of the video anomaly scoring module: regressing each dimension of the video feature data at different scales using three fully connected layers to obtain video segment scores; selecting the score with the highest score at each scale as the score at that scale; and averaging the highest scores at different scales to obtain the video anomaly score.

[0064] like Figure 5 As shown, for each dimension of the video feature data at the three scales obtained in the previous steps (i.e., feature data with a dimension of 1*2048), three fully connected layers are used to finally regress to a 1*1 video segment score, with each score ranging from 0 to 1. Within each scale, the score with the highest score is selected as the score for that scale. The average of the highest scores across the three scales is then used to obtain the anomaly score for the video. When the anomaly score is greater than 0.5, the example is set as an anomaly label 1; otherwise, it is set as an anomaly label 0.

[0065] In specific implementation, the weakly supervised video anomaly detection method provided in the embodiments of the present invention may further include, while training the anomaly judgment model, using the sum of sparse loss, smoothing loss, cross-entropy loss and spatiotemporal feature loss based on KL divergence as the loss function of the anomaly judgment model.

[0066] Specifically, sparse loss refers to constraining the abnormal sparsity in the loss function, and the specific formula is as follows:

[0067]

[0068] Among them, l sp (x) represents the sparse loss, x represents the video anomaly score, m represents the number of instances contained in the video, N represents the set batch size, and n represents the number of samples.

[0069] Smoothing loss refers to applying a smoothing constraint to outlier scores in a video, aiming to prevent abrupt score changes and make the scores more continuous. The specific formula is as follows:

[0070]

[0071] Among them, l sm (x) represents the smoothing loss.

[0072] Cross-entropy is a way to measure the difference between a predicted value and the actual value, and the formula is as follows:

[0073]

[0074] Among them, l ce (x) is the cross-entropy loss, xi For the i-th input, y i To predict the score, l i This is the actual score.

[0075] After fusing multi-scale temporal and spatial features, three different scales of video segment features are obtained. The video frame corresponding to the feature represented by each segment at scale 3 has the largest value among the three scales. When scoring each video packet, the segment with the highest score at scale 3 after spatiotemporal feature fusion is used. This segment is obtained by pooling four segments from scale 1 into two segments from scale 2, and then pooling them back to scale 3. Therefore, the segment with the highest score corresponds to two scale 2 segments and four scale 1 segments. The two segments from scale 2 and the four scale 1 segments are scored separately and averaged to obtain three scores for the scale 3 segment corresponding to the scale 2 and scale 1 segments, named max_score3, max_score2, and max_score1, respectively. The spatiotemporal feature loss is achieved by approximating the three scores using the KL divergence formula, as follows:

[0076]

[0077]

[0078] Among them, l kl (x) represents the spatiotemporal feature loss based on KL divergence proximity, q represents the prediction score divided into 8 segments, P1 represents the prediction score divided into 16 segments, and P2 represents the prediction score divided into 32 segments.

[0079] The formula is used twice in the function of this invention. The first time, P(x) is the score of max_score3 and q(x) is the score of max_score1. The second time, P(x) is the score of max_score3 and q(x) is the score of max_score2.

[0080] Therefore, the final loss function of this invention is L = l sp +l sm +l ce +l kl .

[0081] In specific implementation, in the weakly supervised video anomaly detection method provided in the embodiments of the present invention, step S104 extracts features from the video to be tested and inputs them into a trained anomaly judgment model for anomaly judgment. Specifically, it may include: segmenting the video to be tested to obtain multiple video segments; extracting features from the video segments using a pre-trained I3D model; inputting the features of the video segments into the trained anomaly judgment model, fusing multi-scale temporal and spatial features, and obtaining anomaly scores for the video to be tested.

[0082] This invention performs multi-scale spatiotemporal feature fusion on video features after feature extraction, increasing the semantic representation of the features. The fused spatiotemporal features are then used in the loss function to train the model. Without changing the training data content, this improves the semantic representation of the extracted features, allowing the model to learn sufficient video information even with limited training data. Furthermore, the use of multi-scale spatiotemporal features in the loss function further enhances model convergence, making the model more robust. Testing shows that the frame-level accuracy of the weakly supervised video anomaly detection method provided by this invention reaches 97.62%.

[0083] Based on the same inventive concept, this invention also provides a weakly supervised video anomaly detection device. Since the principle of this device in solving the problem is similar to that of the aforementioned weakly supervised video anomaly detection method, the implementation of this device can refer to the implementation of the weakly supervised video anomaly detection method, and the repeated parts will not be described again.

[0084] In specific implementation, the weakly supervised video anomaly detection device provided in this embodiment of the invention, such as... Figure 6 As shown, it specifically includes:

[0085] Feature extraction unit 11 is used to extract video features;

[0086] Model building unit 12 is used to build an anomaly judgment model that includes a multi-scale temporal feature fusion module, a multi-scale spatial feature fusion module, and a video anomaly scoring module;

[0087] Model training unit 13 is used to train the anomaly detection model using the extracted video features;

[0088] The model inference unit 14 is used to extract features from the video under test and input them into the trained anomaly detection model for anomaly detection.

[0089] In the weakly supervised video anomaly detection device provided in this embodiment of the invention, the interaction of the four units can fuse features of the video at multiple scales in time and space, increase the model's learning range and convergence, improve the model's understanding of video features, learn sufficient video information, make the model more robust, and enable the model to better utilize existing features and automatically judge anomalies in the video without changing the amount of training data, thus greatly improving the efficiency of video anomaly detection.

[0090] For more detailed information on the working process of each of the above units, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0091] Accordingly, this invention also discloses a weakly supervised video anomaly detection device, including a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the weakly supervised video anomaly detection method disclosed in the foregoing embodiments.

[0092] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0093] Furthermore, the present invention also discloses a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, it implements the aforementioned weakly supervised video anomaly detection method.

[0094] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0096] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0098] In summary, the weakly supervised video anomaly detection method provided by this invention includes: extracting video features; constructing an anomaly judgment model comprising a multi-scale temporal feature fusion module, a multi-scale spatial feature fusion module, and a video anomaly scoring module; training the anomaly judgment model using the extracted video features; and extracting features from the test video and inputting them into the trained anomaly judgment model for anomaly judgment. This weakly supervised video anomaly detection method, by fusing multi-scale temporal and spatial features of the video, increases the model's learning range and convergence, improves the model's understanding of video features, and allows it to learn sufficient video information, making the model more robust. Furthermore, without changing the amount of training data, the model can better utilize existing features to automatically judge video anomalies, greatly improving the efficiency of video anomaly detection. In addition, this invention also provides corresponding devices, equipment, and computer-readable storage media for the weakly supervised video anomaly detection method, further enhancing the practicality of the method. These devices, equipment, and computer-readable storage media have corresponding advantages.

[0099] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The weakly supervised video anomaly detection method, apparatus, device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A weakly supervised video anomaly detection method, characterized in that, include: Extract video features; Construct an anomaly detection model that includes a multi-scale temporal feature fusion module, a multi-scale spatial feature fusion module, and a video anomaly scoring module; The anomaly detection model is trained using the extracted video features; The process of training the multi-scale temporal feature fusion module includes: performing at least two average pooling operations on the extracted video features to obtain the temporal features of the video at different scales. The training process of the multi-scale spatial feature fusion module includes: performing at least two spatial convolutions on each of the obtained temporal features to generate at least two feature data obtained by convolution at different scales, and splicing the generated feature data to obtain video feature data that fuses multiple scale temporal and spatial dimensions. The training process for the video anomaly scoring module includes: performing regression on each dimension of the video feature data at different scales using three fully connected layers to obtain a regression of 1. The video segment score is 1, with each score ranging from 0 to 1; within each scale, the score with the highest score is selected as the score for that scale; the average of the highest scores across different scales is used to obtain the video's anomaly score. Feature extraction is performed on the video to be tested, and the feature is input into the trained anomaly detection model for anomaly detection.

2. The weakly supervised video anomaly detection method according to claim 1, characterized in that, Extracting video features, including: The video is segmented to obtain multiple video clips; The video features of the video segment are extracted using a pre-trained I3D model.

3. The weakly supervised video anomaly detection method according to claim 2, characterized in that, Training the anomaly detection model also includes: The sum of sparse loss, smoothing loss, cross-entropy loss, and spatiotemporal feature loss based on KL divergence is used as the loss function of the anomaly detection model.

4. The weakly supervised video anomaly detection method according to claim 3, characterized in that, Feature extraction is performed on the video to be tested and input into the trained anomaly detection model for anomaly detection, including: The video to be tested is segmented to obtain multiple video segments. Features of the video segment to be tested are extracted using a pre-trained I3D model; The features of the video segment to be tested are input into the trained anomaly detection model, and the multi-scale temporal and spatial features are fused to obtain the anomaly score of the video segment to be tested.

5. A weakly supervised video anomaly detection device, characterized in that, include: The feature extraction unit is used to extract video features; The model building unit is used to build an anomaly detection model that includes a multi-scale temporal feature fusion module, a multi-scale spatial feature fusion module, and a video anomaly scoring module. The model training unit is used to train the anomaly detection model using the extracted video features. Specifically, the training of the multi-scale temporal feature fusion module includes: performing at least two average pooling operations on the extracted video features to obtain temporal features at different scales; the training of the multi-scale spatial feature fusion module includes: performing at least two spatial convolutions at different scales on each of the obtained temporal features to generate at least two feature data obtained from convolutions at different scales, and concatenating the generated feature data to obtain video feature data that integrates multi-scale temporal and spatial dimensions; and the training of the video anomaly scoring module includes: performing regression on each dimension of the video feature data at different scales using three fully connected layers to obtain a regression of 1. The video segment score is 1, with each score ranging from 0 to 1; within each scale, the score with the highest score is selected as the score for that scale; the average of the highest scores across different scales is used to obtain the video's anomaly score. The model inference unit is used to extract features from the video under test and input them into the trained anomaly detection model for anomaly detection.

6. A weakly supervised video anomaly detection device, characterized in that, It includes a processor and a memory, wherein the processor implements the weakly supervised video anomaly detection method as described in any one of claims 1 to 4 when executing a computer program stored in the memory.

7. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the weakly supervised video anomaly detection method as described in any one of claims 1 to 4.

Images