A Spatiotemporal Detection Method for Weakly Supervised Video Anomaly Behavior Based on Pre-trained Large Models

Abstract

This invention relates to a weakly supervised spatiotemporal detection method for abnormal video behavior based on a pre-trained large model, belonging to the field of image detection. The method involves constructing an ST-VadCLIP model; inputting training videos into the ST-VadCLIP model for training; using a frozen CLIP image encoder to extract global image features and spatial local image features, and using a frozen CLIP text encoder to extract global and local text label features; performing attention aggregation on the local image features and fusing them with the global image features to obtain video frame-level features; feeding the video frame-level features into a binary classification auxiliary detection head for coarse classification, and then calculating the cosine similarity with the global text label features to obtain the frame-level temporal anomaly confidence; selecting frames where anomalies occur based on the frame-level temporal anomaly confidence, calculating the cosine similarity between the spatial local image features and local text label features of these frames, and combining this with prior rules for abnormal activity motion metrics to obtain the spatial localization result of the abnormal event.

Description

Technical Field

[0001] This invention relates to the field of video abnormal behavior detection technology, and to a method for spatiotemporal detection of video abnormal behavior, specifically a weakly supervised spatiotemporal detection method for video abnormal behavior based on a pre-trained large model. Background Technology

[0002] In recent years, weakly supervised video anomaly detection has attracted increasing attention due to its broad application prospects. For example, with the help of video anomaly detection algorithms, more powerful intelligent video surveillance systems and video content review systems can be developed. The core requirement of weakly supervised video anomaly detection is that the anomaly detector should generate frame-level anomaly confidence scores based on only video-level annotations. Currently, most research in this field follows the following process: the initial step is to extract frame-level features using a pre-trained visual model, such as a 3D convolutional neural network; then, these features are input into a binary classifier based on multi-instance learning for model training; and the final step is to use the anomaly confidence scores predicted by the model to detect anomalous behavior. For example, the methods in Sultani et al.'s paper "Sultani W, Chen C, Shah M. Real-world anomaly detection in surveillance videos. IEEE conference on computer vision and pattern recognition. 2018" and Zhou et al.'s paper "Zhou H, Yu J, Yang W. Dual memory units with uncertainty regulation for weakly supervised video anomaly detection. AAAI conference on artificial intelligence. 2023" are both based on the above process. With the development of large-scale visual-language models, methods using these models in weakly supervised video anomaly detection have gradually emerged, such as CLIP-TSA, which uses the CLIP model to extract image features, and VadCLIP, which uses CLIP for visual-text alignment to retrieve anomalies. Previous weakly supervised video anomaly detection tasks assumed that only video-level labels were used for model training, enabling the model to detect which frames in the video exhibited anomalies. Compared to weakly supervised video anomaly detection, spatiotemporal detection of weakly supervised video anomalies is a more challenging task. This task assumes that the model should detect anomalies at a finer level, i.e., in the spatial dimension, while keeping the supervision signal unchanged. This requires the model not only to detect frames where anomalies occur in the time series but also to locate the spatial positions of the anomalous events. Summary of the Invention

[0003] The technical problem to be solved by this invention is:

[0004] To avoid the shortcomings of existing technologies, this invention provides a spatiotemporal detection method for weakly supervised video abnormal behavior based on a pre-trained large model, in order to solve the problem that existing technologies cannot effectively apply large models containing rich knowledge to the video domain.

[0005] According to embodiments of this disclosure, a spatiotemporal detection method for weakly supervised video anomaly behavior based on a pre-trained large model is provided, the method comprising:

[0006] An ST-VadCLIP model is constructed, comprising a CLIP image encoder, a CLIP text encoder, a local attention focusing module, a temporal modeling module, a temporal localization module, and a spatial localization module; wherein...

[0007] The CLIP image encoder and the CLIP text encoder are respectively frozen CLIP image encoder and frozen CLIP text encoder. The temporal modeling module includes a temporal Transformer encoder with fixed distance relationship attention weights. The local attention focusing module generates attention-weighted focusing local image features by measuring the degree of motion change in local regions. The temporal localization module includes a binary classification auxiliary detection head and a text label feature similarity detection head. The spatial localization module uses local text description labels and motion change priors to jointly locate abnormal behavior regions.

[0008] The training video is input into the ST-VadCLIP model for training. The global and local image features of the training video are extracted using the frozen CLIP image encoder, and the label features of the global text labels and local localized text labels corresponding to the training video are extracted using the frozen CLIP text encoder.

[0009] The image features are processed using the local attention aggregation and the temporal modeling to obtain video frame-level features;

[0010] Using video frame-level features and global text labels with trainable cue parameters, cosine similarity is calculated to obtain anomaly confidence. At the same time, a binary classification auxiliary detection head is set to help the network distinguish between normal and abnormal video frame-level features.

[0011] Frames that may exhibit abnormal behavior are located using anomaly confidence. The spatial localization module is used to locate the spatial location of the abnormal behavior, including calculating cosine similarity using local image features and local text label features, and combining motion change priors to jointly locate the location of the abnormal behavior.

[0012] The prediction results for the training video are obtained based on the video frame-level features and the text label features.

[0013] The local image feature aggregation module of this disclosure is used to focus image features of abnormal regions from different spatial scales; the temporal modeling module improves the performance of the Transformer structure in weakly supervised video anomaly detection by fixing the prior distance attention weight; and the frozen ST-VadCLIP model is effectively adapted to the spatiotemporal detection task of weakly supervised video abnormal behavior by combining the normal and abnormal description labels generated by the large language model with the prior rules of abnormal activity motion measurement.

[0014] The step of extracting image features from the training video using the frozen CLIP image encoder further includes:

[0015] Every preset frame, one frame is sampled and sent to the CLIP image encoder to obtain the image features.

[0016] Each complete image frame, when passed through the CLIP image encoder, is divided into multiple non-overlapping blocks representing each region of the image according to the principle of Vision Transformer. When extracting global image features, all blocks are input into the CLIP image encoder. When extracting local image features, the blocks representing the specified region are input into the CLIP image encoder to obtain the image features representing the local region.

[0017] The step of extracting label features of text labels corresponding to the training video using the frozen CLIP text encoder includes:

[0018] The text labels are converted into category vectors using a tokenizer;

[0019] The category vectors are concatenated with learnable cues to obtain a vector sequence;

[0020] The CLIP text encoder extracts the vector sequence to obtain the tag features.

[0021] The step of processing the local image features using the local attention aggregation to obtain video frame-level features includes:

[0022] The local image features are compressed and aggregated using the local attention aggregation.

[0023] The video frame-level features are obtained by adding the local aggregated image features and the global image features.

[0024] The temporal Transformer encoder calculates attention weights by the distance between two frames, rather than using the dot product attention of the traditional Transformer.

[0025] The timing positioning module includes:

[0026] The aggregated video frame-level features are compared with the global text labels to calculate the cosine similarity, and then the anomaly confidence is obtained by passing the Softmax normalization function.

[0027] The binary classification auxiliary detection head includes a fully connected layer with one neuron.

[0028] The spatial positioning module includes:

[0029] Based on the anomaly confidence level, a threshold is set. If the threshold is exceeded, it indicates that there is abnormal behavior in the frame, and it is extracted to the spatial positioning module.

[0030] The cosine similarity between the local image features of possible anomalous frames and the local text description features obtained through a large language model is calculated to obtain the spatial anomaly confidence.

[0031] Local image features contain two scales, while local text description features contain parts describing abnormal behavior and parts describing things that might appear in a normal scene;

[0032] Motion change prior is used to calculate the degree of change of each local block compared to the blocks at the same position in the two frames before and after it. A threshold is set to filter blocks with small changes and the anomaly confidence of these blocks is set to 0.

[0033] The bounding box of anomalous behavior is located based on the spatial anomaly confidence result.

[0034] The method for calculating the degree of change in each block includes:

[0035] Select a local block location, calculate the L2 norm by subtracting the features of the block from the features of the blocks in the previous and next frames;

[0036] This result is used to measure the degree of change in the current region. If it is less than the threshold, it means that the current region has not changed much in these three frames.

[0037] The method also includes:

[0038] The test video is input into the trained ST-VadCLIP model to obtain the prediction result of the test video.

[0039] The step of inputting the training video into the ST-VadCLIP model for training also includes:

[0040] For the temporal localization module, for both the anomaly confidence and the auxiliary detection head, the multi-instance learning mechanism is used to select the average of K high anomaly confidence values ​​from the abnormal and normal videos as the video-level prediction of the test video, and then the binary cross-entropy between the video-level prediction and the real label is used to calculate the classification loss.

[0041] For all global text label features, calculate the average similarity between each text label feature and other text label features, and use it as the text separation loss;

[0042] The total loss function is calculated based on the classification loss and the text separation loss.

[0043] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0044] In the embodiments of this disclosure, the attention aggregation mechanism of local image features can capture the features of small-sized abnormal behaviors in the video through the above-described method for spatiotemporal detection of abnormal behaviors in video; distance attention weights improve the performance of the Transformer temporal adapter when there is a lack of strong supervision signals; text label features with learnable prompts can effectively complete the weakly supervised temporal detection task of abnormal behaviors, and the auxiliary detection head can improve the performance of the model; local video features and local text label features utilize the powerful generalization ability of the CLIP model to locate abnormal regions in space, and motion change priors improve the accuracy of spatial abnormal behavior detection by introducing artificial priors; the cross-modal alignment mechanism based on multi-instance learning realizes optimization under weak supervision, thereby preserving the pre-trained knowledge as much as possible. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0046] Figure 1 This diagram illustrates the steps of a method for spatiotemporal detection of abnormal video behavior according to an exemplary embodiment of this disclosure.

[0047] Figure 2 A framework diagram of the ST-VadCLIP model in an exemplary embodiment of this disclosure is shown;

[0048] Figure 3 This diagram illustrates a method for detecting abnormal video behavior in an exemplary embodiment of this disclosure.

[0049] Figure 4This illustration shows a qualitative diagram of temporal anomalous behavior detection on the UCF-Crime dataset in an exemplary embodiment of this disclosure;

[0050] Figure 5 This illustration shows a qualitative diagram of spatial anomalous behavior detection on the UCF-Crime dataset in an exemplary embodiment of this disclosure. Detailed Implementation

[0051] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0052] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0053] This example implementation first provides a method for detecting abnormal behavior in videos. (See reference...) Figure 1 As shown, the method for detecting abnormal behavior in a video may include steps S101 to S106.

[0054] Step S101: Construct the VadCLIP model, which includes a CLIP image encoder, a CLIP text encoder, a local attention focusing module, a temporal modeling module, a temporal localization module, and a spatial localization module; wherein,

[0055] The CLIP image encoder and the CLIP text encoder are respectively frozen CLIP image encoder and frozen CLIP text encoder. The temporal modeling module includes a temporal Transformer encoder with fixed distance relationship attention weights. The local attention focusing module generates attention-weighted focusing local image features by measuring the degree of motion change in local regions. The temporal localization module includes a binary classification auxiliary detection head and a text label feature similarity detection head. The spatial localization module uses local text description labels and motion change priors to jointly locate abnormal behavior regions.

[0056] Step S102: Input the training video into the VadCLIP model for training, use the frozen CLIP image encoder to extract global and local image features of the training video, and use the frozen CLIP text encoder to extract the label features of global text labels and local localized text labels corresponding to the training video.

[0057] Step S103: Process the image features using the local attention aggregation and the temporal modeling to obtain video frame-level features;

[0058] Step S104: Calculate cosine similarity using video frame-level features and global text labels with trainable cue parameters to obtain anomaly confidence. At the same time, set up a binary classification auxiliary detection head to help the network distinguish between normal and abnormal video frame-level features.

[0059] Step S105: Locate frames that may exhibit abnormal behavior using anomaly confidence, and locate the spatial location of the abnormal behavior using the spatial localization module, including calculating cosine similarity using local image features and local text label features, and combining motion change priors to jointly locate the location of the abnormal behavior.

[0060] Step S106: Obtain the prediction result of the training video based on the video frame-level features and the text label features.

[0061] The above-described method for spatiotemporal detection of video anomalies utilizes the attention aggregation mechanism of local image features to capture the features of small-sized anomalies in the video; distance attention weights improve the performance of the Transformer temporal adapter in the absence of strong supervision signals; text label features with learnable cues can effectively complete the weakly supervised temporal detection of anomalies, and the auxiliary detection head can improve the model's performance; local video features and local text label features leverage the powerful generalization ability of the CLIP model to locate spatial anomalies, and motion change priors improve the accuracy of spatial anomaly detection by introducing human priors; and a cross-modal alignment mechanism based on multi-instance learning achieves optimization under weak supervision, thereby preserving as much pre-trained knowledge as possible.

[0062] Below, we will refer to Figure 1 The method for spatiotemporal detection of abnormal video behavior described in this example embodiment will be explained in more detail for each step of the method for spatiotemporal detection of abnormal video behavior described above.

[0063] This application presents a novel paradigm for weakly supervised video anomaly detection. It constructs an anomaly detection model (i.e., the ST-VadCLIP model) by directly utilizing a frozen pre-trained large model (i.e., the CLIP model), without any pre-training or fine-tuning process. The pre-trained large model used in this application is the OpenAI open-source CLIP (Contrastive Language-Image Pre-Training) model. It is a large-scale image-text cross-modal contrastive learning model, consisting of a transformer-based image encoder and a transformer-based text encoder. It is trained through contrastive learning on 400 million high-quality image-text pairs, giving CLIP high image recognition accuracy and strong generalization ability, enabling it to complete numerous downstream tasks without additional training or parameter fine-tuning.

[0064] like Figure 2 As shown, this application fully leverages the advantages of pre-trained large models and constructs a dual-branch network to achieve temporal and spatial detection of abnormal behavior. The temporal detection branch utilizes image features and text label features to obtain anomaly confidence, while the spatial detection branch fully utilizes the generalization ability of CLIP, using the similarity between local image features and local text label features to locate the position of abnormal behavior in the image. By leveraging the advantages of the dual-branch approach, this application effectively transfers pre-trained knowledge to the weakly supervised video abnormal behavior detection task, simultaneously achieving temporal and spatial localization of video abnormal behavior under weak supervision. Figure 3 The diagram shown is a flowchart of the video abnormal behavior spatiotemporal detection method of this application. The technical solution adopted by this application to solve its technical problem includes the following steps:

[0065] In step S101, the training video is input to obtain global and local image features:

[0066] Given videos in the training set, the CLIP image encoder of the frozen pre-trained large model is first used to extract frame-level global image features X. clip and local image features X patch .

[0067] Specifically, given videos in the training set, the CLIP image encoder, using a frozen pre-trained large model, is first used as the video frame feature extractor, resulting in a 512-dimensional feature set. For global image feature extraction, the entire frame image is directly input into the CLIP image encoder to obtain a single image feature. For local image feature extraction, the first 2D convolutional layer of the CLIP image encoder divides the image into 14×14 patches, then re-divides these patches into 2×2 adjacent, non-overlapping blocks, which are then input into subsequent layers of the encoder to obtain local image features that divide the original image into 7×7 blocks. Due to computational resource limitations, for a given training video, one frame is sampled every 16 frames and fed into the image encoder during feature extraction. This ultimately yields the video frame-level feature X. clip In this application, the longest supported temporal length for video input is 256, which is equivalent to the frame-level image feature X. clip The maximum length is 256. For cases exceeding the maximum time series length, the sequence is directly truncated into multiple segments with a maximum length of 256. For cases shorter than the maximum length, zeros are added to complete the sequence.

[0068] In step S102, the corresponding label is input, and the label features are obtained.

[0069] For the global text labels and local text labels used for spatial localization of abnormal behavior in the video, the CLIP text encoder of the frozen pre-trained large model is used to extract label features t. out and t patch .

[0070] Specifically, global text labels, typically words or phrases, are too concise to adequately summarize anomalous events. Therefore, we augmented the number of text labels using a large language model. Specifically, we queried the large language model ChatGPT3.5, providing synonyms for both anomalous and normal text labels. To learn the robust transferability of text embeddings, learnable cues were added to the original label information. Specifically, the original text labels were first converted into category vectors using a tokenizer, as follows:

[0071] t = Tokenizer(Label) (1)

[0072] Here, "Label" represents discrete text labels, such as combat, shooting, traffic accident, etc. Then, t is combined with l learnable cues {c1,...,c...} l These vectors are concatenated to form a complete sentence vector sequence, which serves as the input to the text encoder. The specific vector sequence is shown below.

[0073] t p ={c1, ... , t, ... , cl} (2)

[0074] This vector sequence is then incorporated into positional encoding to obtain positional information, and subsequently fed into the CLIP text encoder of a pre-trained large model to generate video label features t. out .

[0075] For local text description labels used for spatial localization of abnormal behavior, they are also generated using the large language model ChatGPT3.5. A fixed-prompt query model is used to obtain fine-grained local text descriptions for normal and abnormal behavior images, respectively. These text descriptions are then input into the CLIP text encoder to generate local text description label features t. patch .

[0076] In step S103, local attention aggregation and temporal modeling are used to process the image features to obtain video frame-level features.

[0077] The purpose of extracting local image features is to focus on small-scale anomalous events. Global image features are insufficient for representing many small-scale anomalous events; therefore, this application uses local image features to compensate for this problem. To focus on the region where the anomalous event occurs, this application designs a local attention aggregation method. This method introduces an artificial prior, namely, that when an anomalous event occurs, the corresponding region of the image changes significantly. For example, when a fight, explosion, or car accident occurs, it is a continuously changing dynamic event.

[0078] Specifically, to describe the degree of change in local image regions, this application designs the following method: Take a local feature X from the t-th frame of region number i. patch_i t Subtract the feature value from the features of the previous and next frames at the same location, and then calculate the L2 norm. The result is the degree of change in the region. This operation can be described by the following formula:

[0079] Mo i t =|| 2X patch_i t - X patch_i t+1 - X patch_i t-1 || L2 (3)

[0080] When the image in this region changes little, the feature similarity between the preceding and following frames after passing through the CLIP image encoder is high and remains stable. In this case, the result of formula 3 after subtraction is close to 0. When the image in this region changes much, the feature similarity between the preceding and following frames is low. In this case, the result of formula 3 after subtraction is larger than the previous case.

[0081] Since directly processing local image features would consume more computational resources, this application reduces computational complexity by aggregating local features into a single feature. For a total of 7×7 local blocks, based on the calculated degree of change M, this application selects the 12 regions with the highest degree of change for aggregation, discarding the rest. The change degree Mo, normalized using softmax, is used as the attention weight to weight and aggregate the region image features to obtain the local aggregated feature. The calculation process is shown in the following formula:

[0082] Attention t =Softmax(TOP(Mo t ))(4)

[0083] X as t =(Attention) t ) T X Mo t

[0084] Where X Mo t The patch image features of the 12 regions with the highest Mo scores in frame t are aggregated to obtain the final local aggregated feature X. as t and global image features X clip t The features are then added together to obtain the final video image features.

[0085] Since the image encoder of a pre-trained large model is trained at the image level, it lacks the ability to capture temporal relationships. This application designs a temporal modeling module to process image features lacking contextual temporal relationships and capture these relationships. Because weakly supervised anomaly detection tasks lack sufficient supervisory information, directly using the Transformer as a temporal modeler is ineffective, as has been verified in other methods. To improve this, this application designs an attention matrix based on the relative distance between two frames for capturing temporal relationships.

[0086] Specifically, H dis The attention matrix, obtained by calculating the distance between two frames, can be calculated using the following formula:

[0087]

[0088] The distance relationship between the i-th and j-th frames is determined solely by their relative temporal positions, and σ is a hyperparameter controlling the range of influence of this distance relationship. According to this formula, the closer the relative distance between two frames, the higher the score, the larger the adjacency matrix value, and the more information exchanged.

[0089] The video image features are input into two distance attention Transformers, and finally the video frame-level features X after temporal relationship modeling is obtained.

[0090] In step S104, the confidence level of video frame-level feature anomalies is calculated.

[0091] The calculation of frame-level anomaly confidence is divided into two modules: an auxiliary detection head module and a video-text label alignment module.

[0092] The auxiliary detection head module consists of a fully connected layer. It takes video frame-level features X as input and outputs frame-level anomaly confidence, ranging from 0 to 1. This output is not used as the final temporal anomaly detection result, but is only used to assist the neural network in separating normal and abnormal frames.

[0093] The video-text label alignment module uses video frame-level features and global text label features for alignment, calculating similarity to obtain anomaly confidence results. The global text label contains multiple normal and anomalous labels. The average of all normal label features is used to obtain a total normal category label feature, which is then concatenated with other anomalous label features. The video frame-level features are added to the global frame features extracted by the original CLIP image encoder, and then a cosine similarity matrix is ​​calculated with the text label features, as shown in the following formula:

[0094]

[0095] The similarity matrix M represents the similarity between video frames and text labels of all categories, where the vector S = {s1,...,s...} m} represents the similarity between a video frame and all category labels. All vectors are normalized using the Softmax function, as shown in the following formula:

[0096]

[0097] The final confidence probability p of choosing the normal category normal Calculate 1-p normal The frame-level anomaly confidence score p is obtained.

[0098] In step S105, the frames that may exhibit abnormal behavior are located using the anomaly confidence level, and the spatial localization module is used to locate the spatial location of the abnormal behavior.

[0099] The frame-level anomaly confidence p is used to locate the location of anomaly events in time sequence. A threshold is set, and if p is greater than the threshold, the frame is considered to have an anomaly event, and further spatial anomaly event detection is performed.

[0100] Spatial anomaly detection is divided into two steps. The first step calculates the similarity between local image features and local text description features, and then uses the generalization performance of CLIP to detect suspicious anomaly regions, referencing practices used in industrial image anomaly detection. To improve the detection rate, this application uses 4×4 local image patch features in addition to the existing 7×7 local image patch features. The cosine similarity between the local image features and the local text description features is calculated and normalized using the Softmax function (referencing Equations 6 and 7). The probabilities of all anomaly categories are summed to obtain the anomaly confidence map for the current region.

[0101] Since the 7×7 region partition and the 4×4 region partition are different in size, this application uses the resize function in the OpenCV library to extend the 4×4 spatial anomaly confidence map to 7×7 through bilinear interpolation, and then adds it to the original 7×7 spatial anomaly confidence map with weights to obtain the spatial anomaly confidence map.

[0102] Due to factors such as video clarity and semantic ambiguity in text description tags, the resulting spatial anomaly confidence map may have a high false positive rate. To reduce this rate, this application introduces the motion change prior proposed earlier. Specifically, the change degree matrix Mo is taken, and a threshold is set. Values ​​exceeding this threshold are set to 1, and values ​​below it are set to 0, resulting in the motion mask matrix Mask, as shown in the following equation:

[0103]

[0104] Among them Mo ij The region in the i-th row and j-th column of the change degree matrix is ​​represented by this formula. The Mask matrix is ​​obtained by multiplying it with the spatial confidence map mentioned above to obtain the final spatial confidence map S.

[0105] In step S106, the prediction result of the training video is obtained.

[0106] For temporal anomaly detection, a multi-instance learning mechanism is used to select the average of K high anomaly confidence values ​​from both anomalous and normal videos as the video-level prediction. Then, the classification loss L is calculated using the binary cross-entropy between the video-level prediction and the ground truth label. bce1 Similarly, the auxiliary detection head uses the same loss function to calculate the classification loss L. bce2 It is used to help distinguish between normal and abnormal video features.

[0107] For spatial anomaly detection, the final spatial confidence map S is 7×7 in size. This application uses the OpenCV library function `resize` to enlarge it to the size of the original video, transforming it into an anomaly heatmap. Then, the OpenCV library function `findCounters` is used to find the region with the largest area in the anomaly heatmap, which is used as the final bounding box for the anomaly event. After being enlarged to the original video size, the overlap (IoU) can be calculated between the anomaly event bounding boxes at the original video coordinates.

[0108] To differentiate between normal and abnormal text label features, this application designs an additional text separation loss function. For all global text label features, the average similarity between each text label feature and other text label features is calculated, which serves as the text separation loss L. dis The higher the similarity between normal and abnormal text features, the larger the loss function result, encouraging the network to move towards the abnormal text. The final total loss function is L. bce1 +L bce2 +L dis .

[0109] The effects of this application will be further described below with reference to simulation experiments.

[0110] 1. Simulation conditions:

[0111] The simulation in this application was performed using PyTorch software on a server containing an NVIDIA RTX 3090 graphics card.

[0112] 2. Simulation content and result analysis:

[0113] This application was compared with eight existing methods (including one semi-supervised method, one fully supervised method, and six weakly supervised methods) in a comparative simulation on the commonly used UCF-Crime dataset.

[0114] The simulation data used in this application is UCF-Crime, which is the first proposed weakly supervised anomaly monitoring video dataset. It contains 1900 videos from 13 anomaly categories, with 1610 videos in the training set and 290 videos in the test set.

[0115] Table 1 shows the comparison results between the proposed method and existing methods on the UCF-Crime dataset.

[0116]

[0117] Simulations were conducted on this application and existing semi-supervised, fully supervised, and weakly supervised video anomaly detection methods under the same experimental settings and environment. The quantitative evaluation and abundance value comparisons on the UCF-Crime dataset are shown in Table 1. As can be seen from Table 1, the method in this application significantly outperforms the semi-supervised, fully supervised, and weakly supervised methods. More precisely, the method in this application achieves an average accuracy of 88.08% on UCF-Crime, outperforming its best competitors CLIP-TSA and DMU by 0.5% and 1.33%, respectively. Furthermore, in terms of the temporal overlap index (i.e., the sum of the traditional localization index overlap TIOU divided by the total number of frames), the method in this application outperforms the comparison methods by more than 7%. Figure 4 The image shows a qualitative illustration of temporal anomaly detection on the UCF-Crime dataset; as shown... Figure 5 The figure shown is a qualitative diagram of spatial anomaly detection on the UCF-Crime dataset.

[0118] The above-described methods for detecting abnormal behavior in videos demonstrate that the attention aggregation mechanism of local image features can capture the features of small-sized abnormal behaviors in videos; distance attention weights improve the performance of the Transformer temporal adapter when strong supervision signals are lacking; text label features with learnable cues can effectively complete the task of weakly supervised temporal detection of abnormal behavior, and the auxiliary detection head can improve the model's performance; local video features and local text label features utilize the powerful generalization ability of the CLIP model to locate abnormal regions in space, and motion change priors improve the accuracy of spatial abnormal behavior detection by introducing artificial priors; and the cross-modal alignment mechanism based on multi-instance learning achieves optimization under weak supervision, thereby preserving as much pre-trained knowledge as possible.

[0119] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0120] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A spatiotemporal detection method for weakly supervised video anomaly behavior based on a pre-trained large model, characterized in that, include: Construct an ST-VadCLIP model, which includes a CLIP image encoder, a CLIP text encoder, a local attention focusing module, a temporal modeling module, a temporal localization module, and a spatial localization module; the spatial localization module includes: Based on the anomaly confidence level, a threshold is set. If the threshold is exceeded, it indicates that there is abnormal behavior in the frame, and it is extracted to the spatial positioning module. The cosine similarity between the local image features of possible anomalous frames and the local text description features obtained through a large language model is calculated to obtain the spatial anomaly confidence. Local image features contain two scales, while local text description features contain parts describing abnormal behavior and parts describing things that might appear in a normal scene; Motion change prior is used to calculate the degree of change of each local block compared to the blocks at the same position in the two frames before and after it. A threshold is set to filter blocks with small changes and the anomaly confidence of these blocks is set to 0. Locate the bounding box of anomalous behavior based on the spatial anomaly confidence score; The CLIP image encoder and the CLIP text encoder are respectively frozen CLIP image encoder and frozen CLIP text encoder. The temporal modeling module includes a temporal Transformer encoder with fixed distance relationship attention weights. The local attention focusing module generates attention-weighted focusing local image features by measuring the degree of motion change in local regions. The temporal localization module includes a binary classification auxiliary detection head and a text label feature similarity detection head. The spatial localization module uses local text description labels and motion change priors to jointly locate abnormal behavior regions. The training video is input into the ST-VadCLIP model for training. The global and local image features of the training video are extracted using the frozen CLIP image encoder, and the label features of the global text labels and local localized text labels corresponding to the training video are extracted using the frozen CLIP text encoder. The method for calculating the degree of change of each local block includes: Select a local block location, calculate the L2 norm by subtracting the features of the block from the features of the blocks in the previous and next frames; This result is used to measure the degree of change in the current region. If it is less than the threshold, it means that the current region has not changed much in these three frames. The image features are processed using the local attention aggregation and the temporal modeling to obtain video frame-level features; Using video frame-level features and global text labels with trainable cue parameters, cosine similarity is calculated to obtain anomaly confidence. At the same time, a binary classification auxiliary detection head is set to help the network distinguish between normal and abnormal video frame-level features. Frames that may exhibit abnormal behavior are located using anomaly confidence. The spatial localization module is used to locate the spatial location of the abnormal behavior, including calculating cosine similarity using local image features and local text label features, and combining motion change priors to jointly locate the location of the abnormal behavior. The prediction results for the training video are obtained based on the video frame-level features and the text label features.

2. The spatiotemporal detection method for weakly supervised video anomaly behavior based on a pre-trained large model according to claim 1, characterized in that, The step of extracting image features from the training video using the frozen CLIP image encoder further includes: Every preset frame, one frame is sampled and sent to the CLIP image encoder to obtain the image features; Each complete image frame, when passed through the CLIP image encoder, is divided into multiple non-overlapping blocks representing each region of the image according to the principle of Vision Transformer. When extracting global image features, all blocks are input into the CLIP image encoder. When extracting local image features, the blocks representing the specified region are input into the CLIP image encoder to obtain the image features representing the local region.

3. The spatiotemporal detection method for weakly supervised video anomaly behavior based on a pre-trained large model according to claim 2, characterized in that, The step of extracting label features of text labels corresponding to the training video using the frozen CLIP text encoder includes: The text labels are converted into category vectors using a tokenizer; The category vectors are concatenated with learnable cues to obtain a vector sequence; The CLIP text encoder extracts the vector sequence to obtain the tag features.

4. The spatiotemporal detection method for weakly supervised video anomaly behavior based on a pre-trained large model according to claim 3, characterized in that, The step of processing the local image features using the local attention aggregation to obtain video frame-level features includes: The local image features are compressed and aggregated using the local attention aggregation. The video frame-level features are obtained by adding the local aggregated image features and the global image features.

5. The spatiotemporal detection method for weakly supervised video anomaly behavior based on a pre-trained large model according to claim 4, characterized in that, The temporal Transformer encoder calculates attention weights by the distance between two frames, rather than using the dot product attention of the traditional Transformer.

6. The spatiotemporal detection method for weakly supervised video anomaly behavior based on a pre-trained large model according to claim 5, characterized in that, The timing positioning module includes: The aggregated video frame-level features are compared with the global text labels to calculate the cosine similarity, and then the anomaly confidence is obtained by passing the Softmax normalization function. The binary classification auxiliary detection head includes a fully connected layer with one neuron.

7. The spatiotemporal detection method for weakly supervised video anomaly behavior based on a pre-trained large model according to claim 6, characterized in that, The method also includes: The test video is input into the trained ST-VadCLIP model to obtain the prediction result of the test video.

8. The spatiotemporal detection method for weakly supervised video abnormal behavior based on a pre-trained large model according to claim 7, characterized in that, The step of inputting the training video into the ST-VadCLIP model for training also includes: For the temporal localization module, for both the anomaly confidence and the auxiliary detection head, a multi-instance learning mechanism is used to select the average of K high anomaly confidence values ​​from the abnormal and normal videos as the video-level prediction of the test video. Then, the binary cross-entropy between the video-level prediction and the true label is used to calculate the classification loss. For all global text label features, calculate the average similarity between each text label feature and other text label features, and use it as the text separation loss; The total loss function is calculated based on the classification loss and the text separation loss.

Images