A video anomaly localization method, system, medium and device

Abstract

The application discloses a video anomaly positioning method, system, medium and equipment. The application captures the evolution trend of inter-frame abnormal behavior through a time sequence adapter connected in series on the output side of a CLIP visual encoder with parameter freezing, injects time information into spatial features, and makes up for the defects of static large model video modeling; the first abnormal confidence, time sequence features and text embedding are weighted and fused through a fusion layer to generate dynamic text features, solving the problem of low cross-modal alignment accuracy caused by static prompt semantic rigidity and background interference; the category label is taken as supervision, multi-dimensional heterogeneous loss is optimized jointly, parameter updating from coarse granularity to fine granularity under weak supervision is realized; by locking the core frame, when the interval between adjacent core frames is less than the preset interval, the time sequence adapter and the binary classifier are locally fine-tuned according to the intermediate frame, the jagged split and false alarm of the second abnormal confidence are eliminated, and high-precision, uninterrupted abnormal start and end boundary positioning and category recognition are realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision and video analysis technology, and in particular to a video anomaly localization method, system, medium, and device. Background Technology

[0002] With the acceleration of urbanization and the explosive growth in the deployment of surveillance cameras, anomaly detection in surveillance videos has become a core research direction in the field of computer vision. Its goal is to automatically identify sudden events that deviate from normal behavioral logic in video sequences, such as violent conflicts, traffic accidents, and fires. This technology has extremely high practical value in areas such as intelligent security monitoring, public safety assurance, and the operational safety assessment of autonomous vehicles.

[0003] Traditional anomaly detection methods primarily rely on fully supervised learning and unsupervised learning. Fully supervised learning requires significant manpower for frame-level annotation; unsupervised learning, which only uses normal samples for modeling, exhibits poor stability in anomaly detection. To balance annotation costs and detection performance, Weakly-Supervised Video Anomaly Detection (WSVAD) has emerged. During the training phase, it only requires video-level labels (indicating whether the entire video contains anomalies) and eliminates the need for frame-level annotation, significantly reducing the difficulty of data preparation.

[0004] In recent years, large-scale visual language models (VLMs), represented by the Contrastive Language Image Pre-trained Model (CLIP), have brought new breakthroughs to this field. CLIP, through pre-training on massive amounts of image-text pairs, has acquired powerful visual semantic association capabilities, enabling the model to utilize textual information to guide the understanding and localization of visual features. However, how to effectively transfer this static cross-modal alignment capability to video anomaly detection tasks with dynamic temporal features remains a hot topic and a challenge in current technological research.

[0005] Currently, existing technologies mainly fall into two categories: one is the traditional weakly supervised method based on temporal feature enhancement, which relies on the Multiple Instant Learning (MIL) framework and uses 3D convolutional networks such as I3D to extract visual features from video segments, distinguishing normal and abnormal segments through feature modulus or contrast loss. A common drawback of this type of method is that its detection performance highly depends on the statistical distribution of pure visual features, completely lacking understanding and utilization of the semantics of abnormal event categories. Therefore, it is prone to false positives and false negatives in scenes with severe background interference or complex lighting changes. The other category is the visual-language association scheme based on the CLIP model, which introduces a frozen CLIP dual encoder to extract cross-modal features and then concatenates a temporal modeling module (such as local attention, graph convolution, etc.) with it, combined with a learnable cueing mechanism and multiple instance learning for weakly supervised training. While these solutions have initially achieved visual-semantic association, they still have shortcomings such as CLIP's native architecture being designed for static images, which makes it difficult to accurately capture subtle temporal changes when processing long-term dynamic events, even with an external temporal module; static category labels (such as "robbery") are difficult to cover the visual diversity of real-world scenes, and normal background frames can cause serious semantic interference, leading to a decrease in cross-modal matching accuracy; under weak supervision, the model tends to capture the most prominent feature segments, resulting in a jagged and fragmented output anomaly score curve that divides continuous abnormal events into scattered segments, causing blind spots and missed detections.

[0006] Therefore, how to overcome the inherent deficiencies of visual-language pre-trained models in video temporal modeling under weak supervision, solve the problems of insufficient semantic expression ability of static text prompts and background noise interference, and eliminate the jagged fragmentation and missed detection of frame-level anomaly localization curves caused by coarse-grained video-level annotation, so as to achieve high-precision and uninterrupted frame-level anomaly event start and end boundary localization, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] Therefore, it is necessary to provide a video anomaly localization method, system, medium, and device to address the aforementioned technical problems.

[0008] The following technical solution is adopted in this specification: This manual provides a method for locating video anomalies, specifically including: An anomaly localization model is constructed, which includes a parameter-frozen CLIP visual encoder and a language encoder, a temporal adapter connected in series on the output side of the visual encoder, a binary classifier, and a learnable cue template containing a stitching layer and a fusion layer; the stitching layer is connected in series on the input side of the language encoder, and the fusion layer is connected in series on the output side of the language encoder.

[0009] Multiple consecutive video segments of the training video are input into a visual encoder to obtain a spatial feature matrix representing the complete training video. The spatial feature matrix is ​​combined with a preset learnable temporal location encoding matrix and then input into a temporal adapter. The evolution trend of inter-frame abnormal behavior is captured through a multi-head self-attention mechanism to obtain a temporal feature matrix. The temporal feature matrix is ​​input into a binary classifier to obtain the first anomaly confidence.

[0010] The preset anomaly category name is input into the concatenation layer to obtain the category vector. The category vector is then input into the language encoder to obtain the text embedding containing the category label. The temporal feature matrix, the first anomaly confidence score, and the text embedding are input into the fusion layer to obtain the dynamic text features.

[0011] Using the category labels as supervision, the video-level Top-K classification loss, frame-level constraint loss, and alignment loss between the temporal features and dynamic text features are jointly optimized to update the temporal adapter, learnable cue template, and binary classifier parameters.

[0012] The anomaly localization model, updated with the training video input parameters, obtains the second anomaly confidence score for each frame. The core frame is then locked using the second anomaly confidence score of each frame and a preset threshold. When the interval between adjacent core frames is less than the preset interval, the temporal adapter and binary classifier are locally reverse-tuned based on the intermediate frames between the adjacent core frames. Based on the finely tuned temporal adapter and binary classifier, combined with the visual encoder, the final anomaly localization model is constructed.

[0013] Furthermore, the multiple consecutive video segments are obtained by sampling the training video in a non-overlapping equal-division manner, and the tails that are less than a fixed length are discarded or zero-padding is performed.

[0014] Furthermore, the learnable temporal location encoding matrix has the same dimension as the spatial feature matrix.

[0015] Furthermore, the timing adapter consists of at least one Transformer layer, which is composed of a multi-head self-attention mechanism and a feedforward neural network residual cascade, with each layer organized by layer normalization and residual connections.

[0016] Furthermore, the dynamic text features obtained by combining the temporal feature matrix, the first anomaly confidence score, and the text embedding into the input fusion layer specifically include: Using the first anomaly confidence score as a weight, the temporal feature matrix is ​​weighted by transpose dot product aggregation to obtain the visual context vector corresponding to the current video segment instance.

[0017] The visual context vector is copied and expanded according to the number of categories, then added to the text embedding, and finally processed through a feedforward network and a residual shortcut to obtain dynamic text features.

[0018] Furthermore, the binary classifier consists of a fully connected layer and a Sigmoid activation function; First anomaly confidence level The expression is: ; The visual context vector corresponding to the current video segment instance The expression is: ; in For the first Abnormal confidence of a video clip, It is the Sigmoid activation function. This is the weight matrix. For the first Spatial feature matrix of a video segment This is the bias of the same binary classifier.

[0019] Further, the cosine similarity between each temporal feature in the temporal feature matrix and its corresponding dynamic text feature is calculated; for each cosine similarity, the dynamic text feature corresponding to each cosine similarity is determined to be either positive or negative based on the category label; for the column corresponding to the positive class, the positive class error between the cosine similarity above the preset similarity threshold and the preset positive class target value is calculated; for the column corresponding to the negative class, all cosine similarities in that column are reduced to the negative class target value, and the negative class error between each cosine similarity in that column and the preset negative class target value is calculated; all positive class errors and negative class errors are weighted and aggregated to obtain the alignment loss.

[0020] The frame-level constraint loss includes a smoothing penalty term for the confidence difference between adjacent frames and a sparse regularization term that causes the score of normal frames to approach zero.

[0021] This specification provides a video anomaly localization system, specifically including: The model building module is used to build an anomaly localization model that includes a CLIP visual encoder and a language encoder with parameter freezing, a temporal adapter connected in series on the output side of the visual encoder, a binary classifier, and a learnable cue template containing a stitching layer and a fusion layer.

[0022] The feature extraction and temporal adaptive fusion module is used to extract spatial feature matrices from multiple consecutive video segments of the training video. After combining the spatial feature matrix with the learnable temporal location encoding matrix, a temporal feature matrix is ​​obtained through a temporal adapter; then, the first anomaly confidence is obtained through a binary classifier.

[0023] The instance-aware dynamic text feature construction module uses a concatenation layer of learnable cue templates to concatenate preset anomaly category names into category vectors, and then uses a language encoder to map the category vectors into text embeddings containing category labels; through a fusion layer of learnable cue templates, the temporal feature matrix, the first anomaly confidence, and the text embeddings are fused to obtain dynamic text features.

[0024] The reverse fine-tuning and final model construction module is used to update the anomaly localization model after training video input parameters, obtain the second anomaly confidence of each frame; lock the core frame by the second anomaly confidence of each frame and a preset threshold; when the interval between adjacent core frames is less than the preset interval, perform local reverse fine-tuning on the temporal adapter and binary classifier according to the intermediate frame between the adjacent core frames, and construct the final anomaly localization model based on the fine-tuned temporal adapter and binary classifier, combined with the visual encoder.

[0025] The reverse fine-tuning prediction module is used to input the training video into the initially trained anomaly localization model, perform inference, obtain the anomaly probability curve, and obtain pseudo-labeled anomaly frames based on the anomaly probability curve; and to locally reverse fine-tune the output layers of the temporal adapter and binary classifier based on the pseudo-labeled anomaly frames, update the model parameters until the model converges or reaches the maximum number of iterations, and obtain the final anomaly localization model.

[0026] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0027] This specification provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0028] The beneficial effects of this invention are as follows: Compared with the prior art, this invention captures the evolution trend of inter-frame abnormal behavior by using a temporal adapter connected in series with the output side of the CLIP visual encoder with parameter freezing, injecting temporal information into spatial features to compensate for the defects of static large-scale video modeling; by using a fusion layer to weightedly fuse the first anomaly confidence, temporal features, and text embedding to generate dynamic text features, it solves the problem of low cross-modal alignment accuracy caused by the rigidity of static prompt semantics and background interference; using category labels as supervision, it jointly optimizes multi-dimensional heterogeneous loss, realizing parameter updates from coarse-grained to fine-grained under weak supervision; by locking the core frame, when the interval between adjacent core frames is less than the preset interval, it performs local reverse fine-tuning of the temporal adapter and binary classifier based on the intermediate frame, eliminating the jagged fragmentation and missed detection of the second anomaly confidence, and realizing high-precision, uninterrupted anomaly start and end boundary localization and category recognition. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention; Figure 2 This is a schematic diagram of the frame-level anomaly confidence output curve and temporal domain fine-tuning effect in an actual explosion test video according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the frame-level anomaly confidence output curve and temporal domain fine-tuning effect in an actual fighting test video according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the frame-level anomaly confidence output curve and temporal domain fine-tuning effect in an actual robbery test video according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the frame-level anomaly confidence output curve and temporal domain fine-tuning effect under actual normal test video in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this invention.

[0031] This invention provides a video anomaly localization method, comprising seven steps: constructing an anomaly localization model; extracting spatial feature matrices from video segments and combining them with a preset learnable temporal location encoding matrix; obtaining a temporal feature matrix through a temporal adapter; obtaining a first anomaly confidence score through a binary classifier; generating dynamic text features through a learnable cue template and a language encoder; updating model parameters by jointly optimizing video-level multidimensional heterogeneous loss with category labels as supervision; updating the model with training video input parameters and performing local reverse fine-tuning on the temporal adapter and binary classifier; and constructing the final anomaly localization model based on the fine-tuned temporal-fit binary classifier combined with a visual encoder. The method utilizes a temporal adapter connected in series with the output of a parameter-frozen CLIP visual encoder. By capturing the evolution trend of inter-frame anomalous behavior and injecting temporal information into spatial features, the defects of static large-scale video modeling are compensated. Through a fusion layer, the first anomaly confidence, temporal features, and text embedding are weighted and fused to generate dynamic text features, solving the problems of low cross-modal alignment accuracy caused by the rigidity of static prompt semantics and background interference. With category labels as supervision, multi-dimensional heterogeneous loss is jointly optimized, realizing parameter updates from coarse-grained to fine-grained under weak supervision. By locking the core frame, when the interval between adjacent core frames is less than the preset interval, the temporal adapter and binary classifier are locally reversed and fine-tuned according to the intermediate frame to eliminate the jagged fragmentation and false negatives of the second anomaly confidence, realizing high-precision and uninterrupted anomaly start and end boundary localization and category recognition.

[0032] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] A video anomaly localization method specifically includes the following steps: Step 1: Construct an anomaly localization model, which includes a parameter-frozen CLIP visual encoder and a speech encoder, a temporal adapter connected in series on the output side of the visual encoder, a binary classifier, and a learnable cue template containing a stitching layer and a fusion layer; the stitching layer is connected in series on the input side of the speech encoder, and the fusion layer is connected in series on the output side of the speech encoder.

[0034] Step 2: Input multiple consecutive video segments of the training video into the visual encoder to obtain a spatial feature matrix representing the complete training video; combine the spatial feature matrix with a preset learnable temporal location encoding matrix and input it into the temporal adapter to capture the evolution trend of inter-frame abnormal behavior through a multi-head self-attention mechanism to obtain a temporal feature matrix; input the temporal feature matrix into a binary classifier to obtain the first anomaly confidence.

[0035] Furthermore, multiple consecutive video segments are obtained by non-overlapping equal-sampling of the video to be detected, and the tails that are less than a fixed length are discarded or zero-padding.

[0036] Preferably, the learnable temporal location encoding matrix and the spatial feature matrix have the same dimension.

[0037] Preferably, the sequence adapter consists of at least one Transformer layer, which is composed of a multi-head self-attention mechanism and a feedforward neural network residual cascade, with each layer organized by layer normalization and residual connections.

[0038] Step 3: Input the preset anomaly category name into the concatenation layer to obtain the category vector, input the category vector into the language encoder to obtain the text embedding containing the category label; input the temporal feature matrix, the first anomaly confidence score and the text embedding into the fusion layer to obtain the dynamic text features.

[0039] Specifically, the dynamic text features are obtained by combining the temporal feature matrix, the first anomaly confidence score, and the text embedding into the input fusion layer, including: By using confidence scores as weights to perform a weighted aggregation of the transposed dot product of the temporal feature matrix, the visual context vector corresponding to the current video segment instance is obtained.

[0040] The visual context vector is copied and expanded according to the number of categories, added to the learnable text embedding, and then passed through a feedforward network and residual shortcut to obtain instance-aware dynamic text features.

[0041] Preferably, the binary classifier consists of a fully connected layer and a Sigmoid activation function, with an anomaly confidence level... The expression is: ; The visual context vector corresponding to the current video clip instance The expression is: ; in For the first Abnormal confidence of a video clip, It is the Sigmoid activation function. This is the weight matrix. For the first Spatial feature matrix of a video segment This is the bias of the same binary classifier.

[0042] Step 4: Using category labels as supervision, jointly optimize the video-level Top-K classification loss, frame-level constraint loss, and alignment loss between temporal features and dynamic text features, and update the temporal adapter, learnable cue template, and binary classifier parameters.

[0043] Specifically, the cosine similarity between each temporal feature in the temporal feature matrix and its corresponding dynamic text feature is calculated. For each cosine similarity, the dynamic text feature corresponding to each cosine similarity is determined to be either positive or negative based on the category label. For the column corresponding to the positive class, the positive class error between the cosine similarity above the preset similarity threshold and the preset positive class target value is calculated. For the column corresponding to the negative class, all cosine similarities in that column are reduced to the negative class target value, and the negative class error between each cosine similarity in that column and the preset negative class target value is calculated. All positive and negative class errors are weighted and aggregated to obtain the alignment loss. The frame-level constraint loss includes a smoothing penalty term for the confidence difference between adjacent frames and a sparse regularization term that causes the score of normal frames to approach zero.

[0044] Example The overall processing technology and data flow in this embodiment mainly consist of four core stages: feature extraction and temporal adaptive fusion stage, instance-aware dynamic text feature construction stage, multi-task joint training stage, and reverse fine-tuning prediction stage.

[0045] In hardware or computing environment deployment, this embodiment receives the surveillance video source to be detected and performs block editing and parallel feature mapping through a computer vision processing unit. The overall model architecture jointly optimizes three heterogeneous loss functions during training, while during the inference deployment phase, a cascaded temporal threshold correction module performs boundary normalization on the output probability sequence, ultimately outputting frame-level accurate start and end times of abnormal events and event category labels on the timeline, such as... Figure 1 As shown, the specific steps include the following: S1. Feature Extraction and Temporal Adaptive Fusion This is used to address the technical problem of large image models lacking video temporal dimension perception in weakly supervised scenarios.

[0046] S1.1: Video Dynamic Segmentation and Segment Sampling Input an unedited long video. First, it is divided into two parts using a time-domain equal-segmentation sampling process. Each video segment consists of a fixed number of non-overlapping video frames. In this embodiment, each video segment is composed of a fixed number of consecutive video frames to ensure the alignment of the entire network computation.

[0047] S1.2: Spatial Feature Mapping of the Frozen Base This Each video segment is sequentially input into a pre-trained visual-language large-scale model, specifically a CLIP model visual encoder with frozen parameters. Because the large-scale model has been pre-trained against text on massive image data, it can output highly robust low-dimensional spatial high-order semantic features. Each segment is mapped to a feature vector, which are then concatenated to obtain the spatial feature matrix of the entire video. ,in For feature dimensions.

[0048] S1.3: Temporal Adaptive Modeling with Integrated Position Coding Since the visual encoder of a large model can only process static spatial information, this scheme connects a trainable temporal adapter after the visual encoder. First, a trainable temporal adapter is constructed... Learnable temporal location encoding matrices of the same dimension, and compared with Element-wise addition is performed to introduce explicit absolute time-axis topological information into static features.

[0049] S1.4: Temporal Context Fusion of Multi-Head Self-Attention Mechanism The feature matrix with injected positional encoding is fed into the Transformer block inside the temporal adapter. The core of this block consists of a multi-head self-attention (MHSA) mechanism and a feedforward neural network (FFN) residual cascade. The feature matrix is ​​used to calculate pairwise correlations along the temporal dimension, capturing the evolution trend of abnormal behavior between consecutive frames, such as running or falling. Both the MHSA layer and the feedforward network layer are wrapped in layer normalization and passed between layers through residual connections, ultimately outputting a video feature matrix that incorporates long-range temporal dependencies. .

[0050] S2. Instance-Aware Dynamic Text Feature Construction It is used to overcome the shortcomings of traditional static text tags in terms of insufficient matching accuracy when faced with weak supervision and "normal background noise".

[0051] S2.1: Concatenation of initial text embeddings and learnable context vectors Defining the tasks involved in video anomaly detection Specific anomaly category names are provided, such as "fighting," "car accident," and "normal." This approach abandons traditional fixed text descriptions and instead appends a set of continuous, randomly initialized, learnable context vectors to each category label at the front end. The combined learnable cues are then fed into a pre-trained CLIP text encoder, mapping them to a multimodal shared feature space to obtain the initial category text embedding matrix. .

[0052] S2.2: Visual Context Dot Product Aggregation Based on Weakly Supervised Confidence To enable text features to adaptively perceive the background environment of a specific video, this scheme utilizes a lightweight binary classifier consisting of a fully connected layer and a sigmoid activation function. The video features obtained in S1.4 are then processed... Perform a segment-by-segment scan to calculate a coarse-grained confidence score vector for each video segment indicating whether it is an anomaly. Then, the confidence vector As a weighting coefficient, it is used in conjunction with the video feature matrix. Performing the transpose dot product operation, the adaptive weighted aggregation of visual features on the time axis is completed as follows: ; This S was extracted This refers to the unique, pure, and anomalous visual context of the video segment, which excludes a large amount of normal background interference.

[0053] S2.3: Multimodal Dynamic Residual Fusion Dynamically generated visual context With static initial category text embedding Multimodal fusion is performed. The specific implementation process involves: […]. Copy and expand to If the number of rows is consistent, element-wise concatenation and addition are performed, followed by feature alignment and non-linear activation (using the GELU function) through a two-layer linear feedforward network. Finally, the result is compared with the original data through a residual shortcut. Then add them together again. Through this dynamic fusion process, the final output is instance-aware dynamic prompt text features. This feature has adaptive scaling capabilities, dynamically adjusting the semantic focus based on the current video frame to suppress false positives.

[0054] S3. Multi-task joint training By constructing fine-grained alignment mapping matrices and heterogeneous loss functions, the entire network parameters are driven to converge under a weakly supervised framework.

[0055] S3.1: Construction of Cross-Modal Multi-Instance Alignment Mapping Matrix Video feature matrix containing temporal context Instance-aware dynamic prompt text features The cosine similarity is measured in pairs, and the formula is as follows:

[0056] The generated matrix This is a multi-instance alignment mapping matrix, where each element... Intuitively reflects the first The video clip and the first Fine-grained cross-modal matching probabilities between texts of different exception categories.

[0057] S3.2: Design of Joint Loss Function for Multiple Tasks To train accurate frame-level discrimination under weak supervision with only video-level coarse labels, this invention designs a three-way parallel optimized loss function: Video-level anomaly binary classification loss : Employ the Top-K selection strategy from Multiple Instance Learning (MIL). During training, extract the top K anomaly prediction scores from the entire video. The average value of each segment is used as the anomaly probability prediction for the entire video, and the standard cross-entropy loss is calculated by comparing it with the video-level ground truth labels.

[0058] Frame-level anomaly binary classification loss The frame-by-frame confidence sequence output by the binary classifier in S2.2 Smoothness and sparsity constraints (BCE Loss) are applied to force the model's output frame-level scores to have significant discriminative power.

[0059] Fine-grained category alignment loss Alignment mapping matrix generated using S3.1 For each cosine similarity in its mapping matrix, the dynamic text feature corresponding to each cosine similarity is determined to be either positive or negative based on the category label. For the column corresponding to the positive class, the positive class error between the cosine similarity above the preset similarity threshold and the preset positive class target value is calculated. For the column corresponding to the negative class, all cosine similarities in that column are reduced to the negative class target value, and the negative class error between each cosine similarity in that column and the preset negative class target value is calculated. All positive and negative class errors are weighted and aggregated to obtain the alignment loss.

[0060] Finally, the three losses mentioned above are weighted and summed using preset weights to form a joint loss function for backpropagation gradient updates of the temporal adapter, learnable cue vector, and classifier parameters.

[0061] S4. Reverse Fine-tuning Prediction This is used to address the technical issues of jagged, fragmented output results and missed boundary reporting in the weak supervision output stage of inference localization.

[0062] S4.1: Basic Inference and High-Confidence Core Backbone Frame Locking During testing or inference in actual security deployments, the long video to be tested is input into the trained model, and the multi-instance alignment mapping matrix is ​​used. The binary classifier and the output function jointly produce a continuous frame-level anomaly confidence curve. A high hard threshold is set. ,like Filter out those with scores greater than 100 in the curve. The frame was identified as a high-confidence core anomaly frame.

[0063] S4.2: Temporal Continuity Threshold Determination and Blind Spot Bridging Repair Based on prior knowledge that anomalous events in the physical world, such as theft and arson, occur continuously and non-jump in the time domain, this scheme sets a time domain continuity threshold. , The unit is frames or seconds. The algorithm sequentially scans the time interval between two adjacent high-confidence core anomaly frames. ,like If so, it is determined that the two core abnormal frames belong to the same consecutive abnormal event.

[0064] At this point, the algorithm initiates a reverse bridging repair process, automatically modifying and re-marking all intermediate "normal blind zone frames" that were missed during the weak supervision stage due to insignificant features between the two core abnormal frames as "pseudo-label abnormal frames".

[0065] S4.3: Frame-level pseudo-tag adaptive reverse fine-tuning The novel frame-level pseudo-label sequence with complete and continuous temporal boundaries obtained after S4.2 correction is used as a strong supervision signal to perform local, batch-wise frame-level fine-tuning of the model's final classification output layer and temporal adapter.

[0066] S4.4: Final Precision Positioning Output After fine-tuning, the model was re-inferred from the video segment. The final frame-level confidence curve output this time eliminated jagged breaks and fragmentation, and the waveform boundaries highly matched the start and end points of real-world anomalous events. The system ultimately mapped the time periods and mapping matrix of continuous anomalous regions. The category name with the strongest response is output as the final high-precision frame-level localization result.

[0067] like Figures 2 to 5As shown, the frame-by-frame anomaly confidence scores generated on a series of test videos are visualized. Examples include anomaly events: explosions, fights, robberies, and a normal video. In each graph, the top three images show video frames, with red borders indicating frames where an anomaly event is occurring. The line graph below represents the confidence level of the anomaly event. The horizontal axis represents the video timeline, and the vertical axis represents the anomaly confidence score, ranging from 0 to 1. Higher values ​​indicate a higher probability of an anomaly event occurring. The pink area in the line graph corresponds to the actual time period of the anomaly event. As shown, the anomaly confidence values ​​obtained by our model closely match the actual situation. It can be seen that when various anomaly events occur, the anomaly score rises sharply; when the anomaly event ends, the anomaly score drops rapidly to a lower range. For normal videos, our method provides stable predictions throughout the video. The anomaly confidence scores for all frames in normal videos are below 0.005, close to 0. These results demonstrate that the present invention can powerfully and accurately locate various types of anomalies in benchmark videos, promptly indicating the onset of anomalies, while consistently maintaining extremely low anomaly scores for normal video sequences.

[0068] This specification provides a video anomaly localization system, including: The model building module is used to build an anomaly localization model that includes a CLIP visual encoder and a language encoder with parameter freezing, a temporal adapter connected in series on the output side of the visual encoder, a binary classifier, and a learnable cue template containing a stitching layer and a fusion layer.

[0069] The feature extraction and temporal adaptive fusion module is used to extract spatial feature matrices from multiple consecutive video segments of the training video. After combining the spatial feature matrix with the learnable temporal location encoding matrix, a temporal feature matrix is ​​obtained through a temporal adapter; then, the first anomaly confidence is obtained through a binary classifier.

[0070] The instance-aware dynamic text feature construction module uses a concatenation layer of learnable cue templates to concatenate preset anomaly category names into category vectors, and then uses a language encoder to map the category vectors into text embeddings containing category labels; through a fusion layer of learnable cue templates, the temporal feature matrix, the first anomaly confidence, and the text embeddings are fused to obtain dynamic text features.

[0071] The multi-task joint training module is used to jointly optimize the video-level Top-K classification loss, frame-level constraint loss, and alignment loss between temporal feature matrix and dynamic text features, with class labels as supervision, and to update the temporal adapter, learnable cue template and binary classifier parameters.

[0072] The reverse fine-tuning and final model construction module is used to update the anomaly localization model after training video input parameters, obtain the second anomaly confidence of each frame, lock the core frame by the second anomaly confidence of each frame and the preset threshold, and when the interval between adjacent core frames is less than the preset interval, perform local reverse fine-tuning on the temporal adapter and binary classifier according to the intermediate frame between adjacent core frames, and construct the final anomaly localization model based on the fine-tuned temporal adapter and binary classifier and the visual encoder.

[0073] For specific limitations regarding a video anomaly localization system, please refer to the limitations of a video anomaly localization method described above, which will not be repeated here. Each module in the aforementioned video anomaly localization system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0074] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0075] This specification provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A video anomaly localization method, characterized in that, Specifically, it includes: An anomaly localization model is constructed, which includes a parameter-frozen CLIP visual encoder and a language encoder, a temporal adapter connected in series on the output side of the visual encoder, a binary classifier, and a learnable cue template containing a stitching layer and a fusion layer. The splicing layer is connected in series in front of the input side of the speech encoder, and the fusion layer is connected in series in front of the output side of the speech encoder; Multiple consecutive video segments from the training video are input into the visual encoder to obtain a spatial feature matrix representing the complete training video. The spatial feature matrix is ​​combined with a preset learnable temporal location encoding matrix and then input into a temporal adapter. The evolution trend of inter-frame abnormal behavior is captured through a multi-head self-attention mechanism to obtain the temporal feature matrix. The time-series feature matrix is ​​input into a binary classifier to obtain the first anomaly confidence score. The preset anomaly category name is input into the concatenation layer to obtain the category vector. The category vector is then input into the language encoder to obtain the text embedding containing the category label. The temporal feature matrix, the first anomaly confidence score, and the text embedding are input into the fusion layer to obtain the dynamic text features. Using the category labels as supervision, the video-level Top-K classification loss, frame-level constraint loss, and alignment loss between the temporal feature matrix and dynamic text features are jointly optimized to update the temporal adapter, learnable cue template, and binary classifier parameters. The anomaly localization model, updated with the training video input parameters, yields the second anomaly confidence score for each frame; the core frame is then locked using the second anomaly confidence score of each frame and a preset threshold. When the interval between adjacent core frames is less than the preset interval, all image frames with insignificant features between adjacent core frames are forcibly modified and re-labeled as pseudo-labeled abnormal frames, resulting in a pseudo-labeled abnormal frame sequence. The pseudo-labeled abnormal frame sequence is used as a supervision signal to perform local reverse fine-tuning on the temporal adapter and binary classifier. Based on the fine-tuned temporal adapter and binary classifier, combined with the visual encoder, the final anomaly localization model is constructed.

2. The video anomaly localization method as described in claim 1, characterized in that: The multiple consecutive video segments are obtained by sampling the training video in a non-overlapping, equal-division manner, and the tails that are less than a fixed length are discarded or zero-padding is performed.

3. The video anomaly localization method as described in claim 1, characterized in that: The learnable temporal location encoding matrix has the same dimension as the spatial feature matrix.

4. The video anomaly localization method as described in claim 1, characterized in that: The timing adapter consists of at least one Transformer layer, which is composed of a multi-head self-attention mechanism and a feedforward neural network residual cascade, with each layer organized by layer normalization and residual connections.

5. The video anomaly localization method as described in claim 1, characterized in that, The dynamic text features are obtained by embedding the temporal feature matrix, the first anomaly confidence score, and the text into the input fusion layer, specifically including: Using the first anomaly confidence as a weight, the temporal feature matrix is ​​weighted by transpose dot product to obtain the visual context vector corresponding to the current video segment instance. The visual context vector is copied and expanded according to the number of categories, then added to the text embedding, and finally processed through a feedforward network and a residual shortcut to obtain dynamic text features.

6. The video anomaly localization method as described in claim 5, characterized in that: The binary classifier consists of a fully connected layer and a Sigmoid activation function. First anomaly confidence level The expression is: ； The visual context vector corresponding to the current video segment instance The expression is: ； in For the first Abnormal confidence of a video clip, It is the Sigmoid activation function. This is the weight matrix. For the first Temporal feature matrix of each video segment, This is the bias of the same binary classifier.

7. The video anomaly localization method as described in claim 1, characterized in that: The alignment loss is obtained by calculating the cosine similarity between each temporal feature in the temporal feature matrix and its corresponding dynamic text feature; for each cosine similarity, the dynamic text feature corresponding to each cosine similarity is determined to be either positive or negative based on the category label; for the column corresponding to the positive class, the positive class error between the cosine similarity above the preset similarity threshold and the preset positive class target value is calculated; for the column corresponding to the negative class, all cosine similarities in that column are reduced to the negative class target value, and the negative class error between each cosine similarity in that column and the preset negative class target value is calculated; all positive class errors and negative class errors are weighted and aggregated to obtain the alignment loss. The frame-level constraint loss includes a smoothing penalty term for the confidence difference between adjacent frames and a sparse regularization term that causes the score of normal frames to approach zero.

8. A video anomaly localization system, characterized in that, include: The model building module is used to build an anomaly localization model that includes a CLIP visual encoder and a speech encoder with parameter freezing, a temporal adapter connected in series with the output side of the visual encoder, a binary classifier, and a learnable cue template containing a stitching layer and a fusion layer; the stitching layer is connected in series with the input side of the speech encoder, and the fusion layer is connected in series with the output side of the speech encoder. The feature extraction and temporal adaptive fusion module is used to input multiple consecutive video segments of the training video into the visual encoder to obtain a spatial feature matrix representing the complete training video. The spatial feature matrix is ​​combined with a preset learnable temporal location encoding matrix and then input into a temporal adapter. The evolution trend of inter-frame abnormal behavior is captured through a multi-head self-attention mechanism to obtain the temporal feature matrix. The time-series feature matrix is ​​input into a binary classifier to obtain the first anomaly confidence score. The instance-aware dynamic text feature construction module is used to input the preset anomaly category name into the concatenation layer to obtain the category vector, input the category vector into the language encoder to obtain the text embedding containing the category label; and input the temporal feature matrix, the first anomaly confidence and the text embedding into the fusion layer to obtain the dynamic text features. The multi-task joint training module is used to jointly optimize the video-level Top-K classification loss, frame-level constraint loss, and alignment loss between the temporal feature matrix and dynamic text features, using the category labels as supervision, and to update the temporal adapter, learnable cue template, and binary classifier parameters. The reverse fine-tuning and final model building module is used to update the anomaly localization model with the input parameters of the training video to obtain the second anomaly confidence of each frame; and to lock the core frame by the second anomaly confidence of each frame and a preset threshold. When the interval between adjacent core frames is less than the preset interval, all image frames with insignificant features between adjacent core frames are forcibly modified and re-labeled as pseudo-labeled abnormal frames, resulting in a labeled abnormal frame sequence. The pseudo-labeled abnormal frame sequence is used as a supervision signal to perform local reverse fine-tuning on the temporal adapter and binary classifier. Based on the fine-tuned temporal adapter and binary classifier, combined with the visual encoder, the final anomaly localization model is constructed.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

10. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 7.

Images