Joint video anomaly detection method and device based on cascade Transformer
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-23
- Publication Date
- 2026-04-03
Smart Images
Figure CN121789104A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and in particular to a joint video anomaly detection method and apparatus based on cascaded Transformers. Background Technology
[0002] In applications such as intelligent transportation and industrial monitoring, abnormal events can threaten the safety of individuals and public property, causing varying degrees of damage and potentially leading to significant social impact in severe cases. In the early stages when electronic monitoring equipment was less sophisticated, traditional methods relied heavily on manpower for patrols to continuously monitor the surrounding environment and be alert for abnormal events. However, with the widespread adoption of intelligent monitoring equipment and the rapid development of artificial intelligence, traditional manual inspection methods have gradually been phased out due to their low efficiency and high cost. They have been replaced by computer vision-based video anomaly detection technology, which aims to automatically identify abnormal events in videos that deviate from normal behavioral patterns and has become an important research direction in the field of intelligent security. Among these, video anomaly detection methods based on weakly supervised learning have received widespread attention and become a research hotspot due to their advantages such as low annotation costs and strong adaptability. In recent years, with the rapid development of the large-scale visual-language pre-trained model CLIP, this model has demonstrated excellent performance in feature extraction and modeling the deep-level correlation between visual information and linguistic semantics, providing effective solutions for multiple downstream tasks. However, since CLIP was initially applied primarily to the image domain, its migration to the video domain requires careful consideration of the temporal dependencies between video frames. Given the significant advantages of the self-attention mechanism in the Transformer architecture in modeling contextual dependencies, some researchers have lightweighted its encoder structure and applied it to temporal modeling tasks of video frame sequences. However, due to the significant uncertainty and variability in the duration of anomalous events in the video over time, a single global self-attention mechanism is insufficient to effectively capture and model anomalous events across different time ranges, thus adversely affecting the accuracy of anomalous event detection. Simultaneously, it is necessary to fully explore the cross-modal correlation between visual information and linguistic semantics to further improve the performance of anomaly detection tasks through the auxiliary role of textual information.
[0003] Furthermore, current mainstream weakly supervised video anomaly detection methods typically rely on centralized training, where all video data is transmitted to a central server for unified processing. However, in practical applications, this method struggles to adequately protect data privacy, thus limiting its potential application in privacy-sensitive scenarios. Given the sensitivity of video content and the need for privacy protection, centrally collecting anomalous videos from multiple institutions or owners for training becomes extremely difficult, especially in multi-party collaborative scenarios where different institutions may be unable to share raw data due to privacy protection, trade secrets, or legal and regulatory restrictions.
[0004] Therefore, a joint video anomaly detection method and apparatus based on cascaded Transformers is provided. Summary of the Invention
[0005] The purpose of this invention is to provide a joint video anomaly detection method and apparatus based on cascaded Transformers, which can effectively capture the long-term and short-term temporal dependencies in videos, explore the connection between visual information and linguistic semantics, enhance the auxiliary role of text information in detection, and, combined with a federated learning environment, achieve a certain degree of protection of data privacy to overcome the shortcomings of existing technologies where all video data is transmitted to a central server for unified processing, making it difficult to fully guarantee data privacy.
[0006] To achieve the above objectives, the present invention provides the following technical solutions: In a first aspect, the present invention provides a joint video anomaly detection method based on cascaded Transformers, comprising the following steps: S1, the original video is sliced into single frames, and features of each frame are extracted and fused into a unified feature unit to obtain the input features; S2, the input features are input to the cascaded Transformer temporal modeling module to perform long-short-term temporal dependency modeling and obtain temporal modeling features; the cascaded Transformer temporal modeling module includes a local attention submodule and a global attention submodule; S3, based on the cross-attention prompt generation mechanism, combines global anomaly category information and local visual context information to generate text information features; S4. Perform a dot product operation on the temporal modeling features and text information features to obtain a feature alignment map. Calculate the prediction probability scores for multiple categories based on the feature alignment map and calculate the loss. Optimize the model parameters using the loss. S5, in a federated learning environment, local models are trained based on the optimized model parameters. After each round of training, each client uploads its trained local model parameters to the central server for integration to obtain aggregate parameters. The aggregate parameters are then used to update the model to obtain the aggregate model. The performance of the aggregate model is evaluated, and the global model is selected. At the same time, the aggregate parameters are distributed to each client, and each client performs the next round of global training based on the aggregate parameters. S6 tests the global model and evaluates the degree of anomaly of video frames by calculating the similarity between video frames and normal categories based on feature alignment maps, thereby obtaining an anomaly score for the video frames and realizing the detection of video anomalies.
[0007] Furthermore, the specific process of S2 is as follows: The input features are fed into a local attention submodule, which is a single-head self-attention Transformer encoder structure with a local window, to determine the location of the input features. The formula is as follows: ; in, The length of the input features, and the position of each input feature. The corresponding local attention window size is M is the attention mask matrix, and each element... Indicates from position Arrive at the location Whether self-attention is allowed; location and If they are within the same local window, then ,otherwise ; Local self-attention is achieved using the attention mask matrix M to obtain local temporal modeling features. The calculation formula is as follows: ; in, , For input features, , , Calculate the key vector respectively Query vector Sum value vector The generated weight matrix.
[0008] Local temporal modeling features obtained from local self-attention Temporal modeling features are obtained by inputting them into a submodule of a global single-head self-attention Transformer encoder structure without local windows. .
[0009] Furthermore, the specific process of S3 is as follows: Obtain the original global category label y, and then perform word segmentation using CLIP's tokenizer to obtain the category. ; The obtained category tokens are fed into CLIP's text encoder to generate corresponding category embedding vectors. And generate key vectors through transformation. Sum value vector ; Based on time-series modeling features exist Calculate the average value of the dimensions, and then... Transform into query vector Query vector Key vector Sum value vector A set of cue words is generated through a cross-attention mechanism. ; Set of category tokens and prompt words The fused components are then concatenated and fused to obtain the final result, which is then fed into CLIP's text encoder to generate text information features. .
[0010] Furthermore, the specific process of S4 is as follows: Temporal modeling features and text information features Performing a dot product operation yields the feature alignment map F; Based on a weakly supervised setup, only video-level labels are acquired. A Top-K mechanism is introduced to process the feature alignment map F, and the average value is calculated to obtain the confidence score of each video relative to each category. The predicted scores for each category are calculated using the following formula: ; in, This means the current video belongs to the [number]th [section]. The probability of each category It is a temperature hyperparameter; The loss is calculated using the cross-entropy function. And use the optimizer to update the parameters.
[0011] Furthermore, the specific process of S5 is as follows: The central server distributes the optimized model parameters to each client. Each client trains a local model based on the optimized model parameters. The client receives and loads the parameters for the current global iteration. ; After each training round, the client uploads the parameters of the local model to the central server, which then integrates the information from all clients to obtain the aggregate parameters. This generates the parameters for the next round of the initial global model. ; Based on different datasets, the area under the ROC curve or the average precision is used as the evaluation metric to evaluate the performance of the aggregated model on the entire test dataset, and the global model is selected. The central server will use the parameters of the initial global model obtained in the first round. Distribute to each client for the next round of global training.
[0012] Furthermore, the parameters for generating the next round's initial global model... The calculation formula is: ; in, Indicates the client The length of the dataset.
[0013] Furthermore, the time-series modeling features The calculation formula is:
[0014] in, , These are the local temporal modeling features obtained through local self-attention. , , Calculate the key vector respectively Query vector Sum value vector The generated weight matrix.
[0015] Secondly, the present invention also provides a joint video anomaly detection system based on cascaded Transformers, comprising: Extraction and fusion module: used to cut the original video into single frames, extract features from each frame and fuse them into a unified feature unit to obtain the input features; The first generation module is used to input the input features into the cascaded Transformer temporal modeling module to perform long-short-term temporal dependency modeling and obtain temporal modeling features; the cascaded Transformer temporal modeling module includes a local attention submodule and a global attention submodule; The second generation module is used for a cross-attention-based prompt generation mechanism, which combines global anomaly category information with local visual context information to generate text information features. The computation optimization module, in a federated learning environment, is used to perform a dot product operation on temporal modeling features to obtain a feature alignment map, calculate the prediction probability scores of multiple classes based on the feature alignment map, calculate the loss, and optimize the model parameters through the loss. Training module: Used to train local models based on optimized model parameters. After each round of training, each client uploads its trained local model parameters to the central server for integration to obtain aggregate parameters. The aggregate parameters are then used to update the model to obtain the aggregate model. The performance of the aggregate model is evaluated, and the global model is selected. At the same time, the aggregate parameters are distributed to each client, and each client performs the next round of global training based on the aggregate parameters. Detection module: Used to test the global model. Based on the feature alignment map, it evaluates the degree of anomaly of video frames by calculating the similarity between video frames and normal categories, and obtains the anomaly score of the video frame to realize the detection of video anomalies.
[0016] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned joint video anomaly detection method based on cascaded Transformers.
[0017] Fourthly, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned joint video anomaly detection method based on cascaded Transformers.
[0018] Compared with the prior art, the present invention has the following beneficial technical effects: This invention provides a joint video anomaly detection method based on cascaded Transformers. A cascaded Transformer temporal modeling module is designed, consisting of a local attention submodule and a global attention submodule. These two submodules work together to accurately capture temporal dependencies in the video. The local attention submodule focuses on short-term local temporal features, efficiently acquiring real-time correlations between video frames by calculating self-attention mechanisms within a limited window, ensuring the model can keenly capture rapidly changing abnormal behaviors. The global attention submodule, on the other hand, takes a global perspective, covering the entire video sequence and capturing long-term global temporal patterns, enabling the model to identify different durations of abnormal events along the timeline. This combined local and global temporal modeling approach enhances the model's ability to detect different types of abnormal events, significantly improving detection accuracy and robustness. This invention employs a cross-attention-based prompt generation mechanism and trains in a federated learning environment, effectively improving model performance while ensuring data privacy protection. On one hand, the cross-attention mechanism can deeply integrate visual information and linguistic semantics to generate accurate textual information features, strengthening the auxiliary role of text in anomaly detection, thereby improving the model's understanding and recognition of abnormal events. On the other hand, the federated learning framework allows each client to train the model locally, uploading only the parameters to the central server for aggregation. This avoids sharing the original data and effectively protects data privacy. This approach not only enhances the model's anomaly detection capabilities in complex scenarios but also expands its practical application in privacy-sensitive areas, providing an efficient and secure solution for scenarios such as intelligent security. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of a joint video anomaly detection method based on cascaded Transformers in an embodiment of the present invention.
[0020] Figure 2 This is a flowchart illustrating the structure of the cascaded Transformer temporal modeling module proposed in a joint video anomaly detection model based on cascaded Transformers in an embodiment of the present invention. Figure 3 This is a schematic diagram of the cross-attention-based prompt generation mechanism module introduced in a joint video anomaly detection model based on cascaded Transformers in an embodiment of the present invention.
[0021] Figure 4 This is a video anomaly score curve for the shooting category during the testing phase of this invention.
[0022] Figure 5 This is a video anomaly score curve for the explosion category during the testing phase of this invention.
[0023] Figure 6 This is a video anomaly score curve for the arrest category during the testing phase of this invention.
[0024] Figure 7 This is a graph showing the abnormal video score curves for the normal category during the testing phase of this invention. Detailed Implementation
[0025] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.
[0026] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.
[0027] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0028] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a communication connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0029] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.
[0030] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0031] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0032] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0033] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.
[0034] This invention provides a joint video anomaly detection method based on cascaded Transformers, such as... Figure 1 The diagram shown is a flowchart of a joint video anomaly detection method based on cascaded Transformers according to the present invention, which specifically includes the following steps: S1, the original video is sliced into single frames, and features of each frame are extracted and fused into a unified feature unit to obtain the input features; In a more specific embodiment of the present invention, the original video is extracted frame by frame, and each frame is input into the CLIP model image encoder for feature extraction. Subsequently, a specific fusion method is used to merge the features of every 16 frames to obtain a unified feature representation, which serves as the final input feature. .
[0035] S2, input features The input is fed into the cascaded Transformer temporal modeling module to perform long-short-term temporal dependency modeling and obtain temporal modeling features; the cascaded Transformer temporal modeling module includes a local attention submodule and a global attention submodule; In a more specific embodiment of the present invention, such as Figure 2 As shown, the input features obtained in S1 The input is fed into the cascaded Transformer temporal modeling module to model the short- and long-term temporal dependencies of video frames and obtain temporal modeling features. The cascaded Transformer temporal modeling module includes a local attention submodule for modeling local temporal relationships and a global attention submodule for capturing global dependencies.
[0036] The specific implementation steps are as follows: Step 2.1: Extracting input features from the CLIP image encoder The input is fed into the local attention submodule for temporal dependency modeling. The local attention submodule is a submodule of a Transformer encoder structure based on a window attention mechanism, which computes self-attention within each local window to process local temporal information. First, the input features are determined. Position, formula: ; in, The length of the input features, and the position of each input feature. The corresponding local attention window size is M is the attention mask matrix, and each element... Indicates from position Arrive at the location Whether self-attention is allowed; location and If they are within the same local window, then ,otherwise ; Next, the attention mask matrix M is used to calculate local self-attention: ; in, , For input features, , , Calculate the key vector respectively Query vector Sum value vector The generated weight matrix.
[0037] Step 2.2: Apply local self-attention The input is fed into a submodule of a global single-head self-attention Transformer encoder structure without local windows for further processing and to obtain the final feature representation with long short-term temporal dependencies. :
[0038] in , These are the local temporal modeling features obtained through local self-attention. , , Calculate the key vector respectively Query vector Sum value vector The generated weight matrix.
[0039] S3, based on the cross-attention prompt generation mechanism, combines global anomaly category information and local visual context information to generate text information features; In a more specific embodiment of the present invention, such as Figure 3 As shown, by utilizing a cross-attention-based cue generation mechanism and combining global activity category information with local visual context information, more accurate text information features are generated. ,in This represents the number of category labels.
[0040] The specific implementation steps are as follows: Step 3.1: Use CLIP's tokenizer to segment global activity category tags. Perform word segmentation to obtain categories. ; Step 3.2: Input the obtained category tokens into CLIP's text encoder to generate the corresponding category embedding vectors. Then, through mapping operations, the key vector is further transformed to generate a key vector. Sum value vector ; Step 3.3: Contextual temporal modeling features with long and short-term temporal dependencies Firstly, in terms of dimensions Averaging is performed on the above to obtain Then map it to a query vector Combined with the key vector mapped by the category embedding obtained in step 3.2 AND value vector A set of cue words that integrates visual information is generated through a cross-attention mechanism. .
[0041]
[0042] in, , It is a feature for contextual temporal modeling with long and short-term temporal dependencies. In dimensions The result is obtained by averaging over the above. For category embedding vectors, , , Calculate the key vector respectively Query vector Sum value vector The generated weight matrix; Step 3.4: Combine the category tokens obtained in Step 3.1 with the set of prompt words generated in Step 3.3 using visual information guidance. The sequences are spliced and merged to form an enhanced text input sequence. To preserve and differentiate the structural order information of each part, position embedding vectors are further added to the concatenated sequence to clarify the position information of each token. Finally, The text information features are generated by the text encoder of CLIP as input. .
[0043] S4. Perform a dot product operation on the temporal modeling features and text information features to obtain a feature alignment map. Calculate the prediction probability scores for multiple categories based on the feature alignment map and calculate the loss. Optimize the model parameters using the loss. In a more specific embodiment of the present invention, the temporal modeling features obtained in step 2 are... Textual information features based on visual information obtained in step 3 Perform a dot product operation to obtain the feature alignment map. The Top-k strategy is used to obtain the matching degree between each video and various tags. Then, the predicted probability of each type of label is calculated. The model parameters are optimized by calculating the difference between the predicted values and the actual labels using the cross-entropy loss function.
[0044] Step 4.1: Apply the temporal modeling features obtained after cascading Transformer modeling in Step 2. And the text information in step 3 and your features The alignment map between video frames and different activity categories is calculated using the following formula. ;
[0045] Step 4.2: Given the weakly supervised setup, only video-level labels can be obtained during training. Therefore, a Top-K mechanism is used, and the average of these labels is calculated to evaluate the confidence level between the current video and each category, denoted as . Next, the multi-category prediction scores are calculated using the following formula.
[0046]
[0047] in, This indicates that the current video belongs to the [number]th [section]. The probability of the category, This is a temperature hyperparameter. The loss was then calculated using the cross-entropy loss function. and utilize optimizers (such as Update the model parameters.
[0048] S5, in a federated learning environment, local models are trained based on the optimized model parameters. After each round of training, each client uploads its trained local model parameters to the central server for integration to obtain aggregate parameters. The aggregate parameters are then used to update the model to obtain the aggregate model. The performance of the aggregate model is evaluated, and the global model is selected. At the same time, the aggregate parameters are distributed to each client, and each client performs the next round of global training based on the aggregate parameters. In a more specific embodiment of the present invention, within the federated learning framework, each client independently trains its local model. After completing one round of local training, the client uploads its model parameters to a central server for unified aggregation. The central server evaluates the performance of the aggregated global model using either the area under the ROC curve (Receiver Operations Curve) or Average Precision (AP) as evaluation metrics, depending on the dataset, and selects the best-performing model as the global model. Subsequently, the parameters of this global model are distributed to each client for the next round of local training and optimization iterations.
[0049] The specific implementation steps are as follows: Step 5.1: The central server distributes the optimized model parameters to each client. Each client trains a local model based on the optimized model parameters. The client receives and loads the parameters for the current global round. ; After each training round, the client uploads the parameters of the local model to the central server, which then integrates the information from all clients to obtain the aggregate parameters. This generates the parameters for the next round of the initial global model. ; Parameters of the next round of initial global model The calculation formula is as follows: ; in Indicates the client The length of the dataset; Step 5.3: Based on different datasets, evaluate the performance of the fused global model on the entire test dataset using the area under the ROC curve (Receiver Operating Characteristic curve) or Average Precision (AP) as evaluation metrics to determine the performance of the current global model and the optimal global model. Subsequently, the central server aggregates the global model parameters obtained from this global round. Distribute to each client for the next round of training and optimization.
[0050] Step 6: Test using the global model with superior evaluation results. Since feature alignment maps of video frames and each category have already been obtained... The degree of anomalousness of a video frame can be assessed by calculating its similarity to a normal category. A higher similarity indicates that the video frame is more likely to belong to the normal category, while a lower similarity may indicate anomalies. The anomalousness score of a video frame can be quantified by subtracting 1 from the similarity value with the normal category. Figures 4 to 7 The figures show the prediction results of video anomaly scores for some categories during the testing phase of the proposed joint video anomaly detection method based on cascaded Transformers. As shown in the figures, frames with black borders in the upper half of the video frame graph represent anomalous frames, while the black curve in the lower half of the graph represents the anomaly score, and the gray rectangular areas mark the range of actual anomalous frames. The degree of anomalousness of a video frame can be assessed by calculating its similarity to a normal category. A higher similarity indicates that the video frame is more likely to belong to the normal category, while a lower similarity may indicate anomalies. The anomalousness score of a video frame can be quantified by subtracting 1 from the similarity value with the normal category. Figures 4 to 7The figures show the prediction results of video anomaly scores for some categories in the testing phase of the proposed joint video anomaly detection method based on cascaded Transformers. As shown in the figures, frames with black borders in the upper half of the video frame section represent anomalous frames, while the black curves in the lower half of the graph represent anomaly scores, and the gray rectangular areas mark the range of actual anomalous frames.
[0051] In another embodiment of the present invention, a joint video anomaly detection system based on cascaded Transformers is also provided to implement a joint video anomaly detection method based on cascaded Transformers, comprising: Extraction and fusion module: used to cut the original video into single frames, extract features from each frame and fuse them into a unified feature unit to obtain the input features; The first generation module is used to input the input features into the cascaded Transformer temporal modeling module to perform long-short-term temporal dependency modeling and obtain temporal modeling features; the cascaded Transformer temporal modeling module includes a local attention submodule and a global attention submodule; The second generation module is used for a cross-attention-based prompt generation mechanism, which combines global anomaly category information with local visual context information to generate text information features. The computation optimization module, in a federated learning environment, is used to perform a dot product operation on temporal modeling features to obtain a feature alignment map, calculate the prediction probability scores of multiple classes based on the feature alignment map, calculate the loss, and optimize the model parameters through the loss. Training module: Used to train local models based on optimized model parameters. After each round of training, each client uploads its trained local model parameters to the central server for integration to obtain aggregate parameters. The aggregate parameters are then used to update the model to obtain the aggregate model. The performance of the aggregate model is evaluated, and the global model is selected. At the same time, the aggregate parameters are distributed to each client, and each client performs the next round of global training based on the aggregate parameters. Detection module: Used to test the global model. Based on the feature alignment map, it evaluates the degree of anomaly of video frames by calculating the similarity between video frames and normal categories, and obtains the anomaly score of the video frame to realize the detection of video anomalies.
[0052] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described joint video anomaly detection method based on cascaded Transformers.
[0053] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described joint video anomaly detection method based on cascaded Transformers.
[0054] This invention is described based on flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to specific embodiments. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowcharts and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0055] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0056] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0057] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Those skilled in the art can modify the technical solutions described in the above embodiments, or make equivalent substitutions for some of the technical features; and all such modifications and substitutions should fall within the protection scope of the present invention.
Claims
1. A joint video anomaly detection method based on cascaded Transformers, characterized in that, Includes the following steps: S1, the original video is sliced into single frames, and features of each frame are extracted and fused into a unified feature unit to obtain the input features; S2, the input features are input to the cascaded Transformer temporal modeling module to perform long-short-term temporal dependency modeling and obtain temporal modeling features; the cascaded Transformer temporal modeling module includes a local attention submodule and a global attention submodule; S3, based on the cross-attention prompt generation mechanism, combines global anomaly category information and local visual context information to generate text information features; S4. Perform a dot product operation on the temporal modeling features and text information features to obtain a feature alignment map. Calculate the prediction probability scores for multiple categories based on the feature alignment map and calculate the loss. Optimize the model parameters using the loss. S5, in a federated learning environment, local models are trained based on the optimized model parameters. After each round of training, each client uploads its trained local model parameters to the central server for integration to obtain aggregate parameters. The aggregate parameters are then used to update the model to obtain the aggregate model. The performance of the aggregated model is evaluated, the global model is selected, and the aggregated parameters are distributed to each client. Each client then performs the next round of global training based on the aggregated parameters. S6 tests the global model and evaluates the degree of anomaly of video frames by calculating the similarity between video frames and normal categories based on feature alignment maps, thereby obtaining an anomaly score for the video frames and realizing the detection of video anomalies.
2. The joint video anomaly detection method based on cascaded Transformers according to claim 1, characterized in that, The specific process of S2 is as follows: The input features are fed into a local attention submodule, which is a single-head self-attention Transformer encoder structure with a local window, to determine the location of the input features. The formula is as follows: ； in, The length of the input features, and the position of each input feature. The corresponding local attention window size is M is the attention mask matrix, and each element... Indicates from position Arrive at the location Whether self-attention is allowed; location and If they are within the same local window, then ,otherwise ; Local self-attention is achieved using the attention mask matrix M to obtain local temporal modeling features. The calculation formula is as follows: ； in, , For input features, , , Calculate the key vector respectively Query vector Sum value vector The generated weight matrix. Local temporal modeling features obtained from local self-attention Temporal modeling features are obtained by inputting them into a submodule of a global single-head self-attention Transformer encoder structure without local windows. .
3. The joint video anomaly detection method based on cascaded Transformers according to claim 2, characterized in that, The specific process of S3 is as follows: Obtain the original global category label y, and then perform word segmentation using CLIP's tokenizer to obtain the category. ; The obtained category tokens are fed into CLIP's text encoder to generate corresponding category embedding vectors. And generate key vectors through transformation. Sum value vector ; Based on time-series modeling features exist Calculate the average value of the dimensions, and then... Transform into query vector Query vector Key vector Sum value vector A set of cue words is generated through a cross-attention mechanism. ; Set of category tokens and prompt words The fused components are then concatenated and fused to obtain the final result, which is then fed into CLIP's text encoder to generate text information features. .
4. The joint video anomaly detection method based on cascaded Transformers according to claim 3, characterized in that, The specific process of S4 is as follows: Temporal modeling features and text information features Performing a dot product operation yields the feature alignment map F; Based on a weakly supervised setup, only video-level labels are acquired. A Top-K mechanism is introduced to process the feature alignment map F, and the average value is calculated to obtain the confidence score of each video relative to each category. The predicted scores for each category are calculated using the following formula: ； in, This means the current video belongs to the [number]th [section]. The probability of each category It is a temperature hyperparameter; The loss is calculated using the cross-entropy function. And use the optimizer to update the parameters.
5. The joint video anomaly detection method based on cascaded Transformers according to claim 4, characterized in that, The specific process of S5 is as follows: The central server distributes the optimized model parameters to each client. Each client trains a local model based on the optimized model parameters. The client receives and loads the parameters for the current global iteration. ; After each training round, the client uploads the parameters of the local model to the central server, which then integrates the information from all clients to obtain the aggregate parameters. This generates the parameters for the next round of the initial global model. ; Based on different datasets, the area under the ROC curve or the average precision is used as the evaluation metric to evaluate the performance of the aggregated model on the entire test dataset, and the global model is selected. The central server will use the parameters of the initial global model obtained in the first round. Distribute to each client for the next round of global training.
6. The joint video anomaly detection method based on cascaded Transformers according to claim 5, characterized in that, The parameters for generating the next round of the initial global model The calculation formula is: ； in, Indicates the client The length of the dataset.
7. The joint video anomaly detection method based on cascaded Transformers according to claim 2, characterized in that, The time-series modeling features The calculation formula is: in, , These are the local temporal modeling features obtained through local self-attention. , , Calculate the key vector respectively Query vector Sum value vector The generated weight matrix.
8. A joint video anomaly detection system based on cascaded Transformers, characterized in that, include: Extraction and fusion module: used to cut the original video into single frames, extract features from each frame and fuse them into a unified feature unit to obtain the input features; The first generation module is used to input the input features into the cascaded Transformer temporal modeling module to perform long-short-term temporal dependency modeling and obtain temporal modeling features; the cascaded Transformer temporal modeling module includes a local attention submodule and a global attention submodule; The second generation module is used for a cross-attention-based prompt generation mechanism, which combines global anomaly category information with local visual context information to generate text information features. The computation optimization module, in a federated learning environment, is used to perform a dot product operation on temporal modeling features to obtain a feature alignment map, calculate the prediction probability scores of multiple classes based on the feature alignment map, calculate the loss, and optimize the model parameters through the loss. Training module: Used to train local models based on the optimized model parameters. After each round of training, each client uploads its trained local model parameters to the central server for integration to obtain aggregate parameters. The aggregate parameters are then used to update the model to obtain the aggregate model. The performance of the aggregated model is evaluated, the global model is selected, and the aggregated parameters are distributed to each client. Each client then performs the next round of global training based on the aggregated parameters. Detection module: Used to test the global model. Based on the feature alignment map, it evaluates the degree of anomaly of video frames by calculating the similarity between video frames and normal categories, and obtains the anomaly score of the video frame to realize the detection of video anomalies.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the joint video anomaly detection method based on any one of claims 1-7.
10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the joint video anomaly detection method based on cascaded Transformers as described in any one of claims 1-7.