Video anomaly detection method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides a video anomaly detection method and device, equipment, a medium and a program product, and relates to the technical field of anomaly detection. The method comprises the following steps: extracting cross-scale local features and global features from an input image, and fusing the cross-scale local features and the global features to obtain a corresponding fused feature map; performing feature enhancement on each fusion feature map to obtain a corresponding feature enhancement map; performing feature storage on the feature enhancement image, and matching memory features most related to the current original image based on the stored features, so as to reconstruct a reconstructed feature image corresponding to the current original image based on the memory features; and comparing the difference between the reconstructed feature map and the current original image based on a preset difference threshold so as to perform anomaly detection on the current original image. According to the embodiment of the invention, by combining the local feature extraction capability of the convolutional layer and the long-distance dependency capture capability of the state space model architecture, the calculation efficiency and accuracy of abnormal video detection can be improved at the same time.

Description

Technical Field

[0001] The present application relates to the field of anomaly detection technology, and in particular to a video anomaly detection method, device, equipment, medium and program product. Background Art

[0002] Video anomaly detection is an important research direction in the field of computer vision, which aims to automatically identify abnormal events that are inconsistent with normal behavior patterns from video sequences.

[0003] Traditional methods for detecting abnormal events in videos mainly rely on supervised learning, which requires a large amount of labeled data to train the model. However, in practical applications, abnormal behavior samples are usually scarce and difficult to label, resulting in poor performance of existing anomaly detection schemes.

[0004] In addition, although deep learning methods such as CNN (Convolutional Neural Network) and Transformer have achieved remarkable success in the fields of image and natural language processing, they still have the following problems in video anomaly detection: CNN is good at capturing local features, but it is insufficient in capturing long-distance dependencies and global context information. Therefore, for time series data such as video, CNN is difficult to effectively capture long-term dependencies across frames, resulting in low accuracy in detecting abnormal events. Although Transformer can effectively capture long-distance dependency features, its computational complexity grows quadratically with the length of the input sequence, resulting in huge computational overhead when processing high-resolution videos, especially poor performance when running on resource-constrained devices.

[0005] In summary, there is an urgent need for a video anomaly detection solution that can balance accuracy and computational efficiency. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a video anomaly detection method, device, equipment, medium and program product to improve the accuracy and computational efficiency of video anomaly detection.

[0007] In a first aspect, an embodiment of the present application provides a video anomaly detection method, comprising:

[0008] Based on the dual-path attention aggregation module, the input images of each convolutional layer in the CNN are sequentially extracted with cross-scale local features and global features, and the local features and the global features are fused to obtain a corresponding fused feature map;

[0009] Based on the preset state space feature enhancement model, each fusion feature map is enhanced to obtain a corresponding feature enhancement map;

[0010] Using a memory module to store features of the feature enhancement map output by the CNN encoder, and matching the memory features most relevant to the current original image based on the stored features, so that the CNN decoder reconstructs a reconstructed feature map corresponding to the current original image based on the memory features;

[0011] The difference between the reconstructed feature map and the current original image is compared based on a preset difference threshold to perform abnormality detection on the current original image.

[0012] In the embodiment of the present application, by combining the local feature extraction capability of the convolutional layer and the long-distance dependency capture capability of the state-space model architecture, the computational efficiency and accuracy of abnormal video detection can be improved simultaneously.

[0013] In some possible embodiments, the dual-path attention aggregation module sequentially extracts cross-scale local features and global features from the input images of each convolutional layer in the CNN, including:

[0014] Based on the local feature aggregation module, a multi-scale convolution operation is performed on the input image, and the results of the multi-scale convolution operation are spliced ​​to obtain an initial local feature map;

[0015] Performing pooling operations and convolution operations on the local feature map in sequence, and generating attention weights based on a preset Sigmoid activation function;

[0016] The attention weight is upsampled and then element-wise multiplied with the local feature map, and a convolution operation is performed on the multiplication result to obtain a local feature map corresponding to the input image.

[0017] In an embodiment of the present application, by extracting cross-scale local features of the input feature map, it is ensured that the model can capture detail information and contextual information under different perception fields of view, thereby further improving the accuracy of video anomaly detection.

[0018] In some possible embodiments, the dual-path attention aggregation module sequentially extracts cross-scale local features and global features from the input images of each convolutional layer in the CNN, including:

[0019] Based on the global feature aggregation module, the input image is pooled to obtain the global context feature;

[0020] Performing a convolution operation on the global context feature to obtain a compact global feature map;

[0021] The compact global feature map is feature enhanced by using a preset ReLU activation function, and a convolution operation and an upsampling operation are sequentially performed on the result after feature enhancement to obtain a global feature map corresponding to the input image.

[0022] In an embodiment of the present application, by extracting the global context information of the input image, the model can integrate the global information of the entire input, improve the global perception ability when understanding complex scenes, and thus further improve the accuracy of video anomaly detection.

[0023] In some possible embodiments, fusing the local features and the global features to obtain a corresponding fused feature map includes:

[0024] Performing element-by-element multiplication and fusion on the local features and the global features to obtain a corresponding preliminary fusion feature map;

[0025] An addition operation is performed on the input image and the preliminary fusion feature map to obtain the fusion feature map.

[0026] In an embodiment of the present application, after fusing local features and global features, the underlying local detail information and high-level semantic information are fused through an addition operation to enhance the model's ability to perceive the balance between details and semantics, thereby further improving the accuracy of video anomaly detection.

[0027] In some possible embodiments, the feature enhancement of each fused feature map based on a preset state space feature enhancement model to obtain a corresponding feature enhancement map includes:

[0028] In the remaining convolutional layers except the starting layer of the CNN decoder, the first feature enhancement module is used to perform the first batch normalization and ReLU activation function operation, the convolution operation, the second batch normalization and ReLU activation function operation and the sequence conversion operation on the corresponding fused feature map in sequence to obtain the first sequence feature;

[0029] Performing a linear regression operation, a convolution operation, a SiLU activation function operation, and a state space operation on the first sequence features in sequence to obtain a first sequence enhanced feature;

[0030] Performing a linear regression operation and a SiLU activation function operation on the first sequence features in sequence to obtain a second sequence enhanced feature;

[0031] The first sequence enhanced features and the second sequence enhanced features are multiplied element by element, and a linear regression operation and a sequence inverse transformation are performed on the multiplication results in sequence to obtain a feature enhancement map corresponding to each convolutional layer.

[0032] In an embodiment of the present application, the first feature enhancement module utilizes a state space model to increase the long-distance dependencies of the input features, so that both long-distance dependencies and spatial details can be retained when reconstructing a high-resolution feature map, thereby further improving the accuracy of video anomaly detection while improving the computational efficiency of the model.

[0033] In some possible embodiments, the feature enhancement of each fused feature map based on a preset state space feature enhancement model to obtain a corresponding feature enhancement map includes:

[0034] At the starting layer of the CNN decoder, the second feature enhancement module is used to perform the first batch normalization and ReLU activation function operation, the first convolution operation, the second batch normalization and ReLU activation function operation, the second convolution operation, the tanh activation function operation and the sequence conversion operation on the input features in sequence to obtain the second sequence features;

[0035] Performing a linear regression operation, a convolution operation, a SiLU activation function operation, and a state space operation on the second sequence features in sequence to obtain a third sequence enhanced feature;

[0036] Performing a linear regression operation and a SiLU activation function operation on the second sequence features in sequence to obtain a fourth sequence enhanced feature;

[0037] The third sequence enhanced features and the fourth sequence enhanced features are element-by-element multiplied, and a linear regression operation and a sequence inverse transformation are sequentially performed on the multiplication results to obtain a feature enhancement map corresponding to the starting layer.

[0038] In an embodiment of the present application, the memory features selected by the memory module are continuously convolved, batch normalized, and ReLU operated by first using the second feature enhancement module at the starting layer of the decoder, thereby ensuring that the features maintain a relatively stable distribution and enhancing the nonlinear modeling ability of the network, thereby further improving the accuracy of video anomaly detection and improving the computational efficiency of the model.

[0039] In a second aspect, an embodiment of the present application provides a video anomaly detection device, comprising:

[0040] A feature extraction module is used to extract cross-scale local features and global features from the input images of each convolutional layer in the CNN in turn based on the dual-path attention aggregation module, and fuse the local features with the global features to obtain a corresponding fused feature map;

[0041] A feature enhancement module is used to enhance the features of each fused feature map based on a preset state space feature enhancement model to obtain a corresponding feature enhancement map;

[0042] A feature memory module is used to store the feature enhancement map output by the CNN encoder using the memory module, and match the memory feature most relevant to the current original image based on the stored features, so that the CNN decoder reconstructs the reconstructed feature map corresponding to the current original image based on the memory feature;

[0043] The anomaly detection module is used to compare the difference between the reconstructed feature map and the current original image based on a preset difference threshold to perform anomaly detection on the current original image.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the method described in any embodiment of the first aspect when executing the program.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of a video anomaly detection method provided in an embodiment of the present application;

[0049] Figure 2 The overall architecture diagram of the unsupervised video anomaly detection solution based on the Mamba architecture provided in the embodiment of the present application;

[0050] Figure 3 A schematic diagram of the workflow of the dual-path attention aggregation module provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the workflow of the first feature enhancement module provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the workflow of the second feature enhancement module provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of a video anomaly detection device provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0056] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0057] It should be noted that traditional supervised learning-based methods require a large amount of labeled data and are difficult to adapt to the data distribution in practical applications. For example, CNN performs well in local feature extraction, but is insufficient in capturing long-distance dependencies and global context information; while Transformer can effectively capture long-distance dependencies, its computational complexity grows quadratically with the length of the input sequence, resulting in huge computational overhead when processing high-resolution videos, especially poor performance when running on resource-constrained devices. In order to overcome these problems, state space models (SSMs) such as Mamba have been proposed. These models can process long sequence data with linear complexity, thereby significantly improving computational efficiency. However, the existing SSMs still have certain deficiencies in local feature extraction, making it difficult to fully utilize the local information of images in visual tasks.

[0058] In response to the problems existing in the above-mentioned prior art, an embodiment of the present application provides an unsupervised video anomaly detection method based on the Mamba architecture, which improves the accuracy and computational efficiency of video anomaly detection by effectively combining the local feature extraction capability of the convolutional layer with the long-distance dependency capture capability of the Mamba architecture. The method is particularly suitable for resource-constrained devices and scenarios of high-resolution image processing.

[0059] First, an embodiment of the present application designs a dual-path attention aggregation module (Double PathPolymerize Attention model, DPPAM), which enhances the interaction and fusion of local and global features through a dual-path structure, thereby improving the adaptability of the model to complex video scenes, especially when processing videos containing fine-grained abnormal behaviors, and can more accurately locate abnormal areas.

[0060] Secondly, in response to the limitations of existing convolutional neural networks (CNNs) in capturing long-term dependent features and the high computational complexity of the Transformer structure, the embodiment of the present application designs a first feature enhancement module and a second feature enhancement module based on the Mamba (a type of SSM model) mechanism. These two modules have the ability to capture long-term dependencies with linear time complexity, thereby greatly improving the computational efficiency of video anomaly detection.

[0061] Finally, due to the hardware-aware characteristics of the Mamba architecture, the computational efficiency and scalability of the first feature enhancement module and the second feature enhancement module have been significantly improved. In order to address the problem of scarce abnormal behavior samples and data imbalance, the embodiment of the present application also introduces a memory module, which can store, read and update all normal behavior features in a specific scenario during the training process, thereby efficiently utilizing historical information during the reasoning process and further enhancing the robustness and generalization ability of the detection system.

[0062] like Figure 1 As shown, the embodiment of the present application provides a video anomaly detection method, which may include the steps of:

[0063] S1. Based on the dual-path attention aggregation module, the input images of each convolutional layer in CNN are extracted with cross-scale local features and global features in turn, and the local features and global features are fused to obtain the corresponding fused feature map.

[0064] It should be noted that in the feature extraction stage, step S1 extracts local features and global features from the input continuous video frames through a dual-path attention aggregation module, such as Figure 2 As shown in the figure, in the process of video anomaly detection, each convolutional layer of CNN has a dual-path attention aggregation module (DPPAM). Before entering the detection model, the continuous video frames are converted into an original image corresponding to each frame. Each original image is extracted by the encoder, stored and matched by the memory module, and reconstructed by the decoder. Finally, a reconstructed feature map corresponding to the original image is formed. Figure 3 As shown, the dual-path attention aggregation module includes a local feature aggregation module and a global feature aggregation module.

[0065] In some possible embodiments, step S1 may include:

[0066] S101, performing a multi-scale convolution operation on the input image based on the local feature aggregation module, and splicing the results of the multi-scale convolution operation to obtain an initial local feature map;

[0067] S102, performing pooling operations and convolution operations on the local feature map in sequence, and generating attention weights based on a preset Sigmoid activation function;

[0068] S103, after upsampling the attention weight, multiplying it element-by-element with the local feature map, and performing a convolution operation on the multiplication result to obtain a local feature map corresponding to the input image.

[0069] It should be noted that in the dual-path attention aggregation module, the local feature aggregation module is used to extract cross-scale local features of the input image to ensure that the model can capture detail information and contextual information under different receptive fields.

[0070] See also Figure 3 Specifically, after the input image (the input image of the first convolutional layer is the original image, and the input image of the remaining layers is the output of the previous convolutional layer) enters the local feature aggregation module of each convolutional layer, first, it is convolved through three convolution kernels of different sizes k i For the input image F input Perform multi-scale convolution operations to extract multi-scale local features under different receptive fields:

[0071]

[0072] Where i=1, 2, 3, illustratively, k i They can represent 3×3, 5×5, and 7×7 convolution kernels respectively. Indicates the use of k i Perform convolution operation.

[0073] Then, the multi-scale feature map F obtained above is i Use the channel dimension concatenation strategy to concatenate (Concatenate) to form a new local feature map F local (Preliminary local feature map):

[0074] F local =Concat(F1,F2,F3)

[0075] This strategy allows the spliced ​​features to retain detailed information of different scales and improve the performance of the model when dealing with multi-scale targets and complex scenes. In particular, when dealing with objects of different scales, it can respond to local features more adaptively.

[0076] In order to enhance the model’s ability to focus on key information, an attention mechanism is introduced after feature aggregation. First, the feature map F local Perform a pooling operation to reduce the spatial dimension of the feature map while retaining the global context information to obtain the feature map F pool :

[0077] F pool =P(F local )

[0078] Where P(·) represents the pooling operation.

[0079] Then, the pooled feature map is compressed by a 1×1 convolution operation to obtain a new feature map F attn :

[0080] F attn =Conv 1×1 (F pool )

[0081] And use the Sigmoid activation function to transform the feature map F attn Generate attention weights W attn :

[0082] W attn =σ(F attn )

[0083] It is understandable that the attention weight W attn Indicates the importance of each feature point, which is used to adjust each position of the local feature map, helping the model focus on the most informative area.

[0084] Then, in order to enable the model to more prominently represent important local features and improve the feature expression ability, the attention weight W is first attn Upsample to the initial local feature map F local The same scale, and then the upsampled attention weights are combined with F local Multiply elements by element to adaptively assign different weights to features at different positions:

[0085]

[0086] in, represents element-wise multiplication, and Up(·) represents an upsampling operation.

[0087] Finally, in order to further extract more compact features and make the features more discriminative, the weighted local features F are subjected to 1×1 convolution. local-weight Perform channel compression to obtain the final local feature map F local-final :

[0088] F local-final =Conv 1×1 (F local-weight )

[0089] In some possible embodiments, step S1 may further include:

[0090] S111, performing a pooling operation on the input image based on a global feature aggregation module to obtain a global context feature;

[0091] S112, performing a convolution operation on the global context feature to obtain a compact global feature map;

[0092] S113, using a preset ReLU activation function to perform feature enhancement on the compact global feature map, and sequentially performing convolution operations and upsampling operations on the result after feature enhancement to obtain a global feature map corresponding to the input image.

[0093] It should be noted that in the dual-path attention aggregation module, the global feature aggregation module is used to extract the global context information of the input image, ensuring that the model can integrate the global information of the entire input image, thereby having stronger global perception capabilities when understanding complex scenes.

[0094] Please continue to see Figure 3 After the input image enters the global feature aggregation module of each convolutional layer, first, the input image F input Perform pooling operation to obtain the global context feature F global :

[0095] F global =P(F input )

[0096] Here, P(·) represents the pooling operation.

[0097] Then, in order to retain key global information and eliminate redundant features, the pooled feature map is channel compressed through 1×1 convolution to generate a more compact global feature map (compact global feature map):

[0098] F global-compact =Conv 1×1 (F global )

[0099] Then, the ReLU activation function is used to enhance the nonlinear expression ability of global features:

[0100] F global-relu =ReLU(F global-compact )

[0101] Finally, the features are convolved using a 1×1 convolution and restored to the same spatial dimension as the input image by upsampling, ultimately forming a global feature map F corresponding to the input image. global-final :

[0102] F global-final =Up(Conv 1×1 (F global-compact ))

[0103] In some possible embodiments, step S1 may further include:

[0104] S121, perform element-by-element multiplication and fusion of local features and global features to obtain a corresponding preliminary fusion feature map;

[0105] S122, performing a sum operation on the input image and the preliminary fused feature map to obtain a fused feature map.

[0106] It should be noted that after extracting local features and global features from the input image, these features need to be fused. Specifically, in order to enhance the overall perception ability of the model, the local feature F local-final and the global feature F global-final Multiply element by element to fuse local details with global context information to obtain the fused feature map F fused (Preliminary fusion feature map):

[0107]

[0108] in, Represents element-wise multiplication.

[0109] At the same time, in order to make up for the lack of single-layer feature expression ability, the model can use low-level detail information and high-level semantic information at the same time to improve the overall perception ability of the model. Under the condition of maintaining information consistency, the low-level local detail information and high-level semantic information are fused through the addition operation to enhance the model's balanced perception of details and semantics, and the enhanced fusion feature map F is obtained. enhanced :

[0110] F enhanced =F input ⊕F fused

[0111] Here, ⊕ represents element-by-element addition.

[0112] S2. Based on a preset state-space feature enhancement model, each fused feature map is enhanced to obtain a corresponding feature enhancement map.

[0113] It should be noted that the embodiment of the present application proposes a state-space feature enhancement model based on the Mamba mechanism to strengthen the long-distance dependencies of input features, so as to make up for the shortcomings of convolutional neural networks in capturing long-distance contextual information, while improving the computational efficiency and scalability of the model.

[0114] Among them, the first feature enhancement module is mainly used in the encoding and decoding stages. Compared with the encoding stage, the main difference in the decoding stage is that the convolution operation in the encoding stage is replaced by a deconvolution operation, thereby ensuring that the model can retain both long dependencies and spatial details when reconstructing high-resolution feature maps.

[0115] In some possible embodiments, step S2 may include:

[0116] S201, in the remaining convolution layers except the starting layer of the CNN decoder, respectively use the first feature enhancement module to perform the first batch normalization and ReLU activation function operation, the convolution operation, the second batch normalization and ReLU activation function operation and the sequence conversion operation on the corresponding fused feature map in sequence to obtain the first sequence feature;

[0117] S202, sequentially performing a linear regression operation, a convolution operation, a SiLU activation function operation, and a state space operation on the first sequence features to obtain first sequence enhanced features;

[0118] S203, performing a linear regression operation and a SiLU activation function operation on the first sequence features in sequence to obtain a second sequence of enhanced features;

[0119] S204, performing element-by-element multiplication on the first sequence enhanced features and the second sequence enhanced features, and performing linear regression operation and sequence inverse transformation on the multiplication results in sequence to obtain a feature enhancement map corresponding to each convolutional layer.

[0120] It should be noted that if Figure 2 As shown, a first feature enhancement module is deployed in each of the remaining convolutional layers except the starting layer of the CNN decoder, wherein each first feature enhancement module receives the output of the dual-path attention aggregation module from the same convolutional layer as its own input.

[0121] like Figure 4 As shown, the workflow of the first feature enhancement module is as follows:

[0122] First, by transforming the input feature map (the enhanced fusion feature map F output from step 1) enhanced ) performs convolution operation, and after batch normalization and activation function (ReLU), the feature representation ability is enhanced to obtain the feature map F e :

[0123] Then the two-dimensional feature map is rearranged into a sequence form to obtain the first sequence feature F1.

[0124] F1=RS(F e )

[0125] RS(·) represents the Reshape operation, which here means converting a two-dimensional array into a one-dimensional array.

[0126] Then, through the self-attention mechanism, a global dependency relationship can be established between different spatial positions to enhance the global dependency of features, and finally the feature enhancement graph F corresponding to the current convolutional layer is obtained. strato_mamba :

[0127]

[0128] Among them, RS(·) represents the Reshape operation, which here means converting a one-dimensional array into a two-dimensional array; L(·) represents the Linear operation, SSM represents the state space model (state space operation), and SiLU(·) represents the activation function operation; represents the first sequence enhancement feature, Represents the second sequence enhancement feature.

[0129] In some possible embodiments, step S2 may include:

[0130] S211, at the starting layer of the CNN decoder, using the second feature enhancement module to sequentially perform a first batch normalization and ReLU activation function operation, a first convolution operation, a second batch normalization and ReLU activation function operation, a second convolution operation, a tanh activation function operation and a sequence conversion operation on the input features to obtain a second sequence feature;

[0131] S212, sequentially performing a linear regression operation, a convolution operation, a SiLU activation function operation, and a state space operation on the second sequence features to obtain a third sequence enhanced feature;

[0132] S213, sequentially performing a linear regression operation and a SiLU activation function operation on the second sequence features to obtain a fourth sequence of enhanced features;

[0133] S214, performing element-by-element multiplication on the third sequence enhanced features and the fourth sequence enhanced features, and performing linear regression operation and sequence inverse transformation on the multiplication results in sequence to obtain a feature enhancement map corresponding to the starting layer.

[0134] It should be noted that the second feature enhancement module is mainly used to strengthen the feature fusion after upsampling to ensure that long-term dependencies and spatial details are retained in the decoding stage. The global perception of the network in the decoding stage is further enhanced by inputting the Mamba model through serialization operations, which can not only better reconstruct high-resolution features but also maintain long-term dependencies.

[0135] like Figure 2 As shown, a second feature enhancement module is deployed at the starting layer of the CNN decoder, wherein the second feature enhancement module receives the output of the dual-path attention aggregation module from the same convolutional layer as its own input.

[0136] Different from the first feature enhancement module, the second feature enhancement module first obtains the feature map F through continuous convolution, batch normalization and ReLU operations on the input feature map d ,like Figure 5As shown. Therefore, it can not only ensure that the features maintain a relatively stable distribution and enhance the nonlinear modeling ability of the network, but also effectively alleviate the gradient problem of the model and ensure the stability and convergence of the network under the deep structure. Among them, the tanh activation function is used to bound the output of the network, and the bounded two-dimensional feature map F d Rearrange it into sequence form and get the second sequence feature:

[0137] F2=RS(tanh(F d ))

[0138] Among them, RS(·) represents the Reshape operation and tanh(·) represents the activation function.

[0139] Then, similarly to the first feature enhancement module, after Figure 5 After the relevant workflow in , we finally get the feature enhancement graph F corresponding to the output of the convolutional layer. apex_mamba :

[0140]

[0141] Among them, RS(·) represents the Reshape operation, which here means converting a one-dimensional array into a two-dimensional array; L(·) represents the Linear operation, SSM() represents the state space model (state space operation), and SiLU(·) represents the activation function operation; represents the third sequence enhancement feature, Indicates the fourth sequence enhancement feature.

[0142] S3. Use the memory module to store the feature enhancement map output by the CNN encoder, and match the memory features most relevant to the current original image based on the stored features, so that the CNN decoder reconstructs the reconstructed feature map corresponding to the current original image based on the memory features.

[0143] It should be noted that the memory module acts as a bridge between the encoder and the decoder, and realizes effective management and optimization of information through the storage, updating and transmission of features.

[0144] like Figure 2 As shown, the memory module receives the deep features M from the encoder e (The feature enhancement map output by the last convolutional layer of the encoder) ensures that key information is not lost during the layer-by-layer transmission and is stored as the memory item M m .

[0145] Secondly, the memory module can extract important memory items M m , when the model is forward propagating, the most relevant memory feature M is selected by matching s, thereby improving the robustness and generalization ability of the model.

[0146] M s =Memory(M e ,M m )

[0147] Among them, in the process of matching and selecting the most relevant memory features, the memory module is based on the query features (the deep features of the current original image) and the memory items (the stored M m ) to calculate the similarity and select the most relevant memory feature M based on the similarity matching s In the memory module, the similarity calculation formula between the query feature and the key in the memory is:

[0148]

[0149] Among them, Q is the query feature, K is the memory key value, and d key Is the dimension of the key value.

[0150] In addition, the memory module can be dynamically adjusted and updated according to the current input data. This dynamic adjustment mechanism enhances the model's ability to cope with changes and ensures that the decoder can generate high-quality output through effective feature transmission.

[0151] The updated feature representation of the memory module is:

[0152] F updated =Memory(F′ input ,keys)

[0153] Among them, F′ input is the feature of the current input, and keys is the key value in the memory module.

[0154] S4. Compare the difference between the reconstructed feature map and the current original image based on a preset difference threshold to perform anomaly detection on the current original image.

[0155] It can be understood that in the early stage of model training, by continuously inputting normal video sample images, the model will gradually store the memory features of normal images. At the same time, the difference between the reconstructed feature map and the input original feature map is calculated, and these differences are continuously monitored and iterated. Then, based on these differences, a relatively stable difference threshold between the normal reconstructed image and the normal input image can be formed.

[0156] In the actual detection process, for each input original image, a corresponding reconstructed feature map can be reconstructed based on the memory features. The difference between the current original image and the corresponding reconstructed feature map is calculated and compared with the preset difference threshold. If the difference is greater than the difference threshold, the current input is judged to be abnormal. Otherwise, if the difference is not greater than the difference threshold, the current input is judged to be normal.

[0157] Please refer to Figure 6 , Figure 6 FIG. 1 is a block diagram showing the composition of a video anomaly detection device provided by some embodiments of the present application. It should be understood that the video anomaly detection device is similar to the above-mentioned Figure 1 Corresponding to the method embodiment, each step involved in the above method embodiment can be executed. The specific functions of the video anomaly detection device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0158] Figure 6 The video anomaly detection device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the video anomaly detection device, and the video anomaly detection device includes:

[0159] A feature extraction module 610 is used to extract cross-scale local features and global features from the input images of each convolutional layer in the CNN in turn based on the dual-path attention aggregation module, and fuse the local features with the global features to obtain a corresponding fused feature map;

[0160] A feature enhancement module 620 is used to perform feature enhancement on each fused feature map based on a preset state space feature enhancement model to obtain a corresponding feature enhancement map;

[0161] A feature memory module 630 is used to store the feature enhancement map output by the CNN encoder using the memory module, and match the memory feature most relevant to the current original image based on the stored features, so that the CNN decoder reconstructs the reconstructed feature map corresponding to the current original image based on the memory feature;

[0162] The anomaly detection module 640 is used to compare the difference between the reconstructed feature map and the current original image based on a preset difference threshold to perform anomaly detection on the current original image.

[0163] It can be understood that the above-mentioned device item embodiment corresponds to the method item embodiment of the present invention. A video anomaly detection device provided by the embodiment of the present invention can implement the video anomaly detection method provided by any method item embodiment of the present invention.

[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0165] like Figure 7 As shown, some embodiments of the present application provide an electronic device 700, which includes: a memory 710, a processor 720, and a computer program stored in the memory 710 and executable on the processor 720, wherein the processor 720 reads the program from the memory 710 through a bus 730 and executes the program to implement a method of any embodiment of the video anomaly detection method as described above.

[0166] Processor 720 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 720 can be a microprocessor.

[0167] The memory 710 may be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 720 of the disclosed embodiment may be used to execute instructions in the memory 710 to implement the method shown above. The memory 710 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0168] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon. The computer program is executed by a processor to execute the method described in the method embodiment.

[0169] Some embodiments of the present application further provide a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.

[0170] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0173] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0174] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0175] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0176] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A video anomaly detection method, characterized in that: include: Based on the dual-path attention aggregation module, the input images of each convolutional layer in the CNN are sequentially extracted with cross-scale local features and global features, and the local features and the global features are fused to obtain a corresponding fused feature map; Based on the preset state space feature enhancement model, each fusion feature map is enhanced to obtain a corresponding feature enhancement map; Using a memory module to store features of the feature enhancement map output by the CNN encoder, and matching the memory features most relevant to the current original image based on the stored features, so that the CNN decoder reconstructs a reconstructed feature map corresponding to the current original image based on the memory features; The difference between the reconstructed feature map and the current original image is compared based on a preset difference threshold to perform abnormality detection on the current original image.

2. The video anomaly detection method according to claim 1, characterized in that: The dual-path attention aggregation module sequentially extracts cross-scale local features and global features from the input images of each convolutional layer in the CNN, including: Based on the local feature aggregation module, a multi-scale convolution operation is performed on the input image, and the results of the multi-scale convolution operation are spliced ​​to obtain an initial local feature map; Performing pooling operations and convolution operations on the local feature map in sequence, and generating attention weights based on a preset Sigmoid activation function; The attention weight is upsampled and then element-wise multiplied with the local feature map, and a convolution operation is performed on the multiplication result to obtain a local feature map corresponding to the input image.

3. The video anomaly detection method according to claim 1, characterized in that: The dual-path attention aggregation module sequentially extracts cross-scale local features and global features from the input images of each convolutional layer in the CNN, including: Based on the global feature aggregation module, the input image is pooled to obtain the global context feature; Performing a convolution operation on the global context feature to obtain a compact global feature map; The compact global feature map is feature enhanced by using a preset ReLU activation function, and a convolution operation and an upsampling operation are sequentially performed on the result after feature enhancement to obtain a global feature map corresponding to the input image.

4. The video anomaly detection method according to claim 1, characterized in that: The fusing the local features and the global features to obtain a corresponding fused feature map includes: Performing element-by-element multiplication and fusion on the local features and the global features to obtain a corresponding preliminary fusion feature map; An addition operation is performed on the input image and the preliminary fusion feature map to obtain the fusion feature map.

5. The video anomaly detection method according to claim 1, characterized in that: The feature enhancement of each fused feature map based on the preset state space feature enhancement model to obtain a corresponding feature enhancement map includes: In the remaining convolutional layers except the starting layer of the CNN decoder, the first feature enhancement module is used to perform the first batch normalization and ReLU activation function operation, the convolution operation, the second batch normalization and ReLU activation function operation and the sequence conversion operation on the corresponding fused feature map in sequence to obtain the first sequence feature; Performing a linear regression operation, a convolution operation, a SiLU activation function operation, and a state space operation on the first sequence features in sequence to obtain a first sequence enhanced feature; Performing a linear regression operation and a SiLU activation function operation on the first sequence features in sequence to obtain a second sequence enhanced feature; The first sequence enhanced features and the second sequence enhanced features are multiplied element by element, and a linear regression operation and a sequence inverse transformation are performed on the multiplication results in sequence to obtain a feature enhancement map corresponding to each convolutional layer.

6. The video anomaly detection method according to claim 5, characterized in that: The feature enhancement of each fused feature map based on the preset state space feature enhancement model to obtain a corresponding feature enhancement map includes: At the starting layer of the CNN decoder, the second feature enhancement module is used to perform the first batch normalization and ReLU activation function operation, the first convolution operation, the second batch normalization and ReLU activation function operation, the second convolution operation, the tanh activation function operation and the sequence conversion operation on the input features in sequence to obtain the second sequence features; Performing a linear regression operation, a convolution operation, a SiLU activation function operation, and a state space operation on the second sequence features in sequence to obtain a third sequence enhanced feature; Performing a linear regression operation and a SiLU activation function operation on the second sequence features in sequence to obtain a fourth sequence enhanced feature; The third sequence enhanced features and the fourth sequence enhanced features are element-by-element multiplied, and a linear regression operation and a sequence inverse transformation are sequentially performed on the multiplication results to obtain a feature enhancement map corresponding to the starting layer.

7. A video anomaly detection device, characterized in that: include: A feature extraction module is used to extract cross-scale local features and global features from the input images of each convolutional layer in the CNN in turn based on the dual-path attention aggregation module, and fuse the local features with the global features to obtain a corresponding fused feature map; A feature enhancement module is used to enhance the features of each fused feature map based on a preset state space feature enhancement model to obtain a corresponding feature enhancement map; A feature memory module is used to store the feature enhancement map output by the CNN encoder using the memory module, and match the memory feature most relevant to the current original image based on the stored features, so that the CNN decoder reconstructs the reconstructed feature map corresponding to the current original image based on the memory feature; The anomaly detection module is used to compare the difference between the reconstructed feature map and the current original image based on a preset difference threshold to perform anomaly detection on the current original image.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the video anomaly detection method according to any one of claims 1 to 6 can be implemented when the processor executes the program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video anomaly detection method according to any one of claims 1 to 6 is executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the video anomaly detection method according to any one of claims 1 to 6 is implemented.

Images