Video anomaly detection method and system in combination with cross attention and memory optimization

Abstract

The invention provides a video anomaly detection method and system combining cross attention and memory optimization, and the method comprises the steps: inputting RGB data and optical flow data into an appearance encoder branch and a motion encoder branch respectively; gradually extracting appearance features and motion features of different scales by using a convolution module and a maximum pooling layer; then, taking branch output features of the appearance encoder and the motion encoder as input data of a cross attention mechanism, enriching representation of the appearance features by using the motion features, and realizing fusion of the appearance and the motion features; thirdly, inputting the fused features of the appearance and the movement into a memory module to further optimize the features, and improving the extraction and expression ability of normal features; finally, a future frame decoder is utilized, appearance features of different scales are fully utilized, decoding is carried out step by step, generated future frames are obtained, and the video anomaly detection precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to video anomaly detection, and in particular to a video anomaly detection method and system combining cross-attention and memory optimization. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The development of urban security and digital society has led to the explosive growth of the number of surveillance cameras. Manual video anomaly detection is time-consuming and labor-intensive, and it is difficult to cope with the huge amount of video data. Therefore, it is urgent to conduct research on video anomaly detection. General video anomaly detection algorithms are mainly implemented through statistics, classification, and sparse coding. However, the current general video anomaly detection methods are limited in detection accuracy when faced with massive video data with random distribution and complex types, and the model performance needs to be further improved.

[0004] Compared with general video anomaly detection methods, deep learning video anomaly detection methods can extract video features more accurately, greatly improving the detection accuracy of the model, and have been more widely used in the field of video anomaly detection. At the same time, the attention mechanism can further optimize the feature representation ability of the model and has been widely used in the field of video anomaly detection, effectively improving the accuracy of video anomaly detection.

[0005] Existing literature 1 (Liu W, Luo W, Lian D, et al. Future frame prediction for anomaly detection - A new baseline[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018: 6536 - 6545.), literature 2 (Gong D, Liu L, Le V, et al. Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection[C]. Proceedings of the IEEE International Conference on Computer Vision, Seoul, South Korea, 2019: 1705 - 1714.), literature 3 (Cai R, Zhang H, Liu W, et al. Appearance-motion memory consistency network for video anomaly detection[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(2): 938 - 946.), literature 4 (Zhang X, Fang J, Yang B, et al. Hybrid attention and motion constraint for anomaly detection in crowded scenes[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(5): 2259 - 2274.) and literature 5 (Sun Z, Wang P, Zheng W, et al. Dual GroupGAN: An unsupervised four-competitor (2V2) approach for video anomaly detection[J]. Pattern Recognition, 2024, 153: 1 - 14. article no.110500.) have proposed video anomaly detection algorithms based on deep learning. Among them, document 1 designs a video anomaly detection model based on prediction method, document 2 optimizes the quality of features extracted by neural network through memory module, document 3 designs a neural network architecture with two branches of appearance and motion, and adds motion features to enrich the overall features of the model, and documents 4 and 5 introduce attention mechanism to further integrate the features extracted by deep neural network. Although these five algorithms have improved the detection accuracy of video anomaly detection to a certain extent, the extraction and fusion of appearance and motion features are still not deep enough, which limits the further improvement of model accuracy. .

[0006] Therefore, how to effectively extract representative features and deeply integrate appearance and motion features so that the model can accurately distinguish normal and abnormal features and improve the accuracy of video anomaly detection algorithms is a problem that needs to be solved in this field. Summary of the invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a video anomaly detection method and system that combines cross-attention and memory optimization, uses a cross-attention module to achieve a deep fusion of appearance features and motion features, and adopts a memory module to optimize the feature extraction and expression capabilities, so that the model can more accurately distinguish between normal and abnormal features, thereby improving the accuracy of video anomaly detection.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a video anomaly detection method combining cross-attention and memory optimization, comprising:

[0010] Get continuous RGB data and corresponding optical flow data;

[0011] The acquired continuous RGB data and the corresponding optical flow data are input into a trained video detection model to obtain a predicted future frame, and the predicted future frame is used to perform video anomaly detection to obtain a detection result of the video to be detected;

[0012] The processing of the RGB data and the corresponding optical flow data of the video to be detected by the trained video detection model is specifically as follows:

[0013] Extracting appearance features of the RGB data at different scales, and extracting motion features of the optical flow data;

[0014] The motion feature and the appearance feature are fused using a cross attention mechanism to obtain an appearance-motion fusion feature;

[0015] Optimizing the appearance-motion fusion feature by using a memory module updated by a memory matrix to obtain an optimized appearance-motion fusion feature;

[0016] The appearance-motion fusion feature and the optimized appearance-motion fusion feature are decoded step by step using a future frame decoder branch. During the step-by-step decoding process, appearance features of different scales are introduced for fusion decoding to obtain a predicted future frame.

[0017] In a second aspect, the present invention provides a video anomaly detection system combining cross-attention and memory optimization, comprising:

[0018] An acquisition module, which is configured to: acquire continuous RGB data and corresponding optical flow data;

[0019] A detection module is configured to: input the acquired continuous RGB data and corresponding optical flow data into a trained video detection model to obtain a predicted future frame, perform video anomaly detection using the predicted future frame, and obtain a detection result of the video to be detected;

[0020] The processing of the RGB data and the corresponding optical flow data of the video to be detected by the trained video detection model is specifically as follows:

[0021] Extracting appearance features of the RGB data at different scales, and extracting motion features of the optical flow data;

[0022] The motion feature and the appearance feature are fused using a cross attention mechanism to obtain an appearance-motion fusion feature;

[0023] Optimizing the appearance-motion fusion feature by using a memory module updated by a memory matrix to obtain an optimized appearance-motion fusion feature;

[0024] The appearance-motion fusion feature and the optimized appearance-motion fusion feature are decoded step by step using a future frame decoder branch. During the step-by-step decoding process, appearance features of different scales are introduced for fusion decoding to obtain a predicted future frame.

[0025] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0027] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0028] One or more of the above technical solutions have the following beneficial effects:

[0029] In the present invention, RGB data and optical flow data are input into the appearance encoder and motion encoder branches respectively. In the appearance encoder and motion encoder branches, convolution modules and maximum pooling layers are used to gradually extract appearance features and motion features of different scales; then, the output features of the appearance encoder and motion encoder branches are used as input data of the cross-attention mechanism, and the representation of the appearance features is enriched by the motion features to achieve the fusion of the appearance and motion features; then, the fusion features of the appearance and motion are input into the memory module to further optimize the features, and the extraction and expression capabilities of normal features are improved; finally, the future frame decoder is used to make full use of the appearance features of different scales, and decode step by step to obtain the generated future frames. The present invention uses the cross-attention module to achieve the deep fusion of appearance features and motion features, and uses the memory module to optimize the extraction and expression capabilities of features, so that the model can more accurately distinguish normal and abnormal features, and improve the accuracy of video anomaly detection.

[0030] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 Schematic diagram of the structure of a video anomaly detection model in Embodiment 1 of the present invention;

[0033] Figure 2 Schematic diagram of the structure of the convolution module in the first embodiment of the present invention;

[0034] Figure 3 Schematic diagram of the structure of the cross attention module in the first embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of a memory module reading process in Embodiment 1 of the present invention;

[0036] Figure 5 This is a schematic diagram of a memory module update process in Embodiment 1 of the present invention;

[0037] Figure 6 Schematic diagram of an attention fusion module in Embodiment 1 of the present invention;

[0038] Figure 7 Schematic diagram of a deconvolution module in Embodiment 1 of the present invention;

[0039] FIG8( a ) is an anomaly score visualization 1 (aAUC: 99.9%) in Example 1 of the present invention;

[0040] FIG8( b ) is an anomaly score visualization 2 (aAUC: 100%) in Example 1 of the present invention. DETAILED DESCRIPTION

[0041] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0042] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0043] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0044] Embodiment 1

[0045] This embodiment discloses a video anomaly detection method combining cross-attention and memory optimization, including:

[0046] Step 1: Obtain continuous RGB data and corresponding optical flow data;

[0047] Step 2: Input the acquired continuous RGB data and the corresponding optical flow data into the trained video detection model to obtain the predicted future frame, and use the predicted future frame to perform video anomaly detection to obtain the detection result of the video to be detected;

[0048] Among them, the trained video detection model processes the RGB data and the corresponding optical flow data of the video to be detected, specifically as follows: extracting appearance features of different scales of the RGB data, and extracting motion features of the optical flow data; fusing the motion features and the appearance features using a cross-attention mechanism to obtain appearance-motion fusion features; optimizing the appearance-motion fusion features using a memory module updated by a memory matrix to obtain optimized appearance-motion fusion features; and decoding the appearance-motion fusion features and the optimized appearance-motion fusion features step by step using a future frame decoder branch. In the step-by-step decoding process, appearance features of different scales are introduced for fusion decoding to obtain predicted future frames.

[0049] In step 1 of this embodiment, it also includes: preprocessing the acquired continuous RGB data to generate a normalized RGB data tensor; preprocessing the acquired optical flow data to generate a normalized optical flow data tensor.

[0050] In step 2 of this embodiment, the RGB data tensor and the optical flow data tensor generated in step 1 are input into the video anomaly detection model to generate future frames.

[0051] like Figure 1 As shown, the video anomaly detection model provided by this embodiment includes five parts: an appearance encoder branch, a motion encoder branch, a cross attention module, a memory module and a future frame decoder.

[0052] The processing procedures of the appearance encoder branch, motion encoder branch, cross-attention module, memory module and future frame decoder are described in detail below.

[0053] Step 21: Use the appearance encoder branch to process the RGB data tensor.

[0054] Build an appearance encoder branch to extract appearance features of different scales of the image. The feature extraction process of the appearance encoder branch is implemented by the convolution module, and the feature scale transformation is implemented by the maximum pooling layer. The basic structure of the convolution module is as follows: Figure 2 As shown, it specifically includes Conv3×3, BatchNom, LeakyReLU, Conv3×3, BatchNom and LeakyReLU connected in sequence.

[0055] The appearance encoder branch inputs a set of RGB video frames I∈{I1,I2,…,I t}, in this example, the input size is (4×3)×H×W, and the output is appearance features of 4 different scales: and Wherein, H is the height of the input RGB video frame image, and W is the width of the input RGB video frame image.

[0056] Specifically, the input RGB video frame group is passed through the convolution module, so that the input RGB video frame group (4×3)×H×W) is mapped to the appearance feature f1 of 64×H×W a ;

[0057] The appearance feature f1 a Through three groups of maximum pooling layers and convolution modules in sequence, they are mapped into appearance features f2 of 128×(H / 2)×(W / 2) a , 256×(H / 4)×(W / 4) appearance feature f3 a and the appearance feature f4 of 512×(H / 8)×(W / 8)a .

[0058] Step 22: Process the optical flow data tensor using the motion encoder branch.

[0059] Build a motion encoder branch to extract motion features of different scales of the image. The motion encoder branch inputs the optical flow video frame group O∈{O1,O2,…,O t-1}, in this example, the input size is (3×3)×H×W, and the output is motion features of 4 different scales: and

[0060] Specifically, the optical flow video frame group corresponding to the RGB video frame group is passed through the convolution module, so that the input optical flow video frame group (3×3)×H×W) is mapped to the motion feature f1 of 64×H×W. m ;

[0061] The motion feature f1 m Through three groups of maximum pooling layers and convolution modules, they are mapped into motion features f2 of 128×(H / 2)×(W / 2) respectively. m , 256×(H / 4)×(W / 4) motion features f3 m and the motion feature f4 of 512×(H / 8)×(W / 8) m .

[0062] Step 3: Use the criss-cross attention module to fuse the output of the motion encoder branch and the output of the appearance encoder branch.

[0063] Build a cross-attention module that integrates appearance and motion features, such as Figure 3 As shown. The appearance feature f4 of the appearance encoder branch in step 1 is a The motion feature f4 of the motion encoder branch in step 2 m As the input of the cross attention module, the output is the appearance-motion fusion feature

[0064] Specifically, the appearance feature f4 a Input the query branch of the cross attention module and convert the appearance feature f4 a Reshape with the transposed weight matrix Multiply and reshape to get the features

[0065] Set the motion feature f4 t Input the key branch and value branch of the cross attention module and convert the motion feature f4 m Reshape Respectively and the transposed weight matrix Multiply and reshape to get the features

[0066] The obtained feature f Q and the feature f obtained in step 32 K 、f V Formula (1) is used to calculate the appearance-motion fusion feature f4 am :

[0067]

[0068] Will Reshape with the transposed deformation matrix After multiplication, reshape to obtain the appearance-motion fusion feature f4 am as output.

[0069] Step 4: Utilize the memory module to optimize the appearance-motion fusion features.

[0070] The appearance-motion fusion feature f4 obtained in step 3 am As the input of the memory module, the optimized appearance-motion fusion feature As output, the memory module reads the process as Figure 4 As shown, the appearance-motion fusion feature f4 is used at the same time am Update the memory matrix inside the memory module The memory module update process is as follows: Figure 5 shown.

[0071] Specifically, the appearance-motion fusion feature f4 is obtained am Reshape into query matrix in Each original query item is represented as Each original query item The read weight vector w is obtained by using the memory matrix M through formula (2):

[0072]

[0073] Multiply the read weight vector w by the memory matrix M to get the optimized query item The N optimized query items are concatenated along the first dimension and reshaped to obtain the optimized fusion feature f4 M .

[0074] Each memory unit in the memory matrix M The updated weight vector w' is obtained by using the query matrix F through formula (3):

[0075]

[0076] Using memory unit m j , update the weight vector w' and query matrix F to obtain updated memory units Update memory unit The calculation process is:

[0077]

[0078] Among them, L2 is a regularization operation to ensure the update of memory units Without falling into overfitting, each memory unit is spliced ​​along the first dimension to obtain an updated memory matrix The updated memory matrix will optimize the query terms in the next round of training.

[0079] Step 5: Predict future frames using the future frame decoder.

[0080] The appearance-motion fusion feature f4 obtained in step 3 am And the optimized fusion feature f4 obtained in step 4 M Concatenate along the channel dimension and use it as the input fusion feature of the future frame decoder branch In the process of gradual decoding, appearance features of different scales are added, and the attention fusion mechanism is used to enrich the decoding information and output the future frame

[0081] Specifically, the fusion feature f4 is input into the future frame decoder, and after being processed by the convolution module and the deconvolution module, it is combined with the appearance feature f3 a Concatenate along the channel dimension to get the decoding features

[0082] The decoded feature f3 is input into the attention fusion module to further optimize the decoded feature. The output result of f3 processed by the attention fusion module is processed by the convolution module and the deconvolution module in turn, and then combined with the appearance feature f2. a Concatenate along the channel dimension to get the decoded features Similarly, the decoded feature f2 is input into the attention fusion module to further optimize the decoded feature. The output result of f2 processed by the attention fusion module is processed by the convolution module and the deconvolution module in turn, and then combined with the appearance feature f1. a The decoded feature f1 is obtained by splicing along the channel dimension, and the decoded feature f1 is passed through the attention fusion module and the convolution module to obtain the future frame

[0083] The attention fusion module proposed in this embodiment is as follows: Figure 6As shown, specifically, the input features of the attention fusion module are processed by the adaptive average pooling layer and the adaptive maximum pooling layer respectively, and reshaped and ReLu function activation operations are performed, and the two reshaped feature matrices are added to obtain the channel attention weight; then, the channel attention weight is multiplied by the input features of the attention fusion module to obtain the channel attention optimized features; the channel attention optimized features are processed by the channel average pooling layer and the channel maximum pooling layer respectively, and then spliced ​​along the channel dimension, and the spliced ​​features are passed through the convolution layer and normalized to obtain the spatial attention weight; then, the spatial attention weight is multiplied by the channel attention optimized features to obtain the fused features output by the attention fusion module.

[0084] The deconvolution module is as follows Figure 7 As shown, it specifically includes TConv3×3, BatchNom and LeakyReLU connected in sequence.

[0085] This embodiment performs end-to-end training and testing on the video anomaly detection model, and uses three standard data sets, UCSD Ped2, CUHKAvenue, and ShanghaiTech, for training. The data set settings are shown in Table 1. The network uses an unsupervised method during the training process, and uses the representation of normal data to obtain future frames. During the network testing process, when the input video frame contains anomalies, the quality of the future frame predicted by the network will decrease, and the peak signal-to-noise ratio of the predicted future frame and the real future frame will be significantly different from that of the normal situation, thereby realizing video anomaly detection.

[0086] Table 1: Division of training and test sets for three standard datasets

[0087]

[0088] In order to constrain the network's ability to represent future frames, the appearance loss L is designed a , motion loss L m , constrains the output of the model on appearance information and motion information, and designs the discrete loss L of the memory module s , reducing the redundancy of features within the memory matrix M.

[0089] The appearance loss is directly applied to the future frames output by the future frame decoder Constrain the appearance and set the future frame The corresponding real frame I t+1 The appearance loss L is calculated using formula (5) a :

[0090]

[0091] However, only the appearance loss L is used aConstraining future frames will result in poor temporal continuity between future frames and RGB input frames. Therefore, the motion loss L is introduced. m Frame for the future Constrained on time continuity, motion loss L m See formula (6):

[0092]

[0093] Here, f(·,·) represents the optical flow operation, and the output is the optical flow data corresponding to the input image. The difference between the predicted result and the real result in optical flow is used to constrain the temporal continuity of future frames.

[0094] In addition to the loss constraints on the model output, the model uses query items in the memory module to further constrain the ability to understand normal features in the overall encoding and decoding process. The memory cell with the highest weight The memory cell with the second highest weight Designed a memory module discrete loss L s Constraint, discrete loss L s As shown in formula (7):

[0095]

[0096] The total loss function L is shown in formula (8):

[0097] L=λ a L a +λ m L m +λ s L s (8)

[0098] Among them, λ a , m and λ s They are the appearance loss L a , motion loss L m and discreteness loss L s The weight factors are set to 1, 2 and 0.2 respectively in this embodiment.

[0099] Other training parameter settings are shown in Table 2. Save the trained network parameters.

[0100] Table 2 Overall training parameter settings of the model

[0101] Network parameters set up Optimizer Adam <![CDATA[Learning rate R L > 1e-5 <![CDATA[Weight decay W D > CosineAnnealing <![CDATA[Batch size N B > 4 <![CDATA[Training round N E > 60 <![CDATA[Image size S I > 128×128

[0102] During the test, this embodiment uses the peak signal-to-noise ratio as the future frame prediction error e t Measure the model output for future frames Corresponding to the real frame I t+1 The difference between the future frame prediction error e t See formula (9):

[0103]

[0104] In order to visually compare the abnormality of the frames, the prediction error e is normalized using the maximum and minimum normalization method. t Linearly map to the range [0,1] and calculate the anomaly score s t , anomaly score s t See formula (10):

[0105]

[0106] Anomaly scores t The larger the value, the greater the possibility that the video frame contains anomalies.

[0107] Using the anomaly score s of each frame t , get the area under the ROC curve of the data set a AUC As a model performance evaluation indicator, the area under the ROC curve is AUC See formula (11):

[0108]

[0109] Among them, r FP is the false positive rate, r TP is the true positive rate, a AUC The closer it is to 100%, the higher the accuracy of the video anomaly detection model.

[0110] When performing video anomaly detection, the trained video anomaly detection model is used to output future frames, and the trained weight parameters and the area under the ROC curve a are used to calculate the AUC Evaluate the performance of the video anomaly detection model using the anomaly score s t Visualize the anomalies of the video anomaly detection model on each frame.

[0111] The method of this embodiment has high video anomaly detection accuracy, effectively extracts representative features, and deeply integrates appearance and motion features. It is suitable for video anomaly detection tasks and can accurately distinguish normal and abnormal video frames.

[0112] To prove the effectiveness of the solution of this embodiment, three data sets, UCSD Ped2, CUHKAvenue and ShanghaiTech, are used for testing. The test data set settings are shown in Table 1 to verify the video anomaly detection performance of this embodiment.

[0113] This embodiment is compared with other deep learning-based video anomaly detection algorithms (FFP proposed by Liu et al., MemAE proposed by Gong et al., AMMC proposed by Cai et al., HAAE proposed by Zhang et al., and DualGroupGAN proposed by Sun et al.). To ensure the effectiveness of the comparison of experimental results, the experimental data of the three algorithms are divided as shown in Table 1, and are trained and tested on different data sets. Table 3 shows the a comparison of the six video anomaly detection methods on the three data sets. AUC As can be seen from Table 3, this embodiment has a higher detection accuracy, a AUC The best results were achieved on the UCSD Ped2 dataset, and the second best results were achieved on the CUHKAvenue and ShanghaiTech datasets.

[0114] Table 3 Results of 6 video anomaly detection methods on three datasets AUC Value comparison

[0115]

[0116] In order to intuitively demonstrate the video anomaly detection effect of the present invention, Figure 8(a)-Figure 8(b) The visualization results of some anomaly scores of this embodiment on the UCSD Ped2 test set are shown, where the anomaly example included in FIG8(a) is a vehicle traveling on a pedestrian lane, and the anomaly example included in FIG8(b) is a bicycle traveling on a pedestrian lane. The ordinate of the curve reflects the anomaly score of the frame corresponding to the current abscissa. The closer the anomaly score is to 1, the higher the probability that the frame contains an anomaly. The marked area is the part detected to contain an anomaly.

[0117] From a AUC It can be seen from the visualization curves of numerical values ​​and anomaly scores that the video anomaly detection method and system optimized by the cross-attention and memory modules proposed in this embodiment improve the model's ability to extract appearance and motion features, and realize accurate video anomaly detection function.

[0118] Embodiment 2

[0119] The purpose of this embodiment is to provide a video anomaly detection system combining cross-attention and memory optimization, comprising:

[0120] An acquisition module, which is configured to: acquire continuous RGB data and corresponding optical flow data;

[0121] A detection module is configured to: input the acquired continuous RGB data and corresponding optical flow data into a trained video detection model to obtain a predicted future frame, perform video anomaly detection using the predicted future frame, and obtain a detection result of the video to be detected;

[0122] The processing of the RGB data and the corresponding optical flow data of the video to be detected by the trained video detection model is specifically as follows:

[0123] Extracting appearance features of the RGB data at different scales, and extracting motion features of the optical flow data;

[0124] The motion feature and the appearance feature are fused using a cross attention mechanism to obtain an appearance-motion fusion feature;

[0125] Optimizing the appearance-motion fusion feature by using a memory module updated by a memory matrix to obtain an optimized appearance-motion fusion feature;

[0126] The optimized appearance-motion fusion features are decoded step by step using the future frame decoder branch. During the step-by-step decoding process, appearance features of different scales are introduced for fusion decoding to obtain the predicted future frame.

[0127] In further embodiments, there is also provided:

[0128] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, no further description is given here.

[0129] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0130] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0131] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0132] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0133] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in the first embodiment is implemented.

[0134] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as instructions included in a program module, which are executed in a device on a real or virtual processor of the target to perform the process / method as described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or divided between program modules as needed. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0135] The computer program code for implementing the method of the present invention can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the computer or other programmable data processing device, causes the function / operation specified in the flow chart and / or block diagram to be implemented. The program code can be executed completely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer or completely on a remote computer or server.

[0136] In the context of the present invention, computer program codes or related data may be carried by any appropriate carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, etc. Examples of signals may include electrical, optical, radio, acoustic or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0137] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0138] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A video anomaly detection method combining cross-attention and memory optimization, characterized in that include: Get continuous RGB data and corresponding optical flow data; The acquired continuous RGB data and the corresponding optical flow data are input into a trained video detection model to obtain a predicted future frame, and the predicted future frame is used to perform video anomaly detection to obtain a detection result of the video to be detected; The processing of the RGB data and the corresponding optical flow data of the video to be detected by the trained video detection model is specifically as follows: Extracting appearance features of the RGB data at different scales, and extracting motion features of the optical flow data; The motion feature and the appearance feature are fused using a cross attention mechanism to obtain an appearance-motion fusion feature; The appearance-motion fusion feature is optimized by using a memory module updated by a memory matrix to obtain an optimized appearance-motion fusion feature; The appearance-motion fusion feature and the optimized appearance-motion fusion feature are decoded step by step using a future frame decoder branch. During the step-by-step decoding process, appearance features of different scales are introduced for fusion decoding to obtain a predicted future frame.

2. A video anomaly detection method combining cross-attention and memory optimization as claimed in claim 1, characterized in that: The motion feature and the appearance feature are fused using a cross attention mechanism to obtain an appearance-motion fusion feature, specifically: The appearance feature is multiplied by the query weight matrix and then reshaped to obtain a query tensor containing appearance information; The motion feature is multiplied by the key value weight matrix and then reshaped to obtain a key value tensor containing motion information; The motion feature is multiplied by a numerical weight matrix and then reshaped to obtain a numerical tensor containing motion information; The query tensor is multiplied by the transpose of the key value tensor, and after being processed by an activation function, the resultant is multiplied by the value tensor to obtain an appearance-motion fusion feature.

3. A video anomaly detection method combining cross-attention and memory optimization as claimed in claim 1, characterized in that: During the video detection model training process, the memory matrix of the memory module is updated as follows: reshape the appearance-motion fusion feature into a query matrix; Calculating the cosine similarity between the memory matrix and each original query item in the query matrix, and processing them through an activation function to obtain a read weight vector; Multiply the memory matrix by each read weight vector to obtain the appearance-motion fusion feature after the memory matrix is ​​optimized; Calculate the cosine similarity between each memory unit in the memory matrix and the query matrix, and process them through an activation function to obtain an updated weight vector; The corresponding memory unit is updated according to the memory unit in the memory matrix, the update weight vector and the query matrix to obtain an updated memory matrix.

4. A video anomaly detection method combining cross-attention and memory optimization as claimed in claim 1, characterized in that: The appearance-motion fusion feature and the optimized appearance-motion fusion feature are decoded step by step using the future frame decoder branch. In the step-by-step decoding process, appearance features of different scales are introduced for fusion decoding to obtain a predicted future frame, specifically: Splicing the appearance-motion fusion feature with the appearance-motion fusion feature optimized by the memory matrix in the channel dimension to obtain a first splicing result, and using the first splicing result as an input feature of the future frame decoder branch; Performing convolution and deconvolution operations on the input features of the future frame decoder branch; The result of the deconvolution operation is spliced ​​with the appearance features of the corresponding scale on the channel to obtain a second splicing result. The second splicing result is sequentially subjected to attention fusion, convolution and deconvolution operations, and this step is repeated until the predicted future frame is obtained.

5. A video anomaly detection method combining cross-attention and memory optimization as claimed in claim 4, characterized in that: The result of the deconvolution operation is concatenated with the appearance features of the corresponding scale on the channel to obtain a second concatenated result, and the attention fusion operation is performed on the two concatenated results, specifically: The second concatenation results are concatenated after being processed by global maximum pooling and global average pooling to obtain channel attention weights; Multiplying the channel attention weight by the concatenation result to obtain a channel attention optimization feature; The channel attention optimization features are respectively average pooled and maximum pooled in the channel dimension and then concatenated to obtain a spatial attention weight; Multiply the spatial attention weight by the second concatenation result to obtain a spatial attention optimization feature.

6. A video anomaly detection method combining cross-attention and memory optimization as claimed in claim 1, characterized in that: The loss function of the video detection model consists of an appearance loss function, a motion loss function and a discreteness loss function.

7. A video anomaly detection system combining cross-attention and memory optimization, characterized in that include: An acquisition module is configured to: acquire continuous RGB data and corresponding optical flow data; A detection module is configured to: input the acquired continuous RGB data and corresponding optical flow data into a trained video detection model to obtain a predicted future frame, perform video anomaly detection using the predicted future frame, and obtain a detection result of the video to be detected; The processing of the RGB data and the corresponding optical flow data of the video to be detected by the trained video detection model is specifically as follows: Extracting appearance features of the RGB data at different scales, and extracting motion features of the optical flow data; The motion feature and the appearance feature are fused using a cross attention mechanism to obtain an appearance-motion fusion feature; The appearance-motion fusion feature is optimized by using a memory module updated by a memory matrix to obtain an optimized appearance-motion fusion feature; The appearance-motion fusion feature and the optimized appearance-motion fusion feature are decoded step by step using a future frame decoder branch. During the step-by-step decoding process, appearance features of different scales are introduced for fusion decoding to obtain a predicted future frame.

8. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 6 is completed.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images