Unsupervised scene related video anomaly detection method based on Hash mapping

Abstract

The invention belongs to the field of artificial intelligence and video anomaly detection, and discloses an unsupervised scene related video anomaly detection method based on Hash mapping, and the method comprises the following steps: preprocessing a video; training a scene classifier; foreground RGB features and optical flow features are extracted and fused, and scene classification is carried out; hash mapping is carried out on encoding features of an encoder, and a foreground object Hash value is calculated; calculating a distance between adjacent time step hash values of the same object as a loss training network; constructing a normal mode database; and performing test set anomaly detection. By using the method, the perception capability of the model on detail features such as motion amplitude and direction is effectively improved, the distinguishing capability of the model on normality and abnormity is enhanced, effective binding of a foreground object and a scene where the foreground object is located can be realized, the scene perception capability of the model is improved, and the detection capability of unsupervised video abnormity detection is remarkably improved. The unsupervised scene related video anomaly detection method based on Hash mapping can be widely applied to the field of video anomaly detection.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and video anomaly detection, and in particular relates to an unsupervised scene-related video anomaly detection method based on hash mapping. Background Art

[0002] Video anomaly detection is a technology that conducts in-depth analysis of video data to identify unexpected events in the video. This technology plays an important role in many fields such as smart security, smart transportation, and industrial monitoring. By automatically detecting abnormal events such as violent behavior and traffic accidents, it can significantly reduce the workload of manual monitoring and improve monitoring efficiency.

[0003] In recent years, significant progress has been made in the field of video anomaly detection. However, in actual application scenarios, due to the huge amount of monitoring data and the characteristics of abnormal events that are diverse, difficult to predict, and have extremely low frequency, it is very difficult or even impossible to collect sufficiently comprehensive and representative abnormal data, which greatly limits the application of traditional supervised learning methods. In contrast, it is relatively easy to collect abundant normal data. Therefore, unsupervised video anomaly detection methods have received widespread attention from researchers. This type of method only relies on unlabeled normal data for model training, without the support of abnormal data, and is more valuable in actual scenarios.

[0004] At present, the research methods in the field of video anomaly detection can be roughly divided into three categories: (1) Unsupervised video anomaly detection methods, which are trained only on videos containing normal data and identify abnormal events that deviate from normal behavior by learning normal patterns. (2) Weakly supervised video anomaly detection methods, which are trained using video-level labels and learn models by annotating normal and abnormal video clips. (3) Completely unsupervised video anomaly detection methods, which do not rely on any labeled data, but learn based on the characteristics of the data itself.

[0005] Unsupervised video anomaly detection methods only need to be trained on normal data, avoiding the cost of collecting and labeling abnormal data, and showing higher flexibility in scenarios where abnormal events are difficult to foresee. Compared with weakly supervised methods, unsupervised methods do not need to rely on video-level labels, but detect abnormal behaviors through learning of normal patterns, and have stronger generalization capabilities. Compared with completely unsupervised methods, unsupervised methods can focus more on learning the features of normal patterns, thereby more accurately identifying abnormal events that deviate from normal patterns. Therefore, unsupervised video anomaly detection methods have stronger practicality in actual monitoring scenarios and have occupied a dominant position in past research. The present invention is mainly aimed at the field of unsupervised video anomaly detection, and proposes an unsupervised scene-related video anomaly detection method based on hash mapping.

[0006] Unsupervised video anomaly detection methods that have become popular in recent years mainly include methods based on video clip reconstruction and future frame prediction. Such methods usually train autoencoders to achieve reconstruction of the current video clip or prediction of future frames to learn normal patterns. During the test phase, if the video clip deviates from the normal pattern, a large reconstruction or prediction error will occur. Such methods perform well in capturing short-term anomalies, but are usually based on an assumption that abnormal clips are difficult to accurately reconstruct or predict. However, due to the powerful representation ability of deep neural networks, anomalies can sometimes be well reconstructed or predicted. In order to expand the difference between normal and abnormal, it is usually necessary to introduce additional constraints to weaken the generalization ability of the model. For example, Gong et al. proposed a memory-augmented deep autoencoder that stores the representation of normal patterns in memory modules and retrieves them from the memory modules to reconstruct video frames (Gong D, Liu L, Le V, et al. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 1705-1714). In addition, the anomaly of a video frame is usually caused by a foreground object that occupies a smaller part. Some methods have proposed video anomaly detection methods that focus on analyzing the foreground objects of the video, namely object-level methods. This type of method uses a pre-trained object detector to extract the foreground object and uses it as the research object to ignore the influence of background noise. Compared with the frame-level method, it often performs better in complex scenes. In addition, there may be data imbalance in normal data, that is, some normal events occur less frequently, which can easily lead to model misjudgment. Lu et al. proposed a two-stream framework based on context recovery and knowledge retrieval. By using motion information to predict future frames and encoding and storing the normality representation in a hash table, all normal data are fully utilized, which improves the robustness to data imbalance and enhances the model's understanding of normality (Cao C, Lu Y, Zhang Y.Context recovery and knowledge retrieval: A novel two-stream framework for video anomaly detection [J]. IEEE Transactions on Image Processing, 2024). Although the existing methods are effective, they ignore two important factors: (1) the problem of identifying anomalies that are not significantly different from normal events.Some abnormal behaviors are very similar to normal behaviors. For example, the visual features of the same pedestrian when walking and running may be very similar. It is difficult to effectively distinguish the two behaviors only by relying on RGB features, especially when the movement amplitude is small. Existing models perform poorly when dealing with such subtle differences and are prone to miss detections. (2) The problem of identifying scene-related anomalies. Some anomalies are closely related to the scene. For example, playing football on a field is normal, but it is abnormal on a road. Although object-level methods can effectively reduce the impact of background noise, they usually ignore scene information, resulting in the inability to effectively detect scene-related abnormal events. The present invention aims to solve the above two key unsupervised video anomaly detection problems.

[0007] In response to the above problems, the present invention proposes an unsupervised scene-related video anomaly detection method based on hash mapping, which aims to enable the model to capture detailed features such as motion amplitude and direction by integrating optical flow information, and enhance the model's ability to recognize subtle differences between normal and abnormal. At the same time, the foreground object is effectively bound to the scene information in which it is located, so that the model can effectively detect scene-related abnormal events. Summary of the invention

[0008] In order to solve the above technical problems, the purpose of the present invention is to provide an unsupervised scene-related video anomaly detection method based on hash mapping. This method can fuse video frame features with optical flow features, capture spatiotemporal features and detail features such as motion amplitude and direction, thereby expanding the subtle differences between normal and abnormal. At the same time, this method effectively binds the foreground object to the scene information in which it is located, so that the model can learn to identify abnormal events and scene-related anomalies that are not significantly different from normal. Therefore, this method not only improves the ability to detect anomalies, but also effectively solves the problem of identifying scene-related anomalies, significantly improving the performance of unsupervised video anomaly detection.

[0009] The technical solution of the present invention:

[0010] 1. An unsupervised scene-related video anomaly detection method based on hash mapping, characterized in that it comprises the following steps:

[0011] Step 1: Extract continuous RGB video frames of the input video and generate corresponding optical flow maps using a pre-trained optical flow estimation model; the optical flow estimation model is a FlowNet2 network;

[0012] Step 2: Use the pre-trained object detector YOLOv8 to extract the foreground objects from the RGB video frame and optical flow map obtained in step 1, and use the object tracker ByteTrack to track the foreground objects in time order and organize them into foreground object cubes; at the same time, mask the foreground objects in the RGB video frame to obtain the background image;

[0013] Step 3: Use the feature extraction network loaded with the pre-trained model to extract the RGB features and optical flow features of the foreground object respectively, concatenate and fuse the RGB features and optical flow features, and input them into the hash encoder. The hash encoder is a learnable hash function that maps features to hash values; process the hash values ​​to obtain hash keys; take the same object in adjacent time steps as a positive sample pair, define the loss by calculating the cosine distance between their hash values, and update the hash encoder parameters;

[0014] The feature extraction network loaded with the pre-trained model is the I3D network (Inflated 3D ConvNet) pre-trained on the public dataset Kinetics400, and its last classification layer is removed; assuming that the RGB cube and optical flow cube of the foreground object O at time t are and The RGB cube and optical flow cube of the foreground object are input into the I3D network, and the extracted features are f t RGB and f t Flow , concatenate and fuse the two to get feature f t , the calculation formula is as follows:

[0015]

[0016] in, Represents a splicing operation;

[0017] Then, the fused feature f t Input a hash encoder composed of multiple parallel hash layers to obtain multiple hash values, and all hash values ​​are concatenated to form the final hash value h t , the calculation formula is as follows:

[0018]

[0019] Among them, HashLayer i represents the i-th hash layer, B represents the number of parallel hash layers, Represents the feature f t The hash value obtained after the i-th hash layer, concat represents the concatenation operation;

[0020] By calculating the final hash value h t Process it and get the corresponding hash key k t , the calculation formula is as follows:

[0021]

[0022] Among them, l represents the number of possible values ​​for each discrete value in the generated hash key. represents the floor function;

[0023] The same object in adjacent time steps is regarded as a positive sample pair, and the cosine distance L of its hash value c The calculation formula is as follows:

[0024]

[0025] Among them, · represents the dot product operation, ||·|| represents the Euclidean norm, Δ t Indicates a time interval;

[0026] In addition, the distance between the hash values ​​generated by different hash layers is used as the mutual difference loss The calculation formula is as follows:

[0027]

[0028] Where R represents the length of the hash value generated by the hash layer, B represents the number of parallel hash layers, and ||·|| represents the Euclidean norm;

[0029] The weighted sum of the two is used as the total loss function L, and the hash encoder parameters are updated. The calculation formula is as follows:

[0030]

[0031] Among them, λ m is a hyperparameter of the balanced loss function, which optimizes the hash encoder parameters to map similar foreground events to similar hash values ​​while capturing multiple different features;

[0032] Step 4: Input the background image of the video frame into a scene classifier, which is trained based on predefined scene categories to extract discriminative scene information;

[0033] The scene classifier is a ResNet50 model that has been pre-trained on the public dataset ImageNet. The last classification layer is replaced with an output layer with the same number of scene categories, and the newly added classification layer is trained.

[0034] Step 5: All foreground objects are input into the hash key and hash value generated in step 3, and the corresponding background image is passed through the scene classifier to obtain the scene code; the hash key is used as the index, and the hash value and the scene code are combined as the storage content to build a database that stores all normal patterns. Each entry in the database represents a normal pattern, which stores the representation of the normal pattern and its corresponding scene information;

[0035] Assume that the background image corresponding to the foreground object O at time t is B t, the confidence score of the scene category is obtained through the scene classifier, and further processed to obtain the scene code s t , the calculation formula is as follows:

[0036] s t =one_hot(SceneClassifier(B t )) (7)

[0037] Among them, SceneClassifier is the trained scene classifier, one_hot means that the position with the maximum confidence score in the classifier output is set to 1, and the remaining positions are set to 0;

[0038] Using the hash key as the index, the hash value and the scene code are combined as the storage content and regarded as an entry in the database to build a database that stores all normal modes; each parallel hash layer will generate a hash value, so multiple databases are built; when multiple events are mapped to the same hash key, the average vector of the hash values ​​of these events is calculated and the scene code is bitwise ANDed, and the result is used as the unique entry value of the hash key; for the mth database, the hash key The entries in this database are calculated as follows:

[0039]

[0040] Among them, h m Indicates the hash value stored in the entry corresponding to the current hash key of the mth database. represents the hash value of the foreground object at time t after the hash encoder, s m Indicates the scene code stored in the entry corresponding to the current hash key of the mth database. Represents the scene code corresponding to the foreground object at time t, Represents a hash key is a collection of indices, ∧ represents a bitwise AND operation;

[0041] Step 6: Input the test video into step 1 and step 2 for preprocessing to obtain the RGB cube and optical flow cube of the test event; input the RGB cube and optical flow cube into step 3 to obtain the hash key and hash value, input the corresponding background image into the scene classifier trained in step 4 to obtain the scene code, use the hash key as an index to query the normal mode database to obtain the matching entry, calculate the distance between the test event and the matching entry, and determine whether it is an abnormal event to complete the video anomaly detection;

[0042] The test video is input into step 1 and step 2 for preprocessing to obtain the RGB cube and optical flow cube of the test event; for the test event X t Get the video frame cube and the optical flow cube Enter step 3 to get the hash key corresponding to the event Hash value and scene coding Use the hash key as the index to query the matching normal pattern in the normal pattern database to obtain the matching entry, calculate the distance between the test hash value and the matching entry representation, and the minimum distance to all hash tables is considered as the anomaly score. The calculation formula is as follows:

[0043]

[0044] Among them, h x,n Represents the value of the nth matching hash table entry, Represents event X t In the hash value of the nth hash table, ||·|| represents the Euclidean norm, and min represents the minimum value;

[0045] If there is no matching entry or the bitwise AND result with the scene code is 0, it means that this event does not conform to the normal pattern in a specific scene, and a default high anomaly score P is assigned to it. max , that is, the maximum distance between any two hash values, is calculated as follows:

[0046]

[0047] The beneficial effects of the present invention are as follows: the present invention analyzes the foreground object in the video, extracts and fuses its RGB features and optical flow features, and uses a learnable hash function to map the fused features into a compact hash value, thereby generating a hash key to construct a normal mode database. This method can make full use of all normal data, so that the model can pay attention to more detailed features such as motion amplitude and direction, expand the subtle differences between normal and abnormal, and improve the model's ability to identify insignificant abnormalities. In addition, the present invention maps the foreground object features into hash keys and values ​​through a hash encoder, and performs scene encoding on the corresponding background, uses the hash key as the normal mode database index, and uses the representation of the normal mode and the corresponding scene encoding as the entry storage content, thereby realizing the effective association between the foreground object and the scene in which it is located, enhancing the scene perception ability of the model, and enabling the model to effectively detect scene-related abnormalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the unsupervised scene-related video anomaly detection method based on hash mapping of the present invention;

[0049] Figure 2 It is an overall framework diagram of a specific embodiment of the present invention;

[0050] Figure 3The figure is a flow chart of the test steps of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. The present invention includes but is not limited to the following embodiments.

[0052] like Figure 1 As shown, the present invention provides an unsupervised scene-related video anomaly detection method based on hash mapping, and its specific implementation process is as follows:

[0053] 1. Video preprocessing

[0054] First, we extract continuous RGB video frames of the input video and generate the corresponding optical flow map using the pre-trained optical flow estimation model. The selected optical flow estimation model is the FlowNet2 network. Then, we use the pre-trained object detector YOLOv8 to extract the foreground objects from the video frames and the optical flow map, and use the object tracker ByteTrack to track the same objects in time sequence and organize them into RGB cubes and optical flow cubes respectively. At the same time, we shield the foreground objects in the video frame to obtain the background image.

[0055] 2. Training scene classifier

[0056] Use the scene classifier to classify the background image obtained in step 1. The selected scene classifier is ResNet50 loaded with the ImageNet pre-trained model. The last classification layer is replaced with an output layer with the same number of scene categories, and the newly added classification layer is trained to adapt to the specific scene classification task.

[0057] 3. Extract and fuse foreground RGB features and optical flow features, and classify scenes

[0058] The RGB features and optical flow features of the foreground object are extracted using the feature extraction network loaded with the pre-trained model. The feature extraction network loaded with the pre-trained model is the I3D network pre-trained on the public dataset Kinetics400, and its last classification layer is removed. The RGB cube and optical flow cube of the foreground object are input into I3D. Assume that the RGB cube and optical flow cube of the foreground object O at time t are and The extracted features are and Combining the two, we get f t , the calculation formula is as follows:

[0059]

[0060] in Represents a concatenation operation.

[0061] Assume that the background image corresponding to the foreground object O at time t is B t , the confidence score of the scene category is obtained through the scene classifier, and further processed to obtain the scene code s t , the calculation formula is as follows:

[0062] s t =one_hot(SceneClassifier(B t )) (12)

[0063] Among them, SceneClassifier is the trained scene classifier, and one_hot means setting the position with the maximum confidence score in the classifier output to 1 and the remaining positions to 0.

[0064] 4. Hash map encoder encodes features and calculates the hash value of the foreground object

[0065] The fused feature f t Input a hash encoder composed of multiple parallel hash layers to obtain multiple hash values, and all hash values ​​are concatenated to form the final hash value h t , the calculation formula is as follows:

[0066]

[0067] Among them, HashLayer i represents the i-th hash layer, B represents the number of parallel hash layers, Represents the feature f t The hash value obtained after the i-th hash layer. concat represents the concatenation operation.

[0068] By calculating the final hash value h t After processing, the corresponding hash key k can be obtained t , the calculation formula is as follows:

[0069]

[0070] Where l represents the number of possible values ​​for each discrete value in the generated hash key. Represents the floor function.

[0071] 5. Calculate the distance between the hash values ​​of the same object in adjacent time steps as the loss to train the network

[0072] The same objects in adjacent time steps are taken as positive sample pairs, and the RGB features and optical flow features of the two are extracted and fused respectively. Then, the hash values ​​are input into the hash encoder to obtain the hash values ​​respectively, and the cosine distance L between the hash values ​​is calculated. c , the calculation formula is as follows:

[0073]

[0074] Among them, · represents the dot product operation, ||·|| represents the Euclidean norm, Δ t Indicates a time interval.

[0075] In addition, the distance between the hash values ​​generated by different hash layers is used as the mutual difference loss The calculation formula is as follows:

[0076]

[0077] Where R represents the length of the hash value generated by the hash layer, B represents the number of parallel hash layers, · represents the dot product operation, and ||·|| represents the Euclidean norm.

[0078] The weighted sum of the two is used as the total loss function L, and the hash encoder parameters are updated. The calculation formula is as follows:

[0079]

[0080] Among them, λ m It is a hyperparameter of the balanced loss function, which optimizes the model parameters in the direction of increasing the similarity of hash values ​​generated by similar foreground events and reducing the similarity of hash values ​​generated by different hash layers.

[0081] 6. Build a normal mode database

[0082] All normal videos in the training data set are preprocessed in step 1 to obtain the RGB cube, optical flow cube and background image, and then the fused features and the scene code of the background image are obtained in step 3. The fused features are input to the best hash map encoder obtained through training in step 4 to generate a hash value and process it to obtain a hash key. Using the hash key as the index, the hash value is combined with the scene code to form an entry in the normal mode database, thereby constructing a database containing all normal modes. Each parallel hash layer will generate a hash value, so that multiple databases can be constructed. When multiple events are mapped to the same hash key, the average vector of the hash values ​​of these events is calculated and the scene code is bitwise ANDed, and the result is used as the unique entry content of the hash key. For the mth database, the hash key k t m The entries in the normal mode database are calculated as follows:

[0083]

[0084] Among them, h m Indicates the hash value stored in the entry corresponding to the current hash key of the mth database. represents the hash value of the foreground object at time t after the hash encoder, s m Indicates the scene code stored in the entry corresponding to the current hash key of the mth database. Represents the scene code corresponding to the foreground object at time t, Represents a hash key is a collection of indices, ∧ represents a bitwise AND operation;

[0085] The constructed database contains all normal patterns, and each entry represents a normal pattern, where the representation of the pattern and its corresponding scene category are stored.

[0086] 7. Test set anomaly detection

[0087] like Figure 3 As shown, the video to be detected in the test set is input into step 1 for preprocessing to obtain the RGB cube, optical flow cube and corresponding background image of the test event. t , we can get the video frame cube and the optical flow cube Enter step 3 to get the hash key corresponding to the event Hash value and scene coding Use the hash key as the index to query the matching normal pattern in the normal pattern database built in step 6 to obtain the matching entry and calculate the hash value The distance between the matching entry representation and the minimum distance of all hash tables is taken as the anomaly score of the event The calculation formula is as follows:

[0088]

[0089] Among them, h x,n Represents the value of the nth matching hash table entry, Represents event X t In the hash value of the nth hash table, ||·|| represents the Euclidean norm, and min represents the minimum value;

[0090] If no matching entry is found, or the result of bitwise AND with the matching entry scene code is 0, it means that the event does not conform to the normal mode in the specified scene, and a default high anomaly score P is assigned to it. max , that is, the maximum distance between any two hash values, is calculated as follows:

[0091]

[0092] Where R represents the length of the hash key. The higher the anomaly score, the greater the possibility that the foreground object is abnormal. For each foreground object cube, the anomaly score of the cube is the anomaly score of the frame that constitutes the cube. When there are multiple foreground objects in a video frame, the maximum anomaly score of all foreground objects is taken as the anomaly score of the video frame. When the anomaly score exceeds the preset threshold, the frame is determined to be an abnormal frame, thereby obtaining the result of unsupervised scene-related video anomaly detection.

[0093] In summary, the present invention discloses an unsupervised scene-related video anomaly detection method based on hash mapping. The present invention focuses on analyzing the foreground object in the video, and uses a pre-trained feature extractor to extract the RGB and optical flow features of the foreground object cube respectively and fuse them. The fused features are mapped to hash keys and hash values ​​through a learnable hash function, and the background image corresponding to the foreground object is scene-classified and encoded to extract scene-distinguishing information. The generated hash key is used as an index, and the hash value and scene encoding are combined as database entry storage content to construct a normal mode database, store the normal mode of the training video and its corresponding scene, and realize the effective binding of the foreground object and the scene information in which it is located. In the test phase, the consistency of the test sample with the normal mode is judged by querying the database, and the scene difference between the scene and the matching entry is compared to detect the scene relevance. In this way, the model's perception ability for detailed features such as motion amplitude and direction can be improved, and the model's recognition effect on insignificant anomalies can be enhanced. At the same time, the scene perception ability of the model is significantly improved, so that it can effectively detect scene-related anomalies.

Claims

1. An unsupervised scene-related video anomaly detection method based on hash mapping, characterized in that: The following steps are involved: Step 1: Extract continuous RGB video frames of the input video and generate corresponding optical flow maps using a pre-trained optical flow estimation model; the optical flow estimation model is a FlowNet2 network; Step 2: Use the pre-trained object detector YOLOv8 to extract the foreground objects from the RGB video frame and optical flow map obtained in step 1, and use the object tracker ByteTrack to track the foreground objects in time order and organize them into foreground object cubes; at the same time, mask the foreground objects in the RGB video frame to obtain the background image; Step 3: Use the feature extraction network loaded with the pre-trained model to extract the RGB features and optical flow features of the foreground object respectively, concatenate and fuse the RGB features and optical flow features, and input them into the hash encoder. The hash encoder is a learnable hash function that maps features to hash values; process the hash values ​​to obtain hash keys; take the same object in adjacent time steps as a positive sample pair, define the loss by calculating the cosine distance between their hash values, and update the hash encoder parameters; The feature extraction network loaded with the pre-trained model is an I3D network pre-trained on the public dataset Kinetics400, and its last classification layer is removed; assuming that the RGB cube and optical flow cube of the foreground object O at time t are and The RGB cube and optical flow cube of the foreground object are input into the I3D network, and the extracted features are f t RGB and f t Flow , concatenate and fuse the two to get feature f t , the calculation formula is as follows: in, Represents a splicing operation; Then, the fused feature f t Input a hash encoder composed of multiple parallel hash layers to obtain multiple hash values, and all hash values ​​are concatenated to form the final hash value h t , the calculation formula is as follows: Among them, HashLayer i represents the i-th hash layer, B represents the number of parallel hash layers, Represents feature f t The hash value obtained after the i-th hash layer, concat represents the concatenation operation; By calculating the final hash value h t Process it and get the corresponding hash key k t , the calculation formula is as follows: Among them, l represents the number of possible values ​​for each discrete value in the generated hash key. represents the floor function; The same object in adjacent time steps is regarded as a positive sample pair, and the cosine distance L of its hash value c The calculation formula is as follows: Among them, · represents the dot product operation, ||·|| represents the Euclidean norm, Δ t Indicates a time interval; In addition, the distance between the hash values ​​generated by different hash layers is used as the mutual difference loss The calculation formula is as follows: Where R represents the length of the hash value generated by the hash layer, B represents the number of parallel hash layers, and ||·|| represents the Euclidean norm; The weighted sum of the two is used as the total loss function L, and the hash encoder parameters are updated. The calculation formula is as follows: Among them, λ m is a hyperparameter of the balanced loss function, which optimizes the hash encoder parameters to map similar foreground events to similar hash values ​​while capturing multiple different features; Step 4: Input the background image of the video frame into a scene classifier, which is trained based on predefined scene categories to extract discriminative scene information; The scene classifier is a ResNet50 model that has been pre-trained on the public dataset ImageNet. The last classification layer is replaced with an output layer with the same number of scene categories, and the newly added classification layer is trained. Step 5: All foreground objects are input into the hash key and hash value generated in step 3, and the corresponding background image is passed through the scene classifier to obtain the scene code; the hash key is used as the index, and the hash value and the scene code are combined as the storage content to build a database that stores all normal patterns. Each entry in the database represents a normal pattern, which stores the representation of the normal pattern and its corresponding scene information; Assume that the background image corresponding to the foreground object O at time t is B t , the confidence score of the scene category is obtained through the scene classifier, and further processed to obtain the scene code s t , the calculation formula is as follows: s t =one_hot(SceneClassifier(B t )) (7) Among them, SceneClassifier is the trained scene classifier, one_hot means that the position with the maximum confidence score in the classifier output is set to 1, and the remaining positions are set to 0; Using the hash key as the index, the hash value and the scene code are combined as the storage content and regarded as an entry in the database to build a database that stores all normal modes; each parallel hash layer will generate a hash value, so multiple databases are built; when multiple events are mapped to the same hash key, the average vector of the hash values ​​of these events is calculated and the scene code is bitwise ANDed, and the result is used as the unique entry value of the hash key; for the mth database, the hash key The entries in this database are calculated as follows: Among them, h m Indicates the hash value stored in the entry corresponding to the current hash key of the mth database. represents the hash value of the foreground object at time t after the hash encoder, s m Indicates the scene code stored in the entry corresponding to the current hash key of the mth database. Represents the scene code corresponding to the foreground object at time t, Represents a hash key is a collection of indices, ∧ represents a bitwise AND operation; Step 6: Input the test video into step 1 and step 2 for preprocessing to obtain the RGB cube and optical flow cube of the test event; input the RGB cube and optical flow cube into step 3 to obtain the hash key and hash value, input the corresponding background image into the scene classifier trained in step 4 to obtain the scene code, use the hash key as an index to query the normal mode database to obtain the matching entry, calculate the distance between the test event and the matching entry, and determine whether it is an abnormal event to complete the video anomaly detection; The test video is input into step 1 and step 2 for preprocessing to obtain the RGB cube and optical flow cube of the test event; for the test event X t Get the video frame cube and the optical flow cube Enter step 3 to get the hash key corresponding to the event Hash value and scene coding Use the hash key as the index to query the matching normal pattern in the normal pattern database to obtain the matching entry, calculate the distance between the test hash value and the matching entry representation, and the minimum distance to all hash tables is considered as the anomaly score. The calculation formula is as follows: Among them, h x,n Represents the value of the nth matching hash table entry, Represents event X t In the hash value of the nth hash table, ||·|| represents the Euclidean norm, and min represents the minimum value; If there is no matching entry or the bitwise AND result with the scene code is 0, it means that this event does not conform to the normal pattern in a specific scene, and a default high anomaly score P is assigned to it. max , that is, the maximum distance between any two hash values, is calculated as follows:

Images