Video anomaly detection method and device based on open vocabularies and electronic equipment
Through the video anomaly detection method based on open vocabulary, combined with visual language pre-training model and text encoding, the problem of low detection accuracy in the existing technology is solved, effective detection of abnormal behaviors in the category of unknowns is achieved, and detection is improved Accuracy and robustness.
Patent Information
- Application Number
- CN202411507920.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2024-11-26
AI Technical Summary
The existing video anomaly detection method has low detection accuracy in actual application scenarios. Due to the category settings of the closed set, it is difficult to detect abnormal behaviors of the categories that are not seen in categories.
Using the video anomaly detection method based on open vocabulary, by obtaining abnormal behavior vocabulary input by users, collecting video data in real time, and using visual language pre-trained models for feature extraction and text encoding, combining local and global timing dependencies to perform specific categories classification and category-free detection, enhance the expression ability of text information, and detect known and no category abnormal behaviors.
It improves the accuracy and detection rate of video anomaly detection, supports open vocabulary video anomaly detection, can effectively detect abnormal behaviors of unknown categories in the open world, and enhances the robustness and accuracy of detection.
Smart Images

Figure CN119027861A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method, device and electronic equipment based on open vocabulary. Background Art
[0002] At present, most surveillance video systems rely on manual monitoring. However, after working for a long time, people's attention will be constantly distracted due to fatigue, which can easily lead to false detection and missed detection.
[0003] In the related technology, the video anomaly detection method based on supervised learning is trained on a data set, in which frames are pre-classified as "normal" or "abnormal". It requires a data set with detailed annotations at the frame level. The low efficiency of manual labeling leads to limited data volume, and is restricted by the category setting of the closed set. Video anomaly detection in actual application scenarios is prone to produce more false alarms.
[0004] This shows that the video anomaly detection method in the related art has the technical problem of low detection accuracy in actual application scenarios. Summary of the invention
[0005] The present invention provides a video anomaly detection method, device and electronic device based on open vocabulary, which are used to solve the defect of low detection accuracy of video anomaly detection methods in the prior art in actual application scenarios, and improve the prediction accuracy and prediction detection rate in the open vocabulary video anomaly detection problem.
[0006] The present invention provides a video anomaly detection method based on open vocabulary, comprising the following steps.
[0007] Acquire abnormal behavior vocabulary input by a user; perform real-time video acquisition on target category videos corresponding to the abnormal behavior vocabulary to obtain regional video data; input the regional video data into a visual language pre-training model to obtain specific abnormal behaviors and unseen category abnormal behaviors output by the visual language pre-training model, which includes: extracting features from the regional video data to obtain visual information and text information; performing image encoding on the visual information to obtain frame-level visual features; determining local time dependencies and global temporal dependencies of the frame-level visual features to obtain sequence temporal features; performing vocabulary expansion on the text information based on learnable prompts to obtain enhanced text information; performing text encoding on the enhanced text information to obtain label text features; determining known category anomalies and unseen category anomalies of the regional video data based on the frame-level visual features, the sequence temporal features and the label text features; performing specific category classification on the known category anomalies to obtain specific abnormal behaviors; performing category-free detection on the unseen category anomalies to obtain unseen category abnormal behaviors.
[0008] According to a video anomaly detection method based on open vocabulary provided by the present invention, real-time video acquisition is performed on target category videos corresponding to the abnormal behavior vocabulary to obtain regional video data, including: determining an open set monitoring category corresponding to the abnormal behavior vocabulary; determining a target edge device including the open set monitoring category; based on timestamp and geographic location information, annotating the real-time monitoring video corresponding to the open set monitoring category through the target edge device to obtain the target category video; and performing video acquisition on the target category video to obtain regional video data.
[0009] According to a video anomaly detection method based on open vocabulary provided by the present invention, the vocabulary expansion of the text information based on learnable prompts to obtain enhanced text information includes: obtaining a learnable prompt pre-associated with the text information; and expanding the knowledge graph vocabulary of the text information based on the learnable prompt through a preset semantic network to obtain enhanced text information.
[0010] According to a video anomaly detection method based on open vocabulary provided by the present invention, the local time dependency and the global temporal dependency of the frame-level visual features are determined to obtain the sequence temporal features, including: performing a self-attention operation based on a feature local window on the frame-level visual features to obtain the local time dependency of the frame-level visual features; performing a global cross-window graph convolution operation on the frame-level visual features to obtain the global temporal dependency of the frame-level visual features; and determining the sequence temporal features based on the local time dependency and the global temporal dependency.
[0011] According to a video anomaly detection method based on open vocabulary provided by the present invention, the unseen category anomaly is subjected to category-free detection to obtain unseen category abnormal behavior, including: determining a phrase list of the label text features of the unseen category anomaly, wherein the phrases in the phrase list are composed of scene nouns and action verbs; performing feature extraction on the phrase list to obtain text embedding, and using the text embedding as semantic knowledge; determining a cosine similarity matrix of the frame-level visual features of the unseen category anomaly and the semantic knowledge; normalizing the cosine similarity matrix to obtain anomaly confidence of unseen category anomaly detection; associating the anomaly confidence with the frame-level visual features of the unseen category anomaly to obtain an enhanced image feature representation of the unseen category anomaly; performing category-free detection based on the enhanced image feature representation to obtain unseen category abnormal behavior.
[0012] According to a video anomaly detection method based on open vocabulary provided by the present invention, the known category anomaly is classified into a specific category to obtain a specific abnormal behavior, including: generating a semantic description of a new category anomaly based on the known category anomaly; inputting the semantic description of the new category anomaly into a generative artificial intelligence model to obtain a simulated abnormal video clip output by the generative artificial intelligence model; inserting the simulated abnormal video clip into a preset normal video to obtain a simulated abnormal video; training a video anomaly detection model based on the simulated abnormal video to obtain a trained video anomaly detection model; and classifying the known category anomaly into a specific category through the trained video anomaly detection model to obtain a specific abnormal behavior.
[0013] The present invention also provides a video anomaly detection device based on open vocabulary, comprising the following modules: An acquisition module is used to acquire abnormal behavior vocabulary input by a user; a collection module is used to perform real-time video collection of target category videos corresponding to the abnormal behavior vocabulary to obtain regional video data; a detection module is used to input the regional video data into a visual language pre-training model to obtain specific abnormal behaviors and unseen category abnormal behaviors output by the visual language pre-training model, wherein the detection module is also used to: perform feature extraction on the regional video data to obtain visual information and text information; perform image encoding on the visual information to obtain frame-level visual features; determine the local time dependency and global temporal dependency of the frame-level visual features to obtain sequence temporal features; perform vocabulary expansion on the text information based on learnable prompts to obtain enhanced text information; perform text encoding on the enhanced text information to obtain label text features; determine known category anomalies and unseen category anomalies of the regional video data based on the frame-level visual features, the sequence temporal features and the label text features; perform specific category classification on the known category anomalies to obtain specific abnormal behaviors; The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned methods for detecting anomalies in a video based on open vocabulary is implemented.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the video anomaly detection method based on open vocabulary as described above is implemented.
[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for detecting video anomalies based on open vocabulary.
[0016] The video anomaly detection method, device and electronic device based on open vocabulary provided by the present invention acquire the abnormal behavior vocabulary input by the user to collect the target category video in real time and obtain regional video data, thereby being able to monitor the characteristic abnormal behavior and improve the flexibility and pertinence of video anomaly detection; using the visual language pre-training model to extract features and enrich the semantic features, thereby improving the detection accuracy and detection rate of video anomaly detection, and supporting open vocabulary video anomaly detection, performing image encoding and time series analysis (including local time dependency and global time series dependency) on visual information, being able to capture dynamic changes in the video and improving the accuracy of abnormal behavior detection; performing vocabulary expansion and text encoding on text information, enhancing the expression ability of text information, so as to facilitate the subsequent determination of abnormal categories; combining visual information and text information, being able to make full use of the complementarity of the two modalities, and improving the robustness and accuracy of abnormal behavior detection; and classifying the abnormal behaviors of known categories and detecting the abnormal behaviors of unseen categories at the same time, thereby solving the technical problem that the video anomaly detection method in the prior art has low detection accuracy in actual application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 It is a flow chart of the open vocabulary-based video anomaly detection method provided by the present invention.
[0019] Figure 2 It is a structural diagram of the cloud-edge collaboration provided by the present invention.
[0020] Figure 3 It is a structural schematic diagram of the local timing and global timing provided by the present invention.
[0021] Figure 4 It is a structural schematic diagram of the semantic enhancement module provided by the present invention.
[0022] Figure 5 It is a structural schematic diagram of the abnormal video synthesis module provided by the present invention.
[0023] Figure 6 It is a schematic diagram of the overall structure of the open vocabulary-based video anomaly detection method provided by the present invention.
[0024] Figure 7 It is a structural schematic diagram of a video anomaly detection device based on open vocabulary provided by the present invention.
[0025] Figure 8 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] With the rapid development of the economy, people's requirements for urban life safety are constantly increasing, and it is becoming more and more common to install surveillance cameras in public places. At present, most surveillance video systems rely on manual monitoring. However, after working for a long time, people's attention will be constantly distracted due to fatigue, which can easily lead to false detection and missed detection. In order to overcome the various problems of manual monitoring, surveillance video anomaly detection technology has emerged. Through computer programs, video data is intelligently analyzed, and abnormal events occurring in video scenes can be automatically warned, greatly improving the detection accuracy and efficiency. Video anomaly detection equipment can detect abnormal behaviors or entities that are obviously deviated from normal, such as identifying multiple moving objects when the prior knowledge of video surveillance is limited, or detecting specific events, such as fighting, trampling, traffic accidents, etc.
[0028] Current video anomaly detection methods can be divided into supervised, self-supervised, weakly supervised, and unsupervised learning paradigms. The above methods have certain shortcomings: (1) In supervised learning, the algorithm is trained on a dataset where frames are pre-classified as "normal" or "abnormal", which requires a dataset with detailed annotations at the frame level. The low efficiency of manual annotation leads to limited data volume. (2) In self-supervised methods, the algorithm generates supervision from the input data, bypassing the need for external labels provided by humans. This paradigm is valuable for anomaly detection where labeled abnormal data is scarce, but it is easily limited by the distribution and number of samples, and the accuracy is not high in practical applications. (3) In weakly supervised methods, weak supervision consists of normal and abnormal samples, which can be regarded as a binary classification task, aiming to detect anomalies at the frame level under the constraint of temporal annotation. The annotator can mark the "segments" or short clips where the anomaly is observed in the video as weak labels for training the model. The multi-instance learning method is usually adopted, which regards normal and abnormal videos as negative and positive bags, respectively, and the video clips as instances in the bag. The multi-instance learning method tends to ignore the complex interactions of features over time. (4) Unsupervised methods are mainly reconstruction-based methods. Normal events can be effectively reconstructed from the learned representation, while abnormal or anomalous events deviate significantly from the representation and are therefore more difficult to reconstruct. During training, only normal videos are considered, while during testing, the model is evaluated on both normal and abnormal videos to assess its anomaly detection capabilities. However, the diversity of abnormal events makes its accuracy poor in practical applications. (5) The above learning paradigms are all limited to the category setting of closed sets and are difficult to apply in the open world. Some open video anomaly detection methods cannot detect unseen anomaly types given seen abnormal and normal videos.
[0029] The task of video anomaly detection has an open world demand, that is, it is necessary to detect untrained anomaly types. Traditional self-supervised and unsupervised methods are prone to large false positives, while weak supervision is effective in detecting seen anomalies, but less effective in detecting unseen anomalies. Open set video anomaly detection aims to train models based on normal videos and visible anomaly types, and try to detect unseen anomaly types, but existing methods mainly focus on anomaly detection of seen categories and ignore unseen anomaly categories.
[0030] Optionally, the open vocabulary-based video anomaly detection method of the embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the example of the open vocabulary-based video anomaly detection method of the present embodiment being executed by a server.
[0031] Figure 1 is a flow chart of the open vocabulary-based video anomaly detection method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Acquire abnormal behavior vocabulary input by a user.
[0032] Step 102 : real-time video acquisition is performed on target category videos corresponding to abnormal behavior words to obtain regional video data.
[0033] In an embodiment of the present invention, a public area video acquisition device is pre-deployed, an edge device is started in the cloud, and an open set monitoring category of the edge device is set in the cloud. Users can input any abnormal behavior vocabulary through the cloud interactive interface. The system will adjust the detection strategy according to the input abnormal behavior vocabulary to achieve specific category video anomaly detection; cameras are set in the cloud to cover the monitoring area to achieve 24-hour uninterrupted video acquisition and analysis.
[0034] In an embodiment of the present invention, a surveillance camera is called to transmit the real-time collected public area video data to a background monitoring edge device. At the same time, the edge device uses timestamp and geographic location information to mark the video data to ensure the consistency and traceability of the data.
[0035] Step 103 , inputting the regional video data into the visual language pre-training model, and obtaining the specific abnormal behavior and the unseen category abnormal behavior output by the visual language pre-training model.
[0036] In an embodiment of the present invention, an edge device is called to use a visual language pre-training model to extract features from regional video data. The visual language pre-training model can capture visual information and semantic information in the video, providing rich feature representation for subsequent anomaly detection.
[0037] The above step 103 specifically includes the following steps: Step 1031, extract features from the regional video data to obtain visual information and text information.
[0038] The regional video data is segmented into separate frames for subsequent visual analysis; various visual feature extraction methods (such as color features, texture features, shape features, motion features, etc.) are used to calculate the visual features of each frame.
[0039] For text contained in video frames (such as road signs, license plates, subtitles, etc.), OCR technology can be used to extract them into readable text information; if the video contains audio, speech recognition technology can be used to convert the audio into text information.
[0040] Step 1032, performing image encoding on the visual information to obtain frame-level visual features.
[0041] In an embodiment of the present invention, the weights of the visual language pre-training model inherit the multimodal pre-training neural network (CLIP, Contrastive Language-Image Pre-training) model, and no parameter update is performed during training. The image encoder of the visual language pre-training model extracts visual features from each frame of the video frame by frame to obtain frame-level visual features.
[0042] Among them, the visual language pre-training model can be pre-trained using a large amount of paired data of images and texts to learn the alignment relationship between images and texts. The model has the ability of multimodal learning and can simultaneously understand information from two different modalities, image and text, and establish connections between them.
[0043] Step 1033, determining the local time dependency and the global temporal dependency of the frame-level visual features to obtain the sequence temporal features.
[0044] In an embodiment of the present invention, in order to capture local temporal dependencies, a self-attention operation is introduced on the frame-level features from the frozen image encoder of the visual language pre-training model, and the self-attention calculation is limited to a local window rather than a global scope; in order to further capture global dependencies, a lightweight graph convolutional network (GCN) module is introduced after the local module to capture global dependencies from the perspective of feature similarity and relative distance.
[0045] Through the embodiments of the present invention, local and global temporal modeling modules can be used to capture temporal dependencies from different angles, make up for the difference between the fragment image domain and the video domain, and enhance long-term and short-term temporal dependencies.
[0046] Step 1034, expand the vocabulary of the text information based on the learnable prompt to obtain enhanced text information.
[0047] In an embodiment of the present invention, abnormally relevant, category-specific text descriptions (i.e., learnable prompts) are extracted from a preset semantic network (e.g., ConceptNet) to enhance text information.
[0048] In the context of text processing, learnable cues involve a deep understanding of the text content, enabling the recommendation or generation of words or phrases that are related to the original text but not in it. Vocabulary expansion refers to increasing the vocabulary or phrases in a text to enrich the text content and potentially improve the performance of subsequent tasks such as text classification, sentiment analysis, etc. This can be achieved by adding synonyms, antonyms, related concepts, or contextually relevant words.
[0049] For example, semantic features are extracted from the input text information through a preset semantic network, and additional words or phrases related to the text content are generated based on the semantic features. These generated words or phrases are added to the original text to obtain enhanced text information.
[0050] Step 1035, performing text encoding on the enhanced text information to obtain label text features.
[0051] In the embodiment of the present invention, the text information is enhanced by processing the text encoder of the visual language pre-training model, that is, the enhanced text information is converted into a series of numerical text vectors, and the text vector captures the semantic and syntactic information in the text. Then, the text vector can be used as the input feature of the subsequent task, that is, the label text feature.
[0052] Label text features refer to text features that can be used for subsequent tasks (such as classification, regression, etc.) after being encoded. These features are usually represented in a high-dimensional vector space and have the ability to distinguish different text categories.
[0053] Step 1036, based on the frame-level visual features, the sequence temporal features, and the label text features, determine the known category anomalies and the unseen category anomalies of the regional video data.
[0054] In an embodiment of the present invention, the edge device uses a visual language model to extract features from video data. The model can capture visual information and semantic information (i.e., text information) in the video, provide rich feature representation for subsequent anomaly detection, and extract anomaly-related, category-specific text from ConceptNet to enhance text information; the visual language pre-training model learns the semantic correspondence between vision and language through pre-training on large-scale data, has powerful visual representation capabilities with rich semantics and zero-sample generalization capabilities, captures video frame-level video features, sequence timing features, and label text features, and provides rich feature representation for open vocabulary video anomaly detection.
[0055] In an embodiment of the present invention, an open set training method is used to train the visual language pre-training model. The supervised pre-training for seen abnormal categories improves the visual language pre-training model's ability to recognize specific abnormal behaviors. The open set pre-training for unseen abnormal categories enables the visual language pre-training model to have generalization capabilities and to recognize abnormal behaviors of unseen categories.
[0056] Step 1037, classify the known category anomalies into specific categories to obtain specific abnormal behaviors.
[0057] Step 1038, performing non-class detection on the unseen class anomaly to obtain the unseen class abnormal behavior.
[0058] In an embodiment of the present invention, a visual language pre-training model is used to detect and identify anomalies of known categories and unseen categories, and the open vocabulary video anomaly detection task is decomposed into two complementary task models: category-free detection and specific category classification. The two tasks are jointly optimized, and a semantic knowledge enhancement module is designed to introduce semantic knowledge from the language model for the detection task. By designing an anomaly synthesis module, a forged unseen anomaly video is generated with the help of a visual generation model for the classification task, thereby enhancing the detection capability of normal frames and anomaly frames, and achieving accurate open vocabulary video anomaly detection.
[0059] The analysis results at the edge (including specific abnormal behaviors and abnormal behaviors of unseen categories) will be uploaded to the cloud device for specific category recording, statistics, and alarms; the cloud automatically generates an anomaly detection report, which includes the abnormal behavior category, time and location of occurrence, degree of abnormality, etc., and provides corresponding processing suggestions.
[0060] Through the above steps of the embodiment of the present invention, the abnormal behavior vocabulary input by the user is obtained; the target category video corresponding to the abnormal behavior vocabulary is collected in real time to obtain regional video data; the regional video data is input into the visual language pre-training model to obtain specific abnormal behaviors and unseen category abnormal behaviors output by the visual language pre-training model, which includes: feature extraction of regional video data to obtain visual information and text information; image encoding of visual information to obtain frame-level visual features; determining the local time dependency and global temporal dependency of frame-level visual features to obtain sequence temporal features; vocabulary expansion of text information based on learnable prompts to obtain enhanced text information; text encoding of enhanced text information to obtain label text features; based on frame-level visual features, sequence temporal features and label text features, known category anomalies and unseen category anomalies of regional video data are determined; specific category classification of known category anomalies is performed to obtain specific abnormal behaviors; unclassified detection of unseen category anomalies is performed to obtain unseen category abnormal behaviors. This solves the technical problem that the video anomaly detection method in the prior art has low detection accuracy in actual application scenarios.
[0061] According to a video anomaly detection method based on open vocabulary provided by the present invention, real-time video acquisition is performed on target category videos corresponding to abnormal behavior vocabulary to obtain regional video data, including: Determine the open set monitoring categories corresponding to the abnormal behavior vocabulary; Identify target edge devices that include open set monitoring categories; Based on the timestamp and geographic location information, the real-time monitoring video corresponding to the open set monitoring category is annotated by the target edge device to obtain the target category video; The target category video is collected to obtain regional video data.
[0062] In an embodiment of the present invention, a public area video acquisition device is deployed, an edge device is started on the cloud, an open set monitoring category of the edge device is set on the cloud, and a user can input words of any abnormal behavior through the cloud interactive interface. The system will adjust the detection strategy according to the input words to realize abnormal detection of specific category videos; a camera is set on the cloud to cover the monitoring area to realize 24-hour uninterrupted video acquisition and analysis; during the video acquisition process, the normal operation of the camera and related sensors is ensured to ensure continuous acquisition of video data, and the cloud adjusts the camera deployment position and angle to realize effective monitoring of the monitoring area.
[0063] refer to Figure 2 , Figure 2 It is a structural diagram of the cloud-edge collaboration provided by the present invention, which includes multiple surveillance cameras, multiple edge computing units and a cloud-based master control platform.
[0064] In an embodiment of the present invention, a user inputs abnormal behavior vocabulary into a cloud-based master control platform, and the cloud-based master control platform determines a target edge device (i.e., an edge computing unit) based on an open set monitoring category corresponding to the abnormal behavior vocabulary; real-time monitoring video is acquired from a surveillance camera corresponding to the open set monitoring category through the target edge device; the target edge device labels the regional video data according to the timestamp and geographic location information to obtain a target category video, and returns the target category video to the cloud-based master control platform through the target edge device.
[0065] The surveillance camera is called to transmit the real-time collected public area video data to the background intelligent monitoring edge device. At the same time, the video data is annotated by the edge device with timestamp and geographic location information to ensure data consistency and traceability.
[0066] According to a video anomaly detection method based on open vocabulary provided by the present invention, text information is expanded based on learnable prompts to obtain enhanced text information, including: Get learnable cues pre-associated with text information; The text information is enhanced by expanding the knowledge graph vocabulary based on learnable prompts through a preset semantic network.
[0067] In an embodiment of the present invention, unusually relevant, specific category texts are extracted from a preset semantic network (e.g., ConceptNet) to enhance text information; Among them, ConceptNet is used for text data amplification in the training phase. It performs data amplification on the input text information based on the vocabulary of the knowledge graph to improve the semantic understanding ability of the visual language pre-training model for different similar texts.
[0068] For example, text information is analyzed to obtain text analysis results, where the text analysis results include key entities, events or attributes in the text, and learnable prompts corresponding to the text analysis results are obtained; query statements for ConceptNet are constructed based on the learnable prompts as query keywords; queries are executed in ConceptNet to obtain vocabulary and relationships associated with the prompts; and the original text information is expanded based on the query results. New vocabulary, phrases and relationships associated with the prompts are integrated into the text information to obtain enhanced text information to enhance its expressiveness and information content.
[0069] Through the embodiments of the present invention, text information can be expanded to enhance text feature information of regional video data.
[0070] According to an open vocabulary-based video anomaly detection method provided by the present invention, local time dependency and global temporal dependency of frame-level visual features are determined to obtain sequence temporal features, including: Perform a self-attention operation based on a local feature window on the frame-level visual features to obtain the local temporal dependency of the frame-level visual features; Perform a global cross-window graph convolution operation on the frame-level visual features to obtain the global temporal dependency of the frame-level visual features; Based on the local time dependency and the global time dependency, the sequence timing characteristics are determined.
[0071] refer to Figure 3 , Figure 3 It is a structural schematic diagram of the local timing and the global timing provided by the present invention; like Figure 3 As shown, the abnormal video, normal video or synthetic abnormal video is input into the image encoder of the visual language pre-training model to obtain the image features, and then the image features are subjected to the self-attention operation of the local feature window and the graph convolution operation of the global feature across the window.
[0072] In an embodiment of the present invention, in order to capture local temporal dependencies, a self-attention operation is introduced on the frame-level features of the frozen image encoder from the visual language pre-training model, and the self-attention calculation is limited to a local window rather than a global scope; in order to further capture global dependencies, a lightweight graph convolutional network module is introduced after the local module to capture global dependencies from the perspective of feature similarity and relative distance.
[0073] The local window part (local module) uses a self-attention operation, introduces a self-attention operation on the frame-level features of the frozen image encoder from the visual language pre-training model, and limits the self-attention calculation to the local window rather than the global scope. Specifically, the frame-level features are divided into windows of equal length and overlap (image blocks), and the self-attention calculation is limited to each window, and there is no information exchange between windows. Such an operation has a local receptive field similar to convolution and has a low computational complexity. The self-attention operation can refer to the following formula (1): (1)
[0074] in, represents the attention function, They represent query, key, and value respectively, and are the features obtained by passing the image features through different linear layers. express The characteristic dimension of represents the transposed matrix, represents the normalized exponential function.
[0075] Based on the feature similarity and feature relative distance of frame-level visual features, the global temporal dependency is determined.
[0076] Among them, the global cross-window (global module) part adopts graph convolution (GCN) operation, and a lightweight graph convolution network module is introduced after the local module to capture global dependencies. The graph convolution network module is used to model global dependencies from the perspective of feature similarity and relative distance. The feature similarity branch is used to generate the similarity relationship adjacency matrix of the graph convolution network module. The frame-by-frame cosine similarity is used to calculate the adjacency matrix. The feature similarity can be specifically referred to in the following formula (2): (2)
[0077] in, and are the feature vectors of different image slices, and Represents the eigenvectors and The bi-norm (also called the Euclidean norm or length) of represents the transposed matrix, Represents the cosine similarity of different image slices, that is, the cosine similarity adjacency matrix.
[0078] The position distance branch is used to capture long-range dependencies based on the position distance between every two frames, the neighboring adjacency matrix, and the feature relative distance are specifically shown in the following formula (3): (3)
[0079] in, Respectively represent different image block indexes, represents the manually set hyperparameters, Represents the proximity adjacency matrix.
[0080] According to the cosine similarity adjacency matrix ( ) and the neighboring adjacency matrix ( ) is used to construct the similarity relationship adjacency matrix of the graph model, and a graph convolution operation is performed to obtain the features of global association.
[0081] In some embodiments, the features obtained by the local self-attention operation (local temporal dependency) and the features obtained by the global graph convolution operation (global temporal dependency) are concatenated, added, or otherwise fused to obtain sequence temporal features; the sequence temporal features can be used for subsequent sequence modeling tasks such as classification, regression, or prediction.
[0082] Through the embodiments of the present invention, local and global temporal dependencies of frame-level visual features in video or time series data can be captured more effectively, thereby improving the performance of related tasks.
[0083] In some embodiments, a visual language pre-trained model is used to detect and identify anomalies of known and unseen categories, and the open vocabulary video anomaly detection task is decomposed into two complementary task models: category-free detection and category-specific classification. The two tasks are jointly optimized, and a semantic knowledge enhancement module is designed to introduce semantic knowledge from the language model for the detection task. An anomaly synthesis module is designed to generate forged unseen anomaly videos with the help of a visual generation model for the classification task, thereby enhancing the detection capability of normal frames and anomaly frames and achieving accurate open vocabulary video anomaly detection.
[0084] According to a video anomaly detection method based on open vocabulary provided by the present invention, a category-free detection is performed on anomalies of unseen categories to obtain abnormal behaviors of unseen categories, including: Determine a phrase list of label text features with no category anomalies, wherein the phrases in the phrase list are composed of scene nouns and action verbs; Perform feature extraction on the phrase list to obtain text embedding, and use the text embedding as semantic knowledge; Determine the cosine similarity matrix between frame-level visual features and semantic knowledge of unseen category anomalies; Normalize the cosine similarity matrix to obtain the anomaly confidence of unseen category anomaly detection; Associating the anomaly confidence with the frame-level visual features of the unseen category anomalies to obtain enhanced image feature representations of the unseen category anomalies; Class-free detection is performed based on enhanced image feature representation to obtain abnormal behaviors of unseen categories.
[0085] refer to Figure 4 , Figure 4 It is a structural schematic diagram of the semantic enhancement module provided by the present invention.
[0086] like Figure 4 As shown in the figure, the inner product of the input text features and image features is performed, normalized with softmax, and the abnormality confidence is obtained, which is then associated (summed) with the image features to obtain the enhanced image feature representation.
[0087] In the embodiment of the present invention, text semantic knowledge is introduced to assist visual detection, and the The phrase list represented by is composed of nouns (scenes) and verbs (actions). The text encoder of CLIP is used to extract text embedding as semantic knowledge, specifically referring to the following formula (4); (4)
[0088] in, Represents semantic knowledge, represents the text encoder of the vision-language pre-trained model, represents a language tokenizer that converts words into vectors, Represents a list of phrases.
[0089] In order to effectively integrate this textual semantic knowledge into visual information to enhance anomaly detection, a cross-modal injection strategy is designed. This strategy encourages visual signals to seek relevant semantic knowledge and integrate it into the process. For details, please refer to the following formula (5): (5)
[0090] in, represents the abnormal confidence, represents the normalized exponential function, represents the frame-level visual features, Represents semantic knowledge, represents the transposed matrix, Represents a hyperparameter.
[0091] In the embodiment of the present invention, the connection and , calculate the cosine similarity matrix of visual information (i.e., frame-level visual features) and semantic knowledge, and use Normalization is performed to generate anomaly confidences for class-agnostic detection (i.e., category-free detection), which are then associated with image features to obtain enhanced image feature representation.
[0092] According to an open vocabulary-based video anomaly detection method provided by the present invention, known category anomalies are classified into specific categories to obtain specific abnormal behaviors, including: Generate semantic descriptions of new categories of anomalies based on known categories of anomalies; Inputting the semantic description of the new category of anomalies into the generative artificial intelligence model, and obtaining a simulated anomaly video clip output by the generative artificial intelligence model; Inserting the simulated abnormal video clip into the preset normal video to obtain the simulated abnormal video; The video anomaly detection model is trained based on the simulated abnormal video to obtain a trained video anomaly detection model; The trained video anomaly detection model is used to classify known category anomalies into specific categories to obtain specific abnormal behaviors.
[0093] refer to Figure 5 , Figure 5 It is a structural schematic diagram of the abnormal video synthesis module provided by the present invention.
[0094] like Figure 5 As shown, the abnormal behavior vocabulary "fighting" is input into a preset language model to generate a semantic description; the semantic description is input into a preset video generation model to obtain multiple synthetic abnormal videos; the multiple synthetic abnormal videos are inserted into normal activity videos to obtain simulated abnormal video clips.
[0095] In the embodiment of the present invention, an abnormal video synthesis module is designed to generate a series of pseudo-new anomalies based on potential anomaly categories with the help of a large-scale visual generation model for classification tasks. The specific process is as follows: Generate semantic descriptions of potential new anomaly categories using a predetermined template, and use a generative artificial intelligence (AIGC) model to generate corresponding images or short video clips, including but not limited to Gen-2, Vidu, CogVideo and other video generation models. For details, please refer to the following formula (6): (6)
[0096] in, represents the generated simulated abnormal video clip, represents a generative artificial intelligence model, Represents a text description used to guide the generation of a video.
[0097] Then, the generated video Insert it into a normal video in the database to simulate an abnormal video clip. Then, use the video containing the abnormal video clip to train the video anomaly detection model.
[0098] refer to Figure 6 , Figure 6 It is a schematic diagram of the overall structure of the open vocabulary-based video anomaly detection method provided by the present invention, which includes image input (abnormal video, normal video and synthetic abnormal video), inputting learnable prompts and abnormal behavior vocabulary (for example, abuse, arrest and assault) into ConceptNet for knowledge graph vocabulary expansion, obtaining text input (i.e., enhanced text information), CLIP image encoder, CLIP text encoder, temporal adaptation module, semantic knowledge enhancement module, decoder, detection task and classification task.
[0099] Reference Figure 6 As shown in the figure, the edge device uses a visual language model to extract features from video data. The model can capture the visual and semantic information in the video, provide rich feature representation for subsequent anomaly detection, and extract anomaly-related, category-specific text from ConceptNet to enhance text information; the visual language pre-training model learns the semantic correspondence between vision and language through pre-training on large-scale data, has powerful visual representation capabilities with rich semantics and zero-sample generalization capabilities, captures video frame-level video features, sequence timing features, and label text features, and provides rich feature representation for open vocabulary video anomaly detection.
[0100] The following is an example of the open vocabulary-based video anomaly detection method provided by the present invention in a practical application scenario.
[0101] The present invention provides an intelligent security monitoring device based on open vocabulary video anomaly detection. The device aims to solve the problem that video anomaly detection methods are usually limited to closed set settings under weak supervision and encounter detection difficulties in open world applications where there are unseen anomaly categories in test data. The present invention designs an intelligent security monitoring device for public areas, a video anomaly detection method for unseen categories and specific categories based on a visual language model, an open set training method and anomaly detection analysis system.
[0102] This device uses surveillance cameras to take pictures. The imaging system is set up above the public area to capture the video stream, and the edge computing board is used for intelligent analysis. The analysis results are uploaded to the cloud for recording and statistics. At the same time, the alarm category can also be set through the cloud and sent to each edge computing unit.
[0103] In addition, the device designs an open vocabulary video anomaly detection method, uses a visual language pre-training model, calls the ConceptNet network to expand text data, enhances semantic context information, designs a semantic knowledge enhancement module, integrates text knowledge into visual signals with the help of a visual language model, and designs an abnormal data synthesis module. With the help of a visual generation model, pseudo unseen abnormal videos are generated to improve the accuracy of classification tasks. Semantic knowledge and synthetic abnormalities effectively expand the model's detection capabilities in seen and unseen category anomalies. The device adopts an open set video anomaly detection training method. First, in the training phase, a normal video that does not contain any abnormal clips and an abnormal video of a seen category are used to train the model. In the inference phase, abnormal videos of known and unseen categories are simultaneously predicted to meet the needs of the open world. In combination with a video acquisition and analysis device and a video anomaly detection method, the present invention designs and implements a smart security monitoring device based on open vocabulary video anomaly detection, and proposes the use of the zero-sample generalization advantage of the visual language pre-training model CLIP to further improve the accuracy of video anomaly detection of unseen categories. This device realizes integrated services such as public area video surveillance data collection, seen and unseen category anomaly detection, and cloud-edge collaborative management. It has extremely high prediction accuracy and prediction detection rate in the problem of open vocabulary video anomaly detection, and has practical application value.
[0104] The purpose of the present invention is to provide an intelligent security monitoring device based on open vocabulary video anomaly detection, including public area intelligent security monitoring equipment, an unseen category and specific category video anomaly detection method based on a visual language model, an open set training method and anomaly detection analysis system, which adopts open vocabulary detection technology, video anomaly detection technology, visual language model pre-training technology, and anomaly detection quality analysis technology.
[0105] The present invention provides an intelligent security monitoring device based on open vocabulary video anomaly detection, including a public area intelligent security monitoring device, an unseen category and specific category video anomaly detection method based on a visual language model, an open set training method and an anomaly detection analysis system, the method comprising: Step S1: Deploy the video acquisition device in the public area, start the edge device in the cloud, set the open set monitoring category of the edge device in the cloud, and the user can enter any abnormal behavior vocabulary through the cloud interactive interface. The system will adjust the detection strategy according to the input vocabulary to realize the abnormal detection of specific category videos; set the camera in the cloud to cover the monitoring area to realize 24-hour uninterrupted video acquisition and analysis.
[0106] Step S2: The surveillance camera transmits the real-time collected public area video data to the background smart monitoring edge device. At the same time, the edge device uses timestamp and geographic location information to mark the video data to ensure data consistency and traceability.
[0107] Step S3: The edge device uses a visual language model to extract features from the video data. The model can capture the visual and semantic information in the video, provide rich feature representation for subsequent anomaly detection, and extract anomaly-related, category-specific text from ConceptNet to enhance text information.
[0108] Step S4: Utilize local and global temporal modeling modules to capture temporal dependencies from different perspectives, bridge the gap between the fragment image domain and the video domain, and enhance long-term and short-term temporal dependencies.
[0109] Step S5: Train the model using an open set training method, perform supervised pre-training on seen abnormal categories to improve the model's ability to recognize specific abnormal behaviors; perform open set pre-training on unseen abnormal categories to enable the model to have generalization capabilities and be able to recognize abnormal behaviors of unseen categories.
[0110] Step S6: The analysis results at the edge will be uploaded to the cloud device for specific category recording, statistics, and alarms; the cloud automatically generates an anomaly detection report, which includes the abnormal behavior category, occurrence time, location, degree of anomaly, etc., and provides corresponding processing suggestions.
[0111] Further, in step S1, a public area video acquisition device is deployed, the edge device is started in the cloud, and an open set monitoring category of the edge device is set in the cloud. The user can input any abnormal behavior vocabulary through the cloud interactive interface. The system will adjust the detection strategy according to the input vocabulary to realize specific category video anomaly detection; the cloud sets cameras to cover the monitoring area to realize 24-hour uninterrupted video acquisition and analysis; during the video acquisition process, ensure that the camera and related sensors are working normally to ensure continuous acquisition of video data, and the cloud adjusts the camera deployment position and angle to realize effective monitoring of the monitoring area.
[0112] Furthermore, in step S2, the surveillance camera transmits the real-time collected public area video data to the background smart monitoring edge device. At the same time, the edge device uses timestamp and geographic location information to mark the video data to ensure the consistency and traceability of the data.
[0113] Furthermore, in step S3, the edge device uses a visual language model to extract features from the video data. The model can capture visual and semantic information in the video, provide rich feature representation for subsequent anomaly detection, and extract anomaly-related, category-specific text from ConceptNet to enhance text information. The visual language pre-training model learns the semantic correspondence between vision and language through pre-training on large-scale data. It has powerful visual representation capabilities with rich semantics and zero-sample generalization capabilities, captures video frame-level video features, sequence timing features, and label text features, and provides rich feature representation for open vocabulary video anomaly detection.
[0114] Furthermore, in step S4, in order to capture local temporal dependencies, a self-attention operation is introduced on the frame-level features from the CLIP frozen image encoder, limiting the self-attention calculation to a local window rather than a global scope; in order to further capture global dependencies, a lightweight graph convolutional network (GCN) module is introduced after the local module to capture global dependencies from the perspective of feature similarity and relative distance.
[0115] Furthermore, in step S5, a pre-trained visual language model is used to detect and identify anomalies of known and unseen categories, and the open vocabulary video anomaly detection task is decomposed into two complementary task models: category-free detection and category-specific classification. The two tasks are jointly optimized, and semantic knowledge is introduced from the language model for the detection task by designing a semantic knowledge enhancement module. By designing an anomaly synthesis module, forged unseen anomaly videos are generated with the help of a visual generation model for the classification task, thereby enhancing the detection capability of normal frames and anomaly frames, and achieving accurate open vocabulary video anomaly detection.
[0116] Furthermore, in step S6, the analysis results at the edge will be uploaded to the cloud device for specific category recording, statistics, and alarms; the cloud automatically generates an anomaly detection report, which includes the abnormal behavior category, occurrence time, location, degree of anomaly, etc., and provides corresponding processing suggestions.
[0117] The intelligent security monitoring device based on open vocabulary video anomaly detection designed by the present invention includes an intelligent security monitoring device for public areas, an unseen category and specific category video anomaly detection method based on a visual language model, an open set training method and an anomaly detection and analysis system, which has the following advantages: (1) forming a complete set of video anomaly detection and analysis equipment with integrated hardware and software and cloud-edge collaboration, realizing a complete collection-detection-analysis system; (2) adopting an integrated system built by high-definition camera-edge computing unit-cloud server, the cloud server uniformly sends the configuration, and the edge sends the analysis results to the cloud server after the analysis, which has the function of efficiently managing all monitoring cameras and all edge computing units. Advantages; (3) Use the visual language pre-training model for feature extraction and enrich the semantic features, thereby improving the detection accuracy and detection rate of video anomaly detection, and supporting open vocabulary video anomaly detection; (4) The training framework for open vocabulary can improve the expressiveness of model features, thereby enhancing the generalization performance of the model and improving the model's recognition accuracy for different seen and unseen anomaly categories; (5) Use the semantic knowledge enhancement module to integrate textual knowledge into the visual signal with the help of the visual language model, effectively expanding the model's detection capabilities for seen and unseen category anomalies; (6) Design an anomaly synthesis module to generate pseudo unseen anomaly videos with the help of the visual generation model to improve the accuracy of classification tasks.
[0118] The video anomaly detection device based on an open vocabulary provided by the present invention is described below. The video anomaly detection device based on an open vocabulary described below and the video anomaly detection method based on an open vocabulary described above can be referred to each other.
[0119] refer to Figure 7 , Figure 7 70 is a schematic diagram of the structure of the open vocabulary-based video anomaly detection device provided by the present invention, which includes an acquisition module 701, a collection module 702 and a detection module 703.
[0120] An acquisition module 701 is used to acquire abnormal behavior words input by a user; The acquisition module 702 is used to perform real-time video acquisition of target category videos corresponding to abnormal behavior words to obtain regional video data; The detection module 703 is used to input the regional video data into the visual language pre-training model to obtain the specific abnormal behavior and the unseen category abnormal behavior output by the visual language pre-training model, wherein the detection module is further used to: Extract features from regional video data to obtain visual information and text information; Encode the visual information into images to obtain frame-level visual features; Determine the local temporal dependency and global temporal dependency of frame-level visual features to obtain sequence temporal features; Expanding the vocabulary of text information based on learnable cues to obtain enhanced text information; Perform text encoding on the enhanced text information to obtain label text features; Based on frame-level visual features, sequence temporal features, and label text features, known category anomalies and unseen category anomalies in regional video data are determined; Classify known category anomalies into specific categories to obtain specific abnormal behaviors; Perform category-free detection on unseen category anomalies to obtain unseen category abnormal behaviors.
[0121] Specifically, the above-mentioned video anomaly detection device based on open vocabulary provided by the present invention can implement all the method steps implemented by the above-mentioned video anomaly detection method embodiment based on open vocabulary, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0122] Figure 8 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 8 As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the video anomaly detection method based on open vocabulary, which includes: obtaining abnormal behavior vocabulary input by the user; performing real-time video acquisition on the target category video corresponding to the abnormal behavior vocabulary to obtain regional video data; inputting the regional video data into the visual language pre-training model to obtain specific abnormal behavior and unseen category abnormal behavior output by the visual language pre-training model, which includes: extracting features from the regional video data to obtain visual information and text information; performing image encoding on the visual information to obtain frame-level visual features; determining the local time dependency and the global time dependency of the frame-level visual features to obtain sequence time features; performing vocabulary expansion on the text information based on the learnable prompt to obtain enhanced text information; performing text encoding on the enhanced text information to obtain label text features; determining known category anomalies and unseen category anomalies of the regional video data based on the frame-level visual features, the sequence time features and the label text features; performing specific category classification on the known category anomalies to obtain specific abnormal behaviors; performing category-free detection on the unseen category anomalies to obtain unseen category abnormal behaviors.
[0123] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video anomaly detection method based on open vocabulary provided by the above methods, which includes: obtaining abnormal behavior vocabulary input by a user; performing real-time video acquisition on target category videos corresponding to the abnormal behavior vocabulary to obtain regional video data; inputting the regional video data into a visual language pre-training model to obtain specific abnormal behaviors and unseen category abnormal behaviors output by the visual language pre-training model, which includes: The method is to extract features based on the data to obtain visual information and text information; perform image encoding on the visual information to obtain frame-level visual features; determine the local time dependency and global temporal dependency of the frame-level visual features to obtain sequence temporal features; perform vocabulary expansion on the text information based on learnable prompts to obtain enhanced text information; perform text encoding on the enhanced text information to obtain label text features; based on the frame-level visual features, sequence temporal features and label text features, determine the known category anomalies and unseen category anomalies of the regional video data; perform specific category classification on the known category anomalies to obtain specific abnormal behaviors; perform category-free detection on the unseen category anomalies to obtain unseen category abnormal behaviors.
[0125] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the video anomaly detection method based on open vocabulary provided by the above methods, the method comprising: obtaining an abnormal behavior vocabulary input by a user; performing real-time video acquisition on a target category video corresponding to the abnormal behavior vocabulary to obtain regional video data; inputting the regional video data into a visual language pre-training model to obtain specific abnormal behaviors and unseen category abnormal behaviors output by the visual language pre-training model, which includes: extracting features from the regional video data to obtain visual information and text information; performing image encoding on the visual information to obtain frame-level visual features; determining the local time dependency and the global time sequence dependency of the frame-level visual features to obtain sequence time sequence features; performing vocabulary expansion on the text information based on learnable prompts to obtain enhanced text information; performing text encoding on the enhanced text information to obtain label text features; determining known category anomalies and unseen category anomalies of the regional video data based on frame-level visual features, sequence time sequence features and label text features; performing specific category classification on the known category anomalies to obtain specific abnormal behaviors; performing category-free detection on the unseen category anomalies to obtain unseen category abnormal behaviors.
[0126] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A video anomaly detection method based on open vocabulary, characterized in that: include: Obtain abnormal behavior words input by the user; Performing real-time video acquisition on target category videos corresponding to the abnormal behavior vocabulary to obtain regional video data; Inputting the regional video data into a visual language pre-training model to obtain specific abnormal behaviors and unseen abnormal behaviors output by the visual language pre-training model, including: Extracting features from the video data of the region to obtain visual information and text information; Performing image encoding on the visual information to obtain frame-level visual features; Determining the local time dependency and the global time sequence dependency of the frame-level visual features to obtain a sequence time sequence feature; Expanding the vocabulary of the text information based on the learnable prompt to obtain enhanced text information; Performing text encoding on the enhanced text information to obtain label text features; Determine known category anomalies and unseen category anomalies of the regional video data based on the frame-level visual features, the sequence temporal features, and the label text features; Classifying the known category anomalies into specific categories to obtain specific abnormal behaviors; Performing non-class detection on the unseen category anomaly to obtain unseen category abnormal behavior.
2. The open vocabulary based video anomaly detection method according to claim 1, characterized in that: The real-time video acquisition of the target category video corresponding to the abnormal behavior vocabulary to obtain regional video data includes: Determine an open set monitoring category corresponding to the abnormal behavior vocabulary; determining a target edge device comprising the open set monitoring category; Based on the timestamp and geographic location information, the real-time monitoring video corresponding to the open set monitoring category is annotated by the target edge device to obtain the target category video; The target category video is captured to obtain regional video data.
3. The open vocabulary based video anomaly detection method according to claim 1, characterized in that: The step of performing vocabulary expansion on the text information based on the learnable prompt to obtain enhanced text information includes: obtaining a learnable prompt pre-associated with the text information; The text information is expanded through a preset semantic network based on the learnable prompts through the knowledge graph vocabulary to obtain enhanced text information.
4. The open vocabulary based video anomaly detection method according to claim 1, characterized in that: The determining of the local time dependency and the global time sequence dependency of the frame-level visual features to obtain the sequence time sequence features includes: Performing a self-attention operation based on a feature local window on the frame-level visual features to obtain a local time dependency of the frame-level visual features; Performing a global cross-window graph convolution operation on the frame-level visual features to obtain a global temporal dependency relationship of the frame-level visual features; Based on the local time dependency and the global timing dependency, a sequence timing feature is determined.
5. The open vocabulary based video anomaly detection method according to claim 1, characterized in that: The performing non-class detection on the unseen class anomaly to obtain the unseen class abnormal behavior includes: Determine a phrase list of the label text features of the unseen category anomaly, wherein the phrases in the phrase list are composed of scene nouns and action verbs; Performing feature extraction on the phrase list to obtain text embedding, and using the text embedding as semantic knowledge; Determining a cosine similarity matrix between the frame-level visual features of the unseen category anomaly and the semantic knowledge; Normalizing the cosine similarity matrix to obtain anomaly confidence of unseen category anomaly detection; Associating the anomaly confidence with the frame-level visual features of the unseen category anomaly to obtain an enhanced image feature representation of the unseen category anomaly; Based on the enhanced image feature representation, category-free detection is performed to obtain abnormal behaviors of unseen categories.
6. The open vocabulary based video anomaly detection method according to claim 1, characterized in that: The specific category classification of the known category anomalies to obtain specific abnormal behaviors includes: Generate a semantic description of a new category of anomalies based on the known category of anomalies; Inputting the semantic description of the new category of anomalies into a generative artificial intelligence model to obtain a simulated anomaly video clip output by the generative artificial intelligence model; Inserting the simulated abnormal video clip into a preset normal video to obtain a simulated abnormal video; Training a video anomaly detection model based on the simulated abnormal video to obtain a trained video anomaly detection model; The known category anomalies are classified into specific categories through the trained video anomaly detection model to obtain specific abnormal behaviors.
7. A video anomaly detection device based on open vocabulary, characterized in that: include: An acquisition module, used to acquire abnormal behavior vocabulary input by a user; A collection module, used for real-time video collection of target category videos corresponding to the abnormal behavior vocabulary to obtain regional video data; A detection module is used to input the regional video data into a visual language pre-training model to obtain specific abnormal behaviors and unseen abnormal behaviors output by the visual language pre-training model, wherein the detection module is further used to: Extracting features from the video data of the region to obtain visual information and text information; Performing image encoding on the visual information to obtain frame-level visual features; Determining the local time dependency and the global time sequence dependency of the frame-level visual features to obtain a sequence time sequence feature; Expanding the vocabulary of the text information based on the learnable prompt to obtain enhanced text information; Performing text encoding on the enhanced text information to obtain label text features; Determine known category anomalies and unseen category anomalies of the regional video data based on the frame-level visual features, the sequence temporal features, and the label text features; Classifying the known category anomalies into specific categories to obtain specific abnormal behaviors; Performing non-class detection on the unseen category anomaly to obtain unseen category abnormal behavior.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the open vocabulary-based video anomaly detection method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the open vocabulary-based video anomaly detection method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the open vocabulary-based video anomaly detection method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Method, device and apparatus for processing monitoring video data and medium
CN109194913A
Video abnormal behavior detection method
CN117351392A
Feature enhancement and fusion-based weak supervision video anomaly detection method and system
CN118470608A
Zero sample behavior recognition method based on multi-mode prompt learning
CN118823644A
Abnormality detection system, information processing device, and abnormality detection method
JP2019053384A
Cited By
Abnormal behavior detection method and device and electronic equipment
CN120673303A
Anomaly detection method and system
EP4807693A1
