A monitoring video abnormal event prediction method and system for precursor analysis

Abstract

The application discloses a kind of precursor analysis's monitoring video abnormal event prediction method and system.Utilize visual language model to extract abnormal event data automatically, construct structured semantic tuple label data set;Integrate semantic tuple and establish "environment-object-behavior-label" four-layer dynamic knowledge base exception tree;Design layered semantic retrieval mechanism: convert visual media into text description and encode as vector, match the best node in the specific level of exception tree through vector similarity, return the child node as the candidate of next stage.Using this mechanism, the context and key objects are retrieved layer by layer to narrow the scope, and finally based on behavior information, a prompt word enhancement model is constructed to analyze and generate abnormal event prediction.The application changes from the traditional way of generating future frames at the pixel level to semantic level reasoning, significantly reduces hardware dependence, realizes online real-time early warning on edge devices, provides white-box explanation of complete reasoning path, and enhances prediction credibility.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and behavior recognition, and specifically to a method and system for predicting abnormal events in surveillance videos based on precursor analysis. Background Technology

[0002] In modern intelligent surveillance systems, the ability to predict anomalies before they occur is a highly challenging yet crucial capability. Although technologies such as visual language models have made revolutionary progress in video understanding in recent years, existing video anomaly detection methods remain essentially reactive. The core of these technologies lies in identifying anomalies that have already occurred or are occurring; this "post-event detection" model significantly limits their practical effectiveness in critical application scenarios requiring prevention, such as public safety, industrial production, and traffic management.

[0003] To fill this critical gap, Video Anomaly Prediction (VAP) has emerged and become an important research direction. The core objective of VAP technology is to provide early warnings by identifying and analyzing precursory patterns that foreshadow impending anomalies. However, the fundamental challenge of this technology lies in accurately identifying extremely subtle behavioral cues and environmental indicators that occur seconds or even minutes before an anomalous event.

[0004] To address this challenge, traditional VAP (Virtual Anomaly Prediction) techniques primarily rely on future frame generation. These methods attempt to synthesize future video frames using generative models and then perform anomaly analysis on these generated future contents. However, this paradigm suffers from three insurmountable limitations: First, it suffers from low computational efficiency, requiring high-fidelity video frame synthesis at the pixel level, which incurs enormous computational overhead and makes real-time performance difficult to meet. Second, it relies heavily on historical data, with model training heavily dependent on a large number of well-labeled "precursor-anomaly" event pairs, resulting in severely insufficient predictive ability for novel or rare anomalies. Third, it lacks semantic interpretability; even if the model predicts successfully, it cannot provide a deep semantic explanation of "why" the anomaly occurred. Its prediction process functions like a "black box," limiting decision-makers' trust in and effective use of the prediction results.

[0005] One current technique is a video anomaly prediction method based on dual-channel feature enhancement, proposed in the paper "Video Anomaly Prediction: Problem, Dataset, and Method." This technique aims to address the limitation of traditional video anomaly detection techniques, which can only respond retrospectively. It constructs a dataset specifically for VAP tasks, and its core is a dual-channel deep learning model. This technique has verified the feasibility of predicting anomalies in advance by analyzing video content. The drawbacks of this technique are the lack of semantic interpretability and causal reasoning ability. Essentially, it is an end-to-end "black box" model that predicts by learning the statistical correlation of video features over time. The model can determine that an anomaly is about to occur, but it cannot provide a deep logical explanation of why the anomaly will occur. Predictive ability is limited by data-driven pattern matching; the model's predictive power relies entirely on visual patterns learned from the training data. This means that it only learns the representation of video pixel features, rather than the true causal relationships or logical chains behind the events. Therefore, its predictive performance will significantly decrease for anomalies that have not appeared in the training data or whose patterns are rare, resulting in insufficient generalization ability.

[0006] The second existing technology is the forward-backward scene-conditional autoencoder technique proposed in the paper "Scene-dependent prediction in latent space for video anomaly detection and anticipation" for semi-supervised video anomaly detection and prediction. This technique aims to simultaneously solve the problem of video anomaly detection and prediction, with a particular focus on the industry-wide challenge of scene-dependent anomalies. The core is to construct a forward-backward prediction model. The forward network predicts future frames based on past frames to detect current anomalies, while the backward network, based on the future frames predicted by the forward network and some observed frames, predicts past observed frames to predict future anomalies. Furthermore, a scene-conditional variational autoencoder is introduced, enabling the model to learn normal behavior patterns in specific scenes, effectively handling scene-dependent anomalies. The drawbacks of this technique are twofold. First, its prediction is still based on pixel-level future frame generation, leading to significant computational overhead and difficulty in generating high-quality, long-term future videos, thus limiting its predictive capabilities. Second, this approach is essentially an anomaly discrimination method based on reconstruction error, focusing on inconsistencies at the visual representation level rather than the inherent semantic logic and causal relationships of events. Furthermore, the technique used in this method to predict future frames relies on real and already occurring anomalous frames for calculation, making it impossible to perform actual video anomaly prediction.

[0007] The third existing technology is an event-level video anomaly prediction technique proposed in the paper "AssistPDA: An Online Video Surveillance Assistant for Video Anomaly Prediction, Detection, and Analysis". This technology aims to achieve early warning of potential anomalies, rather than just post-event detection. It uses Qwen2-VL as the backbone network, combining a visual encoder and a large language model. The visual encoder is responsible for extracting visual features from the real-time video stream, while the large language model processes these visual features and the text features of the user query to perform prediction, detection, and analysis, and generate a natural language response. The drawback of this technology is that its core capabilities heavily rely on the pre-training quality and generalization ability of the underlying Visual-Language Model (VLM) and Large Language Model (LLM). If the pre-training data does not adequately cover the precursor patterns of specific types or rare anomalies, the model's prediction performance may be affected when facing unseen or rare scenes, resulting in limited generalization ability. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing methods and propose a method and system for predicting abnormal events in surveillance videos based on precursor analysis. The main problems addressed by this invention include: First, meeting the real-time computational efficiency requirements of surveillance systems and overcoming the high latency bottleneck caused by pixel-level prediction and frame generation in traditional methods. Second, providing a clear and logically coherent reasoning path for the causes of anomalies, enhancing the credibility of prediction results and user understanding of model decisions. Third, effectively addressing rare or novel abnormal events, overcoming the limitations of purely data-driven methods in knowledge reuse and pattern expansion.

[0009] To address the aforementioned problems, this invention proposes a method for predicting abnormal events in surveillance video based on precursor analysis, the method comprising:

[0010] We use a visual language model to automatically extract data from anomaly event datasets and build a structured anomaly feature text annotation dataset consisting of semantic tuples.

[0011] The semantic tuples in the abnormal feature text annotation dataset are integrated to establish a dynamic, four-layer hierarchical knowledge base composed of semantic nodes, denoted as the anomaly tree;

[0012] Design a hierarchical semantic retrieval mechanism. The mechanism transforms the input visual media into a text description and encodes it into a vector through a visual language model. Then, it calculates the vector similarity to match the best semantic node among the candidate nodes at a specific level of the anomaly tree and returns all its child nodes as a candidate set for the next stage of retrieval.

[0013] The hierarchical semantic retrieval mechanism is used to retrieve environmental context nodes and key object nodes from the anomaly tree layer by layer to narrow down the candidate range. Then, based on the retrieved specific behavioral information, prompt words are constructed to enhance the visual language model's analysis of the surveillance video, and finally, anomaly event prediction is generated.

[0014] Preferably, the step of automatically extracting data from the abnormal event dataset using a visual language model and establishing a structured abnormal feature text annotation dataset composed of semantic tuples specifically involves:

[0015] The abnormal event dataset includes publicly available abnormal event datasets on the network and abnormal event datasets from surveillance videos owned by the location where the abnormal event prediction is deployed.

[0016] Video sequences that show a premonitory behavioral window that can be observed and analyzed before the actual occurrence of the abnormal event are selected from the abnormal event dataset to form a premonitory video dataset. Each of its data units is a tuple. ,in This represents a complete video sequence containing three stages: normal, precursor, and abnormal. Yes A textual description of the abnormal events that occurred;

[0017] Visual Language Model (VLM) is used for each video sequence Premonition fragments Deep analysis and automated annotation are performed, using prompt words to guide the VLM to extract information from four interrelated dimensions, and the results are organized into a structured semantic tuple; the automated annotation process is described by the following formula:

[0018] ;

[0019] Where VLM represents the visual language model function; and the text description of the abnormal event. As part of the prompt words, the VLM model is guided to generate classification labels; the output is a structured semantic tuple containing four-dimensional textual descriptions:

[0020] Context ( Environment Context: Describes the static scene and background information in which the video takes place;

[0021] Key objects ( Key Objects: Identify all dynamic or static entities in the scene that are highly relevant to potential anomalies, and describe their attributes and relative positions;

[0022] Specific behaviors ( Specific Behaviors: Detailed description of key objects Behavioral sequences and interaction patterns, capturing those subtle movements that deviate from the norm;

[0023] Abnormal tags ( Anomaly Label: Based on a comprehensive understanding of the environment, objects, and behaviors, it performs fine-grained classification of potential anomalies.

[0024] The precursor video dataset The structured semantic tuples formed by all video sequences constitute the anomaly feature text annotation dataset.

[0025] Preferably, the step of integrating the semantic tuples in the abnormal feature text annotation dataset to establish a dynamic, four-layer hierarchical knowledge base composed of semantic nodes, denoted as an anomaly tree, specifically involves:

[0026] The text in each structured semantic tuple is summarized using a Large Language Model (LLM), as follows:

[0027] ;

[0028] Here, LLM stands for Large Language Model Function, whose output is a highly summarized core phrase, including a summary of the environmental context. Key Object Summary Summary of specific behaviors Summary of abnormal tags ;

[0029] A pre-trained text encoder is used to process the text. Converting it to a high-dimensional vector is shown in the following equation:

[0030] ;

[0031] in It is the pre-trained text encoder. The input consists of four independent dimensions: "environmental context", "key objects", "specific behaviors", and "anomaly labels". Abstract, , It is the corresponding dimension A high-dimensional vector;

[0032] In the three independent dimensions of "environmental context", "key objects", and "specific behaviors" Internally, clustering is performed separately, and the clustering adopts an adaptive similarity threshold strategy: for a high-dimensional vector Calculate its relationship with the dimension it belongs to. Previously existing nodes high-dimensional vector Cosine similarity between them:

[0033] ;

[0034] when At that time, using high-dimensional vectors Create a new node; otherwise, put the vector... Merge into node In China; among them It is a dimension Preset similarity threshold; no clustering is required for the "abnormal label" dimension;

[0035] After clustering, a set of environment context summary nodes is formed. Key object summary node set Specific behavior summary node set , collection of abnormal tag summary nodes ;

[0036] The , , , Connecting the anomalies according to the pre-defined causal logic and their corresponding relationships forms a complete anomaly tree; formally represented by the following formula:

[0037] ;

[0038] The arrow → represents a directed connection between nodes, indicating causal relationships and conditional dependencies; each complete path from the root node to a leaf node represents a causal chain; the tree structure essentially encodes a conditional probability model, whose joint probability distribution is expressed as follows:

[0039] ;

[0040] in These respectively represent "environmental context", "key object", "specific behavior", and "anomaly label". "Represents the known information in the probability formula" Reasoning under certain circumstances The probability of occurrence.

[0041] Preferably, the hierarchical semantic retrieval mechanism is designed such that the input visual media is converted into a text description and encoded into a vector through a visual language model. Then, the best semantic node is matched among the candidate nodes at a specific level of the anomaly tree by calculating the vector similarity, and all its child nodes are returned as a candidate set for the next stage of retrieval. Specifically:

[0042] The general formal definition of a hierarchical semantic retrieval mechanism is as follows:

[0043] ;

[0044] in, Represents a semantic retrieval function; Represents the input visual media, suitable for static single-frame images. It is used to analyze static environments or objects, and is also suitable for dynamic video clips. It is used to analyze time-related behaviors; Textual cues representing the current analysis level are used to guide the Visual Language Model (VLM) to focus its attention on specific information dimensions. This represents the set of candidate nodes for the current retrieval operation, i.e., a subset of nodes at a specific level of the anomaly tree. When searching for the optimal node at the "environmental context" level, since there is no prior information obtained from the retrieval, For the The complete collection;

[0045] It is the output tuple; Is Found in the set, related to input visual media exist The node that best matches the semantics under guidance; yes The set of all direct child nodes in the anomaly tree will be used as the basis for the next inference stage. ;

[0046] The retrieval process of the hierarchical semantic retrieval mechanism is as follows:

[0047] Visual Language Model (VLM) Reception and Convert visual media into text descriptions , represented as ;

[0048] Using the pre-trained text encoder The text description Mapping to the same high-dimensional vector space as the anomaly tree nodes yields the query vector. , represented as ;

[0049] By calculating the query vector and Each candidate node in the set Embedded vector The cosine similarity between nodes is used to find the semantically best-matching node. :

[0050] ;

[0051] The above formula finds the node that best matches the query intent by maximizing cosine similarity. ;

[0052] In determining the node with the best semantic match Then, query the anomaly tree and obtain all its direct child nodes to form a set. .

[0053] Preferably, the hierarchical semantic retrieval mechanism is used to retrieve environmental context nodes and key object nodes layer by layer from the anomaly tree to narrow down the candidate range. Then, based on the retrieved specific behavioral information, prompt words are constructed to enhance the visual language model's analysis of the surveillance video clips, ultimately generating anomaly event predictions. Specifically:

[0054] The video to be analyzed is ,right The video is split into multiple segments every 2 seconds. , , Total number of video clips; each video clip Four frames are sampled evenly, with each frame being a keyframe.

[0055] Phase 1, Contextual Analysis and Constraints: Considering the fixed location of the surveillance camera, the semantic retrieval function is invoked. Treatment of video clip analysis The first keyframe Processing:

[0056] ;

[0057] During this stage of the call, Parameters correspond to a single frame image , The parameters correspond to the user-input environment description prompts. , The parameter corresponds to the entire root node of the anomaly tree, that is... The complete set; the function returns two results after execution: one is the environment node that best matches the current scene. (For example, "inside the store"); secondly, the environmental node. The set of direct child nodes ,express All associated key object nodes; (For example, it may include "customers", "cashiers", "people wearing hoodies", but excludes irrelevant objects such as "cars" and "pedestrians") will be used as the candidate set for the next stage;

[0058] Phase Two, Context-Aware Key Object Recognition: In a Defined Environment Under the constraints, the goal of this stage is to All video clips The system identifies key objects most relevant to potential anomalies; to balance efficiency and timeliness, the semantic retrieval function is invoked. video clips The first keyframe Processing:

[0059] ;

[0060] During this stage of the call, Parameters correspond to a single frame image , The parameters correspond to the candidate set of objects that are highly relevant to the environment, output from the previous stage. ; The parameters correspond to user-inputted prompts specifically designed for object recognition. The function returns two results after execution: one is in the environment node. The object node that best matches the current object under the constraints (For example, identifying a "person wearing a hoodie" in a "store interior" environment); secondly, the object node. The set of direct child nodes ,express All specific behavior nodes associated with it;

[0061] Phase Three, Behavioral Analysis and Final Prediction: Since behavior itself is a dynamic process with a temporal sequence, this phase analyzes all complete video segments. No longer call Instead of functions, it uses the Visual Language Model (VLM) for analysis;

[0062] First, construct the prompt words. :

[0063] Extract the set of outputs from the previous stage. All specific actions under Nodes, each specific behavior The child nodes of a node are a set of exception labels for abnormal events that may occur under this specific behavior. (e.g., specific behavior nodes) "Quickly approach the cashier and draw your weapon" corresponds to a set of possible abnormal event nodes. "Robbery", "violence"), forming " "Combination; all" "Combination to form prompt words" (For example, "If a person wearing a hoodie quickly approaches the cashier and pulls out a weapon, an unusual event such as a robbery or fight may occur.")

[0064] Then use the prompt words Guide the Visual Language Model (VLM) and generate prediction results:

[0065] ;

[0066] The final output consists of two items: The final confirmation description of the specific observed behavior (e.g., "In a store setting, a person wearing a hoodie is rapidly approaching the cashier and pulling out a weapon, which could be an unusual event such as robbery or violence."). The final predicted label for the most likely anomalous event (e.g., "[robbery, violence]").

[0067] Accordingly, the present invention also provides a surveillance video anomaly event prediction system based on precursor analysis, comprising:

[0068] The dataset construction unit is used to automatically extract data from the abnormal event dataset using a visual language model and build a structured abnormal feature text annotation dataset consisting of semantic tuples.

[0069] The anomaly tree construction unit integrates the semantic tuples in the anomaly feature text annotation dataset to establish a dynamic, four-layer hierarchical knowledge base composed of semantic nodes, denoted as the anomaly tree.

[0070] The retrieval and prediction unit is used to design a hierarchical semantic retrieval mechanism. This mechanism converts the input visual media into text descriptions and encodes them into vectors using a visual language model. Then, it calculates vector similarity to match the best semantic node among candidate nodes at a specific level in the anomaly tree and returns all its child nodes as a candidate set for the next stage of retrieval. The hierarchical semantic retrieval mechanism is used to retrieve environmental context nodes and key object nodes layer by layer from the anomaly tree to narrow down the candidate range. Then, based on the retrieved specific behavioral information, prompt words are constructed to enhance the visual language model's analysis of the surveillance video clips, and finally, anomaly event predictions are generated.

[0071] Implementing this invention has the following beneficial effects:

[0072] This invention achieves three core breakthroughs by constructing an anomaly tree knowledge base and a cascaded semantic reasoning framework:

[0073] I. Innovations in Computational Efficiency and Real-Time Performance

[0074] Traditional methods rely on pixel-level future frame generation, incurring huge computational overhead and only supporting offline analysis. This invention shifts to semantic-level reasoning, employing a "three-stage cascaded framework" for intelligent resource allocation: the first stage completes environmental context analysis through a single keyframe, utilizes caching strategies to avoid redundant computation, prunes irrelevant branches in the anomaly tree, and reduces the search space by several orders of magnitude; subsequent object recognition and behavior analysis are performed only on highly relevant subsets. This coarse-to-fine progressive reasoning dynamically correlates computational load with content complexity, rather than linearly with video duration, significantly reducing hardware dependence and enabling online real-time alerts on edge devices.

[0075] II. Deep Interpretability and Causal Reasoning

[0076] Traditional deep learning models are "black boxes," lacking logical chains. The "anomaly tree" of this invention encodes the causal path of anomalies using a four-layer structure: "environment → object → behavior → label." When the system issues an alert, the complete reasoning path can be traced, generating a white-box explanation report. This transparent reasoning greatly enhances credibility, enabling decision-makers to assess the reliability of predictions and take targeted preventative measures.

[0077] III. Excellent generalization and dynamic expansion capabilities

[0078] Existing technologies are limited by the "precursor-anomaly" pairs in the training data, making it difficult to handle novel anomalies. This invention separates the knowledge base from the inference process: when a new anomaly pattern is added, it only needs to be automatically summarized and encoded from the precursor video clips, incrementally integrated into the anomaly tree as nodes, without requiring model retraining. Inference is based on semantic similarity matching in a high-dimensional vector space, enabling the system to "learn by analogy." This architecture effectively handles new, unseen combinations of environment, object, and behavior, adapting to complex real-world scenarios. Attached Figure Description

[0079] Figure 1 This is a general schematic diagram of a method for predicting abnormal events in surveillance video based on precursor analysis, according to an embodiment of the present invention.

[0080] Figure 2 This is a structural diagram of a surveillance video anomaly event prediction system based on precursor analysis, according to an embodiment of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] This invention proposes a fundamental paradigm shift: from pixel-level visual prediction to semantic-level logical reasoning. Its core technological insight lies in the fact that anomalous events in surveillance scenarios do not occur randomly, but rather follow identifiable patterns determined by the interaction of environmental context, existing object entities, and observed behavior. By structuring and hierarchically modeling this knowledge and utilizing it for guided reasoning, more efficient, accurate, and fully interpretable anomaly prediction can be achieved without relying on video frame generation.

[0083] Figure 1 This is a general schematic diagram of a method for predicting abnormal events in surveillance video based on precursor analysis, according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0084] S1. The visual language model is used to automatically extract data from the abnormal event dataset and a structured abnormal feature text annotation dataset consisting of semantic tuples is established.

[0085] S2, integrate the semantic tuples in the abnormal feature text annotation dataset to establish a dynamic, four-layer hierarchical knowledge base composed of semantic nodes, denoted as the abnormal tree;

[0086] S3. Design a hierarchical semantic retrieval mechanism. The mechanism converts the input visual media into text descriptions and encodes them into vectors through a visual language model. Then, it calculates the vector similarity to match the best semantic node among the candidate nodes at a specific level of the anomaly tree and returns all its child nodes as a candidate set for the next stage of retrieval.

[0087] S4. Using the hierarchical semantic retrieval mechanism, environmental context nodes and key object nodes are retrieved layer by layer from the anomaly tree to narrow down the candidate range. Then, based on the retrieved specific behavioral information, prompt words are constructed to enhance the visual language model's analysis of the surveillance video, and finally, anomaly event prediction is generated.

[0088] Step S1 is as follows:

[0089] The abnormal event dataset includes publicly available abnormal event datasets on the network and abnormal event datasets from surveillance videos owned by the location where the abnormal event prediction is deployed.

[0090] S1-1, Filter and construct a data source focused on precursor pattern analysis from the aforementioned anomalous event dataset. The filtering focuses on retaining video sequences that exhibit a window of observable and analyzable precursory behavior before the anomalous event officially occurs. These precursory behaviors may include unusual loitering by people, suspicious placement of objects, or subtle changes in the environment. This ultimately forms the precursor video dataset. Each of its data units is a tuple. ,in This represents a complete video sequence containing three stages: normal, precursor, and abnormal. Yes The textual description of the abnormal events that occurred during the analysis; this step ensures the data quality and relevance of subsequent analyses and is the fundamental guarantee of the effectiveness of the entire method.

[0091] S1-2, the selected video data is transformed from its original pixel format into structured semantic knowledge: a powerful, pre-trained visual language model (VLM) is used, specifically the Qwen2.5-VL model in this embodiment, for each video sequence. Premonition fragments Deep analysis and automated annotation are performed, using prompt words to guide the VLM to extract information from four interrelated dimensions, and the results are organized into a structured semantic tuple; the automated annotation process is described by the following formula:

[0092] ;

[0093] Where VLM represents the visual language model function; and the text description of the abnormal event. As part of the prompt words, the VLM model is guided to generate classification labels; the output is a structured semantic tuple containing four-dimensional textual descriptions:

[0094] Context ( Environment Context: Describes the static scene and background information in which the video takes place; for example, "a dimly lit parking lot with multiple cars parked" or "the interior of a busy store with shelves and a cash register." This provides a basis for understanding the rationale behind subsequent actions.

[0095] Key objects ( Key Objects: Identify all dynamic or static entities in the scene that are highly correlated with potential anomalies and describe their attributes and relative positions; for example, “a man wearing a black hoodie” and “a dark-colored car parked near the exit”.

[0096] Specific behaviors ( Specific Behaviors: Detailed description of key objects The system tracks behavioral sequences and interaction patterns to capture subtle movements that deviate from the norm; for example, "the man paces back and forth between vehicles and frequently observes his surroundings" or "two customers are having a heated argument in the shelving area and are pushing each other."

[0097] Abnormal tags ( Anomaly Label: Based on a comprehensive understanding of the environment, objects, and behaviors, it performs fine-grained classification of potential abnormal events; for example, predicting them as "theft," "fighting," or "arson."

[0098] The precursor video dataset The structured semantic tuples formed by all video sequences constitute the anomaly feature text annotation dataset.

[0099] Step S2 is as follows:

[0100] Using a Large Language Model (LLM), specifically Qwen2.5 in this embodiment, the text in each structured semantic tuple is summarized, as follows:

[0101] ;

[0102] Here, LLM stands for Large Language Model Function, whose input is the raw, relatively long text description generated in S1, and whose output is a highly summarized core phrase, including a summary of the environmental context. Key Object Summary Summary of specific behaviors Summary of abnormal tags For example, the phrase "a male individual wearing a dark hoodie with his face partially obscured is loitering near the cashier" can be refined into "a person wearing a hoodie" or "suspiciously loitering." This process significantly improves the signal-to-noise ratio of semantic features, providing cleaner and more standardized input for subsequent embedding and clustering, and ensuring the quality and consistency of knowledge base nodes.

[0103] S2-2, employing a pre-trained text encoder, in this embodiment using a Sentence Transformer, to process the... Converting it to a high-dimensional vector is shown in the following equation:

[0104] ;

[0105] in It is the pre-trained text encoder. The input consists of four independent dimensions: "environmental context", "key objects", "specific behaviors", and "anomaly labels". Abstract, , It is the corresponding dimension A high-dimensional vector;

[0106] In the three independent dimensions of "environmental context", "key objects", and "specific behaviors" Internally, clustering is performed separately, and the clustering adopts an adaptive similarity threshold strategy: for a high-dimensional vector Calculate its relationship with the dimension it belongs to. Previously existing nodes high-dimensional vector Cosine similarity between them:

[0107] ;

[0108] when At that time, using high-dimensional vectors Create a new node; otherwise, put the vector... Merge into node In China; among them It is a dimension A preset similarity threshold; based on experimental verification, differentiating thresholds are set for different levels (e.g., , , This approach yields the best results because it accurately reflects the differences in semantic granularity across different levels: environmental descriptions are relatively broad, while behavioral descriptions are more specific and nuanced. The "abnormal label" dimension represents specific abnormal behaviors with small differences in semantic granularity, thus requiring no additional clustering.

[0109] After clustering, a set of environment context summary nodes is formed. Key object summary node set Specific behavior summary node set , collection of abnormal tag summary nodes ;

[0110] S2-3, the aforementioned , , , Connecting the anomalies according to the pre-defined causal logic and their corresponding relationships forms a complete anomaly tree; formally represented by the following formula:

[0111] ;

[0112] The arrow → represents a directed connection between nodes, indicating causal relationships and conditional dependencies; each complete path from the root node to a leaf node represents a causal chain; the tree structure essentially encodes a conditional probability model, whose joint probability distribution is expressed as follows:

[0113] ;

[0114] in These respectively represent "environmental context", "key object", "specific behavior", and "anomaly label". "Represents the known information in the probability formula" Reasoning under certain circumstances The probability of occurrence. This structure enables the system to perform hierarchical probabilistic inference during reasoning, greatly enhancing the robustness and accuracy of predictions.

[0115] Step S3 is as follows:

[0116] S3-1, the hierarchical semantic retrieval mechanism, aims to provide a unified interface for efficient and accurate semantic node matching and retrieval within the anomaly tree knowledge base built in S2, based on input visual media and specific analysis objectives. Its modular design allows for flexible invocation at different stages of the subsequent S4 inference framework. The general formal definition of this mechanism is as follows:

[0117] ;

[0118] in, Represents a semantic retrieval function; Represents the input visual media, suitable for static single-frame images. It is used to analyze static environments or objects, and is also suitable for dynamic video clips. It is used to analyze time-related behaviors; Textual cues representing the current analysis level are used to guide the Visual Language Model (VLM) to focus its attention on specific information dimensions. This represents the set of candidate nodes for the current retrieval operation, i.e., a subset of nodes at a specific level of the anomaly tree. When searching for the optimal node at the "environmental context" level, since there is no prior information obtained from the retrieval, For the The entire set; this makes the retrieval process limited and focused, rather than a global blind search.

[0119] It is the output tuple; Is Found in the set, related to input visual media exist The node that best matches the semantics under guidance; yes The set of all direct child nodes in the anomaly tree will be used as the basis for the next inference stage. ;

[0120] S3-2, the retrieval process of the hierarchical semantic retrieval mechanism is as follows:

[0121] Visual Language Model (VLM) Reception and Convert visual media into text descriptions , represented as ;

[0122] Using the pre-trained text encoder In this embodiment, SentenceTransformer is used to transform the text description. Mapping to the same high-dimensional vector space as the anomaly tree nodes yields the query vector. , represented as ;

[0123] By calculating the query vector and Each candidate node in the set Embedded vector The cosine similarity between nodes is used to find the semantically best-matching node. :

[0124] ;

[0125] The above formula finds the node that best matches the query intent by maximizing cosine similarity. This vector space-based matching is more robust and accurate than simple keyword matching.

[0126] In determining the node with the best semantic match Then, query the anomaly tree and obtain all its direct child nodes to form a set. .

[0127] Step S4 is as follows:

[0128] The video to be analyzed is ,right The video is split into multiple segments every 2 seconds. , , Total number of video clips; each video clip Four frames are sampled evenly, with each frame being a keyframe.

[0129] S4-1, Phase One, Environmental Context Analysis and Constraints: This phase aims to quickly identify the macroscopic environment in which the video is located, providing a strong prior constraint for subsequent analysis. Considering the fixed location of the surveillance camera, the semantic retrieval function is invoked. Treatment of video clip analysis The first keyframe Processing is performed to maximize computational efficiency:

[0130] ;

[0131] During this stage of the call, Parameters correspond to a single frame image , The parameters correspond to the user-input environment description prompts. , The parameter corresponds to the entire root node of the anomaly tree, that is... The complete set; the function returns two results after execution: one is the environment node that best matches the current scene. (For example, "inside the store"); secondly, the environmental node. The set of direct child nodes ,express All associated key object nodes; (For example, it may include "customers", "cashiers", and "people wearing hoodies", but exclude irrelevant objects such as "cars" and "pedestrians") will be used as the candidate set for the next stage, which greatly reduces the subsequent search space and thus significantly improves the accuracy and efficiency of subsequent steps.

[0132] S4-2, Phase Two, Context-Aware Key Object Recognition: In a defined environment Under the constraints, the goal of this stage is to All video clips The system identifies key objects most relevant to potential anomalies; to balance efficiency and timeliness, the semantic retrieval function is invoked. video clips The first keyframe Processing:

[0133] ;

[0134] During this stage of the call, Parameters correspond to a single frame image , The parameters correspond to the candidate set of objects that are highly relevant to the environment, output from the previous stage. ; The parameters correspond to user-inputted prompts specifically designed for object recognition. The function returns two results after execution: one is in the environment node. The object node that best matches the current object under the constraints (For example, identifying a "person wearing a hoodie" in a "store interior" environment); secondly, the object node. The set of direct child nodes ,express All the specific behavioral nodes associated with it provide a highly focused hypothesis space for the final behavioral analysis.

[0135] S4-3, Phase Three, Behavior Analysis and Final Prediction: Since behavior itself is a dynamic process with a temporal sequence, this phase analyzes all complete video segments. No longer call Instead of functions, it uses the Visual Language Model (VLM) for analysis;

[0136] First, construct the prompt words. :

[0137] Extract the set of outputs from the previous stage. All specific actions under Nodes, each specific behavior The child nodes of a node are a set of exception labels for abnormal events that may occur under this specific behavior. (e.g., specific behavior nodes) "Quickly approach the cashier and draw your weapon" corresponds to a set of possible abnormal event nodes. "Robbery", "violence"), forming " "Combination; all" "Combination to form prompt words" (For example, "If a person wearing a hoodie quickly approaches the cashier and pulls out a weapon, an unusual event such as a robbery or fight may occur.")

[0138] Then use the prompt words The visual language model (VLM) is guided and prediction results are generated. In this embodiment, the visual language model (VLM) adopts the Qwen2.5-VL model:

[0139] ;

[0140] The final output consists of two items: The final confirmation description of the specific observed behavior (e.g., "In a store setting, a person wearing a hoodie is rapidly approaching the cashier and pulling out a weapon, which could be an unusual event such as robbery or violence."). The system then assigns a final predicted label (e.g., "[robbery, violence]") to the most likely anomalous events. At this point, the entire hierarchical reasoning process is complete, and the system outputs an interpretable, early warning of anomalies.

[0141] Accordingly, the present invention also provides a system for predicting abnormal events in surveillance video based on precursor analysis, such as... Figure 2 As shown, it includes:

[0142] Dataset construction unit 1 is used to automatically extract data from the abnormal event dataset using a visual language model and to build a structured abnormal feature text annotation dataset consisting of semantic tuples.

[0143] Anomaly tree construction unit 2 integrates the semantic tuples in the anomaly feature text annotation dataset to establish a dynamic, four-layer hierarchical knowledge base composed of semantic nodes, denoted as an anomaly tree;

[0144] The retrieval prediction unit 3 is used to design a hierarchical semantic retrieval mechanism. This mechanism converts the input visual media into text descriptions and encodes them into vectors through a visual language model. Then, it calculates the vector similarity to match the best semantic node among the candidate nodes at a specific level of the anomaly tree and returns all its child nodes as a candidate set for the next stage of retrieval. The hierarchical semantic retrieval mechanism is used to retrieve environmental context nodes and key object nodes from the anomaly tree layer by layer to narrow down the candidate range. Then, based on the retrieved specific behavioral information, prompt words are constructed to enhance the visual language model's analysis of the surveillance video clips, and finally, anomaly event predictions are generated.

[0145] Therefore, this invention achieves three core breakthroughs by constructing an anomaly tree knowledge base and a cascaded semantic reasoning framework:

[0146] I. Innovations in Computational Efficiency and Real-Time Performance

[0147] Traditional methods rely on pixel-level future frame generation, incurring huge computational overhead and only supporting offline analysis. This invention shifts to semantic-level reasoning, employing a "three-stage cascaded framework" for intelligent resource allocation: the first stage completes environmental context analysis through a single keyframe, utilizes caching strategies to avoid redundant computation, prunes irrelevant branches in the anomaly tree, and reduces the search space by several orders of magnitude; subsequent object recognition and behavior analysis are performed only on highly relevant subsets. This coarse-to-fine progressive reasoning dynamically correlates computational load with content complexity, rather than linearly with video duration, significantly reducing hardware dependence and enabling online real-time alerts on edge devices.

[0148] II. Deep Interpretability and Causal Reasoning

[0149] Traditional deep learning models are "black boxes," lacking logical chains. The "anomaly tree" of this invention encodes the causal path of anomalies using a four-layer structure: "environment → object → behavior → label." When the system issues an alert, the complete reasoning path can be traced, generating a white-box explanation report. This transparent reasoning greatly enhances credibility, enabling decision-makers to assess the reliability of predictions and take targeted preventative measures.

[0150] III. Excellent generalization and dynamic expansion capabilities

[0151] Existing technologies are limited by the "precursor-anomaly" pairs in the training data, making it difficult to handle novel anomalies. This invention separates the knowledge base from the inference process: when a new anomaly pattern is added, it only needs to be automatically summarized and encoded from the precursor video clips, incrementally integrated into the anomaly tree as nodes, without requiring model retraining. Inference is based on semantic similarity matching in a high-dimensional vector space, enabling the system to "learn by analogy." This architecture effectively handles new, unseen combinations of environment, object, and behavior, adapting to complex real-world scenarios.

[0152] The above provides a detailed description of a method and system for predicting abnormal events in surveillance video based on precursor analysis, as provided in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for predicting abnormal events in surveillance video based on precursor analysis, characterized in that, The method includes: We use a visual language model to automatically extract data from anomaly event datasets and build a structured anomaly feature text annotation dataset consisting of semantic tuples. The semantic tuples in the abnormal feature text annotation dataset are integrated to establish a dynamic, four-layer hierarchical knowledge base composed of semantic nodes, denoted as the anomaly tree; Design a hierarchical semantic retrieval mechanism. The mechanism transforms the input visual media into a text description and encodes it into a vector through a visual language model. Then, it calculates the vector similarity to match the best semantic node among the candidate nodes at a specific level of the anomaly tree and returns all its child nodes as a candidate set for the next stage of retrieval. The hierarchical semantic retrieval mechanism is used to retrieve environmental context nodes and key object nodes from the anomaly tree layer by layer to narrow down the candidate range. Then, based on the retrieved specific behavioral information, prompt words are constructed to enhance the visual language model's analysis of the surveillance video, and finally, anomaly event prediction is generated.

2. The method for predicting abnormal events in surveillance video based on precursor analysis as described in claim 1, characterized in that, The process involves automatically extracting data from the abnormal event dataset using a visual language model and establishing a structured abnormal feature text annotation dataset composed of semantic tuples. Specifically: The abnormal event dataset includes publicly available abnormal event datasets on the network and abnormal event datasets from surveillance videos owned by the location where the abnormal event prediction is deployed. Video sequences that show a premonitory behavioral window that can be observed and analyzed before the actual occurrence of the abnormal event are selected from the abnormal event dataset to form a premonitory video dataset. Each of its data units is a tuple. ,in This represents a complete video sequence containing three stages: normal, precursor, and abnormal. Yes A textual description of the abnormal events that occurred; Visual Language Model (VLM) is used for each video sequence Premonition fragments The analysis and automated annotation process involves using prompt words to guide the VLM to extract information from four interrelated dimensions and organizing the results into a structured semantic tuple. The automated annotation process is described by the following formula: ； Where VLM represents the visual language model function; and the text description of the abnormal event. As part of the prompt words, it guides the VLM model to generate classification labels; The output is a structured semantic tuple containing four-dimensional textual descriptions: Context : Describes the static scene and background information in which the video takes place; Key objects Identify all dynamic or static entities in a scene that are highly correlated with potential anomalies, and describe their attributes and relative positions; Specific actions : Describe the key objects in detail Behavioral sequences and interaction patterns, capturing those subtle movements that deviate from the norm; Abnormal tags Based on a comprehensive understanding of the environment, objects, and behaviors, fine-grained classification of potential abnormal events is performed. The precursor video dataset The structured semantic tuples formed by all video sequences constitute the anomaly feature text annotation dataset.

3. The method for predicting abnormal events in surveillance video based on precursor analysis as described in claim 2, characterized in that, The semantic tuples in the abnormal feature text annotation dataset are integrated to establish a dynamic, four-layer hierarchical knowledge base composed of semantic nodes, denoted as an anomaly tree, specifically as follows: The text in each structured semantic tuple is summarized using a Large Language Model (LLM), as follows: ； Here, LLM stands for Large Language Model Function, whose output is a highly summarized core phrase, including a summary of the environmental context. Key Object Summary Summary of specific behaviors Summary of abnormal tags ; A pre-trained text encoder is used to process the text. Converting it to a high-dimensional vector is shown in the following equation: ； in It is the pre-trained text encoder. The input consists of four independent dimensions: "environmental context", "key objects", "specific behaviors", and "anomaly labels". Abstract, , It is the corresponding dimension A high-dimensional vector; In the three independent dimensions of "environmental context", "key objects", and "specific behaviors" Internally, clustering is performed separately, and the clustering adopts an adaptive similarity threshold strategy: for a high-dimensional vector Calculate its relationship with the dimension it belongs to. Previously existing nodes high-dimensional vector Cosine similarity between them: ； when At that time, using high-dimensional vectors Create a new node; otherwise, put the vector... Merge into node In China; among them It is a dimension Preset similarity threshold; no clustering is required for the "abnormal label" dimension; After clustering, a set of environment context summary nodes is formed. Key object summary node set Specific behavior summary node set , collection of abnormal tag summary nodes ; The , , , Connecting the anomalies according to causal logic and their corresponding relationships forms a complete anomaly tree; formally represented by the following formula: ； The arrow → represents a directed connection between nodes, indicating causal relationships and conditional dependencies; each complete path from the root node to a leaf node represents a causal chain.

4. The method for predicting abnormal events in surveillance video based on precursor analysis as described in claim 3, characterized in that, The proposed hierarchical semantic retrieval mechanism transforms the input visual media into a text description and encodes it as a vector using a visual language model. Then, it calculates vector similarity to match the best semantic node among candidate nodes at a specific level of the anomaly tree, and returns all its child nodes as a candidate set for the next stage of retrieval. Specifically: The general formal definition of a hierarchical semantic retrieval mechanism is as follows: ； in, Represents a semantic retrieval function; Represents the input visual media, suitable for static single-frame images. It is also suitable for dynamic video clips. ; A text prompt representing the current analysis level; This represents the set of candidate nodes for the current retrieval operation, i.e., a subset of nodes at a specific level of the anomaly tree, when searching for the optimal node at the "environment context" level. For the The complete collection; It is the output tuple; Is Found in the set, related to input visual media exist The node that best matches the semantics under guidance; yes The set of all direct child nodes in the anomaly tree will be used as the basis for the next inference stage. ; The retrieval process of the hierarchical semantic retrieval mechanism is as follows: Visual Language Model (VLM) Reception and Convert visual media into text descriptions , represented as ; Using the pre-trained text encoder The text description Mapping to the same high-dimensional vector space as the anomaly tree nodes yields the query vector. , represented as ; By calculating the query vector and Each candidate node in the set Embedded vector The cosine similarity between nodes is used to find the semantically best-matching node. : ； The above formula finds the node that best matches the query intent by maximizing cosine similarity. ; In determining the node with the best semantic match Then, query the anomaly tree and obtain all its direct child nodes to form a set. .

5. The method for predicting abnormal events in surveillance video based on precursor analysis as described in claim 4, characterized in that, The hierarchical semantic retrieval mechanism is used to retrieve environmental context nodes and key object nodes layer by layer from the anomaly tree to narrow down the candidate range. Then, based on the retrieved specific behavioral information, prompt words are constructed to enhance the visual language model's analysis of surveillance video clips, ultimately generating anomaly event predictions. Specifically: The video to be analyzed is ,right The video is split into multiple segments every few seconds. , , Total number of video clips; each video clip Several keyframes are sampled evenly; Phase 1, Contextual Analysis and Constraints: Invoking the semantic retrieval function Treatment of video clip analysis The first keyframe Processing: ； During this stage of the call, Parameters correspond to a single frame image , The parameters correspond to the user-input environment description prompts. , The parameter corresponds to the entire root node of the anomaly tree, that is... The complete set; the function returns two results after execution: one is the environment node that best matches the current scene. Secondly, the environmental node The set of direct child nodes ,express All associated key object nodes; This will serve as the candidate set for the next stage; Phase Two, Context-Aware Key Object Recognition: In a Defined Environment Under the constraints, the goal of this stage is to All video clips Identify the key objects most relevant to potential anomalies; Call the semantic retrieval function video clips The first keyframe Processing: ； During this stage of the call, Parameters correspond to a single frame image , The parameters correspond to the candidate set of objects that are highly relevant to the environment, output from the previous stage. ; The parameters correspond to user-inputted prompts specifically designed for object recognition. The function returns two results after execution: one is in the environment node. The object node that best matches the current object under the constraints Secondly, the object node The set of direct child nodes ,express All specific behavior nodes associated with it; Phase Three, Behavioral Analysis and Final Prediction: This phase analyzes all complete video clips. No longer call Instead of functions, it uses the Visual Language Model (VLM) for analysis; First, construct the prompt words. : Extract the set of outputs from the previous stage. All specific actions under Nodes, each specific behavior The child nodes of a node are a set of exception labels for abnormal events that may occur under this specific behavior. ,form" "Combination; all" "Combination to form prompt words" ; Then use the prompt words Guide the Visual Language Model (VLM) and generate prediction results: ； The final output consists of two items: A final confirmation description of the specific observed behavior; The final predicted label for the most likely abnormal event.

6. A system for predicting abnormal events in surveillance video based on precursor analysis, characterized in that, The system includes: The dataset construction unit is used to automatically extract data from the abnormal event dataset using a visual language model and build a structured abnormal feature text annotation dataset consisting of semantic tuples. The anomaly tree construction unit integrates the semantic tuples in the anomaly feature text annotation dataset to establish a dynamic, four-layer hierarchical knowledge base composed of semantic nodes, denoted as the anomaly tree. The retrieval and prediction unit is used to design a hierarchical semantic retrieval mechanism. This mechanism converts the input visual media into text descriptions and encodes them into vectors using a visual language model. Then, it calculates vector similarity to match the best semantic node among candidate nodes at a specific level in the anomaly tree and returns all its child nodes as a candidate set for the next stage of retrieval. The hierarchical semantic retrieval mechanism is used to retrieve environmental context nodes and key object nodes layer by layer from the anomaly tree to narrow down the candidate range. Then, based on the retrieved specific behavioral information, prompt words are constructed to enhance the visual language model's analysis of the surveillance video clips, and finally, anomaly event predictions are generated.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images