A video anomaly detection and semantic understanding method oriented to logical reasoning
By generating structured event descriptions and constructing event logic chains through a multimodal large model, and combining it with a multi-agent debate mechanism, the problem of insufficient logical reasoning in existing technologies is solved, and efficient and interpretable video anomaly detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGZHOU RES INST OF XIAN UNIV OF ELECTRONIC SCI & TECH
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-05
AI Technical Summary
Existing video anomaly detection technologies lack logical reasoning capabilities in complex open-world scenarios, making it difficult to understand the causal logic and contextual dependencies behind events. This results in a high false alarm rate, a lack of interpretability, and difficulty in identifying latent anomalies.
Structured atomic event descriptions are generated through a multimodal large model, an event logic chain is constructed and logical consistency constraints are imposed, a multi-agent debate mechanism is introduced for reasoning and judgment, and anomaly scoring is performed by combining video observation logic and pre-defined normative logic.
It improves the detectability of anomaly detection, reduces the false alarm rate, provides interpretable judgment criteria, and reduces resource consumption and deployment and maintenance costs.
Smart Images
Figure CN122157151A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of video anomaly detection technology, specifically relating to a video anomaly detection and semantic understanding method oriented towards logical reasoning. Background Technology
[0002] Video anomaly detection (VAD) is a core task in the field of intelligent surveillance, aiming to automatically identify events in videos that do not conform to expected patterns or social norms. With the development of deep learning technology, existing mainstream technologies can be mainly divided into the following two categories:
[0003] Generative methods based on reconstruction or prediction: These methods assume that anomalous samples cannot be well reconstructed or predicted by a model trained only on normal samples. Commonly used models include autoencoders (AE), generative adversarial networks (GANs), and U-Net. During testing, anomalies are determined by calculating pixel-level reconstruction error or prediction error for future frames.
[0004] Discriminative methods based on weakly supervised learning: These methods utilize the Multiple Instant Learning (MIL) framework to classify videos into positive packets (abnormal videos) and negative packets (normal videos). By extracting spatiotemporal features of the video (such as I3D, C3D, and VideoMAE features), a binary classifier or regression network is trained to classify segments whose feature magnitudes significantly deviate from the normal distribution as anomalous.
[0005] Despite the progress made by existing technologies on specific benchmark datasets, there are still insurmountable limitations when facing complex real-world open-world scenarios.
[0006] The lack of logical inference ability: Existing models are essentially based on "visual similarity" or "statistical bias." They can easily identify "vigorous movement" (such as fighting or explosions), but cannot understand the causal logic and contextual dependencies behind the event. For example, "running in a corridor" could be due to "fire escape" (normal) or "chasing and playing" (abnormal). Existing technologies cannot combine contextual information such as "whether there is smoke" or "whether there is an alarm" to make logical inferences, resulting in a high false alarm rate.
[0007] Shallow Semantic Understanding: Existing methods mostly remain at the physical signal level (such as optical flow changes and pixel gradients), making it difficult to rise to the semantic intent level. For "latent anomalies" with small movements but serious nature (such as tailgating, thieves stealing with tweezers, and security personnel leaving their posts), existing feature amplitude-based methods are prone to missing detection because their visual characteristics are very similar to normal behavior.
[0008] Lack of Interpretability: Deep learning models are often "black boxes," outputting only an anomaly probability value (0-1). When the system alarms, security personnel cannot know the basis for the system's judgment. Furthermore, they can fabricate violations that do not appear in the video and are unable to correct themselves. Summary of the Invention
[0009] The purpose of this invention is to provide a video anomaly detection and semantic understanding method oriented towards logical reasoning.
[0010] The technical solution for achieving the objective of this invention is a video anomaly detection and semantic understanding method oriented towards logical reasoning, the method comprising the following steps:
[0011] Step A: Generate structured atomic event descriptions based on multimodal large models, transforming continuous pixel streams into discrete, standardized semantic units;
[0012] Step B, event logic chain modeling and semantic hierarchical analysis, transforms discrete atomic events into dynamic graphs with temporal and causal relationships, enabling the understanding from isolated events to event chains;
[0013] Step C, Logical Consistency Constraints and Scoring Mechanism, quantifies the degree of anomaly by calculating the semantic distance between the "video observation logic" and the "pre-defined normative logic";
[0014] Step D: Multi-agent debate, which involves adversarial reasoning and debate among multiple roles to generate a judgment with a complete reasoning path.
[0015] A further preferred embodiment is that step A includes the following steps;
[0016] Step A1: Video temporal segmentation and keyframe extraction. The input surveillance video stream V is segmented into a semantically coherent sequence of short segments using an adaptive shot boundary detection algorithm based on color histogram difference. ; and for each segment Extract the intermediate frames or frames with the largest changes in optical flow as keyframes. ;
[0017] Step A2: Structured prompts, set dedicated prompt words, and extract video clips. For each video clip, output standard JSON format data that must include core fields;
[0018] Step A3: Auxiliary attribute extraction. While extracting video clips, guide the model to output meta-attribute labels for the actions.
[0019] A further preferred embodiment is that, in step A2, the core fields include subject, action, object, environment, and time sequence;
[0020] In step A3, the output action includes the range of motion, concealment, and multi-person interaction status.
[0021] A further preferred embodiment is that step B includes the following steps:
[0022] Step B1: Constructing the temporal cause-effect graph and defining the directed graph. , where the set of nodes Composed of atomic events with structured hints;
[0023] Temporal edges are constructed for two event nodes with adjacent timestamps. and Directly establish directed edges This represents the natural evolutionary sequence of events along the timeline;
[0024] Causal edge construction leverages the logical reasoning capabilities of large language models to analyze whether there is an inherent causal dependency between adjacent or similar events and establishes causal edges.
[0025] Step B2, Multidimensional Semantic Hierarchy Mapping: The nodes in the graph are semantically abstracted from bottom to top and divided into three levels to facilitate subsequent rule matching.
[0026] A further preferred embodiment is that in step B2, the three layers include a physical layer, an interaction layer, and an intent layer;
[0027] The physical layer is used to describe basic limb movements;
[0028] The interaction layer describes the interaction between the subject and the environment or objects.
[0029] The intent layer infers underlying motivations based on context, causal chains, and environmental attributes.
[0030] A further preferred embodiment is that step C includes the following steps:
[0031] Step C1: Construct a standardized knowledge base, build a hierarchical text rule base, and use a model to convert all rule texts into high-dimensional vectors and store them in a vector database;
[0032] Step C2: Logical consistency check algorithm;
[0033] Logical chain vectorization encodes the event logical chain description text generated from the video into query vectors. ;
[0034] Rule retrieval retrieves the Top-K rule vectors in the canonical knowledge base that are most semantically relevant to the current behavior. ;
[0035] Natural Language Inference (NLI) uses the "logical chain description" as a premise and the "rule text" as a hypothesis to determine the relationship between the two.
[0036] A further preferred approach is that its NLI model determines the relationship between the two as including: contradiction, implication, and neutrality;
[0037] The contradiction is that the video behavior directly violates the rules and is assigned a high abnormal score;
[0038] Implied meaning is that the video behavior conforms to the rules and is assigned a low anomaly score;
[0039] Neutrality means that the video behavior is unrelated to the rules.
[0040] A further preferred embodiment is that step D includes the following steps:
[0041] Step D1: Define the roles of the agents, including the agent on the side of justice, the agent on the side of justice, and the referee agent;
[0042] A positive-squared intelligent agent is used to prove that the behavior in the video is normal, based on facts and to find a logical and reasonable explanation.
[0043] The opposing intelligent agent is used to prove that the behavior in the video is abnormal, and to find evidence of violations based on facts and laws and regulations;
[0044] The referee agent is used to remain neutral, not generate new viewpoints, and to review the validity of the arguments presented by both sides, while making a judgment.
[0045] Step D2: Adversarial debate, where both agents initially state their viewpoints based on the structured event logic chain input.
[0046] In multiple rounds of debate, both sides' intelligent agents must cite specific atomic event node IDs in the video logic chain or specific clauses in the specification knowledge base. If one side is pointed out to have made a factual error by the other side, it must revise its views in the next round of debate.
[0047] After the set number of debate rounds, the referee agent summarizes all the verified valid arguments and generates a natural language judgment statement containing the complete reasoning path, and finally weights the abnormal confidence scores.
[0048] Compared with the prior art, the present invention has the following positive effects: The present invention structures the video content into an event sequence and establishes the logical relationship between events, so that the judgment no longer depends solely on the visual differences of a single segment, but can be judged by combining behavioral combinations and contextual conditions. It can form combined evidence based on event elements such as card swiping status, distance relationship between people, and passage time interval, thereby improving detectability.
[0049] When outputting abnormal scores, the corresponding event and rule basis are given, explaining which key event fragments triggered the conclusion and which rules are inconsistent with it, which facilitates security personnel to review and handle the matter and reduces the review and communication costs.
[0050] The system employs multi-role reasoning and evidence citation constraints, requiring different conclusions to be supported by structured events and retrieved rules. Inferences lacking factual support or inconsistent with event records can be pointed out and corrected during the interaction process, thereby reducing false alarms and avoiding resource consumption caused by invalid alarms.
[0051] By decoupling anomaly detection from the rule base, adaptation can be achieved by updating the text rules in the rule base when scenarios change or management specifications are adjusted, without having to recollect a large number of samples and retrain the model, thereby reducing deployment and maintenance costs. Attached Figure Description
[0052] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:
[0053] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation
[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0055] Example
[0056] See Figure 1 As shown, a video anomaly detection and semantic understanding method oriented towards logical reasoning includes the following steps:
[0057] Step A: Generate structured atomic event descriptions based on a multimodal large model, transforming a continuous pixel stream into discrete, standardized semantic units; Step A includes the following steps;
[0058] Step A1: Video temporal segmentation and keyframe extraction. The input surveillance video stream V is segmented into a semantically coherent sequence of short segments using an adaptive shot boundary detection algorithm based on color histogram difference. In practical applications, if there are no obvious camera transitions, a sliding window mechanism (e.g., a window size of 16 frames and a step size of 8 frames) is used for segmentation; and each segment is then segmented... Extract the intermediate frames or frames with the largest changes in optical flow as keyframes. Used for subsequent image understanding.
[0059] Step A2: Structured prompts. Set dedicated prompt words with strict constraints, requiring multimodal large models (such as Qwen3-VL) to abandon subjective modifications and focus only on objective facts;
[0060] It extracts video segments and outputs standard JSON format data for each video segment, which must include the core fields. The standard requirement for the extraction model is that the output of standard JSON format data for each video segment must include the following five core fields (i.e., "atomic event quintuple").
[0061] Furthermore, in step A2, its core fields include subject, action, object, environment, and timing;
[0062] Subject: The person or object that performs the action (e.g., "security guard in blue uniform", "black SUV").
[0063] Action: A specific physical description (e.g., "forcibly push away", "jump over", "squat down to pick up").
[0064] Object: The direct recipient or object of an action (such as a "turnstile", "fence", or "wallet on the ground").
[0065] Environment: Key spatiotemporal attributes of the scene (such as "dimly lit corridor" or "the interior of a bank counter").
[0066] Time: The relative timestamp or duration of an event.
[0067] Step A3: Auxiliary attribute extraction. While extracting video clips, guide the model to output meta-attribute labels for the actions.
[0068] In step A3, the output action includes the range of motion, concealment, and multi-person interaction status.
[0069] Among these, attributes such as amplitude of movement (High / Low), concealment (True / False), and multi-person interaction status (Yes / No) serve as auxiliary features to enhance the perception of subtle anomalies.
[0070] Step B, event logic chain modeling and semantic hierarchical analysis, transforms discrete atomic events into dynamic graphs with temporal and causal relationships, enabling the understanding from isolated events to event chains; Step B includes the following steps:
[0071] Step B1: Constructing the temporal cause-effect graph and defining the directed graph. , where the set of nodes Composed of atomic events with structured hints;
[0072] Temporal edges are constructed for two event nodes with adjacent timestamps. and Directly establish directed edges This represents the natural evolutionary sequence of events along the timeline;
[0073] Causal edge construction leverages the logical reasoning capabilities of large language models to analyze whether there is an inherent causal dependency between adjacent or similar events and establishes causal edges; for example, if =“Man throws bricks” =“The glass door breaks”, although there may be a slight time interval between the two, there is a strong causal relationship, therefore a causal edge is established. .
[0074] Step B2, multi-dimensional semantic hierarchy mapping, involves bottom-up semantic abstraction of the nodes in the graph and dividing them into three levels to facilitate subsequent rule matching. In step B2, the three levels include a physical layer, an interaction layer, and an intent layer.
[0075] The physical layer describes basic limb movements; it only describes basic limb movements (such as "arms raised high" or "moving quickly"), corresponding to the lowest level of visual perception, and does not contain value judgments.
[0076] The interaction layer is used to describe the interaction between the subject and the environment or objects; describing the interaction between the subject and the environment or objects (such as "picking locks", "passing items", "following others") introduces object constraints and begins to have social significance.
[0077] The intent layer infers underlying motivations based on context, causal chains, and environmental attributes. These underlying motivations (such as "theft," "vandalism," "illegal intrusion," and "emergency rescue") are crucial for identifying anomalies.
[0078] Step C, Logical Consistency Constraints and Scoring Mechanism, quantifies the degree of anomaly by calculating the semantic distance between the "video observation logic" and the "pre-defined normative logic." Step C includes the following steps:
[0079] Step C1: Construct a standardized knowledge base. Build a hierarchical text rule base and use a model to convert all rule texts into high-dimensional vectors and store them in a vector database; the model is the Sentence-BERT model.
[0080] Its normative knowledge base includes general social norms, such as "prohibition of violent damage to public and private property" and "prohibition of attacking others," as well as scenario-specific rules, such as "non-staff members are prohibited from entering the warehouse" and "cards must be swiped before entering the turnstile."
[0081] Step C2: Logical consistency check algorithm;
[0082] Logical chain vectorization encodes the event logical chain description text generated from the video (e.g., "Man did not swipe card -> Close to the person in front -> Quickly pass through the gate") into a query vector. ;
[0083] Rule retrieval retrieves the Top-K rule vectors in the canonical knowledge base that are most semantically relevant to the current behavior. ;
[0084] Natural Language Inference (NLI) uses a logical chain description as a premise and a rule text as a hypothesis, employing an NLI model to determine the relationship between the two. The NLI model determines the relationship as follows: contradiction, implication, and neutrality.
[0085] The contradiction is that the video behavior directly violates the rules (such as the premise of "entry without swiping a card", assuming that "entry requires swiping a card"), and is assigned a higher abnormal score;
[0086] Implied meaning is that the video behavior conforms to the rules (such as the premise of "entering after swiping the card") and is assigned a lower abnormal score;
[0087] Neutrality means that the video behavior is unrelated to the rules.
[0088] Step D: Multi-agent debate. This step generates a complete reasoning path judgment through adversarial reasoning and debate among multiple agents. This step incorporates game theory concepts to address the "illusion" and "misjudgment" problems that are prone to occur with single-agent models, ensuring the robustness of the detection results.
[0089] Step D includes the following steps:
[0090] Step D1: Define the agent persona, including the Defender, Prosecutor, and Judge.
[0091] The Cube intelligent agent is used to prove that the behavior in the video is normal. The CoT (CoT) is based on facts and seeks a logical and reasonable explanation (e.g., maintenance, misoperation, emergency avoidance, moving of items, etc.).
[0092] The opposing agent is used to prove that the behavior in the video is abnormal. The CoT (CoT) searches for evidence of violations based on facts and laws and regulations (e.g., aggressive actions, unauthorized entry, damage to property, etc.).
[0093] The referee agent is used to remain neutral, not generate new viewpoints, and to review the validity of the arguments presented by both sides, while making a judgment.
[0094] Step D2, Adversarial Debate, Opening Statement: Both agents state their initial viewpoints based on the structured event logic chain of the input.
[0095] In a multi-round debate (Rebuttal Round), both agents must cite the specific atomic event node ID in the video logic chain (e.g., "based on event..."). Action characteristics...) or specific clauses in the normative knowledge base, when one party is pointed out to have made a factual error by the other party, it must revise its viewpoint in the next round of debate; Self-Reflection: If one party is pointed out to have made a factual error by the other party, that agent must revise its viewpoint in the next round. This mechanism can effectively eliminate the fabrication phenomenon when a single model "interprets a picture".
[0096] After the set number of debate rounds (such as 3 or 5 rounds), the referee agent summarizes all the verified valid arguments, generates a natural language judgment statement containing the complete reasoning path, and finally weights the abnormal confidence scores.
[0097] Compared with the prior art, the present invention has the following positive effects: The present invention structures the video content into an event sequence and establishes the logical relationship between events, so that the judgment no longer depends solely on the visual differences of a single segment, but can be judged by combining behavioral combinations and contextual conditions. It can form combined evidence based on event elements such as card swiping status, distance relationship between people, and passage time interval, thereby improving detectability.
[0098] When outputting abnormal scores, the corresponding event and rule basis are given, explaining which key event fragments triggered the conclusion and which rules are inconsistent with it, which facilitates security personnel to review and handle the matter and reduces the review and communication costs.
[0099] The system employs multi-role reasoning and evidence citation constraints, requiring different conclusions to be supported by structured events and retrieved rules. Inferences lacking factual support or inconsistent with event records can be pointed out and corrected during the interaction process, thereby reducing false alarms and avoiding resource consumption caused by invalid alarms.
[0100] By decoupling anomaly detection from the rule base, adaptation can be achieved by updating the text rules in the rule base when scenarios change or management specifications are adjusted, without having to recollect a large number of samples and retrain the model, thereby reducing deployment and maintenance costs.
[0101] The standard parts used in this embodiment can be purchased directly from the market, and the non-standard structural parts described in the instruction manual can also be processed without any doubt based on existing technical common sense. At the same time, the connection methods of each component adopt mature conventional methods in the existing technology, and the machinery, parts and equipment all adopt conventional models in the existing technology, so they will not be described in detail here.
[0102] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, these obvious variations or modifications derived from the essential spirit of the present invention still fall within the scope of protection of the present invention.
Claims
1. A video anomaly detection and semantic understanding method oriented towards logical reasoning, characterized in that: The method includes the following steps: Step A: Generate structured atomic event descriptions based on multimodal large models, transforming continuous pixel streams into discrete, standardized semantic units; Step B, event logic chain modeling and semantic hierarchical analysis, transforms discrete atomic events into dynamic graphs with temporal and causal relationships, enabling the understanding from isolated events to event chains; Step C, Logical Consistency Constraints and Scoring Mechanism, quantifies the degree of anomaly by calculating the semantic distance between "video observation logic" and "pre-defined normative logic"; Step D: Multi-agent debate, which involves adversarial reasoning and debate among multiple roles to generate a judgment with a complete reasoning path.
2. The video anomaly detection and semantic understanding method for logical reasoning according to claim 1, characterized in that: In step A, Includes the following steps; Step A1: Video temporal segmentation and keyframe extraction. The input surveillance video stream V is segmented into a semantically coherent sequence of short segments using an adaptive shot boundary detection algorithm based on color histogram difference. For each segment, the middle frame or the frame with the largest change in optical flow is extracted as the keyframe. ; Step A2: Structured prompts, set dedicated prompt words, and extract video clips. For each video clip, output standard JSON format data that must include core fields; Step A3: Auxiliary attribute extraction. While extracting video clips, guide the model to output meta-attribute labels for the actions.
3. The video anomaly detection and semantic understanding method for logical reasoning according to claim 2, characterized in that: In step A2, the core fields include subject, action, object, environment, and time sequence; In step A3, the output action includes the range of motion, concealment, and multi-person interaction status.
4. The video anomaly detection and semantic understanding method for logical reasoning according to claim 3, characterized in that: Step B includes the following steps: Step B1: Constructing the temporal cause-effect graph and defining the directed graph. , where the set of nodes Composed of atomic events with structured hints; Temporal edges are constructed for two event nodes with adjacent timestamps. and Directly establish directed edges This represents the natural evolutionary sequence of events along the timeline; Causal edge construction leverages the logical reasoning capabilities of large language models to analyze whether there is an inherent causal dependency between adjacent or similar events and establishes causal edges. Step B2, Multidimensional Semantic Hierarchy Mapping: The nodes in the graph are semantically abstracted from bottom to top and divided into three levels to facilitate subsequent rule matching.
5. The video anomaly detection and semantic understanding method for logical reasoning according to claim 4, characterized in that: In step B2, the three layers include the physical layer, the interaction layer, and the intent layer; The physical layer is used to describe basic limb movements; The interaction layer describes the interaction between the subject and the environment or objects. The intent layer infers underlying motivations based on context, causal chains, and environmental attributes.
6. The video anomaly detection and semantic understanding method for logical reasoning according to claim 5, characterized in that: Step C includes the following steps: Step C1: Construct a standardized knowledge base, build a hierarchical text rule base, and use a model to convert all rule texts into high-dimensional vectors and store them in a vector database; Step C2: Logical consistency check algorithm; Logical chain vectorization encodes the event logical chain description text generated from the video into query vectors. ; Rule retrieval retrieves the Top-K rule vectors in the canonical knowledge base that are most semantically relevant to the current behavior. ; Natural Language Inference (NLI) uses the "logical chain description" as a premise and the "rule text" as a hypothesis to determine the relationship between the two.
7. The video anomaly detection and semantic understanding method for logical reasoning according to claim 6, characterized in that: Its NLI model determines the relationship between the two as including: contradiction, implication, and neutrality; The contradiction is that the video behavior directly violates the rules and is assigned a high abnormal score; Implied meaning is that the video behavior conforms to the rules and is assigned a low anomaly score; Neutrality means that the video behavior is unrelated to the rules.
8. The video anomaly detection and semantic understanding method for logical reasoning according to claim 7, characterized in that: Step D includes the following steps: Step D1: Define the roles of the agents, including the agent on the side of justice, the agent on the side of justice, and the referee agent; A positive-squared intelligent agent is used to prove that the behavior in the video is normal, based on facts and to find a logical and reasonable explanation. The opposing intelligent agent is used to prove that the behavior in the video is abnormal, and to find evidence of violations based on facts and laws and regulations; The referee agent is used to remain neutral, not generate new viewpoints, and to review the validity of the arguments presented by both sides, while making a judgment. Step D2: Adversarial debate, where both agents initially state their viewpoints based on the structured event logic chain input. In multiple rounds of debate, both sides' intelligent agents must cite specific atomic event node IDs in the video logic chain or specific clauses in the specification knowledge base. If one side is pointed out to have made a factual error by the other side, it must revise its views in the next round of debate. After the set number of debate rounds, the referee agent summarizes all the verified valid arguments and generates a natural language judgment statement containing the complete reasoning path, and finally weights the abnormal confidence scores.