Zero-shot long video understanding method based on structured event and event relationship reasoning

Abstract

The application discloses a zero-shot long video understanding method based on structured events and event relationship reasoning. The method comprises the following steps: performing hierarchical and time sequence division on an input video to obtain coarse-grained nodes and fine-grained nodes, and then using a visual language model to obtain corresponding coarse-grained semantic descriptions and fine-grained semantic descriptions; screening and merging suspected abnormal intervals to obtain a set of merged abnormal intervals; merging, deduplicating and evidence screening the fine-grained semantic descriptions corresponding to each abnormal interval to obtain a compressed abnormal evidence set, and then using a large language model to generate structured abnormal events; modeling the relationships of the abnormal events to form causal clues and abnormal evolution chains of the events; constructing a global semantic memory based on the coarse-grained semantic descriptions; and generating a video-level abnormal understanding result by fusing local events, event relationships and global semantics. The application improves the coherence, explainability and zero-shot generalization capability of the long video abnormal understanding result.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a zero-sample long video understanding method based on structured events and event relationship reasoning. Background Technology

[0002] With the rapid development of intelligent security, video surveillance, and large-scale multimodal perception technologies, video anomaly understanding technology has gradually expanded from anomaly detection in short clips, single perspectives, and single modalities to hierarchical anomaly semantic understanding for long videos, multiple perspectives, and complex scenes. In long video anomaly understanding scenarios, anomaly events often span a long time span, include multiple evolutionary stages, and exhibit characteristics such as large temporal spans, strong event correlations, and dynamic complexity of scenes, which places higher demands on long-process temporal modeling and global semantic understanding.

[0003] Video anomaly understanding, as an important research direction in intelligent security, public safety monitoring, and video content analysis, initially focused on video anomaly detection tasks, emphasizing the localization of anomalies, screening of anomaly segments, and classification of anomaly categories. Traditional methods typically target short video clips or fixed time windows, using visual feature extraction, temporal modeling, or anomaly score prediction to determine the presence of abnormal behavior in the video. However, with the rapid development of large-scale surveillance networks, long-term video storage, and multimodal large-scale model technologies, the demands of practical applications for anomaly event analysis have evolved from basic judgments of "whether it is abnormal and where it is abnormal" to a comprehensive understanding of the anomaly event's occurrence process, its evolution through different stages, causal relationships, and video-level semantic interpretation. Therefore, zero-shot hierarchical video anomaly understanding for long-form video scenarios is gradually becoming an important development direction in the field of intelligent video analysis.

[0004] Because long-video anomaly understanding tasks require addressing issues such as long-range temporal dependencies, fragmented anomaly segments, event stage evolution, cross-segment semantic associations, global context fusion, and generalization to unknown scenarios, related research is still in the exploratory stage and has not yet formed a fully mature and unified technical framework. Existing technologies are usually extensions of traditional video anomaly detection, video description generation, or short-segment anomaly understanding, employing methods such as sliding window segmentation, segment-level visual language description, anomaly score filtering, and language model summarization inference to achieve video anomaly analysis. These methods can generate anomaly segment descriptions or video-level summaries to a certain extent, but they still have shortcomings in modeling the intrinsic relationships, causal evolution, and global consistency between multi-stage anomaly events in long videos.

[0005] In existing technologies, hierarchical methods for video anomaly understanding and long-video anomaly analysis have been proposed. For example, Holmes-VAU promotes the development of video anomaly understanding from local segment description to hierarchical semantic understanding by constructing a multi-granularity anomaly understanding annotation system at the segment, event, and video levels. VADTree adopts a hierarchical temporal decomposition approach, organizing long videos into coarse-grained and fine-grained time nodes to improve the ability to locate and generate descriptions of long-video anomaly segments. Methods such as VAU-R1 attempt to construct a unified reasoning framework for anomaly understanding through supervised fine-tuning or reinforcement fine-tuning. In addition, related methods such as Hawk, VADER, LAVAD, and GtS have explored aspects such as anomaly narrative generation, causal relationship modeling, zero-shot anomaly detection, and hierarchical anomaly localization. These solutions are usually based on visual language models and large language models, and achieve modeling of anomaly segments and their semantic information through video segment description, feature fusion, cue learning, or task-specific training.

[0006] Analysis reveals that existing solutions still have certain limitations. First, many methods primarily rely on fragment-level descriptions or sliding window modeling, which easily fragments continuous anomalous events in long videos into multiple isolated segments, making it difficult to form stable, structured event representations. Second, existing methods lack explicit modeling of the temporal continuity, semantic similarity, entity overlap, and event evolution patterns between anomalous events, making it difficult to accurately determine whether there are preludes, escalations, continuations, escapes, or a shared event thread between multiple anomalous stages. Third, some methods rely on supervised fine-tuning or category-specific training on anomalous datasets, limiting their generalization ability when facing unknown anomalous scenarios, cross-dataset applications, or complex monitoring environments. Furthermore, existing methods underutilize the global video context, easily leading to inconsistencies between local segment understanding and overall video semantics, thus affecting the coherence and credibility of anomalous interpretations.

[0007] As mentioned earlier, existing video anomaly understanding technologies are mostly in the exploratory stage of extending from traditional anomaly detection to semantic understanding, and overall there are still several shortcomings, mainly including the following aspects: (1) Existing technologies generally adopt segment-level description or sliding window modeling to divide long videos into several relatively independent time segments for analysis, failing to fully consider the continuous evolution characteristics of abnormal events in long time-series videos. In actual security scenarios, abnormal events are often not single instantaneous behaviors, but are gradually composed of multiple stages such as approach, conflict, escalation, escape, and handling. When using fixed windows or isolated segments for modeling, it is easy to fragment the complete abnormal process, making it difficult for the system to accurately understand the overall development context and cross-stage relationships of abnormal events. Existing methods generally adopt sliding window time-series modeling or isolated segment-level semantic generation, which can only analyze local video segments independently. This not only makes it difficult to effectively capture long-range event dependencies across long videos and cannot fully depict the overall evolution context of abnormal events, but also results in huge computational overhead, low inference efficiency, and insufficient real-time performance when processing long videos due to repeated calculations of a large number of overlapping windows, making it difficult to meet the needs of efficient analysis in actual security scenarios.

[0008] (2) Existing technologies in the process of anomaly semantic modeling usually lack stable structured event representations. Most methods mainly rely on visual language models to generate fragment-level text descriptions, and then use large language models for simple summarization or classification. They fail to uniformly organize information such as event time intervals, event summaries, anomaly category candidates, key entities, anomaly causes, and time evidence chains. Due to the lack of structured expression, it is difficult to effectively compare and correlate different anomaly fragments, which can easily lead to inconsistencies in video-level anomaly understanding results, insufficient evidence support, and unclear explanatory logic.

[0009] (3) Existing solutions lack sufficient modeling of the relationships and causal dependencies between abnormal events, making it difficult to explicitly determine whether there are relationships such as temporal continuity, semantic continuity, behavioral escalation, foreshadowing, the same event thread, or escape / ending between different abnormal stages. Furthermore, some methods rely on supervised fine-tuning or task-specific training on abnormal datasets, which limits their generalization ability when facing unknown abnormal categories, complex monitoring environments, or cross-scenario applications. These problems are particularly prominent in long videos, complex events, and multi-stage abnormal evolution scenarios, restricting the application effectiveness of existing technologies in intelligent security and real video surveillance analysis.

[0010] (4) Existing methods often lack explicit modeling of the relationships and causal dependencies between abnormal events. Abnormal events in real monitoring scenarios are usually formed by multiple temporally related and logically progressive stages (such as robbery events, which include a series of processes such as approaching, threatening, taking the item, and escaping). However, existing methods can only output fragmented local abnormal descriptions and cannot achieve event-level causal reasoning and logical connection, resulting in logical breaks, poor coherence, and weak interpretability in the abnormal understanding results.

[0011] (5) Existing mainstream video anomaly understanding methods rely heavily on supervised training or parameter fine-tuning specific to anomaly scenarios. The model's capabilities are limited to known anomaly types and fixed scenario distributions. It lacks the ability to adapt to unknown anomalies, new scenarios, and new behavioral patterns. Its generalization performance is severely limited, making it difficult to adapt to complex and ever-changing actual security monitoring environments with diverse anomaly types.

[0012] In summary, existing technologies for understanding complex long-form video anomalies exhibit significant shortcomings in temporal modeling, event reasoning, and model generalization. A systematic solution has yet to be developed to address core issues such as the lack of long-range temporal dependencies, insufficient modeling of event causal relationships, inefficient sliding window computation, and weak generalization capabilities in these scenarios. In particular, a hierarchical video anomaly understanding technology that can simultaneously address global temporal consistency, event causal correlation, efficient reasoning, and zero-shot generalization is lacking. Therefore, for zero-shot hierarchical anomaly understanding scenarios in long-form videos, it is still necessary to propose a novel technical solution that can simultaneously adapt to long-range temporal differences, event relationship dependencies, local detailed evidence, and global semantic fusion requirements, in order to improve the accuracy, coherence, and interpretability of understanding anomalies in complex surveillance videos. Summary of the Invention

[0013] The purpose of this invention is to overcome the shortcomings of the prior art and provide a zero-shot long video understanding method based on structured event and event relationship reasoning. This method includes the following steps: The input video is divided into hierarchical temporal segments to obtain coarse-grained nodes and fine-grained nodes. The coarse-grained nodes are used to represent the overall scene background and long-term process changes, while the fine-grained nodes are used to capture local abnormal actions, character interactions, or abnormal details. The coarse-grained nodes and the fine-grained nodes are respectively input into the visual language model to obtain the corresponding coarse-grained semantic description and fine-grained semantic description; Filter and merge suspected abnormal intervals to obtain a merged set of abnormal intervals; The fine-grained semantic descriptions corresponding to each abnormal interval are merged, deduplicated, and evidence filtered to obtain a compressed abnormal evidence set. Then, a large language model is used to generate structured abnormal events. Relationship modeling is performed on the structured abnormal events to form structured event causal clues and abnormal evolution chains; Based on coarse-grained semantic description, a global semantic memory is constructed. The global semantic memory is used to describe the overall video environment, background changes, and long-term event development. By integrating local events, event relationships, and global semantics, video-level anomaly understanding results are generated.

[0014] Compared with existing technologies, the advantages of this invention lie in providing a zero-shot hierarchical video anomaly understanding method for long video scenarios. This method aggregates fine-grained anomaly fragments into structured anomaly events and further introduces event relationship modeling and local-global semantic fusion mechanisms, achieving a layer-by-layer analysis from fragment-level description to event-level reasoning, and then to video-level understanding. This invention leverages structured event representation to enhance the organization of anomaly evidence. By explicitly modeling the temporal continuity, semantic similarity, entity overlap, and anomaly evolution patterns between events, it strengthens the understanding of the development process and causal relationships of anomaly events. Furthermore, by introducing coarse-grained global semantic memory, this invention fuses local anomaly evidence with the overall video background and long-range process information, thereby improving the coherence, interpretability, and zero-shot generalization ability of long video anomaly understanding results.

[0015] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0017] Figure 1 This is a flowchart of a zero-sample long video understanding method based on structured events and event relationship reasoning according to an embodiment of the present invention. Detailed Implementation

[0018] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0019] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0020] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0021] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0023] In summary, this invention proposes a zero-shot hierarchical video anomaly understanding method for long-form video scenarios. In the process of long-form video anomaly understanding, the method first performs hierarchical temporal segmentation of the input video, distinguishing between coarse-grained nodes representing the overall scene background and long-term process information, and fine-grained nodes capturing local abnormal actions, character interactions, and key evidence. Under a unified video semantic understanding framework, a visual language model is used to generate semantic descriptions of different granularities, and fine-grained anomaly fragments are further aggregated into structured anomaly events, forming an event-level representation that includes event time intervals, event summaries, anomaly category candidates, evidence chains, anomaly causes, and key entities. Simultaneously, using coarse-grained global semantic information as contextual memory, an event relationship modeling mechanism constrains the consistency of different anomaly events in terms of temporal order, semantic association, entity overlap, and behavioral evolution, inferring the continuation, escalation, support, or shared main line relationship between events. Finally, local anomaly events, event relationships, and global semantic memory are fused to generate video-level anomaly categories, anomaly summaries, anomaly time segments, and causal explanation chains. This invention can alleviate problems such as fragmentation of abnormal segments in long videos, inconsistency of global semantics, and insufficient generalization ability across scenarios, thereby improving the accuracy, coherence, and interpretability of the results of abnormal understanding of long videos.

[0024] Specifically, see Figure 1 As shown, the zero-shot long video understanding method based on structured events and event relationship reasoning includes the following steps: Step S1: Perform hierarchical temporal division on the input long video to obtain a coarse-grained node set and a fine-grained node set.

[0025] Given a long video as input: (1) in, Let t represent the video image at frame t, and T represent the total number of frames in the video. First, the long video is divided into hierarchical temporal segments to obtain a coarse-grained set of time nodes. and fine-grained time node set : (2) (3) in, This represents a set of coarse-grained nodes, primarily used to characterize the overall scene background and long-range process changes in the video. This represents a fine-grained set of nodes, primarily used to capture local actions, character interactions, and anomaly details. Indicates the number of coarse-grained nodes. This indicates the number of fine-grained nodes.

[0026] Step S2: For coarse-grained nodes and fine-grained nodes, generate corresponding coarse-grained semantic descriptions and fine-grained semantic descriptions respectively using a visual language model.

[0027] Inputting coarse-grained and fine-grained nodes into the visual language model generates corresponding textual semantic descriptions: (4) (5) in, Representing a visual language model, This represents a set of coarse-grained semantic descriptions. This represents a set of fine-grained semantic descriptions. Coarse-grained descriptions are used to provide global scene context and long-range context, while fine-grained descriptions are used to provide local anomalous actions and key evidence.

[0028] Step S3: Filter and merge suspected abnormal intervals to obtain a merged set of abnormal intervals.

[0029] For example, the abnormal score sequence S is obtained through the VADTree module.

[0030] First, consecutive high-score segments are aggregated into initial anomaly intervals. : (6) in, This indicates a continuous anomalous fragment aggregation operation.

[0031] Then, based on the anomaly threshold And the Top-K strategy for screening candidate outlier intervals: (7) For anomaly intervals that are temporally adjacent, have high anomaly scores, and are semantically continuous, further merging is performed: (8) in, This represents the set of merged abnormal intervals. This indicates semantic continuity based on fine-grained semantic description computation. Indicates based on the abnormal threshold The candidate anomaly intervals obtained by the Top-K strategy, This represents a set of fine-grained semantic descriptions. This step is used to reduce the fragmentation of abnormal events caused by fixed window segmentation.

[0032] Step S4 involves merging, deduplicating, and filtering the fine-grained semantic descriptions corresponding to each abnormal interval to obtain a compressed set of abnormal evidence. Then, a large language model is used to construct a structured abnormal event.

[0033] For each abnormal interval Collect the corresponding fine-grained semantic descriptions within this interval: (9) Then, the fine-grained descriptions are merged, deduplicated, and filtered to obtain a compressed set of anomaly evidence: (10) in, This indicates a semantic merging operation. This indicates a deduplication operation. This indicates that the top K most important pieces of anomalous evidence are selected under the constraint of prompt word length.

[0034] Subsequently, the compressed anomalous evidence is input into the large language model to generate structured anomalous events: (11) in, Representing a large language model; Indicates the time interval of the event. Indicates an event summary. Indicates candidates for anomaly categories. Representing a time-based chain of evidence, Indicates the cause of the abnormality. Indicates a key entity.

[0035] This step transforms fragmented, piecemeal descriptions into unified, stable, and reasonable event-level representations.

[0036] Step S5: Perform relation-aware event dependency modeling on structured anomalous events to form causal clues and anomalous evolution chains for structured events.

[0037] After obtaining the structured event representation generated by the event-level reasoning module, explicit dependency modeling is further performed on anomalous events in the same video. Unlike treating anomalous events as independent local fragments, this invention constructs adjacent event pairs in chronological order and infers the relationship type between events based on rule priority, dynamic temporal constraints, and multidimensional continuity evidence. This module does not additionally call LLM; instead, it utilizes only the event summary, anomalous label candidates, temporal evidence chains, anomalous causes, and key entities from the event-level output for relationship modeling, thereby achieving lightweight and interpretable event-dependent reasoning.

[0038] Specifically, all events are first sorted according to their time intervals, and directed event pairs are constructed only for events that are adjacent in time: (12) For each pair of adjacent events It estimates the time proximity based on the event time interval and adaptively sets dynamic time constraints based on the event duration to adapt to the event density and time scale in different videos.

[0039] At the same time, from the perspective of label relevance Relevance of event context Continuity of action and entity relevance Continuity evidence is extracted from four aspects. Since the structured event representation has been generated by LLM induction, its summary, anomaly cause, action description, and key entities usually contain core keywords of anomalous behavior and scene context. Therefore, this invention does not introduce an additional embedding encoder or trainable relation model, but instead uses keyword matching, tag group classification, and token-level Jaccard similarity for lightweight approximate judgment. This represents the set of tokens after text normalization.

[0040] (13) Specifically, label relevance The consistency of the top-1 anomaly labels of adjacent events determines whether adjacent events belong to the same anomaly category; event context relevance. Textual overlap between summaries of adjacent events and causes of anomalies is estimated to determine whether two events are in similar anomalous contexts; action continuity. Calculated from the action and description fields in the time-based evidence chain, it is used to capture relationships of similar actions, action continuation, or action transfer; entity association. The determination is made by whether the roles and types of the key entities are consistent, which is used to identify whether two events revolve around the same person, vehicle, object, or scene object.

[0041] Ultimately, the four types of continuous evidence were aggregated into: (14) If either piece of evidence is valid, the system considers there to be a potential continuity between the two events.

[0042] Based on dynamic time constraints and continuous evidence, this invention judges the relationship between events in sequence according to a fixed priority, as follows.

[0043] (15) (1) If the latter event includes behaviors such as escaping, evacuating, or leaving quickly, and the former event includes abnormal triggering behaviors such as fighting, attacking, or stealing, and both meet the escape time constraint and continuity evidence, then the event is preferentially determined as followed_by_avoidance.

[0044] (2) If the abnormal intensity of the latter event is higher than that of the former event, and both satisfy the escalation time constraint and the continuity evidence, then it is determined to be escalates_to.

[0045] (3) If the preceding event includes contextual behaviors such as approaching, entering, waiting, observing, preparing, approaching the target or interacting with the door, and the following event reaches a clear abnormality level, and both satisfy the contextual support time constraint and continuity evidence, then it is determined as precedes_and_supports.

[0046] (4) If the event pair does not satisfy the above strong relationship, but shares at least two types of continuous evidence within the main time constraint, it is determined to be same_storyline.

[0047] (5) If an event pair only satisfies the main time constraint, or only has a single piece of continuous evidence, but is insufficient to support a stronger relationship type, it is determined to be weakly_related.

[0048] (6) If none of the above conditions are met, it is determined to be independent.

[0049] Step S6: Construct a global semantic memory based on coarse-grained semantic description.

[0050] To supplement global context information beyond local anomalies, the coarse-grained semantic descriptions are deduplicated, sorted, and compressed to obtain a compressed set of coarse-grained descriptions: (16) Then it is input into a large language model to generate global semantic memory: (17) in, Represents a scene-level summary. This represents a global process summary. Global semantic memory is used to describe the overall video environment, background changes, and the development of long-term events, providing contextual support for video-level anomaly understanding.

[0051] Step S7: Integrate local events, event relationships, and global semantics to generate video-level anomaly understanding results.

[0052] Based on the event anomaly confidence level, event relationship importance, and key entity information, calculate the importance score for each event: (18) in, Indicates the confidence level of an event's anomaly. Indicates the degree of relationship between events. This indicates the key entity reward item.

[0053] Subsequently, the set of major anomalous events was selected based on the event importance score: (19) Finally, the main anomalous events, event relationships, and global semantic memory are fused to generate the final video-level anomaly understanding result: (20) in, This represents a video-level fusion function. This represents the set of event relationships. The final output can be represented as: (twenty one) in, Indicates the final exception category, Indicates the confidence level of the anomaly. Indicates a video-level anomaly summary. Indicates the causal explanation chain of an event. This represents a set of abnormal time segments.

[0054] In summary, compared with the prior art, the present invention has the following distinguishing features: (1) Zero-shot hierarchical anomaly understanding strategy for long videos This invention proposes a zero-shot hierarchical video anomaly understanding strategy for long video scenarios. By dividing long videos into coarse-grained time nodes and fine-grained time nodes, the global scene background and local anomaly details are modeled respectively. Without relying on anomaly-specific supervised fine-tuning, hierarchical anomaly analysis is achieved from segment-level description to event-level reasoning and then to video-level understanding, thereby improving the generalization ability and interpretability of anomaly understanding in complex long video scenarios.

[0055] (2) Anomaly semantic modeling mechanism based on structured event representation This invention further aggregates the selected suspected anomalous fragments into structured anomalous events, rather than simply describing isolated fragments. Each structured anomalous event includes information such as the event's time interval, event summary, candidate anomalous categories, temporal evidence chain, anomalous cause, and key entities. This mechanism can transform fragmented, segment-level semantic descriptions into unified, stable, and reasonable event-level representations, providing a reliable foundation for subsequent event relationship modeling and video-level anomaly understanding.

[0056] (3) Relationship-aware abnormal event dependency modeling method This invention proposes a relation-aware event dependency modeling method. After obtaining multiple structured anomalous events, event pairs are constructed according to chronological order. The method integrates temporal continuity, semantic similarity, entity overlap, action similarity, anomalous label consistency, and anomalous evolution patterns to determine the relationships between different events, such as continuation, escalation, prior and subsequent support, the same event thread, escape / ending, or mutual independence. This approach can explicitly model the development process and causal dependencies of anomalous events in long videos, avoiding the problem of existing methods that only splice segments and lack event logic.

[0057] (4) The mechanism for fusing local abnormal events with global semantic memory.

[0058] This invention designs a local-global fusion video-level anomaly understanding mechanism. It constructs a global semantic memory using coarse-grained semantic descriptions, preserving the overall video scene, background environment, and long-term event development trajectory. Simultaneously, it combines fine-grained anomaly events and event relationship chains to generate the final video-level anomaly category, anomaly summary, anomaly time segments, and event causal explanation chain. This mechanism can alleviate the problems of fragmented local evidence, inconsistent global semantics, and disjointed anomaly explanations in long video anomaly understanding, improving the accuracy, coherence, and credibility of video anomaly understanding results.

[0059] In summary, compared with the prior art, the present invention has the following main advantages: (1) Effectively solved the problem of fragmentation of abnormal events in long videos. Existing technologies typically employ sliding window or fragment-level description methods, which easily divide complete abnormal events into multiple isolated segments. This invention, through hierarchical temporal division and abnormal interval merging mechanisms, organizes temporally adjacent and semantically continuous abnormal segments into complete abnormal events, thereby alleviating the event fragmentation problem in long video anomaly understanding and enabling a more complete expression of the occurrence process and stage changes of abnormal events.

[0060] (2) Abnormal events are represented in a more structured way and are more inferable. Existing technologies typically rely on segment captions or video summaries for anomaly description, lacking a unified event organization format and making it difficult to support subsequent reasoning. This invention further transforms anomaly intervals into structured anomaly events, including information such as event time intervals, event summaries, anomaly category candidates, temporal evidence chains, anomaly causes, and key entities. This transforms the originally fragmented segment-level semantic descriptions into stable, unified, and reasonable event-level representations, thereby improving the accuracy and interpretability of video anomaly understanding.

[0061] (3) Able to explicitly model the relationships and causal dependencies between anomalous events Existing technologies typically only splice or summarize multiple anomalous fragments. This invention further constructs an event relationship modeling mechanism that comprehensively considers factors such as temporal continuity, semantic similarity, entity overlap, action similarity, anomalous label consistency, and anomalous evolution patterns to determine the continuation, escalation, prior and subsequent support, common main thread, escape / ending, or independent relationships between different anomalous events. This mechanism can better characterize the development process of anomalous events, forming clearer causal clues and anomalous evolution chains.

[0062] (4) It can improve the global semantic consistency of long videos. Existing technologies often rely excessively on local segment descriptions when processing long videos, which can easily lead to inconsistencies between local anomaly interpretations and the overall video background. This invention introduces coarse-grained global semantic memory, preserving the overall video scene, background environment, and long-term event development context, and integrates these with local structured anomaly events and their relationships. This results in a more coherent, complete, and semantically consistent video-level anomaly summary and anomaly judgment.

[0063] (5) Stronger generalization ability with zero samples Some existing video anomaly understanding methods rely on supervised fine-tuning on anomaly datasets or task-specific training, which limits their generalization ability when faced with unknown anomaly categories, videos across different datasets, or complex monitoring scenarios. This invention does not rely on anomaly-specific supervised fine-tuning, but instead utilizes visual language models, large language models, structured event reasoning, and rule-based relationship modeling to achieve anomaly understanding. Therefore, it can better adapt to unknown anomaly types and the complex variations in real-world monitoring scenarios.

[0064] (6) The results of anomaly understanding are more interpretable and traceable.

[0065] Existing technologies typically only output anomaly categories or a short video description, failing to explain which segments, evidence, and event relationships the anomaly judgment is based on. This invention ultimately outputs anomaly categories, anomaly confidence levels, video summaries, anomaly time segments, and causal explanation chains. It not only answers "what anomaly occurred," but also further explains "where the anomaly occurred," "what evidence supports it," "how the events are related," and "how the anomaly process evolved," thereby improving the reliability and practicality of anomaly analysis results in intelligent security systems.

[0066] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0067] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0068] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0069] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A zero-shot long video understanding method based on structured events and event relationship reasoning, comprising the following steps: The input video is divided into hierarchical temporal segments to obtain coarse-grained nodes and fine-grained nodes. The coarse-grained nodes are used to represent the overall scene background and long-term process changes, while the fine-grained nodes are used to capture local abnormal actions, character interactions, or abnormal details. The coarse-grained nodes and the fine-grained nodes are respectively input into the visual language model to obtain the corresponding coarse-grained semantic description and fine-grained semantic description; Filter and merge suspected abnormal intervals to obtain a merged set of abnormal intervals; The fine-grained semantic descriptions corresponding to each abnormal interval are merged, deduplicated, and evidence filtered to obtain a compressed abnormal evidence set. Then, a large language model is used to generate structured abnormal events. Relationship modeling is performed on the structured abnormal events to form structured event causal clues and abnormal evolution chains; Based on coarse-grained semantic description, a global semantic memory is constructed. The global semantic memory is used to describe the overall video environment, background changes, and long-term event development. By integrating local events, event relationships, and global semantics, video-level anomaly understanding results are generated.

2. The method according to claim 1, characterized in that, The coarse-grained semantic description and the fine-grained semantic description are respectively represented as: in, Representing a visual language model, This represents a set of coarse-grained semantic descriptions. This represents a set of fine-grained semantic descriptions. Indicates the first A coarse-grained node, Indicates the number of coarse-grained nodes. Indicates the first Fine-grained nodes, This indicates the number of fine-grained nodes.

3. The method according to claim 1, characterized in that, The process of filtering and merging suspected abnormal intervals includes the following steps: Consecutive high-abnormality score segments are aggregated into initial abnormality intervals. : in, This indicates a continuous anomalous fragment aggregation operation. It is an abnormal score sequence; Based on the abnormal threshold And the Top-K strategy for screening candidate outlier intervals: For anomaly intervals that are temporally adjacent, have high anomaly scores, and are semantically continuous, further merging is performed: in, This represents the set of merged abnormal intervals. This indicates semantic continuity based on fine-grained semantic description computation. Indicates based on the abnormal threshold The candidate anomaly intervals obtained by the Top-K strategy, This represents a set of fine-grained semantic descriptions.

4. The method according to claim 1, characterized in that, In the process of modeling the relationships between the structured anomalous events, adjacent event pairs are constructed in chronological order, and the relationship types between the events are inferred based on rule priority, dynamic time constraints, and multidimensional continuity evidence.

5. The method according to claim 4, characterized in that, The relationship types between the events are determined sequentially according to a set priority, as follows: Among them, if the latter event Including behaviors such as escaping, evacuating, or leaving quickly, the preceding event If an event pair includes abnormal triggering behaviors such as fighting, attacking, or stealing, and both events satisfy escape time constraints and continuity evidence, they are preferentially classified as a followed_by_avoidance relationship. If the abnormal intensity of the latter event is higher than that of the former event, and both events satisfy escalation time constraints and continuity evidence, they are classified as escalates_to. If the former event includes contextual behaviors such as approaching, entering, waiting, observing, preparing, approaching a target, or interacting with a door, and the latter event reaches a clear abnormal level, and both events satisfy contextual support time constraints and continuity evidence, they are classified as precedes_and_supports. If the event pair does not satisfy the above strong relationships, but shares at least two types of continuity evidence within the main time constraints, they are classified as a same_storyline relationship. If the event pair only satisfies the main time constraints, or only has a single type of continuity evidence, but does not satisfy the support for a stronger relationship type, they are classified as weakly_related. If none of the above conditions are met, they are classified as independent.

6. The method according to claim 1, characterized in that, The global semantic memory is obtained according to the following steps: The coarse-grained semantic descriptions are deduplicated, sorted, and compressed to obtain a compressed set of coarse-grained descriptions: The compressed coarse-grained description set is input into the visual language model to generate a global semantic memory, represented as: in, Represents a scene-level summary. This represents a summary of the global process.

7. The method according to claim 1, characterized in that, The structured anomaly event includes the event time interval, event summary, anomaly category candidates, time evidence chain, anomaly cause, and key entity information.

8. The method according to claim 1, characterized in that, The video-level anomaly understanding results are obtained according to the following steps: Calculate the importance score for each event, expressed as: in, Indicates the confidence level of an event's anomaly. Indicates the degree of relationship between events. Indicates key entity reward items; The set of anomalous events is filtered based on the event importance score, and is represented as follows: The following formula is used to generate video-level anomaly understanding results: in, This represents a video-level fusion function. Represents a set of event relationships; The final video understanding result is represented as: in, Indicates the final exception category, Indicates the confidence level of the anomaly. This indicates a video-level anomaly summary. This represents the causal explanation chain of an event. This represents a set of abnormal time segments.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images