Intelligent monitoring and early warning system and method with enhanced scene understanding

Abstract

The invention discloses a scene understanding enhanced intelligent monitoring and early warning system and method. The system comprises a scene semantic understanding module, a scene graph construction and relation reasoning module, a scene common sense knowledge base enhancement module and a context sensing anomaly detection module. The scene semantic understanding module is used for extracting scene context, target features and behavior features; the scene graph construction and relation reasoning module is used for organizing the discrete features into a structured scene graph and performing relation reasoning; the scene common sense knowledge base enhancement module is used for injecting scene common sense knowledge; and the context sensing anomaly detection module is used for carrying out anomaly evaluation and early warning by integrating the multi-dimensional information. According to the method, the scene-behavior-object ternary relation model is constructed, and the scene common sense knowledge base is fused, so that the deep semantic understanding of the monitoring video and the anomaly detection of context sensing are realized, the false alarm rate is remarkably reduced, and the early warning capability in a complex scene is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to an intelligent monitoring and early warning system and method with enhanced scene understanding. By integrating video streams, scene semantics, spatiotemporal relationships and common sense knowledge base, it achieves deep understanding and intelligent early warning of the monitoring scene, and is applicable to various application scenarios such as public safety, industrial safety, and smart communities. Background Technology

[0002] With the widespread adoption of video surveillance technology, surveillance cameras have been extensively deployed in public transportation, industrial parks, commercial venues, and communities. Traditional surveillance systems primarily rely on manual monitoring or rule-based anomaly detection algorithms, which suffer from the following technical shortcomings:

[0003] Insufficient semantic understanding: Existing surveillance systems can only identify object categories (such as "person" and "vehicle") and simple behaviors (such as "running" and "falling"), but cannot understand the context in which the behavior occurs. For example, the behavior of "a person running" is normal on a playground, but it may mean a robbery or an emergency in a bank lobby. Traditional systems cannot distinguish such semantic differences.

[0004] High false positive rate: Rule-based anomaly detection methods rely on manually set thresholds and rules, making it difficult to adapt to complex and ever-changing real-world scenarios. For example, defining "rapid movement" as an anomaly can lead to a large number of false positives for normal running behavior, while relaxing the rules can cause real anomalies to be missed.

[0005] Lack of common-sense reasoning: Traditional systems cannot use common-sense reasoning based on the context. For example, on a subway platform, a person standing outside the yellow line for an extended period may pose a risk of jumping onto the tracks; in a parking lot, a person loitering near multiple vehicles may be a precursor to theft. These judgments require a comprehensive analysis combining contextual knowledge and behavioral patterns.

[0006] Weak spatiotemporal correlation analysis: Existing systems mostly analyze single frames or short video clips, lacking the ability to model long-term trajectories of people, multi-target interaction relationships, and cross-camera behavior continuity, making it difficult to identify complex abnormal behaviors such as tailing, gathering, and coordinated crimes.

[0007] Low query and retrieval efficiency: Traditional monitoring systems mainly rely on timestamps and camera numbers for video retrieval, which cannot support natural language semantic queries, such as "find the video of the person wearing red clothes entering Building B yesterday afternoon", resulting in low efficiency of post-event investigation.

[0008] Therefore, there is an urgent need for an intelligent monitoring and early warning system that can deeply understand scene semantics, combine common sense knowledge for reasoning, and support spatiotemporal correlation analysis and natural language query, so as to improve the intelligence level and practical value of the monitoring system. Summary of the Invention

[0009] To address the aforementioned issues, this invention provides an intelligent monitoring and early warning system and method with enhanced scene understanding. By constructing a scene-behavior-object ternary relationship model and integrating a scene common sense knowledge base, it achieves deep semantic understanding of monitoring videos and context-aware anomaly detection, significantly reducing false alarm rates and improving early warning capabilities in complex scenarios.

[0010] The specific plan is as follows:

[0011] An intelligent monitoring and early warning system with enhanced scene understanding includes a scene semantic understanding module, a scene graph construction and relation reasoning module, a scene common sense knowledge base enhancement module, and a context-aware anomaly detection module. The scene semantic understanding module is used to extract scene context, target features, and behavioral features. The scene graph construction and relation reasoning module is used to organize discrete features into a structured scene graph and perform relation reasoning. The scene common sense knowledge base enhancement module is used to inject scene common sense knowledge. The context-aware anomaly detection module is used to comprehensively evaluate and warn of anomalies by integrating multi-dimensional information. Each module realizes information transmission through scene context vectors and progressively enhanced node features, forming an end-to-end intelligent early warning process.

[0012] Furthermore, the scene semantic understanding module is responsible for extracting multi-layered semantic information from the original surveillance video, providing a structured scene representation for subsequent modules; it includes three closely related sub-modules: scene classification, object detection, and behavior recognition; the outputs of these three sub-modules together constitute the multi-layered semantic representation of the scene; specifically, the scene context vector... Provides global scene context and target features Describe the static attributes and behavioral characteristics of each target. It depicts the dynamic behavior patterns of the target; these three types of features are integrated into a unified node representation in the scene graph construction and relation reasoning module, realizing the transformation from discrete targets to structured scene graphs.

[0013] Furthermore, the scene graph construction and relation reasoning module receives the scene context vector output by the scene semantic understanding module. Target feature set and behavioral feature set This discrete semantic information is organized into a structured scene graph representation, and multi-layer relational reasoning is performed through a graph neural network to obtain the final node features. This module enables a deep understanding of the interaction relationships and overall semantics between targets in a scene. Its output is enhanced node features that incorporate relational information, providing a foundation for subsequent knowledge base enhancement and anomaly detection.

[0014] Furthermore, the scene common sense knowledge base enhancement module receives node features output by the scene graph construction module. and scene context vector By injecting normal behavior patterns, abnormal behavior definitions, and object interaction rules from different scenarios into a pre-constructed scene knowledge graph, the system acquires common-sense reasoning capabilities. The core idea of ​​this module is to fuse symbolic scene common-sense knowledge with visual features extracted by a neural network, outputting knowledge-enhanced node features. This is used by the context-aware anomaly detection module.

[0015] Furthermore, the context-aware anomaly detection module receives knowledge-enhanced node features. Scene context vector and scene graph structure This module integrates scene semantics, target relationships, and common-sense knowledge to assess the anomaly of each target and the scene as a whole, outputting risk levels and early warning information. This module serves as the decision-making output module for the entire system, drawing upon the accumulated information from all the aforementioned modules.

[0016] A scene-understanding-enhanced intelligent monitoring and early warning method includes the following steps:

[0017] S1, Scene Semantic Understanding

[0018] S11, Scene Classification and Context Coding

[0019] First, the surveillance video is classified into scenes to identify the scene type of the current monitored area (e.g., subway station, parking lot, factory workshop, shopping mall, campus, etc.). A pre-trained visual Transformer model is used to extract global features from the video frames, and a scene classifier outputs the scene category probability distribution. Scene context vectors are then generated. The calculation formula is:

[0020]

[0021] in, Indicates the input video frame; ( ) represents the visual Transformer feature extraction function; and These are the weight matrix and bias vector of the scene classifier, respectively; Output feature dimensions for ViT; For scene context vectors; This represents the total number of scene categories.

[0022] Scene context vector It not only identifies the scene type, but also serves as global context information for all subsequent modules, ensuring that behavior understanding and anomaly detection are based on the correct scene context;

[0023] S12, Multi-target detection and attribute extraction

[0024] Guided by the scene context, all targets in the video frame are detected and their attributes are extracted; improved YOLO or Faster R-CNN models are used to detect people, vehicles, and objects, and the location, category, and appearance features of each target are extracted.

[0025] For the One detection target, , The total number of detected targets, its feature representation Based on location features Category characteristics and appearance features It is pieced together:

[0026]

[0027] in, Represents the center coordinates of the bounding box of the target. and size ; For category embedding features, the category ID is mapped to a dense vector through the embedding layer; Embed dimensions for categories; These are the deep appearance features extracted from the target region (such as feature vectors extracted by ResNet). For appearance features; This represents a vector concatenation operation; therefore, the total dimension of the target features is... ;

[0028] S13. Spatiotemporal behavioral feature extraction

[0029] To capture the dynamic behavior of the target, a 3D convolutional neural network or video Transformer is used to extract spatiotemporal features from consecutive video frames; for time windows... The video clip inside, number Spatiotemporal behavior characteristics of an individual target The calculation is as follows:

[0030]

[0031] in, Indicates the time window A continuous video frame sequence; ( ) represents a 3D convolutional feature extraction function (such as I3D, SlowFast, etc.), which is located at the target position. Extracting spatiotemporal features from the surrounding environment; This is a behavioral feature vector; This refers to the behavioral characteristic dimension.

[0032] S2, Scene Graph Construction and Relationship Reasoning

[0033] S21, Scene Graph Construction

[0034] Scene diagram From node set and edge set Composition: Nodes represent entities in the scene (people, objects, etc.), and edges represent relationships between entities (spatial relationships, interaction relationships, etc.).

[0035] For detected The first objective is to analyze the target features from the scene semantic understanding module. Behavioral characteristics and scene context vector Perform fusion to generate initial node features; The initial feature calculation formula for each node is:

[0036]

[0037] in, and The weight matrix and bias vector for feature fusion; For node feature dimensions; Represents vector concatenation operation; scene context vector Features are incorporated into each node to ensure that the node representation includes contextual information; superscript This indicates that these are the initial node features before inference in the graph neural network;

[0038] Edges are constructed based on the spatial distance and semantic association between targets; for nodes and nodes edge weight The calculation is as follows:

[0039]

[0040] in, For nodes and The relative position encoding between them (including spatial relationship information such as Euclidean distance and relative angle); Encode the dimension for relative position; The parameter vector calculated for the edge weights; Using the Sigmoid activation function, the edge weights are normalized to... Interval; when Exceeding the preset threshold At the node and Establish an edge between them, and connect the edges Add edge set ;

[0041] S22, Graph Attention Network Reasoning

[0042] After constructing the scene graph, a Graph Attention Network (GAT) is used for multi-layer relation reasoning, updating node features to incorporate information from neighboring nodes; the graph attention network has a total of Layer, number layer( The node feature update formula for ) is:

[0043]

[0044] in, Represents a node The set of neighboring nodes (i.e., nodes with the node) (There exists a set of nodes connected by edges). For the first Layer weight matrix; The activation function (such as ReLU or ELU); The attention coefficient represents the node. For nodes Importance weights; attention coefficients are calculated using the attention mechanism and satisfy normalization conditions. Ensure that the sum of the contribution weights of all neighboring nodes is 1;

[0045] go through After inference using the layered graph attention network, the final node features are obtained. Each Each node feature integrates information from its neighboring nodes, capturing the interaction relationships and overall semantics between targets in the scene; these enhanced node features are then passed to the scene common sense knowledge base enhancement module, further injecting scene common sense knowledge.

[0046] S3, Enhanced Scenario Common Sense Knowledge Base

[0047] S31, Construction of Scene Knowledge Graph

[0048] Scene knowledge graph From entity set Relationship set and triple set Composition: Entities include scene type, behavior type, object type, and risk level; relationships include "scene-normal behavior", "scene-abnormal behavior", and "behavior-risk level".

[0049] For example, a knowledge graph contains the following triples:

[0050] (Subway station, normal behavior, queuing for the train)

[0051] (Subway station, unusual behavior, crossing the yellow line)

[0052] (Crossing the yellow line, risk level: high)

[0053] (Parking lot, unusual behavior, loitering for an extended period of time)

[0054] S32. Knowledge Embedding and Integration

[0055] Knowledge graph embedding methods such as TransE are used to map entities and relations in the knowledge graph to a low-dimensional vector space; for triples (Head entity, relation, tail entity), the embedding vectors satisfy the translation relation:

[0056]

[0057] in, These are the embedding vectors for the head entity, relation, and tail entity, respectively. Embedding dimensions for knowledge; Represents approximate equality relationships in a vector space;

[0058] During anomaly detection, based on the current scene category Based on the detected behavior type, relevant common sense rules are retrieved from the knowledge graph; specifically, for the first... For each target, based on its behavioral characteristics and the identified behavioral category, the attribute (normal / abnormal, risk level, etc.) of that behavior in the current scenario is queried in the knowledge graph to obtain the corresponding knowledge embedding vector. Then through linear mapping Mapping it to the node feature space yields By fusing knowledge embedding vectors with scene graph node features, the semantic representation of nodes is enhanced.

[0059]

[0060] in, The node features output by the graph attention network; The mapped knowledge embedding vector; This is the knowledge fusion weighting coefficient, used to control the intensity of knowledge and information injection; For knowledge-enhanced node features;

[0061] Through knowledge base enhancement, the system can use predefined common sense about scenarios for reasoning. For example, it can determine that "not wearing a safety helmet in a factory workshop" is an abnormal behavior, while "not wearing a safety helmet in an office" is a normal behavior, thus achieving context-aware anomaly detection. Enhanced node features. The semantics, which include visual information, relational information, and common sense knowledge, will serve as the input to the anomaly detection module.

[0062] S4. Context-aware anomaly detection

[0063] S41. Anomaly Score Calculation

[0064] For the One objective, from its enhanced node features Three dimensions of anomaly index are extracted; its comprehensive anomaly score is... By visual abnormality Degree of behavioral abnormality and knowledge anomaly The weighted fusion yields:

[0065]

[0066] in, Assess the overall anomaly score; Let be the weighting coefficient, satisfying This is used to balance the importance of the three dimensions;

[0067] Visual abnormality The matching degree calculation is based on the target's appearance features and scene context, measuring whether the target's appearance attributes (such as location, color, carried items, etc.) are consistent with the current scene; behavioral anomaly degree. The degree of deviation between behavioral characteristics and normal behavioral patterns is calculated by comparing it with the pre-trained distribution of normal behavior; knowledge anomaly. The calculation based on the abnormal behavior definition in the knowledge graph directly reflects the knowledge base's judgment of the abnormality of the behavior in the current scenario.

[0068] Overall scene anomaly By aggregating the anomalies of all targets and the anomalies of relationships between targets, we obtain:

[0069]

[0070] in, The first term is the relation anomaly weighting coefficient. Capture the most unusual target in the scene; second item Capture situations where there are significant differences in the degree of abnormality between targets (such as one person behaving abnormally while those around them are behaving normally, which may indicate conflict or threat). The edge weights (calculated during the scene graph construction phase) are used to weight the difference in anomalies between different target pairs.

[0071] S42. Risk Level Classification and Early Warning

[0072] Based on the overall abnormality score of the scene The system categorizes risks into three levels: low, medium, and high.

[0073] Low risk: Normal monitoring, no warning required;

[0074] Medium risk: Issue alerts and record events;

[0075] High risk: This will immediately trigger an alarm and notify security personnel.

[0076] in, and As a risk threshold, It can be dynamically adjusted according to different scenarios and application requirements. For example, in high-security scenarios (such as airports and nuclear power plants), a lower threshold can be set to improve sensitivity, while in general scenarios, a higher threshold can be set to reduce false alarms.

[0077] This invention also supports a progressive early warning mechanism, which issues an early warning for targets with continuously rising anomalies before they reach a high-risk threshold, thus giving security personnel time to respond.

[0078] Through context-aware anomaly detection, the system can dynamically adjust anomaly judgment criteria based on scene context, significantly reducing the false alarm rate while improving the sensitivity to detect real anomalies. The entire system forms a complete closed loop from video input to early warning output: video frame → scene semantic understanding → scene graph construction → knowledge base enhancement → anomaly detection → risk warning. Each module utilizes scene context vectors... Closely linked to progressively enhanced node characteristics, it enables end-to-end intelligent monitoring and early warning.

[0079] The beneficial effects of this invention are as follows:

[0080] 1. Significantly reduced false alarm rate: Through scene context awareness and common sense knowledge base enhancement, the system can dynamically adjust the anomaly judgment criteria according to the scene context, avoiding misjudging normal behavior as abnormal. Experiments show that compared with traditional rule-based methods, the false alarm rate of this invention is reduced by more than 60%.

[0081] 2. Enhanced understanding of complex scenarios: Through scene graph construction and graph neural network reasoning, the system can understand the interaction relationships between targets and the overall semantics of the scene, and identify complex behavioral patterns such as collaborative crimes and abnormal group behavior, breaking through the limitation of traditional methods that can only identify simple behaviors of single targets.

[0082] 3. No need for extensive scene-specific data training: Through a pre-built scene common sense knowledge base, the system can quickly adapt to new scenes without the need to collect a large amount of labeled data for model training for each scene, which greatly reduces deployment costs and time.

[0083] 4. Supports progressive early warning: Through spatiotemporal behavior trajectory modeling and anomaly trend analysis, the system can issue early warnings in the early stages of abnormal behavior, giving security personnel time to respond and effectively preventing accidents from occurring.

[0084] 5. Enhance the intelligence level of the monitoring system: The system supports natural language semantic query and automatic generation of event descriptions, which significantly improves the retrieval efficiency of monitoring videos and the ability to conduct post-event investigations, and promotes the transformation of the monitoring system from passive recording to proactive early warning and intelligent analysis.

[0085] 6. Wide range of applications: This invention is applicable to various scenarios such as public transportation, industrial safety, smart communities, retail loss prevention, and campus security. It has strong versatility and scalability, and can provide intelligent solutions for security monitoring in different fields. Attached Figure Description

[0086] Figure 1 This is a flowchart of the method of the present invention.

[0087] Figure 2 This is the system architecture diagram of the present invention. Detailed Implementation

[0088] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0089] As shown in the figure, this embodiment provides an intelligent monitoring and early warning system and method with enhanced scene understanding, based on a public transportation scenario (subway station). The specific solution is as follows:

[0090] S1. Scene Setup and Data Acquisition

[0091] A subway station has deployed the intelligent monitoring and early warning system of this invention, covering the platform, entrance, staircases, and waiting area. The system connects to 10 high-definition cameras with a video resolution of 1920×1080 and a frame rate of 25fps. The system processes the video stream in real time, analyzing 5 frames per second. The following analysis of a single frame at a specific moment illustrates the complete processing flow of the system.

[0092] S2, Scene Semantic Understanding

[0093] S21, Scene Classification

[0094] The system performs scene classification on the input video frames. A pre-trained ViT-B / 16 model is used to extract global features, which are then input into the scene classifier. Scene categories include 20 categories such as subway platforms, subway entrances, subway stairs, shopping malls, and parking lots. ).

[0095] Scene context vector calculation:

[0096]

[0097] in, , The ViT output feature dimension is 768. The classification result shows that the current scene is a "subway platform" with a probability of 0.92.

[0098] S22, Multi-target detection

[0099] The system uses the YOLOv8 model to detect all targets in video frames. In a given frame, it detects... One target: 12 people, 2 suitcases, and 1 trash can.

[0100] For the first target (personnel), its characteristics are represented as follows:

[0101]

[0102] in, Represents the center coordinates of the bounding box and size , Embedding features for the category "person" (category embedding dimension) ), The appearance features extracted from ResNet-50 (appearance feature dimension) Therefore, the target feature dimension is... .

[0103] S23. Spatiotemporal behavioral feature extraction

[0104] The system uses the I3D (Inflated3DConvNet) model to extract spatiotemporal behavioral features. The time window is set to... Frame (approximately 0.64 seconds). For the first target, its spatiotemporal behavior characteristics are calculated as follows:

[0105]

[0106] Output features That is, behavioral characteristic dimension The behavior classifier identifies the target's behavior as "rapid movement".

[0107] At this point, the scene semantic understanding module has completed multi-level semantic extraction from the input video and output a scene context vector. A set of target features for 15 targets and behavioral feature set These features will serve as input to the scene graph building module.

[0108] S3, Scene Graph Construction and Relationship Reasoning

[0109] The scene graph construction module receives the output from the scene semantic understanding module, begins to construct the scene graph, and performs relational reasoning.

[0110] S31, Scene Graph Construction

[0111] The system detected Each target constructs a scene graph. The node feature dimension is set to... First, calculate the initial node features. For the first target, its target features are... The dimension is behavioral characteristics The dimension is Scene context The dimension is The initial node features are calculated as follows:

[0112]

[0113] in, The target features, behavior features, and scene context are concatenated and mapped to a 256-dimensional matrix. This process is repeated for all 15 targets to obtain the initial node feature matrix. .

[0114] Edge construction is based on spatial distance and semantic association. For target 1 and target 2, relative position encoding is calculated. ( This includes spatial relationship information such as Euclidean distance, relative angle, and relative dimensions. Edge weight calculation:

[0115]

[0116] in, Calculated Set threshold ,when Edges are established sequentially. This process is repeated for all target pairs to obtain the final scene graph. It contains 15 nodes and 42 edges.

[0117] S32, Graph Attention Network Reasoning

[0118] use Layered graph attention network is used for relational reasoning. Layer 1 ( Node feature update:

[0119]

[0120] Among them, the set of neighboring nodes of target 1 (People who are in close proximity) This is the weight matrix for layer 0. This is the ELU activation function. Attention coefficient calculation:

[0121]

[0122] in, This is the attention parameter vector for layer 0. It is calculated as follows: , , It satisfies the normalization condition.

[0123] By performing the second and third levels of reasoning in sequence, we finally arrive at... After three layers of reasoning, the features of node 1... By incorporating behavioral information from surrounding individuals, the system identifies target 1 as chasing a train whose doors are about to close, while the surrounding individuals (targets 2, 3, and 5) are all in a normal waiting state. After inference through a graph attention network, all nodes yield an enhanced set of node features. This feature set will be passed to the scene common sense knowledge base enhancement module.

[0124] S4, Enhanced Scenario Common Sense Knowledge Base

[0125] The scene common sense knowledge base enhancement module receives node features from the output of the graph attention network. and scene context vector To enhance knowledge.

[0126] S41 Knowledge Graph Retrieval

[0127] The system categorizes scenarios as "subway platform" (from...). The detected behavior "rapid movement" (obtained from the behavior classifier output) is derived from a pre-built knowledge graph. Relevant triples for retrieval:

[0128] (Subway platform, normal behavior, queuing for the train)

[0129] (Subway platform, normal behavior, chasing the train)

[0130] (Subway platform, unusual behavior, crossing the yellow line)

[0131] (Subway platform, unusual behavior, fighting)

[0132] (Chasing the train, risk level: low)

[0133] (Crossing the yellow line, risk level: high)

[0134] For target 1, the system matches the triple (subway platform, normal behavior, chasing the train) and determines that the behavior is normal behavior in the current scenario.

[0135] S42, Knowledge Embedding and Fusion

[0136] The knowledge graph embedding is trained using the TransE method, with embedding dimensions... For the triple (subway platform, normal behavior, chasing the train), the embedding vector satisfies:

[0137]

[0138] in, For the embedding of "subway platform", Embedding of "normal behavior" relationships, The phrase "to catch up with the train" is embedded in the text.

[0139] Knowledge is embedded and integrated into the features of node 1. First, a linear transformation is performed... Mapping to the node feature space: ,in ,get Then feature fusion is performed:

[0140]

[0141] in, These are the knowledge fusion weight coefficients. The fused features. This indicates that the behavior is normal in a subway platform setting, with low knowledge anomaly. This process is repeated for all 15 targets to obtain the knowledge-enhanced node feature set. This feature set will be passed to the context-aware anomaly detection module.

[0142] S5. Anomaly Detection in Text Context

[0143] The context-aware anomaly detection module receives knowledge-enhanced node features. Scene context vector and scene graph structure Anomaly assessment and risk warning are conducted.

[0144] S51, Anomaly Score

[0145] For objective 1, the system utilizes its enhanced node features. Three dimensions of anomaly indicators were extracted:

[0146] Visual abnormality (Based on appearance features) With scene context Matching accuracy calculation, normal appearance, no suspicious items)

[0147] behavioral abnormality (Based on behavioral characteristics) The degree of deviation from normal behavior patterns is calculated; rapid movement is slightly abnormal in general scenarios.

[0148] Knowledge Anomaly (Based on knowledge embedding) Calculations and the knowledge base indicate that chasing trains is normal behavior at subway stations.

[0149] Overall anomaly score:

[0150]

[0151] Among them, the weighting coefficient ,satisfy .

[0152] Anomaly scores were calculated for all 15 targets, resulting in... Calculation of overall scene anomaly:

[0153]

[0154] in, This is the weighting coefficient for the anomaly of the relationship. The relation anomaly term represents the maximum anomaly among all targets. A smaller value indicates that the overall scene is coordinated and the difference in anomaly degree among the targets is not significant.

[0155] S52, Risk Level Assessment

[0156] Set risk threshold .because The system determined the risk to be low, proceeded with normal monitoring, and did not trigger any alerts. The system continued monitoring the next frame of the video.

[0157] S53. Comparative Case: Abnormal Behavior Detection

[0158] To demonstrate the system's ability to detect abnormal behavior, suppose that at another moment in the same scenario, Target 7 (another person)'s behavior is identified as "crossing the yellow line and approaching the track." The system then re-executes the complete processing flow:

[0159] Scene semantic understanding: Scene context vector Still a "subway platform", the behavioral characteristics of target 7 The behavior was identified as "crossing the yellow line".

[0160] Scene Graph Construction and Relationship Reasoning: Node Features of Objective 7 It integrates its location features (near the edge of the track), behavioral features (crossing the yellow line), and scene context. After inference via a graph attention network, it obtains... .

[0161] Knowledge base enhancement: The system retrieves triples (subway platform, abnormal behavior, crossing the yellow line) and (crossing the yellow line, risk level, high) from the knowledge graph to obtain knowledge embeddings. , fusion .

[0162] Anomaly Detection: Calculate the three-dimensional anomaly degree of target 7:

[0163] Visual abnormality (Abnormal location, near a dangerous area)

[0164] behavioral abnormality (Crossing the safety line is clearly abnormal behavior)

[0165] Knowledge Anomaly (The knowledge base explicitly defines this as high-risk abnormal behavior)

[0166] Overall anomaly score:

[0167]

[0168] Assuming other targets have low anomaly rates, the overall scene anomaly rate is: .because The system determined the risk to be medium and immediately triggered an alert, notifying platform security personnel to pay attention to the target and prevent a derailment accident.

[0169] As can be seen from the above embodiments, the present invention, through scene context awareness and common sense knowledge base enhancement, can accurately distinguish between normal and abnormal behaviors of the same type (such as fast movement vs. crossing the yellow line) in the same scene, significantly reducing the false alarm rate, while maintaining high sensitivity to real high-risk behaviors.

[0170] This invention innovatively constructs a unified scene graph representation of scene context, target behavior, and object attributes. It utilizes a graph attention network for multi-layered relational reasoning to achieve a deep understanding of complex interactive relationships. A pre-built knowledge graph containing definitions of normal / abnormal behavior, risk levels, and object interaction rules for various scenarios is constructed. Common sense is injected into the model through knowledge embedding methods, enabling the system to possess scene-aware reasoning capabilities. By integrating three dimensions—visual anomality, behavioral anomality, and knowledge anomality—and dynamically adjusting the anomaly judgment criteria in conjunction with the scene context, a balance between low false alarm rate and high detection sensitivity is achieved. Through long-term temporal behavioral feature extraction and cross-frame target tracking, modeling the movement trajectory and behavioral development trends of personnel, it supports early warning of complex abnormal behaviors such as loitering, tailing, and gathering.

[0171] The above embodiments are merely typical illustrative methods of the present invention, and the scope of protection of the present invention is not limited thereto. All equivalent substitutions and improvements made under the concept of the present invention should fall within the scope of protection. It should be emphasized that any modifications or minor adjustments made by those skilled in the art without departing from the basic principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A scene-understanding enhanced intelligent monitoring and early warning system, characterized in that, It includes a scene semantic understanding module, a scene graph construction and relation reasoning module, a scene common sense knowledge base enhancement module, and a context-aware anomaly detection module; the scene semantic understanding module is used to extract scene context, target features, and behavioral features; The scene graph construction and relation reasoning module is used to organize discrete features into a structured scene graph and perform relation reasoning; the scene common sense knowledge base enhancement module is used to inject scene common sense knowledge. The context-aware anomaly detection module is used to comprehensively assess and warn of anomalies by integrating multi-dimensional information. Each module transmits information through scene context vectors and progressively enhanced node features, forming an end-to-end intelligent early warning process.

2. The scene-understanding enhanced intelligent monitoring and early warning system according to claim 1, characterized in that, The scene semantic understanding module is responsible for extracting multi-layered semantic information from the original surveillance video, providing a structured scene representation for subsequent modules. It comprises three closely related sub-modules: scene classification, object detection, and behavior recognition. The outputs of these three sub-modules together constitute the multi-layered semantic representation of the scene. Specifically, the scene context vector... Provides global scene context and target features Describe the static attributes and behavioral characteristics of each target. Depicting the dynamic behavioral patterns of the target; These three types of features are integrated into a unified node representation in the scene graph construction and relation reasoning module, realizing the transformation from discrete targets to structured scene graphs.

3. The scene-understanding enhanced intelligent monitoring and early warning system according to claim 2, characterized in that, The scene graph construction and relation reasoning module receives the scene context vector output by the scene semantic understanding module. Target feature set and behavioral feature set This discrete semantic information is organized into a structured scene graph representation, and multi-layer relational reasoning is performed through a graph neural network to obtain the final node features. This enables a deep understanding of the interaction relationships between targets and the overall semantics within a scene.

4. The scene-understanding enhanced intelligent monitoring and early warning system according to claim 3, characterized in that, The scene common sense knowledge base enhancement module receives node features output by the scene graph construction module. and scene context vector By injecting normal behavior patterns, abnormal behavior definitions, and object interaction rules from different scenarios into a pre-built scene knowledge graph, the system outputs knowledge-enhanced node features. This is used by the context-aware anomaly detection module.

5. The scene-understanding enhanced intelligent monitoring and early warning system according to claim 4, characterized in that, The context-aware anomaly detection module receives knowledge-enhanced node features. Scene context vector and scene graph structure By integrating scene semantics, target relationships, and common sense knowledge, anomaly assessment is performed on each target and the scene as a whole, and risk level and early warning information are output.

6. A scene-understanding enhanced intelligent monitoring and early warning method, based on the system as described in any one of claims 1-5, characterized in that, Includes the following steps: S1, Scene semantic understanding; S2, Scene Graph Construction and Relationship Reasoning; S3, Enhanced scenario common sense knowledge base; S4. Context-aware anomaly detection.

7. The scene-understanding enhanced intelligent monitoring and early warning method according to claim 6, characterized in that, Step S1 specifically includes: S11, Scene Classification and Context Coding First, the surveillance video is classified to identify the scene type of the current monitored area. A pre-trained visual Transformer model is used to extract global features from the video frames, and a scene classifier outputs the scene category probability distribution. Scene context vectors are then generated. The calculation formula is: in, Indicates the input video frame; ( ) represents the visual Transformer feature extraction function; and These are the weight matrix and bias vector of the scene classifier, respectively; Output feature dimensions for ViT; For scene context vectors; This represents the total number of scene categories. Scene context vector It not only identifies the scene type, but also serves as global context information for all subsequent modules; S12, Multi-target detection and attribute extraction Guided by the scene context, all targets in the video frame are detected and their attributes are extracted; improved YOLO or Faster R-CNN models are used to detect people, vehicles, and objects, and the location, category, and appearance features of each target are extracted. For the One detection target, , The total number of detected targets, its feature representation Based on location features Category characteristics and appearance features It is pieced together: in, Represents the center coordinates of the bounding box of the target. and size ; For category embedding features, the category ID is mapped to a dense vector through the embedding layer; Embed dimensions for categories; For depth appearance features extracted from the target region; For appearance features; This represents a vector concatenation operation; therefore, the total dimension of the target features is... ; S13. Spatiotemporal behavioral feature extraction To capture the dynamic behavior of the target, a 3D convolutional neural network or video Transformer is used to extract spatiotemporal features from consecutive video frames; for time windows... The video clip inside, number Spatiotemporal behavior characteristics of an individual target The calculation is as follows: in, Indicates the time window A continuous video frame sequence; ( ) represents the 3D convolutional feature extraction function, which is located at the target position. Extracting spatiotemporal features from the surrounding environment; This is a behavioral feature vector; This refers to the behavioral characteristic dimension.

8. The scene-understanding enhanced intelligent monitoring and early warning method according to claim 7, characterized in that, Step S2 specifically includes: S21, Scene Graph Construction Scene diagram From node set and edge set Composition: Nodes represent entities in the scene, and edges represent the relationships between entities; For detected The first objective is to analyze the target features from the scene semantic understanding module. Behavioral characteristics and scene context vector Perform fusion to generate initial node features; The initial feature calculation formula for each node is: in, and The weight matrix and bias vector for feature fusion; For node feature dimensions; Represents vector concatenation operation; scene context vector Features are incorporated into each node to ensure that the node representation includes contextual information; superscript This indicates that these are the initial node features before inference in the graph neural network; Edges are constructed based on the spatial distance and semantic association between targets; for nodes and nodes edge weight The calculation is as follows: in, For nodes and Relative position encoding between them; Encode the dimension for relative position; The parameter vector calculated for the edge weights; Using the Sigmoid activation function, the edge weights are normalized to... Interval; when Exceeding the preset threshold At the node and Establish an edge between them, and connect the edges Add edge set ; S22, Graph Attention Network Reasoning After constructing the scene graph, a graph attention network is used for multi-layer relationship reasoning, updating node features to incorporate information from neighboring nodes; the graph attention network has a total of Layer, number The formula for updating the node features of a layer is: in, Represents a node The set of neighboring nodes; For the first Layer weight matrix; For activation functions; The attention coefficient represents the node. For nodes Importance weights; attention coefficients are calculated using the attention mechanism and satisfy normalization conditions. Ensure that the sum of the contribution weights of all neighboring nodes is 1; go through After inference using the layered graph attention network, the final node features are obtained. Each Each node feature integrates information from its neighboring nodes, capturing the interaction relationships and overall semantics between targets in the scene; further, common-sense knowledge of the scene is injected.

9. The scene-understanding enhanced intelligent monitoring and early warning method according to claim 8, characterized in that, Step S3 specifically includes: S31, Construction of Scene Knowledge Graph Scene knowledge graph From entity set Relationship set and triple set Composition: Entities include scene type, behavior type, object type, and risk level; relationships include "scene-normal behavior", "scene-abnormal behavior", and "behavior-risk level". S32. Knowledge Embedding and Integration A knowledge graph embedding method is used to map entities and relations in the knowledge graph to a low-dimensional vector space; for triples The embedding vectors satisfy the translation relationship: in, These are the embedding vectors for the head entity, relation, and tail entity, respectively. Embedding dimensions for knowledge; Represents approximate equality relationships in a vector space; During anomaly detection, based on the current scene category Based on the detected behavior type, relevant common sense rules are retrieved from the knowledge graph; specifically, for the first... For each target, based on its behavioral characteristics and the identified behavioral category, the attribute of that behavior in the current scene is queried in the knowledge graph to obtain the corresponding knowledge embedding vector. Then through linear mapping Mapping it to the node feature space yields By fusing knowledge embedding vectors with scene graph node features, the semantic representation of nodes is enhanced. in, The node features output by the graph attention network; The mapped knowledge embedding vector; This is the knowledge fusion weighting coefficient, used to control the intensity of knowledge and information injection; These are the node features after knowledge enhancement.

10. The scene-understanding enhanced intelligent monitoring and early warning method according to claim 8, characterized in that, Step S4 specifically includes: S41. Anomaly Score Calculation For the One objective, from its enhanced node features Three dimensions of anomaly index are extracted; its comprehensive anomaly score is... By visual abnormality Degree of behavioral abnormality and knowledge anomaly The weighted fusion yields: in, Assess the overall anomaly score; Let be the weighting coefficient, satisfying This is used to balance the importance of the three dimensions; Visual abnormality The matching degree between the target's appearance features and the scene context is calculated to measure whether the target's appearance attributes are consistent with the current scene; behavioral anomaly degree. The degree of deviation between behavioral characteristics and normal behavioral patterns is calculated by comparing it with the pre-trained distribution of normal behavior; knowledge anomaly. The calculation based on the abnormal behavior definition in the knowledge graph directly reflects the knowledge base's judgment of the abnormality of the behavior in the current scenario. Overall scene anomaly By aggregating the anomalies of all targets and the anomalies of relationships between targets, we obtain: in, The first term is the relation anomaly weighting coefficient. Capture the most unusual target in the scene; second item Capture situations where there are significant differences in anomaly levels between targets; The edge weights are used to weight the differences in anomalies between different target pairs; S42. Risk Level Classification and Early Warning Based on the overall abnormality score of the scene The system categorizes risks into three levels: low, medium, and high. Low risk: Normal monitoring, no warning required; Medium risk: Issue alerts and record events; High risk: This will immediately trigger an alarm and notify security personnel. in, and As a risk threshold, .

Images