Airport side road monitoring event detection method based on deep learning

Abstract

The invention discloses an airport side road monitoring event detection method based on deep learning, and the method comprises the steps: processing a video frame, and constructing a dynamic time sequence scene graph containing entity nodes and multi-dimensional relation edges; learning a causal adjacency matrix representing a stable causal relationship between entities through a loss function including a data fitting item and a differentiable directed acyclic graph constraint item based on historical state changes; iteratively generating predicted node hiding states of a plurality of time steps in the future by using a graph neural network layer fusing spatial information and causal information; inputting the current and predicted node states into an event classifier, and respectively generating a detection result of a current event and a pre-judgment result of a future event; according to the method, causal logic of a scene is deeply mined, so that the accuracy and robustness of event detection are remarkably improved, conversion from passive response to active pre-judgment is realized, and interpretable causal attribution support is provided for decision making.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a deep learning-based method for detecting events on airport side roads. Background Technology

[0002] With the rapid development of deep learning technology, its application in video surveillance has become increasingly mature, especially in traffic incident detection, where it has shown great potential. Current mainstream technologies typically follow a multi-stage processing flow: using convolutional neural networks (CNNs)-based object detection models, such as YOLO (You Only Look Once) or Faster R-CNN series algorithms, to perform real-time localization and classification of key dynamic entities such as vehicles and pedestrians in video frames; subsequently, tracking algorithms, such as DeepSORT, combining deep correlation metrics or Kalman filtering techniques, are used to correlate the detected entities across frames to form stable motion trajectories; finally, heuristic rules are set for the kinematic parameters of these entities, such as trajectory, speed, and dwell time, or recurrent neural networks (RNNs) are used to perform temporal modeling of trajectory features to identify predefined events such as illegal parking, abnormal congestion, and crowd gatherings. This technology, to a certain extent, achieves automated monitoring and improves regulatory efficiency.

[0003] However, the aforementioned technologies have significant limitations when dealing with complex scenarios like airport side roads, which are characterized by high dynamics, high density, and strong interactions. First, existing technologies generally lack the ability to understand the deep relationships between entities within the scene. They typically treat each entity as an independent analytical unit, failing to effectively model the complex interactions and constraints between vehicles and passengers, passengers and luggage, and vehicles and specific areas (such as drop-off zones and no-parking zones). Consequently, they struggle to distinguish between seemingly similar but fundamentally different scenarios. Second, the models heavily rely on surface-level statistical correlations rather than underlying causal logic in their decision-making process. For example, a model might learn a strong correlation between "hazard lights flashing" and "illegal parking," but in areas where temporary parking is permitted, this false association can trigger numerous false alarms, as the model fails to understand that "prolonged traffic congestion" is the root cause of the event. Third, existing detection methods are essentially passive responses, only able to identify and issue alerts after an event occurs or persists for a period of time. They lack the ability to dynamically extrapolate and predict the development trend of events, missing the optimal opportunity for proactive intervention. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a deep learning-based method for airport-side road monitoring event detection to address the problems mentioned in the background section.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a deep learning-based method for detecting events on airport-side roads, comprising: Acquire continuous video frames from the surveillance video, process the continuous video frames, and construct a dynamic temporal scene graph representing the airport-side road scene. The dynamic temporal scene graph contains multiple time steps, and the scene graph of each time step consists of a set of nodes representing entities in the scene and a set of multidimensional relationship edges representing the relationships between the entities. Based on the state change data of entity nodes in the dynamic time series scene graph at multiple historical time steps, a causal adjacency matrix representing the stable causal relationship between the state changes of the entities is learned and generated through optimization using a preset loss function that includes data fitting terms and differentiable directed acyclic graph constraint terms. The node hidden state at the current time step is initialized and processed using a graph neural network layer that combines spatial information aggregation and causal information aggregation. This process iterates using the output of the previous prediction time step as the input of the next prediction time step to generate the predicted node hidden state for one or more future time steps. The node hiding state at the current time step and the predicted future node hiding state are respectively input into an event classifier to generate the detection result of the current event and the prediction result of the future event.

[0007] As a preferred embodiment of the deep learning-based airport-side road monitoring event detection method of the present invention, the step of constructing the dynamic temporal scene graph includes: The depth feature map of the video frame is extracted by sharing the backbone network; Pedestrians, vehicles, and luggage as dynamic entity nodes are detected in parallel from the depth feature map using a multi-head target detector, and an initial node feature vector containing appearance features, location encoding, kinematic features, and category embedding is generated for each dynamic entity node. Load predefined static environment nodes that contain geometric information and region types; By calculating the joint feature vector between any two node pairs, and using a relation classifier, relation edges representing spatial proximity, temporal association, or semantic interaction are dynamically generated for the node pairs.

[0008] As a preferred embodiment of the deep learning-based airport-side road monitoring event detection method of the present invention, the step of learning and generating the causal adjacency matrix includes: From the dynamic time-series scene diagram, preset key state variables are selected for each entity node, and a time series data matrix composed of the values ​​of the key state variables at multiple historical time steps is constructed. The data fitting term is used to minimize the error of linearly reconstructing the time series data matrix through the causal adjacency matrix; The differentiable directed acyclic graph constraint term is a function constructed based on matrix exponentiation and trace operation, and its function value reaches its preset minimum value only when the graph represented by the causal adjacency matrix is ​​a directed acyclic graph.

[0009] As a preferred embodiment of the deep learning-based airport-side road monitoring event detection method of the present invention, wherein: the graph neural network layer is a gated graph neural network layer, and its processing includes: By using the first aggregation path based on spatial adjacency and the second aggregation path based on the causal adjacency matrix, information aggregation of the node hidden state at the current time step is performed to obtain spatial aggregation features and causal aggregation features. The spatial aggregation feature and the causal aggregation feature are concatenated to form a joint information feature; The joint information features are fused with the hidden state of the node in the previous layer through a gating mechanism that includes update gates and reset gates to generate the hidden state of the output node in the current layer.

[0010] As a preferred embodiment of the deep learning-based airport-side road monitoring event detection method of the present invention, the method further includes a causal attribution step, which is triggered after a specific event is detected or predicted, including: Calculate the gradient of the output of the event classifier corresponding to the specific event with respect to the hidden state of the node input to the classifier, and backpropagate the gradient to each node in the scene graph to obtain the initial importance score of each node; On the causal graph represented by the causal adjacency matrix, starting from the node with the highest initial importance score, a reverse path is traced along the causal relationship edges to identify one or more causal chains that led to the occurrence of the specific event. By combining the nodes and their state information on the causal chain, an attribution explanation for the specific event is generated.

[0011] As a preferred embodiment of the deep learning-based airport-side road monitoring event detection method of the present invention, the method further includes a counterfactual analysis step, which is performed after the attribution explanation is generated, and includes: Receive instructions from the user to intervene in the state of a cause node in the causal chain; According to the intervention instruction, modify the value in the feature vector of the cause node in the scene graph at the corresponding time step; The modified scene graph is input into the graph neural network layer for processing, a new forward propagation is performed, and a counterfactual event prediction result is generated. The counterfactual event prediction results are compared with the original event prediction results.

[0012] As a preferred embodiment of the deep learning-based airport side road monitoring event detection method described in this invention, the kinematic features in the initial node feature vector are obtained by associating the entity detected in the current frame with the same entity in the historical frame through a lightweight target tracker, and calculating the instantaneous velocity and acceleration information based on its historical trajectory.

[0013] As a preferred embodiment of the deep learning-based airport side road monitoring event detection method of the present invention, the spatial adjacency relationship used in the first aggregation path is obtained by symmetric normalization of the matrix representing the spatial proximity probability between nodes output by the relation classifier.

[0014] As a preferred embodiment of the deep learning-based airport-side road monitoring event detection method described in this invention, the event classifier further includes a graph pooling sub-step before processing the hidden state of nodes. This sub-step generates a fixed-dimensional global graph representation vector representing the entire graph state by performing average pooling or attention-weighted pooling on the hidden state vectors of all nodes in the scene graph at a time step.

[0015] As a preferred embodiment of the deep learning-based airport-side road monitoring event detection method described in this invention, the training process of the method is optimized end-to-end through a joint loss function, which includes: An event classification loss term used to monitor the accuracy of event detection and prediction; A future state prediction loss term used to monitor the accuracy of future node state predictions; In addition, a causal discovery loss term is used to optimize the causal adjacency matrix.

[0016] Compared with existing technologies, the beneficial effects of this solution are: 1. This invention constructs a dynamic temporal scene graph, unifying dynamic entities (pedestrians, vehicles, etc.) and static environmental elements (functional areas) in the scene into nodes, and dynamically generates semantically rich multidimensional relationship edges. This overcomes the limitations of existing technologies that treat entities as isolated units for analysis, and achieves a global and structured understanding of airport-side road scenes. As a result, it can accurately distinguish complex events that appear similar but have different underlying logics (such as normal temporary parking and illegal lane occupation), improving the accuracy and robustness of detection. 2. This invention introduces a causal discovery mechanism. By learning an adjacency matrix that represents the stable causal relationship between changes in entity states, the model's decision-making basis shifts from surface-level data correlation to underlying causal logic. This enhances the model's generalization ability and reliability when facing unseen or deceptive scenarios, and effectively reduces false positives and false negatives caused by misunderstanding the nature of the scenario. 3. In addition, this invention combines a graph neural network that aggregates spatial and causal information and adopts an iterative processing method to realize the dynamic extrapolation and prediction of the state of scenarios at multiple future time steps. It can not only detect events that have already occurred, but also predict the development trend of events in advance (such as impending congestion), realizing a shift from a "passive response" to a "proactive early warning" mode, and providing managers with valuable decision-making lead time. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating the overall process of an airport-side road monitoring event detection method based on deep learning, as described in one embodiment of the present invention. Detailed Implementation

[0018] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0019] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0020] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0021] This invention is described in detail with reference to the schematic diagrams. When describing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0022] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0023] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] Example 1 Reference Figure 1 This is the first embodiment of the present invention, which provides a deep learning-based method for detecting events on airport-side roads, including: S1. Obtain continuous video frames from the surveillance video, process the continuous video frames, and construct a dynamic temporal scene graph representing the airport-side road scene. The dynamic temporal scene graph contains multiple time steps, and the scene graph of each time step consists of a set of nodes representing entities in the scene and a set of multi-dimensional relationship edges representing the relationships between entities. It should be noted that this step aims to transform the raw, unstructured video pixel data stream into a series of structured graph data rich in semantic information, and to use this transformation process as the basis for subsequent causal reasoning and event prediction. Furthermore, regarding the time step Acquired video frames Generate a corresponding scene graph ,in, Let be the set of nodes at that moment. It is an edge set; Specifically, the input video frames (in, and The height and width of each frame are fed into a pre-trained convolutional neural network (CNN) as a shared backbone network. In this embodiment, in addition to the above-mentioned convolutional neural network, neural networks such as ResNet-101 or more advanced SwingTransformer can also be used. It should be explained that the role of this backbone network is to perform deep feature extraction on the image, thereby generating a multi-scale deep feature map containing rich spatial and semantic information. This provides a high-quality visual representation foundation for subsequent entity detection and relationship judgment, and improves the computational efficiency of the entire model. Furthermore, when obtaining multi-scale deep feature maps Subsequently, a multi-head detector is used to perform parallel detection of key dynamic entities in the scene. In this embodiment, the detector is preferably based on a query architecture, such as DETR (DEtection Transformer) or its variants. The advantage of choosing this architecture is that it can output detection results end-to-end, avoiding the cumbersome non-maximum suppression (NMS) post-processing in traditional detectors. The detector's multiple "heads" are responsible for detecting different categories of entities. In the airport-side road application scenario of this invention, these categories of entities mainly include pedestrians, vehicles, and luggage. For the detected... Create a node for a dynamic entity. And generate a corresponding one Initial node feature vectors of dimension This feature vector is a multi-part concatenated vector designed to comprehensively describe the state of the entity: in, The appearance feature is directly taken from the final embedding vector of the Object Query corresponding to the entity output by the DETR decoder. This vector condenses the visual appearance information of the entity. This is a location encoding, which is the normalized coordinates of the entity's bounding box. (Center point coordinates, width, height) are encoded using a sinusoidal position encoding function or a small multilayer perceptron (MLP), enabling the neural network to understand the spatial location of entities in the image; For kinematic features, this is achieved through a lightweight target tracker (e.g., a simple online tracker based on IOU or appearance similarity), by using the current frame. Detected entities and historical frames By associating the same entity with its identity (ID), the short-term motion trajectory of that entity can be constructed. Based on this trajectory, kinematic information such as the instantaneous velocity, acceleration, and orientation angle of the entity at the current moment can be calculated. For example, the instantaneous velocity can be obtained by calculating the bit removal of the entity's center point between the two most recent frames by time interval, while the acceleration can be approximated by performing a first-order time difference on the velocity. This information is crucial for understanding the dynamic behavior of the entity. It should be emphasized that in the first frame of the sequence, this feature vector can be initialized to a zero vector. Category embedding aims to transform the entity category labels (such as "pedestrian", "vehicle", "luggage") output by the detector into a dense vector representation through a learnable embedding layer; In addition to the dynamic entities in the scene, the static environmental elements of the airport side roads also have a key impact on the occurrence of events. Therefore, it is also necessary to load a set of predefined static environmental nodes. These static environmental nodes represent fixed functional areas in the scene, such as "Departure Level - Lane 1", "Arrival Level - Taxi Waiting Area", "Pedestrian Crossing", "No Parking Zone", etc. The geometric information (such as polygon vertex coordinates) and area type (such as "Temporary Parking Area") of these functional areas can be loaded from pre-annotated Geographic Information System (GIS) data or scene configuration files. Furthermore, each static node also has a feature vector, which is composed of the positional encoding of its geometric information and the category embedding of the area type. It is important to emphasize that these static nodes remain unchanged in all time steps, serving as "anchor points" of the scene graph and providing context for understanding the interaction between entities and the environment. Furthermore, after defining the complete node set After (including dynamic entity nodes and static environment nodes), it is necessary to construct the edge set of relationships between node pairs. Unlike traditional graph construction methods that only consider spatial proximity, the present invention employs a dynamic, semantic-based relation generation mechanism for node sets. Any pair of nodes in Treating it as a potential relation candidate, in order to determine whether a node pair exists and what kind of relationship exists, a joint feature vector is first extracted. This vector contains sufficient information to describe the relationship between the two nodes: in, This indicates stopping gradient operations, meaning that node features will not be updated during the relationship determination phase; It is the appearance feature of the region covering the minimum bounding rectangle of the two nodes, and It is a node encoding Relative to node The relative position encoding of direction and distance; Furthermore, the joint feature vector The input is fed into a pre-trained relation classifier (usually an MLP), whose output is a probability distribution representing the confidence level of various possible relation types between the nodes, such as "spatial proximity," "contact," "direction," "belonging to" (e.g., luggage belongs to a pedestrian), "within" (e.g., a vehicle is in a no-parking zone), etc. Then, based on the relation classifier's output, a multi-dimensional, weighted relation edge set can be constructed, which is represented as one or more adjacency matrices, such as a spatial proximity matrix. Its elements The value is the probability of the "spatial proximity" relationship predicted by the relation classifier; It should be noted that by constructing the node feature vectors and relation edge sets described above, the constructed scene graph can be... It not only describes the positional relationships but also reveals the semantic interactions between entities, providing a foundation for subsequent causal analysis and event prediction. S2. Based on the state change data of entity nodes in the dynamic time series scene graph at multiple historical time steps, a causal adjacency matrix representing the stable causal relationship between entity state changes is learned and generated by optimizing a pre-set loss function that includes data fitting terms and differentiable directed acyclic graph constraint terms. It should be noted that this step aims to learn a global causal adjacency matrix that remains stable over a period of time. This is to improve the model's generalization ability, robustness, and interpretability; where the non-zero elements of the matrix... Represents the entity State changes affect entities The changes in state have a direct causal effect; Furthermore, from the generated dynamic temporal scene graph sequence In the middle, for all in the scene Key state variables for causal discovery are extracted from each entity (including dynamic entities and static nodes). These key state variables are predefined according to the entity type and can be directly calculated from the node's feature vector or through the relationships between nodes. For example: For vehicle nodes: their key state variables may include: instantaneous velocity and acceleration directly obtained from the kinematic feature part of their node feature vector; cumulative dwell time in a specific area (such as a no-parking zone) calculated by querying the static region nodes connected by the "within..." relation edge; and the headway calculated by finding another vehicle node that is "spatially adjacent" to them in the scene graph and is located directly in front of them. For pedestrian nodes: their key state variables may include: walking speed obtained from their kinematic features; distance to luggage calculated by finding luggage nodes connected by the "belong to" relation edge; and all other key state variables may be calculated. The total of individual entities Key state variables in the past The values ​​of each historical time step are collected and organized to construct a time series data matrix. The time series data matrix Each line represents the time step all The observed values ​​of the key state variables are listed in column 1, with each column representing the observed values ​​of the key state variables. Key state variables in The trajectory of change within a time step; Furthermore, the causal discovery problem is transformed into a continuously optimized objective, enabling seamless integration with deep learning frameworks. Simultaneously, this is done to learn the causal adjacency matrix. The present invention also constructs a causal discovery loss term. This function is composed of data fitting terms. Constraints of a Differentiable Directed Acyclic Graph (DAG) Linear weighted composition: in, It is a hyperparameter used to balance the importance between data fitting and structural constraints, and its range is [0, 1]. Specifically, this data fitting term To ensure that the learned causal graph can well explain the observed data, this embodiment employs the assumption of a linear structural equation model, namely that the value of a variable can be linearly represented by the values ​​of its direct cause variables. Therefore, this data fitting term aims to minimize the causal adjacency matrix. For time series data matrix The error in linear reconstruction, specifically, the data fitting term, is defined as the least squares loss, typically measured using the Frobenius norm: in, It is the observed time series data matrix. It is the causal adjacency matrix to be learned. yes The transpose of the matrix, product Indicates causal structure Data Linear prediction or reconstruction; The term represents the square of the Frobenius norm of the matrix, which is the sum of the squares of all elements in the matrix. Minimizing this term drives the model to learn a causal adjacency matrix that can linearly predict the "outcome" variable from the "cause" variable. ; Specifically, for the constraint terms of a differentiable directed acyclic graph (DAG) An effective causal graph must be directed acyclic, meaning it does not contain cyclic dependencies such as A leading to B, B leading to C, and C leading to A. Traditional causal discovery algorithms typically use combinatorial search to ensure acyclicity, resulting in significant computational overhead. Therefore, this invention employs a fully differentiable constraint term, transforming the discrete graph structure constraint into a continuously optimizable function. This function is constructed based on matrix exponentiation and trace operations. in, The trace of a matrix is ​​the sum of the elements on its main diagonal. The matrix exponentiation operation is defined as follows: ; Hadamard product, which is the element-wise multiplication of matrices. What we get is a set of elements that are A matrix containing the squares of corresponding elements; It refers to the number of key state variables, which can be represented as a matrix here. The dimension; It should be noted that this function utilizes the relationship between the power of the adjacency matrix of the graph and the number of its paths, if and only if the adjacency matrix... When the represented graph is a directed acyclic graph (DAG), The value is 0 (i.e., its preset minimum value). If any loop exists in the graph, then The value will be greater than 0, because the matrix exponent, trace, and Hadamard product operations are all differentiable; therefore, the entire... Function for Since it is differentiable, during the optimization process, we minimize... Gradient descent can then be used to effectively force the learned adjacency matrix to... It converges to a solution that satisfies the acyclic constraint; Furthermore, by minimizing the aforementioned causal discovery loss term using a gradient-based optimization algorithm such as Adam, the optimal causal adjacency matrix can be obtained. After optimization, the resulting matrix That is, it is a dense real matrix whose elements The absolute value of a variable represents its magnitude. For variables The strength of causal influence is usually determined by setting a threshold, and elements with absolute values ​​less than the threshold are set to 0, thus obtaining a sparse and more interpretable causal adjacency matrix. This matrix can reveal the intrinsic driving logic of entity state evolution in the airport side road scenario, and can provide stable and profound causal prior knowledge for subsequent graph neural network information aggregation and event prediction. S3. Initialize the node hidden state at the current time step. Process it using a graph neural network layer that combines spatial information aggregation and causal information aggregation. This process iterates using the output of the previous prediction time step as the input of the next prediction time step to generate the prediction node hidden state for one or more future time steps. It should be noted that the goal of this step is to use the causal prior knowledge learned in the causal adjacency matrix, combined with the explicit spatial relationships in the scene graph, to dynamically infer and predict the future scene state. To this end, this invention designs a graph neural network (GNN) architecture that can simultaneously model "visible" spatial interactions and "invisible" causal transmission within a unified framework. Furthermore, regarding the current time step , for scene diagram Each node in Its hidden state vector is initialized, and this initial hidden state comes from the initial node feature vector generated for this node in step S1. ; Furthermore, a special graph neural network layer (i.e., a gated causal spatial graph neural network layer) is designed. This layer can aggregate information of neighboring nodes in parallel from two different information dimensions (spatial dimension and causal dimension) and adaptively fuse these two information dimensions using a gating mechanism. In this embodiment, based on the idea of ​​gated graph neural network (GGNN), the graph neural network layer is extended into a dual-path aggregation structure. Specifically, for the first... Layer, which is related to nodes Hidden state update process The aim is to divide the information aggregation process into two parallel paths: First aggregation path (spatial information aggregation): This path aims to model direct interactions between entities based on physical location, that is, using a standard spatial adjacency matrix. To guide the dissemination of information; this matrix It is obtained by symmetric normalization of the matrix representing the probability of "spatial proximity" between nodes output by the relation classifier in step S1; it needs to be explained that the normalization process ( ,in, yes The degree matrix, It is after normalization. This helps stabilize numerical values ​​during information aggregation, where the spatial aggregation feature... Through aggregation nodes The hidden state of the spatial neighbors is used for calculation: in, It is a node Neighbor set on a spatial graph It is the first Learnable weight matrix of layer space path; The second aggregation path (causal information aggregation): This path aims to model the potential causal effects between entity states that span spatial distances; it utilizes the causal adjacency matrix learned in step S2. This causal aggregation feature guides information dissemination. By aggregating nodes The hidden states of nodes whose states have a direct causal influence are calculated as follows: in, These are elements of the causal adjacency matrix, representing nodes. For nodes The strength of causal influence It is the first The learnable weight matrix of the causal path at each layer; it should be noted that the summation in this formula applies to all nodes, because causal relationships may exist between any two entities, without being limited by spatial distance; Furthermore, after obtaining spatial aggregation features and causal aggregation features Then, they are concatenated into a joint information feature vector. Subsequently, through a gating mechanism similar to a gated circular unit (GRU), this new joint information is linked to the hidden state of the previous layer. The output hidden state of the current layer is calculated by fusion. It should be emphasized that this gating mechanism includes an update gate. and a reset door : in, It is the Sigmoid activation function. All of these are learnable parameters; it needs to be explained that the update gate determines how much old state information is retained, while the reset gate determines how much old state information is ignored when calculating candidate states. Furthermore, calculate the candidate hidden states. : in, It is the hyperbolic tangent activation function; Furthermore, by updating the door Linear combination of the hidden states of the previous layer and candidate hidden state The final output hidden state can then be obtained. : It should be noted that, through stacking With such a gated causal spatial graph neural network layer, the model can capture the relationships between nodes. The complex spatiotemporal and causal dependencies of order; In addition, in order to generate predictions for multiple future time steps, the present invention also employs an auto-regressive iterative processing method. Specifically, the current time step Initial node hidden state Input to stack In a layer-gated causal spatial graph neural network, the hidden state of the final node at the current time step is obtained. Subsequently, in order to predict the first time step in the future... In this state, we will output the state of the gated causal spatial graph neural network layer. As input for the next prediction time step, it is again forward-propagated through the same gated causal spatial graph neural network layer to obtain the result. Hidden state of the predicted node at time step It is important to emphasize that this process can be repeated iteratively. Next, to generate the future The predicted hidden state of the node at the nth time step, i.e., the nth time step. The predicted state at the nth future time step is based on the nth The predicted state is generated for each future time step: in, The graph neural network, representing the entire stack, uses this iterative approach to simulate the dynamic process of scene state evolution over time and capture the complete trajectory of event development. S4. Input the node hidden state at the current time step and the predicted future node hidden state into an event classifier to generate the detection result of the current event and the prediction result of the future event. It should be noted that this step aims to decode the node-level representations generated in step S3, which contain rich spatiotemporal and causal information, into scene-level, understandable semantic event tags. Furthermore, before sending node information to the event classifier, a graph pooling sub-step is performed to hide the states of a variable number of nodes in the scene graph at a given time step. They are aggregated into a fixed-dimensional global graph representation vector that represents the state of the entire graph (i.e., the entire scene). It is important to emphasize that this is necessary because scene-level event classifiers require a constant-dimensional input, and this embodiment provides two optional graph pooling implementations: Average pooling is a simple and effective method that directly calculates the arithmetic mean of the final hidden state vectors of all nodes in the scene graph. in, It is the current time step The global graph representation vector, It is the total number of nodes in the scene graph at that time step. It is the node output by step S3. At time step The final hidden state; Attention-weighted pooling: To enable the model to adaptively focus on nodes that are more important for judging the current event (e.g., when judging "illegal parking," more attention is paid to stationary vehicle nodes and no-parking zone nodes), the present invention preferably employs a self-attention-based pooling mechanism, which calculates an attention weight for each node. Then, a weighted sum is calculated based on the weights: in, It is a learnable query vector that represents the "global context". and These are learned weight matrices. and The key vector and value vector obtained after linear transformation of the hidden state of a node; It is the dimension of the key vector, used to scale the dot product result and prevent gradient vanishing; The operation is designed to ensure that the sum of the attention weights of all nodes is 1; It should be noted that, through this graph pooling sub-step, for the current time step... Hidden state of nodes Its global graph representation can be obtained. Similarly, for the future predicted in step S3... Hidden state of nodes at each time step The corresponding predicted global graph representation can also be obtained. ; Furthermore, after obtaining a series of global graph representation vectors representing the current and future scene states... Then, they are each input into an event classifier with shared weights; Specifically, this event classifier In this embodiment, a multilayer perceptron (MLP) is used, consisting of several fully connected layers and nonlinear activation functions (such as ReLU) stacked together, and finally a softmax activation function is applied to output a probability distribution across all predefined event categories, which can be represented as: Furthermore, the global graph representation of the current time step is... Input the classifier to obtain the event probability distribution at the current time step. : in, It is the total number of event categories (e.g., "normal traffic", "illegal parking", "abnormal congestion", "crowd gathering" etc.). It is the current moment. The occurrence of the first The probability of the event type, and the final detection result. That is, the category with the highest probability: Furthermore, the global graph for each future prediction time step is represented as follows: (in The data are then input into the same classifier to obtain the event probability distribution for future time steps. : in, This is expressed as the probability of an event at a future time that belongs to the first type of event; Furthermore, the final prediction result For the future The event category with the highest probability at any given moment: It should be noted that, through the analysis of By outputting all the predictions, a dynamic predictive sequence about the evolution of future events can be obtained; Furthermore, the entire model training process is conducted through a joint loss function. End-to-end optimization is performed. Specifically, the joint loss function It can be represented as: in, The weights of each loss are hyperparameters used to balance the importance of different learning tasks, and ; Furthermore, the training of the event classifier is then determined by the event classification loss term in the joint loss function. To supervise the process, the loss term uses the standard cross-entropy loss, which measures the difference between the event probability distribution predicted by the model and the true event labels. The loss term is composed of the loss from current event detection and the loss from future event prediction. in, It is a weighting coefficient that can be used to assign different levels of importance to predictions of the future at different distances. For example, it can be set to a random number. Increased attenuation coefficient (e.g.) This indicates that the closer the future prediction, the more important it is; The classification loss at the current time step: in, It is a one-hot vector if the real event is a category If the value is 1, then its value is 1; otherwise, it is 0. Specifically, Is the future number Predicted loss at each time step: in, It is the future Real-time event tags; It is the future The moment occurred The probability of a class of events is the predicted value; It should be noted that by minimizing The backpropagation algorithm can be used to update the parameters of all modules, including the event classifier and the upstream graph neural network, so that the model can learn how to accurately identify and predict various airport-side road events from complex scene graphs. Specifically, for the future state prediction loss term This term is used to directly supervise the prediction accuracy of the graph neural network for the hidden states of future nodes in step S3. It does this by minimizing the predicted hidden states of future nodes. Hidden state of real future nodes The difference between them is used to achieve this; in this embodiment, the mean squared error (MSE) loss is used for calculation. in, It is the total number of predicted future time steps. It is the future Minimizing the number of nodes at any given time can force the graph neural network to learn the true dynamic laws governing the evolution of the scene state over time. Specifically, regarding the causal discovery loss item This item In end-to-end training, (The data fitting term) can be considered an auxiliary task, while The (differentiable directed acyclic graph constraint term) serves as a structure regularization term for the causal adjacency matrix. During joint training, this term ensures that the causal relationships upon which the graph neural network depends are stable and acyclic. It should be noted that minimizing the total loss is achieved by using gradient descent optimizers such as Adam. All learnable parameters of the model, including the backbone network, detector, classifier, graph neural network, and causal adjacency matrix, will be jointly optimized. In addition, in order to solve the "black box" problem that is common in deep learning models, when an event alarm is output (for example, predicting that "abnormal congestion" will occur in 10 seconds), it is necessary to automatically trace its root cause and present it in an understandable way, so as to provide a basis for operators' decision-making. This process needs to be triggered after a specific (usually abnormal) event is detected or predicted. Furthermore, for a specific event that has been detected or predicted (e.g., categorized as...), Its output logit value before the Softmax layer of the event classifier is To measure the contribution of each node in the scene graph to the occurrence of this event, it is necessary to calculate... The global graph representation vector input to the classifier gradient At the same time, due to It is determined by the final hidden state of all nodes. The gradient obtained through graph pooling can be backpropagated to each node using the chain rule to obtain the initial importance score for each node. : in, The L1 norm represents the importance score that quantifies the nodes. The final state of the event The higher the score of a node's "contribution," the more directly and critically it is considered to be the "manifest" entity that triggered the event. Furthermore, since the contribution of the apparent entities alone is insufficient, the causal adjacency matrix learned in step S2 also needs to be utilized. The causal graph represented by it starts from the one with the highest initial importance score. The node (or the top score) (nodes) We begin by traversing backwards along the directed edges of the causal graph; Specifically, if there is an edge in the causal graph (Right now ), then Add it to the tracing path and recursively start from Continue tracing backwards until there are no predecessor nodes or the preset maximum tracing depth is reached. Through this process, one or more causal chains that converge to key manifestation nodes can be identified. Furthermore, the system combines all nodes in the identified causal chain with their state information at key moments (e.g., node type, speed, dwell time, etc.) and uses a preset natural language template to generate attribution explanations. For example, for an "abnormal congestion" event, if the causal chain traced back is: "no-stopping zone"... Vehicle A For "Vehicle B", the following explanation can be generated: "Event warning: Abnormal congestion; Cause analysis: "Vehicle A" stayed in the "no-stopping zone" for too long (state), which caused "Vehicle B" behind to have a speed that was consistently below 5km / h (state), resulting in congestion." Furthermore, after generating attribution explanations, counterfactual analysis is performed to conduct decision deduction; Specifically, in this embodiment, a "what-if" interactive analysis method is adopted, allowing operators to verify the effectiveness of their intervention measures. This analysis method is triggered when the user interacts with the system. First, the generated causal chain is displayed to the user, allowing them to virtually intervene in the state of a specific cause node on the chain. For example, the user can issue a command such as, "Suppose we change the speed of vehicle A to 30 km / h" or "Suppose we remove vehicle A from the scene." Then, based on the user's intervention command, the scene graph at the corresponding time step is modified. For instance, if the command is to modify the speed, the feature vector of the vehicle A node is located, and its kinematic features are modified. If the command is to remove a node, all information related to vehicle A is removed from the node set and edge set at that time step. Finally, this "intervened" and modified scene graph is used as the initial state and re-inputted into the system. In step S3, the pre-trained graph neural network model is used to perform a complete forward propagation (including iterative prediction of future states and event classification in step S4). This process generates a sequence of counterfactual event prediction results under virtual intervention. Finally, the counterfactual event prediction results are compared side-by-side with the original (uninterventional) event prediction results. For example, if the original prediction is "85% probability of congestion in 10 seconds", the counterfactual prediction becomes "15% probability of congestion in 10 seconds". Through this comparison, operators can intuitively verify whether their intervention measures (such as sending someone to drive away vehicle A) can effectively alleviate the risk of future congestion, thereby making more scientific decisions.

[0025] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0026] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0027] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0028] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0029] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0030] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A deep learning-based event detection method for airport-side road monitoring, characterized in that, include: Acquire continuous video frames from the surveillance video, process the continuous video frames, and construct a dynamic temporal scene graph representing the airport-side road scene. The dynamic temporal scene graph contains multiple time steps, and the scene graph of each time step consists of a set of nodes representing entities in the scene and a set of multidimensional relationship edges representing the relationships between the entities. Based on the state change data of entity nodes in the dynamic time series scene graph at multiple historical time steps, a causal adjacency matrix representing the stable causal relationship between the state changes of the entities is learned and generated through optimization using a preset loss function that includes data fitting terms and differentiable directed acyclic graph constraint terms. The node hidden state at the current time step is initialized and processed using a graph neural network layer that combines spatial information aggregation and causal information aggregation. This process iterates using the output of the previous prediction time step as the input of the next prediction time step to generate the predicted node hidden state for one or more future time steps. The node hiding state at the current time step and the predicted future node hiding state are respectively input into an event classifier to generate the detection result of the current event and the prediction result of the future event.

2. The airport-side road monitoring event detection method based on deep learning as described in claim 1, characterized in that, The steps for constructing the dynamic temporal scene graph include: The depth feature map of the video frame is extracted by sharing the backbone network; Pedestrians, vehicles, and luggage as dynamic entity nodes are detected in parallel from the depth feature map using a multi-head target detector, and an initial node feature vector containing appearance features, location encoding, kinematic features, and category embedding is generated for each dynamic entity node. Load predefined static environment nodes that contain geometric information and region types; By calculating the joint feature vector between any two node pairs, and using a relation classifier, relation edges representing spatial proximity, temporal association, or semantic interaction are dynamically generated for the node pairs.

3. The airport-side road monitoring event detection method based on deep learning as described in claim 1, characterized in that, The steps of learning and generating the causal adjacency matrix include: From the dynamic time-series scene diagram, preset key state variables are selected for each entity node, and a time series data matrix composed of the values ​​of the key state variables at multiple historical time steps is constructed. The data fitting term is used to minimize the error of linearly reconstructing the time series data matrix through the causal adjacency matrix; The differentiable directed acyclic graph constraint term is a function constructed based on matrix exponentiation and trace operation, and its function value reaches its preset minimum value only when the graph represented by the causal adjacency matrix is ​​a directed acyclic graph.

4. The airport-side road monitoring event detection method based on deep learning as described in claim 1, characterized in that, The graph neural network layer is a gated graph neural network layer, and its processing includes: By using the first aggregation path based on spatial adjacency and the second aggregation path based on the causal adjacency matrix, information aggregation of the node hidden state at the current time step is performed to obtain spatial aggregation features and causal aggregation features. The spatial aggregation feature and the causal aggregation feature are concatenated to form a joint information feature; The joint information features are fused with the hidden state of the node in the previous layer through a gating mechanism that includes update gates and reset gates to generate the hidden state of the output node in the current layer.

5. The airport-side road monitoring event detection method based on deep learning as described in any one of claims 1 to 4, characterized in that, The method also includes a causal attribution step, which is triggered upon detection or prediction of a specific event, including: Calculate the gradient of the output of the event classifier corresponding to the specific event with respect to the hidden state of the node input to the classifier, and backpropagate the gradient to each node in the scene graph to obtain the initial importance score of each node; On the causal graph represented by the causal adjacency matrix, starting from the node with the highest initial importance score, a reverse path is traced along the causal relationship edges to identify one or more causal chains that led to the occurrence of the specific event. By combining the nodes and their state information on the causal chain, an attribution explanation for the specific event is generated.

6. The airport-side road monitoring event detection method based on deep learning as described in claim 5, characterized in that, The method further includes a counterfactual analysis step, performed after the attribution explanation is generated, including: Receive instructions from the user to intervene in the state of a cause node in the causal chain; According to the intervention instruction, modify the value in the feature vector of the cause node in the scene graph at the corresponding time step; The modified scene graph is input into the graph neural network layer for processing, a new forward propagation is performed, and a counterfactual event prediction result is generated. The counterfactual event prediction results are compared with the original event prediction results.

7. The airport-side road monitoring event detection method based on deep learning as described in claim 2, characterized in that, The kinematic features in the initial node feature vector are obtained by associating the entity detected in the current frame with the same entity in the historical frames using a lightweight target tracker, and calculating the instantaneous velocity and acceleration information based on its historical trajectory.

8. The airport-side road monitoring event detection method based on deep learning as described in claim 4, characterized in that, The spatial adjacency relationship used in the first aggregation path is obtained by symmetric normalization of the matrix representing the spatial proximity probability between nodes output by the relationship classifier.

9. The airport-side road monitoring event detection method based on deep learning as described in claim 1, characterized in that, Before processing the hidden state of nodes, the event classifier also includes a graph pooling sub-step. This sub-step generates a fixed-dimensional global graph representation vector that represents the state of the entire graph by performing average pooling or attention-weighted pooling on the hidden state vectors of all nodes in the scene graph at a time step.

10. The airport-side road monitoring event detection method based on deep learning as described in claim 1, characterized in that, The training process of the method is optimized end-to-end using a joint loss function, which includes: An event classification loss term used to monitor the accuracy of event detection and prediction; A future state prediction loss term used to monitor the accuracy of future node state predictions; In addition, a causal discovery loss term is used to optimize the causal adjacency matrix.

Images