Vision-space-time multi-mode fusion detection method for linkage target in airport flying area

Abstract

The invention relates to a vision-space-time multi-mode fusion detection method for linkage targets in an airport flying area, belongs to the technical field of intelligent multi-mode perception and space-time data analysis of large airports, and solves the problem that high-precision detection of tiny targets is difficult to realize in a complex environment in the prior art. Obtaining and outputting space-time diagram features; s2, constructing a linkage target semantic graph, and obtaining and outputting semantic graph features; s3, visual-space-time fusion features are generated, an optimized model is obtained through training, and fusion features are obtained through the optimized model and serve as the visual-space-time fusion features; and S4, inputting the obtained fusion features into a downstream identification module to complete target identification and classification.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent multimodal perception and spatiotemporal data analysis for large airports, and in particular to a vision-spatiotemporal multimodal fusion detection method for linked targets in airport flight zones. Background Art

[0002] Airport airfield operations involve the coordinated operation of multiple dynamic targets, including aircraft, ground handling vehicles, support equipment, and personnel. As the core area for aircraft takeoff, landing, taxiing, and transit, airfield security surveillance is crucial to airport operational efficiency and aviation safety. However, existing surveillance technologies face multiple challenges in practical application. First, airfields cover a vast area and experience variable lighting conditions (e.g., day-night cycles, rain, snow, fog, and haze). Traditional object detection algorithms based on a single visual modality, such as visible light cameras, are susceptible to environmental noise, resulting in a high rate of missed detection of small targets (e.g., ground signs, small equipment, or personnel). Second, the spatiotemporal correlations among dynamic targets are not fully explored. For example, existing methods often use independent target detection and tracking frameworks, which make it difficult to characterize the spatiotemporal coupling between targets. Furthermore, in airfield scenarios, multiple targets, including aircraft, ground handling vehicles, support equipment, and personnel, frequently interact with each other. Traditional methods ignore these interactions and struggle to capture their inherent connections.

[0003] Therefore, coordinated target surveillance in airport flight zones is an urgent problem to be solved. The emergence of multimodal fusion representation technology has provided a new approach to this problem. By combining multi-source sensor data, such as spatiotemporal trajectories, the limitations of single-modality perception can be overcome. For example, spatiotemporal trajectories provide precise spatial coordinate information. However, existing fusion methods are mostly limited to feature-level splicing or decision-level weighting, failing to achieve deep coupling of cross-modal spatiotemporal correlations. In airport flight zone scenarios, the coordinated behavior of dynamic targets exhibits significant temporal dependencies and spatial constraints (e.g., aircraft taxi paths and ground vehicle avoidance rules). Relying solely on visual features or discrete time series modeling cannot meet the requirements of high-precision situational awareness. Furthermore, for the recognition of small targets, existing methods often rely on high-resolution image input or super-resolution reconstruction. However, due to the information redundancy of consecutive video frames, high-precision recognition is difficult to achieve in large-scale flight zone monitoring systems.

[0004] The Chinese invention patent application, publication number CN111339313A, titled "A knowledge base construction method based on multimodal fusion," discloses the establishment of a topic entity and relationship extraction model based on word vector space and conditional random fields. However, it does not consider and solve the problem that the linkage behavior of dynamic targets in the airport flight area scenario has significant temporal dependence and spatial constraints.

[0005] In this context, there is an urgent need for a visual-spatiotemporal multimodal fusion detection method for flight zone linkage targets. By deeply integrating visual appearance features, spatiotemporal motion laws and multimodal complementary information, high-precision detection and linkage behavior analysis of small targets in complex environments can be achieved, providing technical support for the upgrade of airport intelligent surveillance systems. Summary of the Invention

[0006] In view of the above problems, the present invention provides a visual-spatiotemporal multimodal fusion detection method for airport flight zone linkage targets, which solves the technical problem in the existing technology of how to improve the airport surveillance multi-source heterogeneous data fusion capability and cross-modal spatiotemporal feature alignment capability.

[0007] According to an embodiment of the present invention, a method for detecting airport flight zone linkage targets using a visual-temporal multimodal fusion method includes the following steps: Step S1: Constructing a spatiotemporal graph of aircraft trajectories: Based on the flight zone map nodes, obtain the spatiotemporal trajectory clusters of aircraft, and use each trajectory in the spatiotemporal trajectory cluster as a spatiotemporal graph node. Dynamic edges are constructed based on the spatiotemporal relationship of the tracks, and a spatiotemporal graph encoder is constructed. The adjacency matrix of the spatiotemporal graph nodes is dynamically learned to obtain and output spatiotemporal graph features. Step S2: Constructing a linkage target semantic graph: The locations of targets appearing in the frame images of the airport scene surveillance video are used as semantic target nodes. Semantic edges are constructed based on the business logic relationships between targets. Multi-scale spatiotemporal features are extracted through a spatiotemporal coupled visual encoder to obtain and output semantic graph features. Step S3: Generate visual-spatiotemporal fusion features: Construct a spatiotemporal graph-semantic graph attention alignment module and an objective function, perform training to obtain an optimized model, and input the real-time collected aircraft four-dimensional spatiotemporal trajectory data and airport scene surveillance video into the optimized model to obtain fusion features as visual-spatiotemporal fusion features; Step S4: Input the obtained fusion features into the downstream recognition module to complete the recognition and classification of the target.

[0008] Optionally, step S1 specifically includes: Step S1-1, ADS-B trajectory data collection and preprocessing, real-time sampling to obtain four-dimensional space-time trajectory data, including the geographic coordinates, pressure altitude and start and end time of each four-dimensional space-time trajectory; Step S1-2, generating continuous four-dimensional space-time trajectory clusters, clustering the four-dimensional space-time trajectories that satisfy the space-time continuity constraint into space-time trajectory clusters, and using the trajectories in the space-time trajectory clusters as space-time graph nodes; Step S1-3: Dynamic edge weight calculation, establishing edge weights between nodes in the spatiotemporal graph in the spatiotemporal trajectory cluster; Step S1-4: Construct a spatiotemporal graph encoder, dynamically encode the spatiotemporal trajectory cluster using a graph attention network, and obtain spatiotemporal graph features.

[0009] Optionally, step S1-3 specifically includes: Enhanced geographic distance calculation to measure the spatiotemporal distance between nodes in a spatiotemporal graph; Calculate the motion correlation factor between nodes of the space-time graph through the velocity vector sequence; Based on the spatiotemporal distance metric and motion correlation factor between spatiotemporal graph nodes, a composite function of spatiotemporal distance and motion correlation is established to calculate the edge weights between spatiotemporal graph nodes in a spatiotemporal trajectory cluster.

[0010] Optionally, step S1-4 specifically includes: Construct a spatiotemporal graph encoder based on a multi-head graph attention network, including an attention mechanism that builds an adjacency matrix between nodes in the spatiotemporal graph; Dynamically update the adjacency matrix between nodes in the spatiotemporal graph through residual connections and layer normalization; The spatiotemporal trajectory clusters are fed into the spatiotemporal graph encoder to obtain the spatiotemporal graph features.

[0011] Optionally, step S2 specifically includes: Step S2-1: obtaining an airport scene surveillance video by using a camera device deployed in the airport scene area, and obtaining a frame image set of the airport scene surveillance video frame; Step S2-2: Aircraft operation business logic is linked and established, and interaction rules are formulated based on the aircraft targets and ground guidance vehicle targets that appear in the airport scene surveillance video; Step S2-3: Build the ground support business logic, including developing interaction rules for refueling operations, catering services, and ground maintenance. Step S2-4: constructing a semantic edge graph based on the rules established above to obtain a semantic edge graph set of the frame image set; Step S2-5: construct a spatiotemporal coupled visual encoder including a multi-layer temporal diffusion graph convolution module, process the obtained semantic edge graph set of the frame image set, and obtain semantic graph features.

[0012] Optionally, step S2-2 specifically includes: Taking the parking locations of refueling trucks and aircraft as semantic target nodes, interaction rules are formulated for refueling operations. The interaction rules for catering services are formulated using the number of catering carts in docking, the number of catering carts that can be docked simultaneously by the aircraft's doors, and the parking locations of catering carts as semantic target nodes. The staff's location and aircraft parking location are used as semantic target nodes, semantic edges are constructed based on ground maintenance operations, and interaction rules are formulated for ground maintenance.

[0013] Optionally, step S3 specifically includes: Step S3-1: construct a spatiotemporal graph-semantic graph attention alignment module to process the spatiotemporal graph features and the semantic graph features to obtain fused features; Step S3-2: construct a feature pyramid multi-level self-supervisory loss as the objective function; Step S3-3: Minimize the multi-level self-supervisory loss of the feature pyramid for training to obtain an optimized model; Step S3-4: Input the real-time collected aircraft four-dimensional spatiotemporal trajectory data and airport scene surveillance video into the optimized model, process them to obtain fusion features as visual-spatiotemporal fusion features.

[0014] Compared with the existing technology, the visual-temporal-spatial multimodal fusion detection method for airport flight zone linkage targets provided by the present invention has at least the following beneficial effects.

[0015] (1) Collaborative fusion of multi-dimensional data: This invention breaks through the limitations of traditional single-modal analysis. Through dynamic interactive modeling of spatiotemporal graphs and semantic graphs, it achieves deep coupling of multi-source heterogeneous data such as aircraft trajectories, video targets, and business rules, significantly improving the perception and reasoning capabilities of multi-agent linkage behaviors in complex scenarios.

[0016] (2) Deep embedding of business logic: Converting airport operating rules into computable feature constraints, making the feature learning process naturally adaptable to industry standards, enhancing the business interpretability of algorithm decisions, and supporting dynamic adjustments to quickly adapt to changes in operating modes of different airports.

[0017] (3) Refined spatiotemporal perception: Through cross-modal feature alignment and attention guidance mechanism, high-precision spatiotemporal semantic perception is achieved, effectively improving the timeliness and accuracy of target positioning, anomaly detection and risk warning, and providing reliable technical support for the airport's full-area intelligent management and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention can be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1Schematic diagram of the principle framework of the visual-spatiotemporal multimodal fusion detection method for airport flight zone linkage targets provided according to an embodiment of the present invention.

[0020] Figure 2 The present invention provides a flowchart of a method for visual-spatiotemporal multimodal fusion detection of airport flight zone linkage targets according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features therein can be combined with each other without conflict.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0023] The purpose of the present invention is to address the issues of airport monitoring multi-source heterogeneous data fusion capability and cross-modal spatiotemporal feature alignment capability, adopt the solution idea of ​​"multi-dimensional data collaborative fusion, deep embedding of business logic, and refined spatiotemporal perception", and provide a visual-spatiotemporal multimodal fusion detection method for airport flight zone linkage targets. Through dynamic interactive modeling of spatiotemporal graphs and semantic graphs, it can achieve deep coupling of multi-source heterogeneous data such as aircraft trajectories, video targets, and business rules, so that the feature learning process naturally adapts to industry standards, enhances the business interpretability of algorithm decision-making, achieves high-precision spatiotemporal semantic perception, and effectively improves the timeliness and accuracy of target positioning, anomaly detection, and risk warning, providing reliable technical support for the full-area intelligent management and control of airports.

[0024] The following describes in detail the visual-spatiotemporal multimodal fusion detection method for airport flight zone linkage targets provided according to an embodiment of the present invention with reference to the accompanying drawings to illustrate the effectiveness of the method.

[0025] like Figure 1 and Figure 2 As shown, the visual-temporal-spatial multimodal fusion detection method for airport flight zone linkage targets provided in accordance with an embodiment of the present invention includes the following steps.

[0026] Step S1: Constructing a spatiotemporal graph of aircraft trajectories: Based on the flight zone map nodes, obtain the spatiotemporal trajectory cluster of the aircraft, and use each trajectory in the spatiotemporal trajectory cluster as a spatiotemporal graph node. Based on the spatiotemporal relationship of the trajectory, construct dynamic edges, and construct a spatiotemporal graph encoder. Dynamically learn the adjacency matrix of the spatiotemporal graph nodes to obtain and output the spatiotemporal graph features. The step S1 specifically includes the following steps.

[0027] Step S1-1: ADS-B trajectory data collection and preprocessing, real-time sampling to obtain 4D space-time trajectory data, including the geographic coordinates, pressure altitude, start and end time of each 4D space-time trajectory. Based on the flight zone map node, the 4D space-time trajectory data of the aircraft is obtained in real time through the ADS-B (Automatic Dependent Surveillance-Broadcast) receiving equipment deployed in the airport surface area. The 4D space-time trajectory data can be expressed as:

[0028] in, For the Four-dimensional space-time trajectory data, represents the index of the four-dimensional space-time trajectory, For the The WGS-84 geographic coordinates of the four-dimensional space-time trajectory, For the The pressure height of the four-dimensional space-time trajectory, Indicates the The start and end times of the four-dimensional space-time trajectory are UTC timestamps, Indicates the The starting time of the four-dimensional space-time trajectory, Indicates the The end time of the four-dimensional space-time trajectory, is the set of all four-dimensional space-time trajectories.

[0029] Step S1-2: Generate continuous four-dimensional space-time trajectory clusters. Judge each four-dimensional space-time trajectory and cluster the four-dimensional space-time trajectories that meet the space-time continuity constraint into space-time trajectory clusters. Use the trajectories in the space-time trajectory clusters as space-time graph nodes.

[0030] Use the improved DBSCAN (density-based spatial clustering algorithm) spatiotemporal clustering algorithm and set the dynamic neighborhood radius:

[0031] in, is the average ground speed within the sliding window, is the estimated turning radius, is the adjustment coefficient of the airport operation scenario, Represents the UTC timestamp interval. This dynamic neighborhood radius is the optimization limit of the DBSCAN algorithm. The dynamic neighborhood radius can be used to judge 4D space-time trajectories.

[0032] For the set of all four-dimensional space-time trajectories obtained in step S1-1 For each four-dimensional spacetime trajectory in , set the spacetime continuity constraint:

[0033]

[0034] in, Respectively represent Four-dimensional space-time trajectory data, Indicates the The start and end time of the trajectory, is a hyperparameter that controls the spatial distance, is a hyperparameter that controls the temporal distance.

[0035] Judge each four-dimensional space-time trajectory and cluster the four-dimensional space-time trajectories that meet the above space-time continuity constraints into space-time trajectory clusters , a trajectory in the spatiotemporal trajectory cluster As a node in the space-time graph, Represents the index of the space-time graph node. The four-dimensional space-time trajectories that do not meet the above space-time continuity constraints are discarded.

[0036] Step S1-3: Dynamic edge weight calculation, establishing the edge weights between the nodes in the spatiotemporal graph in the spatiotemporal trajectory cluster. The specific process is as follows.

[0037] First, the Haversine formula is used to enhance the spatial and temporal distance metric for geographic distance calculation:

[0038] in, represents the spatiotemporal distance metric function, Indicates the nodes of the spatiotemporal graph, Indicates the nodes of the spatiotemporal graph, is the Haversine distance, is the time control coefficient, express The corresponding time, express Corresponding time

[0039] Next, the motion correlation factor is calculated from the velocity vector sequence :

[0040] in, represents the cosine function, represents the inverse cosine function, Indicates the The speed of the space-time graph node, Indicates the The speed of a node in the space-time graph.

[0041] Next, a composite function of spatiotemporal distance and motion correlation is established to calculate the edge weights between spatiotemporal graph nodes in a spatiotemporal trajectory cluster:

[0042] in, Representing a spatiotemporal graph node The edge weights between represents the temperature coefficient, Represents the exponential function.

[0043] Step S1-4: Construct a spatiotemporal graph encoder, dynamically encode the spatiotemporal trajectory cluster using a graph attention network, and obtain spatiotemporal graph features. The specific process is as follows.

[0044] Establish an adjacency matrix update mechanism for spatiotemporal graph nodes to obtain the adjacency matrix attention mechanism between spatiotemporal graph nodes in spatiotemporal trajectory clusters:

[0045] in, represents the activation function, For the Layer learnable parameter matrix, A node in the spatiotemporal graph In the The hidden state of the layer, A node in the spatiotemporal graph In the The hidden state of the layer, represents a constant that controls the dot product result, Indicates the Spatiotemporal graph nodes of layers and Attention mechanism of the adjacency matrix.

[0046] Through the above method, we can obtain The adjacency matrix attention mechanism of all spatiotemporal graph nodes in the spatiotemporal trajectory cluster of the layer constitutes the Attention mechanism for the adjacency matrix of all spatiotemporal graph nodes in the spatiotemporal trajectory cluster of the layer .

[0047] Dynamic update of the adjacency matrix attention mechanism is achieved through residual connections and layer normalization:

[0048] in, is the normalization layer, is the random dropout function, Indicates the Attention mechanism for the adjacency matrix of nodes in the spatiotemporal graph of a layer.

[0049] In this way, a spatiotemporal graph encoder based on the Multi-head GAT is constructed.

[0050] The spatiotemporal trajectory cluster obtained in step S1-2 (i.e., a collection of spatiotemporal graph nodes), input the spatiotemporal graph encoder based on the Multi-head GAT constructed above, encode it, and obtain the spatiotemporal graph features .

[0051] Step S2: Constructing a linkage target semantic graph: The positions of group targets such as aircraft, vehicles, people, and facilities that appear in the frame images of the airport scene surveillance video are used as semantic target nodes. Semantic edges are constructed based on the business logic relationship between targets. Multi-scale spatiotemporal features are extracted through the spatiotemporal coupled visual encoder to output semantic graph features. The step S2 specifically includes the following steps.

[0052] Step S2-1: Obtain airport scene surveillance video by using a camera device deployed in the airport scene area, and obtain a frame image set of the airport scene surveillance video frame. , as the original data.

[0053] Step S2-2: Aircraft operation business logic edge construction. The positions of the aircraft target and the ground guidance vehicle target appearing in the airport scene surveillance video are used as semantic target nodes. Semantic edges are constructed with follow-up relationships and interaction rules are formulated. :

[0054] in, is the position of the preceding aircraft or lead vehicle, is the current position of the aircraft or the lead vehicle, Indexes for aircraft and ground guidance vehicles, Indicates a follow-up relationship. Indicates distance.

[0055] Step S2-3: Build the ground support business logic connection. Ground support business includes refueling operations, catering services, and ground maintenance. The targets involved may include aircraft and refueling trucks, aircraft and catering trucks, and aircraft and personnel.

[0056] Taking the parking location of the refueling truck and the parking location of the aircraft as semantic target nodes, interactive rules are formulated for refueling operations. :

[0057] in, Indicates the parking location of the refueling truck. Indicates the aircraft parking position, Represents the distance coefficient.

[0058] The number of catering carts in docking state, the number of catering carts that can be docked by the aircraft's doors at the same time, and the parking positions of catering carts are used as semantic target nodes to formulate interaction rules for catering services. :

[0059] in, is the number of dining cars in docking state, The number of catering carts that can be docked at the aircraft's door at the same time, Indicates the Parking space for food trucks, Indicates the total number of dining cars, Indicates the index of the dining car.

[0060] The location of the staff and the aircraft parking location are used as semantic target nodes, semantic edges are constructed based on ground maintenance operations, and interaction rules are formulated for ground maintenance. :

[0061] in, Indicates the location of the staff member, Indicates ground level service operation.

[0062] Step S2-4: construct a semantic edge graph based on the above established rules to obtain the semantic edge graph of each frame image:

[0063] in, Indicates the Frame image, , represents a set of frame images of the airport scene surveillance video, Indicates the Semantic edge graph of frame image, Indicates the The node set of the semantic edge graph of the frame image, Indicates the The adjacency matrix of the semantic edge graph of the frame image, Indicates the The edge set of the semantic edge graph of the frame image.

[0064] Based on the semantic edge graphs of each frame image obtained above, a semantic edge graph set of the frame image set is constructed. .in, The node set representing the semantic edge graph of the frame image set, The adjacency matrix representing the semantic edge graph of the frame image set, The edge set representing the semantic edge graph of a set of frame images.

[0065] Step S2-5: Construct a spatiotemporal coupled visual encoder. The temporal diffusion graph convolution module uses stacked deformable diffusion convolution kernels to implement dynamic temporal modeling, resulting in a spatiotemporal coupled visual encoder:

[0066] in, For the Layer temporal diffusion graph convolution module, For the Parameters of the layer-wise temporal diffusion graph convolution module. The spatiotemporally coupled visual encoder consists of multiple layers of temporal diffusion graph convolution modules.

[0067] The semantic edge graph set of the obtained frame image set Input the above spatiotemporal coupled visual encoder to obtain semantic graph features .

[0068] Step S3, generate visual-spatiotemporal fusion features: construct a spatiotemporal graph-semantic graph attention alignment module and objective function, train and obtain an optimized model, and obtain fusion features through the optimized model as visual-spatiotemporal fusion features. Establish a spatial mapping relationship between the aircraft position coordinates and visual features, introduce an attention mechanism to achieve pixel-level alignment of spatiotemporal features and visual features, and output the fusion features through a global fusion encoder. , and construct a feature pyramid multi-level self-supervised loss. This step S3 specifically includes the following steps.

[0069] Step S3-1: Construct a spatiotemporal graph-semantic graph attention alignment module (i.e., a global fusion encoder) to process the spatiotemporal graph features and the semantic graph features to obtain fused features.

[0070] Based on the obtained spatiotemporal graph features and semantic graph features , build a spatiotemporal graph-semantic graph attention alignment module to obtain fusion features:

[0071] in, is the query of the attention mechanism, They are the key and value of the attention mechanism respectively. represents the fusion feature, Represents a constant that controls the result of the dot product. The value of comes from the spatiotemporal graph features , The value of comes from the semantic graph feature .

[0072] The spatiotemporal graph features and semantic graph features , input the constructed spatiotemporal graph-semantic graph attention alignment module, obtain and output the fusion feature .

[0073] Step S3-2: Based on the fusion features , construct the feature pyramid multi-level self-supervisory loss as follows, as the objective function:

[0074] in, Indicates the The training parameters of the pyramid feature reconstruction model are frozen. represents the frame-level semantic segmentation network of the corresponding scale, whose training parameters are frozen. For the The pyramid constraint coefficient decays exponentially according to the pyramid level. is the pyramid level index and , is the total number of pyramid levels.

[0075] Thus, a visual-temporal multimodal fusion detection model for airport flight zone linkage targets is obtained.

[0076] Step S3-3: Train the established visual-temporal multimodal fusion detection model for the airport flight zone linkage target. The training goal is to minimize the above objective function, that is, to minimize the multi-level self-supervisory loss of the feature pyramid. The model is trained and iteratively updated to optimize the visual-spatiotemporal multimodal fusion detection model for airport flight zone linkage targets, resulting in an optimized model. This optimization includes optimizing the edge weights between nodes in the spatiotemporal graph.

[0077] Step S3-4: Input the real-time collected aircraft four-dimensional spatiotemporal trajectory data and airport scene surveillance video into the optimized model, process them to obtain fusion features as visual-spatiotemporal fusion features.

[0078] Step S4: The obtained visual-temporal fusion features This information is then fed into the downstream recognition module to complete target identification and classification. Typical visual application scenarios supported by this downstream recognition module include, but are not limited to: aircraft operational safety monitoring (runway incursion warning, taxiway path deviation alert, unauthorized area intrusion detection, wake turbulence separation violation analysis, etc.); ground support operation compliance verification (support vehicle reverse driving identification, equipment docking timeout warning (refueling / catering / loading and unloading), ground personnel intrusion into aircraft taxiing areas, and special vehicle operation range violation monitoring, etc.).

[0079] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0080] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0081] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A visual-temporal multimodal fusion detection method for airport flight zone linkage targets, characterized by: The following steps are involved: Step S1: Constructing a spatiotemporal graph of aircraft trajectories: Based on the flight zone map nodes, obtain the spatiotemporal trajectory clusters of aircraft, and use each trajectory in the spatiotemporal trajectory cluster as a spatiotemporal graph node. Dynamic edges are constructed based on the spatiotemporal relationship of the tracks, and a spatiotemporal graph encoder is constructed. The adjacency matrix of the spatiotemporal graph nodes is dynamically learned to obtain and output spatiotemporal graph features. Step S2: Constructing a linkage target semantic graph: The locations of targets appearing in the frame images of the airport scene surveillance video are used as semantic target nodes. Semantic edges are constructed based on the business logic relationships between targets. Multi-scale spatiotemporal features are extracted through a spatiotemporal coupled visual encoder to obtain and output semantic graph features. Step S3: Generate visual-spatiotemporal fusion features: Construct a spatiotemporal graph-semantic graph attention alignment module and an objective function, perform training to obtain an optimized model, and input the real-time collected aircraft four-dimensional spatiotemporal trajectory data and airport scene surveillance video into the optimized model to obtain fusion features as visual-spatiotemporal fusion features; Step S4: Input the obtained fusion features into the downstream recognition module to complete the recognition and classification of the target.

2. The method for detecting airport flight zone linkage targets using visual-temporal multimodal fusion according to claim 1 is characterized in that: Step S1 specifically includes: Step S1-1, ADS-B trajectory data collection and preprocessing, real-time sampling to obtain four-dimensional space-time trajectory data, including the geographic coordinates, pressure altitude and start and end time of each four-dimensional space-time trajectory; Step S1-2, generating continuous four-dimensional space-time trajectory clusters, clustering the four-dimensional space-time trajectories that satisfy the space-time continuity constraint into space-time trajectory clusters, and using the trajectories in the space-time trajectory clusters as space-time graph nodes; Step S1-3: Dynamic edge weight calculation, establishing edge weights between nodes in the spatiotemporal graph in the spatiotemporal trajectory cluster; Step S1-4: Construct a spatiotemporal graph encoder, dynamically encode the spatiotemporal trajectory cluster using a graph attention network, and obtain spatiotemporal graph features.

3. The method for detecting airport flight zone linkage targets using visual-temporal multimodal fusion according to claim 2 is characterized in that: Step S1-3 specifically includes: Enhanced geographic distance calculation to measure the spatiotemporal distance between nodes in a spatiotemporal graph; Calculate the motion correlation factor between nodes of the space-time graph through the velocity vector sequence; Based on the spatiotemporal distance metric and motion correlation factor between spatiotemporal graph nodes, a composite function of spatiotemporal distance and motion correlation is established to calculate the edge weights between spatiotemporal graph nodes in a spatiotemporal trajectory cluster.

4. The method for detecting airport flight zone linkage targets using visual-temporal multimodal fusion according to claim 2 is characterized in that: Step S1-4 specifically includes: Construct a spatiotemporal graph encoder based on a multi-head graph attention network, including an attention mechanism that builds an adjacency matrix between nodes in the spatiotemporal graph; Dynamically update the adjacency matrix between nodes in the spatiotemporal graph through residual connections and layer normalization; The spatiotemporal trajectory clusters are fed into the spatiotemporal graph encoder to obtain the spatiotemporal graph features.

5. The method for detecting airport flight zone linkage targets using visual-temporal multimodal fusion according to claim 1 is characterized in that: Step S2 specifically includes: Step S2-1: obtaining an airport scene surveillance video by using a camera device deployed in the airport scene area, and obtaining a frame image set of the airport scene surveillance video frame; Step S2-2: Aircraft operation business logic is linked and established, and interaction rules are formulated based on the aircraft targets and ground guidance vehicle targets that appear in the airport scene surveillance video; Step S2-3: Build the ground support business logic, including developing interaction rules for refueling operations, catering services, and ground maintenance. Step S2-4: constructing a semantic edge graph based on the rules established above to obtain a semantic edge graph set of the frame image set; Step S2-5: construct a spatiotemporal coupled visual encoder including a multi-layer temporal diffusion graph convolution module, process the obtained semantic edge graph set of the frame image set, and obtain semantic graph features.

6. The method for detecting airport flight zone linkage targets using visual-temporal multimodal fusion according to claim 1, characterized in that: Step S2-2 specifically includes: Taking the parking locations of refueling trucks and aircraft as semantic target nodes, interaction rules are formulated for refueling operations. The interaction rules for catering services are formulated using the number of catering carts in docking, the number of catering carts that can be docked simultaneously by the aircraft's doors, and the parking locations of catering carts as semantic target nodes. The staff's location and aircraft parking location are used as semantic target nodes, semantic edges are constructed based on ground maintenance operations, and interaction rules are formulated for ground maintenance.

7. The method for detecting airport flight zone linkage targets using visual-temporal multimodal fusion according to claim 1, characterized in that: Step S3 specifically includes: Step S3-1: construct a spatiotemporal graph-semantic graph attention alignment module to process the spatiotemporal graph features and the semantic graph features to obtain fused features; Step S3-2: construct a feature pyramid multi-level self-supervisory loss as the objective function; Step S3-3: Minimize the multi-level self-supervisory loss of the feature pyramid for training to obtain an optimized model; Step S3-4: Input the real-time collected aircraft four-dimensional spatiotemporal trajectory data and airport scene surveillance video into the optimized model, process them to obtain fusion features as visual-spatiotemporal fusion features.

Images