An unmanned aerial vehicle air-ground linkage intelligent inspection system and method based on an STGNN

Abstract

The application discloses an unmanned aerial vehicle air-ground linkage intelligent inspection system and method based on an STGNN, and relates to the technical fields of unmanned aerial vehicle inspection and artificial intelligence. The system comprises a perception layer, a network layer, a decision layer and an application layer. The perception layer collects multi-dimensional heterogeneous inspection data. The network layer realizes low-latency bidirectional transmission. The decision layer is provided with a space-time graph neural network engine, a task scheduling optimization module and a digital twin simulation module, which serve as a core control layer. The method builds a dynamic space-time graph, completes joint representation and prediction of nodes through an STGNN model, generates a collaborative decision in combination with a strategy network and a centralized value network, re-schedules execution and forms a closed loop, and simultaneously completes abnormal diagnosis and knowledge accumulation. The application breaks the information island of air-ground equipment, realizes global collaborative inspection, improves inspection efficiency, abnormal detection accuracy and system self-adaptability, is suitable for traffic, power, security and other multi-inspection scenes, and provides an intelligent solution for urban intelligent management.

Description

Technical Field

[0001] This invention relates to the fields of unmanned aerial vehicle (UAV) inspection and artificial intelligence technology, specifically to an intelligent UAV air-ground linkage inspection system and method based on STGNN (Spatiotemporal Graph Neural Network). Background Technology

[0002] Against the backdrop of the nation's vigorous development of the low-altitude economy and the modernization of urban governance systems and capabilities, smart urban governance places higher demands on comprehensive perception, intelligent decision-making, and rapid response capabilities. Constructing new application scenarios centered on "drones + artificial intelligence (AI)," integrating cutting-edge technologies such as big data, spatiotemporal intelligence, spaceborne remote sensing, and aerial photography fusion modeling, to create a smart monitoring system covering the entire process of urban governance events has become a key path to solving governance pain points. Among these, the innovative application of technologies such as drone swarm collaboration, spatio-temporal graph neural networks (STGNN), and multi-source perception data fusion modeling are the core support for developing an air-ground linked smart inspection and management platform. Its ultimate goal is to achieve integrated fusion of multi-source video perception data, constructing a highly efficient handling model for urban governance events with "second-level perception—minute-level response—hour-level closed loop," providing a low-cost, high-efficiency, and fully closed-loop intelligent solution for urban governance. With the iterative upgrades of drone and artificial intelligence technologies, drone inspections have been widely applied in large-scale infrastructure inspection scenarios such as power line maintenance, urban traffic congestion monitoring, illegal construction investigation, and public security, significantly improving the efficiency and coverage of traditional manual inspections. However, existing drone inspection systems still have many core limitations when adapting to the complex needs of smart urban governance, making it difficult to support truly intelligent collaborative inspections. I. Single-platform operation with limited perspective and capabilities: Current inspections mainly rely on single-platform operation of drones, which have limited payload, endurance, perspective and penetration capabilities (such as for obscured areas), making it difficult to cope with complex environments and all-dimensional inspection needs.

[0003] Second, data silos and low collaboration efficiency: Drones and ground monitoring equipment (such as fixed cameras, ground robots, and IoT sensors) usually operate independently, with data and tasks separated, forming "information silos" and lacking efficient cross-platform information fusion and collaborative decision-making mechanisms.

[0004] Third, static task planning with poor adaptability: Most inspection tasks use pre-set fixed routes or locations, making it impossible to dynamically adjust and reallocate resources based on real-time environmental changes (such as sudden failures or weather effects), equipment status, and task priorities. Fourth, fragmented intelligent analysis: Existing AI analysis is mostly targeted at single data sources (such as only UAV imagery), lacking a framework for unified correlation analysis and in-depth mining of cross-temporal and cross-modal data (visible light, infrared, laser point clouds, vibration data, etc.).

[0005] In recent years, Graph Neural Networks (GNNs) have attracted attention due to their powerful relation modeling capabilities. Spatiotemporal Graph Neural Networks (STGNNs) further integrate temporal and spatial dependencies, achieving success in areas such as traffic prediction and crowd flow analysis. However, currently, there is still a lack of mature and feasible technical solutions for applying them to dynamic systems for collaborative inspection of UAVs and ground equipment, addressing the core challenges of equipment coordination, data fusion, and dynamic decision-making, and realizing true "air-ground linkage" intelligent decision-making. (Invention Content) The purpose of this invention is to address the shortcomings of existing technologies by providing an intelligent air-ground linkage inspection method and system for UAVs based on STGNN. By constructing a dynamic spatiotemporal graph (STGNN) model and integrating spaceborne remote sensing and aerial modeling technologies, it can achieve intelligent scheduling of inspection resources across the entire domain, multi-source data fusion analysis, and collaborative optimization of inspection tasks, thereby improving inspection efficiency, coverage, intelligence level, and system adaptability.

[0006] To achieve the above objectives, this invention provides an intelligent air-ground linkage inspection system for unmanned aerial vehicles based on STGNN, comprising a perception layer, a network layer, a decision layer, and an application layer; wherein, the perception layer is bidirectionally connected to the decision layer via the network layer, and the application layer interacts and communicates with the decision layer. The perception layer consists of several heterogeneous air nodes and ground nodes, and is responsible for collecting multi-dimensional heterogeneous inspection data. The network layer is based on 5G or private network communication technology to realize bidirectional, highly reliable, low-latency transmission of inspection data and control commands between perception layer nodes and between the perception layer and the decision layer. The decision-making layer is equipped with a spatiotemporal graph neural network engine, a task scheduling optimization module, and a digital twin simulation module. The spatiotemporal graph neural network engine abstracts air and ground nodes into graph nodes, generates dynamic edges based on spatial distance between nodes, communication link status, task correlation, and historical interaction data, assigns edge weights, constructs a dynamic spatiotemporal graph with real-time evolution of node attributes, and runs a pre-trained STGNN model to process the dynamic spatiotemporal graph, completing node state prediction, anomaly correlation reasoning, and collaborative decision suggestion generation. The task scheduling optimization module generates the optimal UAV flight path planning scheme and ground equipment scheduling control commands based on the collaborative decision suggestions output by the STGNN model, combined with inspection task priority and resource constraints. The digital twin simulation module constructs a virtual inspection environment synchronized with the physical world, realizing inspection task pre-rehearsal, scheduling strategy evaluation, and STGNN model training. The application layer is used to provide users with a human-computer interaction interface for monitoring inspection tasks, managing abnormal alarms, and generating inspection reports.

[0007] As a further description of the above technical solution: the STGNN model includes an input layer, a spatial attention convolution module, a temporal convolution module, and a graph update module; The input layer receives feature data from the dynamic spatiotemporal graph and performs format conversion, providing the input basis for subsequent modules. The dynamic spatiotemporal graph is represented as follows:

[0008] in: Let F be the node feature matrix, where N is the number of nodes and F is the feature dimension. This is an adjacency matrix, representing spatial connectivity relationships; Let C be the edge feature matrix, and C be the edge feature dimension.

[0009] The spatial attention convolution module employs a multi-head attention mechanism with edge features to learn the importance weights between graph nodes, capturing the spatial dependencies between nodes. The attention coefficient weights are normalized as follows: in, Represents a node The set of neighboring nodes, ; The maximum interaction distance threshold; The temporal recurrent convolution module uses a gated temporal convolutional network to process the sequence of node attribute changes over time, capturing the temporal dependencies of nodes. The gating mechanism is as follows: ; The graph update module uses the output information of the residual connection and layer normalization aggregation spatial attention convolution module and the temporal circular convolution module to update the dynamic spatiotemporal graph, outputting the hidden state representation of each graph node at the next time step. The hidden state representation integrates the node's own historical information, neighbor node association information, and global context information. The update formula is as follows: .

[0010] As a further description of the above technical solution: the aerial node includes at least one type of drone, which is equipped with one or more sensing payloads selected from visible light cameras, infrared imaging devices, and lidar. The node attributes of the drone include remaining battery power, health status, and coverage radius.

[0011] As a further description of the above technical solution: the ground node includes one or more of the following: a high-definition PTZ camera, a ground mobile robot, and a fixed sensor; the fixed sensor includes one or more of the following: a traffic flow detector, a vibration sensor, and an environmental sensor.

[0012] As a further description of the above technical solution: the node attributes of the dynamic spatiotemporal graph include equipment type, spatial location, working status, load capacity and real-time observation data; the edge weights characterize the degree of cooperation or information flow intensity between each device; and the dynamic spatiotemporal graph evolves in real time as the nodes move and the inspection task progresses.

[0013] As a further description of the above technical solution: the resource constraints include UAV battery threshold, airspace flight restrictions, and communication link quality. The UAV battery threshold is the minimum remaining battery power required for the UAV to perform inspection tasks. The airspace flight restrictions are the airspace range and altitude requirements for compliant UAV flight. As a further description of the above technical solution: the decision-making layer is also equipped with a knowledge base, which stores anomaly handling cases, inspection scheduling strategies, and equipment status data, providing data support for the iterative training and optimization of the STGNN model.

[0014] As a further description of the above technical solution: the system is equipped with a closed-loop feedback mechanism, which is used to transmit the device execution status data and environmental feedback data collected by the perception layer back to the spatiotemporal graph neural network engine of the decision layer in real time, drive the spatiotemporal graph neural network engine to update the dynamic spatiotemporal graph and trigger the STGNN model to re-infer, forming an inspection closed loop of "perception-decision-execution-feedback".

[0015] To achieve the above objectives, this invention also provides an intelligent air-to-ground joint inspection method for unmanned aerial vehicles (UAVs) based on STGNN. Utilizing the aforementioned intelligent air-to-ground joint inspection system, the method includes the following steps: S1, Dynamic Spatiotemporal Graph Construction: Each air node and ground node in the perception layer is abstracted as a graph node and its attributes are configured. Then, edges are dynamically generated and weighted based on the spatial distance between graph nodes, communication link status, task relevance, and historical interaction data, thus constructing a real-time evolving dynamic spatiotemporal graph, represented as... ; S2. Joint Representation and Prediction Based on STGNN: The dynamic spatiotemporal graph constructed in step S1 is input into the pre-trained STGNN model. The spatial attention convolution module uses a multi-head attention mechanism with edge features to capture the spatial dependencies between nodes. The temporal recurrent convolution module uses a gated temporal convolution network to mine the temporal evolution of node attributes. Then, the graph update module uses residual connections and layer normalization to aggregate spatiotemporal feature information, update the dynamic spatiotemporal graph, and output the hidden state representation of each graph node at the next time step. The attention coefficient weights of the multi-head attention mechanism are normalized as follows: ; The gating mechanism of gated temporal convolutional networks is as follows: ； The update formula for the graph update module is: ; S3. Multi-agent collaborative decision generation: Based on the node hidden state representation output in step S2, preliminary action suggestions are generated for each air node and ground node through the policy network. Then, the global state is evaluated by the centralized value network in combination with the inspection task priority and resource constraints. The dispersed action suggestions are coordinated and optimized to output the final collaborative decision. S4. Task Scheduling and Execution: The task scheduling optimization module of the decision layer transforms the final collaborative decision of step S3 into control commands, which are then sent to the corresponding air and ground nodes through the network layer. Simultaneously, it collects equipment execution status data and environmental feedback data in real time through a closed-loop feedback mechanism and sends them back to the spatiotemporal graph neural network engine of the decision layer to update the dynamic spatiotemporal graph and form an inspection closed loop. S5. Anomaly Diagnosis and Knowledge Accumulation: When a potential inspection anomaly is detected at any node, the spatiotemporal graph neural network engine of the decision layer performs graph reasoning on the dynamic spatiotemporal graph through the STGNN model to assist in locating the root cause of the anomaly, complete the anomaly handling, and store the anomaly handling cases in the knowledge base configured by the decision layer for iterative training and optimization of the STGNN model.

[0016] As a further description of the above technical solution: the training data of the STGNN model includes historical inspection data, event handling records, and multi-scenario inspection data simulated by the digital twin simulation module. The multi-scenario inspection data includes inspection scenario data under different weather conditions, holidays, and sudden accidents.

[0017] This invention proposes targeted solutions to address the technical shortcomings of existing UAV inspection systems, achieving technical effects such as full-domain collaboration, dynamic intelligence, deep correlation, and resource optimization. This significantly improves the intelligence level, operational efficiency, and scenario adaptability of the inspection system, as detailed below: Full-domain collaborative inspection improves coverage and operational efficiency: Breaking the operational boundaries between aerial and ground inspection equipment, integrating satellite remote sensing and aerial photography data, it enables dynamic allocation and relay execution of inspection tasks, effectively expanding the inspection coverage, eliminating blind spots, and improving overall inspection efficiency by more than 30%. Dynamic intelligent scheduling enhances the system's adaptability: The system can perceive environmental changes, emergencies and equipment operating status in real time, and dynamically adjust the inspection strategy accordingly. It has strong robustness and scenario adaptability, meeting the dynamic inspection needs of multiple scenarios such as power lines, traffic congestion, illegal construction investigation, and public security. Spatiotemporal correlation analysis improves anomaly detection accuracy: By relying on the STGNN model to perform spatiotemporal correlation modeling on multi-source heterogeneous inspection data, it is possible to uncover hidden fault modes and associated abnormal events that are difficult to detect through isolated analysis, and significantly reduce the false alarm and missed alarm rates during the inspection process. Joint optimization of resources to improve system operation efficiency: By jointly optimizing and scheduling the inspection nodes across the entire domain, the energy consumption and workload of each device are balanced, effectively extending the drone's flight cycle, improving the resource utilization rate of the entire inspection system, and extending the overall system operation cycle; Continuous model iteration ensures decision-making accuracy: Anomaly handling cases are stored in a knowledge base, and the STGNN model is iteratively trained using multi-scenario data from digital twin simulations. This continuous optimization ensures the system's decision-making accuracy in complex inspection scenarios. (See attached diagram.) Figure 1 This is a schematic diagram of the overall architecture of the UAV air-ground linkage intelligent inspection system of the present invention.

[0018] Figure 2 This is a schematic diagram of the STGNN model of the present invention.

[0019] Figure 3 This is the overall flowchart of an intelligent air-ground linkage inspection method for unmanned aerial vehicles based on STGNN according to the present invention.

[0020] Figure 4 This is a schematic diagram of the construction of the dynamic spatiotemporal graph of the present invention. Detailed Implementation

[0021] The claims of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, but this does not constitute any limitation on the present invention. Any limited modifications made by any person within the scope of protection of the claims of the present invention shall still be within the scope of protection of the claims of the present invention. Example

[0022] This embodiment provides an intelligent air-to-ground joint inspection system for unmanned aerial vehicles (UAVs) based on STGNN. It adopts a four-layer architecture design: perception layer, network layer, decision layer, and application layer. The core relies on a spatiotemporal graph neural network (STGNN) to achieve collaborative modeling and intelligent decision-making for heterogeneous devices. In scenarios such as traffic congestion inspection, anomaly handling, and traffic management on urban main roads, it can achieve the efficient governance goal of "second-level perception - minute-level response - hour-level closed loop." Its system architecture is as follows: Figure 1 As shown, the system is divided into a perception layer (air nodes, ground nodes), a network layer (5G / private network), a decision layer (spatiotemporal graph neural network engine, task scheduling optimization module, digital twin simulation module), and an application layer (map display, task monitoring, event management, work order generation, information release). The perception layer and the decision layer achieve bidirectional data transmission through the network layer, and the application layer and the decision layer achieve interactive communication. The perception layer, serving as the system's data acquisition terminal, is deployed across the entire urban main road area. It comprises two types of heterogeneous devices: aerial nodes and ground nodes. It is responsible for collecting multi-dimensional heterogeneous inspection data (including cross-modal data such as visible light, infrared, laser point clouds, and vibration), providing fundamental data support for upper-level decision-making. Aerial nodes are drones equipped with visible light cameras, infrared imaging devices, and LiDAR; node attributes include remaining battery power, health status, and coverage radius. Ground nodes include high-definition gimbal cameras, ground mobile robots, traffic flow detectors, vibration sensors, and environmental sensors. The perception layer can integrate spaceborne remote sensing data and aerial photography data to enhance the coverage and data dimensions of the entire inspection area.

[0023] The network layer is based on a converged communication solution of 5G+city private network to achieve bidirectional highly reliable and low-latency transmission of inspection data and control commands between nodes in the perception layer and between the perception layer and the decision layer (communication latency ≤100 ms), ensuring the real-time requirements of collaborative inspection between air and ground equipment.

[0024] The decision-making layer serves as the core control layer, and it deploys a spatiotemporal graph neural network engine, a task scheduling optimization module, a digital twin simulation module, and a knowledge base.

[0025] The spatiotemporal graph neural network engine performs dynamic spatiotemporal graph construction and STGNN model inference. Following the STGNN model calculation formula, it completes node state prediction, anomaly association inference, and collaborative decision-making suggestion generation. The dynamic spatiotemporal graph is represented as follows:

[0026] in: Let F be the node feature matrix, where N is the number of nodes and F is the feature dimension. This is an adjacency matrix, representing spatial connectivity relationships; Let C be the edge feature matrix, and C be the edge feature dimension.

[0027] The task scheduling optimization module generates optimal scheduling instructions based on the STGNN model output and resource constraints such as UAV battery thresholds and airspace flight restrictions. The digital twin simulation module constructs a 3D virtual inspection environment for urban main roads, simulating different scenarios to generate training data for STGNN model training and strategy pre-playing; the knowledge base stores anomaly handling cases, inspection scheduling strategies, and equipment status data, providing data support for the iterative optimization of the STGNN model.

[0028] The application layer is configured with a visual command screen, which provides users with a human-computer interaction interface for monitoring inspection tasks, managing abnormal alarms, and generating inspection reports. It enables real-time display of equipment status and inspection location, as well as automatic push and closed-loop recording of alarm information and handling work orders.

[0029] The system is equipped with a closed-loop feedback mechanism that transmits the device execution status data and environmental feedback data collected by the perception layer back to the spatiotemporal graph neural network engine of the decision layer in real time. This drives the spatiotemporal graph neural network engine to update the dynamic spatiotemporal graph and triggers the STGNN model to re-infer, forming a closed-loop inspection system of "perception-decision-execution-feedback".

[0030] The hierarchical architecture of the STGNN model is as follows: Figure 2 As shown, it includes an input layer, a spatial attention convolution module, a temporal recurrent convolution module, and a graph update module. The input layer receives a dynamic spatiotemporal graph. After the spatial attention convolution module learns the spatial dependency weights between nodes and the temporal recurrent convolution module captures the temporal series features of node attributes, the graph update module aggregates the spatiotemporal features to generate the hidden state representation of each graph node at the next time step. Then, the policy network generates preliminary action suggestions for each node. The value network completes the global state value assessment. Finally, the global collaborative decision is output through the collaborative optimizer. Specifically: The input layer receives feature data from the dynamic spatiotemporal graph and performs format conversion to provide the input basis for subsequent modules; The spatial attention convolution module employs a multi-head attention mechanism with edge features to learn importance weights between graph nodes, capturing the spatial dependencies between nodes; for the th l Layer, number k Each attention head, its query (Q ),key( K ),value( V The projections are represented as follows: ; ; ;

[0031] in, The weight matrix is ​​a learnable weight matrix; It is the bias vector; Dimensions for each attention head; h The total number of attention heads (usually set to 4 or 8).

[0032] The attention coefficient is expressed as: in, The basic attention score is expressed as:

[0033] Represents a node The query vector, This represents the dot product operation; The fusion of spatial constraints and edge features is represented as follows:

[0034] α It is a hyperparameter affected by edge control characteristics; It is an adjacency matrix constraint, expressed as: Attention coefficient weights are normalized as follows:

[0035] in, Represents a node The set of neighboring nodes is defined as follows:

[0036] This is the maximum interaction distance threshold.

[0037] The edge feature is encoded as follows:

[0038] in, The original edge feature vector; Encode weights for edge features; This represents the dimension of the encoded edge features.

[0039] The Temporal Recurrent Convolution module uses a gated temporal convolutional network (Gated TCN) to process the sequence of node attribute changes over time and capture the temporal dependencies of nodes; Each node The time series features are represented as follows:

[0040] Causal dilated convolution is: The attention coefficient weights are normalized as follows: ; ; in, K The kernel size; d expansion factor ( , l (Number of layers); , These are the learnable convolution kernel parameters.

[0041] The gating mechanism is as follows:

[0042] in, This indicates element-wise multiplication.

[0043] The graph update module uses residual connections and layer normalization to aggregate the output information of the spatial attention convolution module and the temporal recurrent convolution module to update the dynamic spatiotemporal graph, outputting the hidden state representation of each graph node at the next time step. This representation integrates the node's own historical information, neighbor node association information, and global context information; The updated formula is: Example 2 This embodiment provides an intelligent air-to-ground joint inspection method for unmanned aerial vehicles (UAVs) based on STGNN. Using the system from Embodiment 1, a typical scenario is intelligent inspection and management of traffic congestion on urban main roads. The process is as follows: Figure 3 As shown, starting with the inspection task trigger (timed / event-triggered), the process sequentially executes dynamic spatiotemporal graph construction, STGNN-based joint representation learning and prediction, multi-agent collaborative decision generation, task scheduling and instruction execution, with real-time collection of status and environmental feedback to complete anomaly diagnosis and knowledge accumulation. The knowledge base data feeds back into the model for iterative optimization, forming a closed-loop process, specifically including the following steps: S1. Dynamic Spatiotemporal Graph Construction: The system detects abnormal traffic flow at intersection A via a ground traffic detector, automatically triggering a congestion inspection task and activating nodes such as drones, roadside cameras, and traffic signal controllers along that road segment and its upstream and downstream areas. Each node is abstracted as a graph node, and node attributes are configured (drone: latitude and longitude, remaining battery power, coverage radius; camera: shooting direction, congestion index; traffic signal controller: timing scheme, operating status, etc.). Based on the spatial adjacency relationships between nodes, traffic flow correlation, and historical interaction data, the system... Define neighbor nodes, dynamically generate edges and assign weights, and construct a dynamic spatiotemporal graph that evolves in real time. ,like Figure 4 As shown, using drones (UAV1, UAV2), cameras (Cam1, Cam2), and signal controllers (SC1) in the inspection scenario as nodes, a dynamic spatiotemporal graph is constructed through abstract mapping. The nodes are connected by edges through location association, power association, task association, etc., which intuitively shows the spatiotemporal relationship between the nodes.

[0044] S2. Joint Representation and Prediction Based on STGNN: The dynamic spatiotemporal graph is input into a pre-trained STGNN model. After input layer format conversion, the spatial attention convolution module learns the importance weights between nodes using a multi-head attention mechanism to capture spatial dependencies between nodes. Nodes in congested areas are assigned high attention coefficients; for example, airborne nodes are made to pay more attention to ground nodes ahead of their flight paths and those that may have occlusion relationships. The temporal trend of traffic flow is analyzed through causal dilated convolution and gating mechanisms in the temporal recurrent convolution module. Finally, the graph update module aggregates the spatiotemporal features and updates them according to the update formula. Complete residual connections and layer normalization, update the spatiotemporal graph, and output the hidden state representation of each graph node at the next time step. This representation integrates its own history, neighbor node information, and global context.

[0045] S3. Multi-agent cooperative decision generation: based on the node hidden state representation output in step S2. Preliminary action suggestions are generated for each air and ground node through a policy network. (e.g., the drone's next flight direction and speed, the ground camera's rotation angle); then, through a centralized value network, combined with the priority of "congestion mitigation" and resource constraints, the initial suggestions are globally optimized to complete the global state value assessment. Finally, the collaborative optimizer outputs the final collaborative decision. (Such as UAV1 patrolling intersection A, UAV2 warning downstream intersection B, signal timing adjustment, etc.).

[0046] S4. Task Scheduling and Execution: The task scheduling optimization module of the decision layer transforms collaborative decisions into precise control commands, which are then distributed to each node via the network layer. When UAV1 detects a rear-end collision at intersection A, it updates the dynamic spatiotemporal graph with this abnormal event as a high-priority attribute. The closed-loop feedback mechanism triggers the STGNN model to re-infer, generating optimization strategies (such as UAV1 continuously monitoring the accident point, dynamically adjusting traffic signal timing, and ground robots providing on-site traffic guidance). S5. Anomaly Diagnosis and Knowledge Accumulation: When a potential inspection anomaly is detected at any node (such as equipment overheating or loose components), the spatiotemporal graph neural network engine of the decision layer uses the STGNN model to perform graph reasoning on the dynamic spatiotemporal graph using graph association information. Combined with the spatiotemporal association data of the accident point node, it determines that the congestion is caused by lane occupancy, assisting traffic police in rapid response. After the accident is handled, the scheduling strategy, handling process, and traffic impact data of this event are stored in the knowledge base. Reinforcement learning is used to iteratively train the STGNN model to optimize the attention coefficient in the model. , Learnable convolution kernel parameters can improve the model's decision-making efficiency for similar scenarios.

[0047] S6, Inspection Closed Loop: After traffic at intersection A returns to normal, the application layer automatically generates an inspection and handling report for this inspection, the system ends the current task, all equipment resumes normal patrol, and the dynamic spatiotemporal map is updated to the normal inspection node status.

[0048] The training data for the STGNN model includes historical inspection data, event handling records, and multi-scenario inspection data simulated by the digital twin simulation module. The multi-scenario inspection data includes inspection scenario data under different weather conditions, holidays, and sudden accidents. The model training process can combine multi-source data from spaceborne remote sensing and aerial photography to improve the model's adaptability to all-domain inspection scenarios.

[0049] The present invention discloses an intelligent air-ground linkage inspection system for unmanned aerial vehicles based on STGNN. It adopts a four-layer architecture design of perception layer, network layer, decision layer and application layer. The core relies on the spatiotemporal graph neural network (STGNN) to realize collaborative modeling and intelligent decision-making of heterogeneous devices. In the scenarios of traffic congestion inspection, anomaly handling and traffic diversion scheduling on urban main roads, it can achieve the efficient governance goal of "second-level perception-minute response-hour closed loop".

[0050] The STGNN-based UAV air-ground linkage intelligent inspection system and method of the present invention are also applicable to scenarios such as power line inspection, illegal construction investigation, public security, and park inspection. Only by adjusting the configuration of the perception layer equipment, STGNN model training data and task priority rules according to the scenario requirements, efficient air-ground linkage intelligent inspection can be achieved, which has strong scenario adaptability and versatility.

[0051] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the present invention.

Claims

1. A drone-based air-to-ground integrated intelligent inspection system based on STGNN, characterized in that: It includes a perception layer, a network layer, a decision layer, and an application layer. The perception layer is bidirectionally connected to the decision layer via the network layer, and the application layer interacts and communicates with the decision layer. The perception layer consists of several heterogeneous air nodes and ground nodes, and is responsible for collecting multi-dimensional heterogeneous inspection data. The network layer is based on 5G or private network communication technology to realize bidirectional, highly reliable, low-latency transmission of inspection data and control commands between perception layer nodes and between the perception layer and the decision layer. The decision-making layer is equipped with a spatiotemporal graph neural network engine, a task scheduling optimization module, and a digital twin simulation module. The spatiotemporal graph neural network engine is used to abstract air and ground nodes into graph nodes, generate dynamic edges and assign edge weights based on the spatial distance between nodes, communication link status, task correlation, and historical interaction data, construct a dynamic spatiotemporal graph with real-time evolution of node attributes, and run a pre-trained STGNN model to process the dynamic spatiotemporal graph to complete node state prediction, anomaly correlation reasoning, and collaborative decision suggestion generation. The task scheduling optimization module is used to generate the optimal UAV flight path planning scheme and ground equipment scheduling control commands based on the collaborative decision suggestions output by the STGNN model, combined with the inspection task priority and resource constraints. The digital twin simulation module is used to construct a virtual inspection environment that is synchronized with the physical world, enabling inspection task rehearsal, scheduling strategy evaluation, and STGNN model training. The application layer is used to provide users with a human-computer interaction interface for monitoring inspection tasks, managing abnormal alarms, and generating inspection reports.

2. The UAV air-to-ground integrated intelligent inspection system according to claim 1, characterized in that: The STGNN model includes an input layer, a spatial attention convolution module, a temporal recurrent convolution module, and a graph update module; The spatial attention convolutional module employs a multi-head attention mechanism with edge features, and the attention coefficient weights are normalized as follows: in, Represents a node The set of neighboring nodes, ; The maximum interaction distance threshold; The temporal cyclic convolution module employs a gated temporal convolutional network, with the gating mechanism as follows: ； The graph update module uses residual connections and layer normalization to update the dynamic spatiotemporal graph. The update formula is as follows: 。 3. The UAV air-to-ground integrated intelligent inspection system according to claim 1, characterized in that: The aerial node includes at least one type of drone, which is equipped with one or more sensing payloads, such as a visible light camera, an infrared imaging device, and a lidar. The node attributes of the drone include remaining battery power, health status, and coverage radius.

4. The UAV air-to-ground integrated intelligent inspection system according to claim 1, characterized in that: The ground node includes one or more of the following: a high-definition PTZ camera, a ground mobile robot, and a fixed sensor; the fixed sensor includes one or more of the following: a traffic flow detector, a vibration sensor, and an environmental sensor.

5. The UAV air-to-ground integrated intelligent inspection system according to claim 1, characterized in that: The node attributes of the dynamic spatiotemporal graph include equipment type, spatial location, working status, load capacity, and real-time observation data; the edge weights represent the degree of cooperation or information flow intensity between the devices; and the dynamic spatiotemporal graph evolves in real time as the nodes move and the inspection task progresses.

6. The UAV air-to-ground integrated intelligent inspection system according to claim 1, characterized in that: The resource constraints include the drone's battery power threshold, airspace flight restrictions, and communication link quality.

7. The UAV air-to-ground integrated intelligent inspection system according to claim 1, characterized in that: The decision-making layer is also equipped with a knowledge base, which is used to store anomaly handling cases, inspection and scheduling strategies, and equipment status data, providing data support for the iterative training and optimization of the STGNN model.

8. The UAV air-to-ground integrated intelligent inspection system according to claim 1, characterized in that: The system is equipped with a closed-loop feedback mechanism, which is used to transmit the device execution status data and environmental feedback data collected by the perception layer back to the spatiotemporal graph neural network engine of the decision layer in real time, drive the spatiotemporal graph neural network engine to update the dynamic spatiotemporal graph and trigger the STGNN model to re-infer, forming a "perception-decision-execution-feedback" inspection closed loop.

9. A UAV-based air-to-ground intelligent inspection method based on STGNN, characterized in that, The UAV air-to-ground integrated intelligent inspection system according to any one of claims 1 to 8 includes the following steps: S1. Dynamic Spatiotemporal Graph Construction: Each air and ground node in the perception layer is abstracted as a graph node and its attributes are configured. Edges are then dynamically generated and weighted based on the spatial distance between graph nodes, communication link status, task relevance, and historical interaction data, constructing a real-time evolving dynamic spatiotemporal graph, represented as follows: ; S2. Joint Representation and Prediction Based on STGNN: The dynamic spatiotemporal graph constructed in step S1 is input into the pre-trained STGNN model. The spatial attention convolution module uses a multi-head attention mechanism with edge features to capture the spatial dependencies between nodes. The temporal recurrent convolution module uses a gated temporal convolution network to mine the temporal evolution of node attributes. Then, the graph update module uses residual connections and layer normalization to aggregate spatiotemporal feature information, update the dynamic spatiotemporal graph, and output the hidden state representation of each graph node at the next time step. The attention coefficient weights of the multi-head attention mechanism are normalized as follows: ； The gating mechanism of gated temporal convolutional networks is as follows: ； The update formula for the graph update module is: ； S3. Multi-agent collaborative decision generation: Based on the node hidden state representation output in step S2, preliminary action suggestions are generated for each air node and ground node through the policy network. Then, the global state is evaluated by the centralized value network in combination with the inspection task priority and resource constraints. The dispersed action suggestions are coordinated and optimized to output the final collaborative decision. S4. Task Scheduling and Execution: The task scheduling optimization module of the decision layer transforms the final collaborative decision of step S3 into control commands, which are then sent to the corresponding air nodes and ground nodes through the network layer. At the same time, the module collects equipment execution status data and environmental feedback data in real time through a closed-loop feedback mechanism and sends them back to the spatiotemporal graph neural network engine of the decision layer to update the dynamic spatiotemporal graph and form an inspection closed loop. S5. Anomaly Diagnosis and Knowledge Accumulation: When a potential inspection anomaly is detected at any node, the spatiotemporal graph neural network engine of the decision layer performs graph reasoning on the dynamic spatiotemporal graph through the STGNN model to help locate the root cause of the anomaly, complete the anomaly handling, and store the anomaly handling cases in the knowledge base configured by the decision layer for iterative training and optimization of the STGNN model.

10. The UAV air-to-ground coordinated intelligent inspection method according to claim 9, characterized in that: The training data for the STGNN model includes historical inspection data, incident handling records, and multi-scenario inspection data simulated by the digital twin simulation module. The multi-scenario inspection data includes inspection scenario data under different weather conditions, holidays, and emergencies.

Images