Inspection methods, devices, equipment, and media based on digital airspace systems

Abstract

This invention discloses an inspection method, apparatus, equipment, and medium based on a digital airspace system. It uses a pre-trained neural network model to extract and calibrate features from multi-source sensing and monitoring data of the area to be inspected, obtaining structured semantic descriptions and time-series measurements. These are then combined with external meteorological spatial data and input into a MoE-based digital airspace expert model for multimodal fusion. The generated multimodal fusion representation is enhanced using a variational autoencoder and input into a multimodal transformer decoder for decoding, resulting in a set of entity triples to construct a digital airspace knowledge graph. An attention enhancement mechanism based on graph neural networks is employed to enhance the node representations in the knowledge graph, which are then input into a conditional generative adversarial network model for multi-task prediction, generating inspection decision suggestions for UAV execution. This effectively improves the utilization rate of multi-source sensing data, thereby increasing the inspection efficiency of the UAV.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to an inspection method, apparatus, equipment and medium based on a digital airspace system. Background Technology

[0002] Digital airspace systems are an indispensable component of UAV power line inspection. They are digital platforms that use ground-based and UAV-mounted sensors to update aerial information in real time, and are the foundation for realizing automated and intelligent UAV inspections.

[0003] Currently, digital airspace systems integrate and process multi-source sensing data from heterogeneous sensors and external sources to provide UAVs with comprehensive environmental perception capabilities. However, these information sources are different, and when used directly, problems such as errors in the sensing data, conflicts between them, and even sensor failures may occur, which can easily mislead inspection tasks and reduce the inspection efficiency of UAVs.

[0004] Therefore, improving the utilization rate of multi-source sensing data used by digital airspace systems for UAV inspection, in order to improve the inspection efficiency of UAVs, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This invention provides an inspection method, apparatus, equipment, and medium based on a digital airspace system, addressing the problem of how to improve the utilization rate of multi-source sensing data used by the digital airspace system for UAV inspection, thereby improving the inspection efficiency of UAVs.

[0006] To address the aforementioned technical problems, the first aspect of this invention provides an inspection method based on a digital airspace system, comprising:

[0007] Acquire multi-source sensing and monitoring data and external meteorological spatial data of the area to be inspected;

[0008] Based on the type of the multi-source sensing and monitoring data, several pre-trained neural network models are used to process the multi-source sensing and monitoring data to obtain structured semantic descriptions and time-series measurement values.

[0009] The external meteorological spatial data, the structured semantic description, and the time-series measurement values ​​are input into a digital airspace expert model based on the MoE architecture for fusion processing to obtain a multimodal fusion representation.

[0010] The multimodal fusion representation is enhanced by a variational autoencoder, and the resulting enhanced multimodal fusion representation is input into a multimodal transformer decoder for decoding to obtain a set of entity triples to construct a digital spatial knowledge graph.

[0011] The digital airspace knowledge graph is processed using an attention enhancement mechanism based on graph neural networks to obtain attention-enhanced node representations, which are then input into a conditional generative adversarial network model for task prediction, generating inspection decision suggestions for the area to be inspected so that the UAV can execute them.

[0012] As one preferred embodiment, the multi-source sensing monitoring data includes unstructured multi-source sensing monitoring data, which includes unstructured point cloud data and unstructured image data; wherein,

[0013] The process involves using several pre-trained neural network models to process the multi-source sensing and monitoring data according to its type, to obtain structured semantic descriptions and time-series measurements, including:

[0014] Geometric features are obtained by extracting features from the unstructured point cloud based on a pre-trained 3D convolutional neural network model and spatial attention mechanism.

[0015] The unstructured image data is used to extract features based on a pre-trained residual neural network model to obtain visual features;

[0016] A cross-modal encoder is used to align and fuse the geometric features and the visual features to generate fused features;

[0017] The fused features are transformed into the structured semantic description by a pre-trained multimodal transformer decoder.

[0018] As one preferred embodiment, the multi-source sensing monitoring data further includes structured multi-source sensing time-series monitoring data; wherein,

[0019] The step of processing the multi-source sensing and monitoring data using several pre-trained neural network models according to the type of the multi-source sensing and monitoring data to obtain structured semantic descriptions and time-series measurement values ​​also includes:

[0020] The structured multi-source sensing time-series monitoring data is transformed into a time-series structure sequence using a sliding time window.

[0021] The temporal structure sequence is input into a pre-trained deep learning network model to calculate the similarity between the temporal structure sequence and the historical temporal structure sequence through an attention mechanism, thereby obtaining a similarity distribution.

[0022] Based on the error characteristics of the sensors corresponding to the structured multi-source sensing time-series monitoring data and the similarity distribution, a data deviation correction matrix is ​​generated;

[0023] The data deviation correction matrix is ​​applied to the structured multi-source sensing time-series monitoring data to obtain corrected time-series measurements.

[0024] As one preferred embodiment, the digital airspace expert model includes several expert network models and a gating network; wherein,

[0025] The training process of the digital airspace expert model includes:

[0026] Historical external meteorological spatial data, historical structured semantic descriptions, and historical time-series measurements are used as training set data, and the training set data is labeled with corresponding multi-task labels; the multi-task labels include airspace status, environmental risk level, regional weather classification, and inspection decision suggestions;

[0027] The training data labeled with multiple task tags is aligned and augmented to form an input dataset, which is then passed to the gating network to obtain the output probability.

[0028] The corresponding expert network model is selected based on the output probability to process the input dataset, and the corresponding output results are obtained by weighted summation based on the output probabilities assigned by the gating network to form a multimodal fusion representation prediction result.

[0029] A loss function is used to evaluate the loss between the multimodal fusion representation prediction result and the multi-task label, and the gradient of the loss with respect to the model parameters is calculated using the backpropagation algorithm to update the parameters of each expert network model and the gating network.

[0030] As one preferred embodiment, the multimodal transformer decoder includes an enhanced self-attention layer, a hierarchical entity recognition layer, and a relation extraction network layer; wherein,

[0031] The process of inputting the obtained multimodal fusion enhanced representation into a multimodal transformer decoder for decoding to obtain a set of entity triples for constructing a digital spatial knowledge graph includes:

[0032] The multimodal fusion enhancement representation is input into the enhancement self-attention layer for processing to capture long-range dependencies in the multimodal fusion enhancement representation;

[0033] The hierarchical entity recognition layer performs entity recognition and extraction on the long-range dependency to obtain basic entities; the basic entities include meteorological elements, geographical features, and equipment status.

[0034] Based on the relationship extraction network layer, the relationships between the basic entities are extracted, and a normalized set of entity triples and their corresponding confidence scores are generated; the relationships include spatial relationships, causal relationships and temporal relationships.

[0035] The entity triple set is transformed into a natural language description using a preset prompt template, and query suggestions are constructed by combining domain knowledge.

[0036] The preset prompt template and the query hint are input into the large language model for knowledge reasoning, and the quality control of each reasoning result is performed using the confidence scores as weights.

[0037] The inference results output by the large language model are integrated to form a unified knowledge system for constructing a digital spatial knowledge graph.

[0038] As one preferred embodiment, the graph neural network includes a gated attention layer, a temporal attention layer, and a residual connection layer; wherein,

[0039] The process of using an attention enhancement mechanism based on graph neural networks to process the digitized spatial knowledge graph to obtain attention-enhanced node representations includes:

[0040] The edges and nodes in the digital spatial knowledge graph are standardized, and a multi-scale neighborhood feature aggregator is constructed for each node to aggregate the structural feature representations of each node from local to global at multiple scales.

[0041] The gated attention layer calculates the task attention score for each of the structural feature representations, and then performs a weighted summation on each of the structural feature representations based on the task attention scores to obtain the fused scale feature representation.

[0042] The temporal attention score of each structural feature representation is calculated based on the temporal attention layer, and the structural feature representations are weighted and summed according to the temporal attention scores to obtain the fused temporal feature representation.

[0043] The residual connection layer processes the fused scale feature representation and the fused temporal feature representation to achieve online updating of node representations. The processing steps of the gated attention layer, the temporal attention layer and the residual connection layer are iteratively executed to obtain attention-enhanced node representations.

[0044] As one preferred embodiment, the conditional generative adversarial network model includes a generator with an encoding layer, a decoding layer, and a multi-task branching layer, as well as a Transformer-based multi-task discriminator; wherein...

[0045] The attention-enhanced node representations are input into a conditional generative adversarial network model for task prediction, generating inspection decision suggestions for the area to be inspected, including:

[0046] The node representation is encoded using the encoding layer to obtain an encoding result, and the encoding result and random noise are mapped to the target output space through the decoding layer to obtain a decoded output result;

[0047] The decoding output is processed by the multi-task branch layer to obtain the airspace status prediction result, regional weather prediction result, environmental risk prediction result, and inspection decision suggestion prediction result for the area to be inspected.

[0048] The multi-task discriminator evaluates the airspace state prediction results, the regional weather prediction results, the environmental risk prediction results, and the inspection decision suggestion prediction results, and adjusts the inspection decision suggestion prediction results based on the evaluation results to obtain inspection decision suggestions for the area to be inspected.

[0049] A second aspect of the present invention provides an inspection device based on a digital airspace system, comprising:

[0050] The data acquisition module is used to acquire multi-source sensing and monitoring data and external meteorological spatial data of the area to be inspected;

[0051] The data processing module is used to process the multi-source sensing and monitoring data using several pre-trained neural network models according to the type of the multi-source sensing and monitoring data, so as to obtain structured semantic descriptions and time-series measurement values.

[0052] The feature fusion module is used to input the external meteorological spatial data, the structured semantic description and the time-series measurement values ​​into a digital airspace expert model based on the MoE architecture for fusion processing to obtain a multimodal fusion representation;

[0053] The graph construction module is used to enhance the multimodal fusion representation through a variational autoencoder, and input the obtained multimodal fusion enhanced representation into a multimodal transformer decoder for decoding to obtain a set of entity triples to construct a digital spatial knowledge graph.

[0054] The decision generation module is used to process the digital airspace knowledge graph using an attention enhancement mechanism based on graph neural networks to obtain attention-enhanced node representations, which are then input into a conditional generative adversarial network model for task prediction, generating inspection decision suggestions for the area to be inspected so that the UAV can execute them.

[0055] A third aspect of the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the inspection method based on a digital airspace system as described above.

[0056] A fourth aspect of the present invention provides a computer-readable storage medium comprising a stored computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the inspection method based on a digital airspace system as described above.

[0057] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following:

[0058] (1) For different types of data, a pre-trained neural network model is used for processing, which can efficiently and accurately extract structured semantic descriptions and time series measurements, avoiding the time and complexity of training the model from scratch; at the same time, meteorological data is taken into consideration, making the decision-making process closer to the actual environment and improving the accuracy and practicality of the decision.

[0059] (2) Through the MoE architecture, the most suitable expert model can be selected for processing according to the characteristics of different data types, realizing the effective fusion of multimodal data and improving the richness and accuracy of data representation; the variational autoencoder is used for enhancement processing, which can further improve the robustness and generalization ability of data representation, and provide a stable and reliable foundation for the subsequent decoding process.

[0060] (3) By using the multimodal transformer decoder to decode the enhanced multimodal fusion representation, entity triples can be accurately extracted, providing strong technical support for the construction of digital airspace knowledge graphs. It also helps to explore the intrinsic connections between data and provide a basis for subsequent decision-making. By processing the digital airspace knowledge graph through the attention mechanism, key nodes and relationships can be highlighted, and the accuracy and relevance of node representation can be improved. Combined with attention-enhanced node representation, the conditional generative adversarial network model can generate inspection decision suggestions that are closer to reality, avoiding the limitations of human experience and improving the level of intelligent decision-making. Attached Figure Description

[0061] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1This is a flowchart of an inspection method based on a digital airspace system provided in a certain embodiment of the present invention;

[0063] Figure 2 This is a structural diagram of an inspection device based on a digital airspace system provided in a certain embodiment of the present invention;

[0064] Figure 3 This is a structural diagram of an electronic device provided in a certain embodiment of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings and examples. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0066] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0067] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0068] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0069] In one embodiment, such as Figure 1 As shown, the first aspect of the present invention provides an inspection method based on a digital airspace system, comprising:

[0070] S1. Acquire multi-source sensing and monitoring data and external meteorological spatial data of the area to be inspected;

[0071] S2. Based on the type of the multi-source sensing and monitoring data, several pre-trained neural network models are used to process the multi-source sensing and monitoring data to obtain structured semantic descriptions and time-series measurement values.

[0072] S3. Input the external meteorological spatial data, the structured semantic description and the time-series measurement values ​​into the digital airspace expert model based on the MoE architecture for fusion processing to obtain a multimodal fusion representation;

[0073] S4. The multimodal fusion representation is enhanced by a variational autoencoder, and the resulting enhanced multimodal fusion representation is input into a multimodal transformer decoder for decoding to obtain a set of entity triples to construct a digital spatial knowledge graph.

[0074] S5. The digital airspace knowledge graph is processed using an attention enhancement mechanism based on graph neural networks to obtain attention-enhanced node representations, which are then input into a conditional generative adversarial network model for task prediction, generating inspection decision suggestions for the area to be inspected so that the UAV can execute them.

[0075] Specifically, this invention uses various sensors on the ground and on drones (such as ground equipment including electromagnetic radiation meters, wireless network cards, climate observation instruments, anemometers, etc., and airborne equipment including wireless network cards, ADS-B (Automatic Dependent Surveillance-Broadcast) devices, lidar, and cameras) to update multi-source heterogeneous sensing and monitoring data of the area to be inspected in real time. Among them, the multi-source sensing and monitoring data includes unstructured multi-source sensing and monitoring data (unstructured multi-source sensing and monitoring data includes unstructured point cloud data and unstructured image data, i.e. point cloud data collected by lidar and high-definition image / video streams acquired by cameras) and structured multi-source sensing time-series monitoring data (i.e., time-series data collected by electromagnetic radiation meters, wireless network cards, climate observation instruments, anemometers, lidar, etc.). This invention also incorporates external meteorological spatial data accessed through Geographic Information Systems (GIS), meteorological platforms, and ground stations: meteorological knowledge data (such as the climate knowledge base built into the digital airspace system and real-time weather data provided by meteorological service APIs), electronic map data, remote sensing data, airspace restriction information (including management information such as no-fly zones, restricted airspace, and temporary flight airspace), and topographic and geomorphological data (including information such as altitude, slope, and water distribution). This invention also considers multi-source heterogeneous sensing data and meteorological data, making the decision-making process more closely aligned with the actual environment, thereby improving the accuracy and practicality of decision-making.

[0076] Based on the type (structured and unstructured) of multi-source sensing and monitoring data, appropriate pre-trained neural network models are selected for processing. For example, convolutional neural networks (CNNs) can be used for feature extraction and classification of image data; for video data, three-dimensional convolutional neural networks (3D CNNs) or recurrent neural networks (RNNs) can be used for processing. The processed data is then subjected to structured semantic description and extraction of time-series measurements to provide a foundation for subsequent data fusion. The pre-trained neural network models have high recognition accuracy and generalization ability, and can accurately process multi-source sensing and monitoring data.

[0077] External meteorological spatial data, structured semantic descriptions, and time-series measurements are preprocessed to ensure consistency in data format and dimensions. This preprocessed data is then input into a digital spatial domain expert model based on the MoE (Mixture of Experts) architecture. This model consists of multiple expert network models and a gating network, each adept at handling specific data types. The gating network selects the appropriate expert network model to process the data, resulting in a multimodal fusion representation. The MoE architecture enhances the model's flexibility and scalability, enabling it to handle various data types. The multimodal fusion representation integrates information from multiple data sources, improving the utilization rate of multi-source sensing data and enhancing the accuracy and reliability of subsequent decision-making.

[0078] The multimodal fusion representation is enhanced using a Variational Autoencoder (VAE): the encoder network of the VAE maps the representation to a probability distribution in the latent space, samples latent variables from this distribution, and the decoder network of the VAE, conditioned on these latent variables, reconstructs the original features of multiple modalities simultaneously, including sensor numerical sequences, image features, point cloud features, etc. Then, by minimizing the joint loss function of reconstruction error and KL divergence, the fusion representation is forced to learn the essential features and intrinsic relationships of different modalities, thus obtaining the enhanced multimodal fusion representation. This invention significantly enhances the integrity and robustness of the representation using this self-supervised learning approach, providing a more reliable feature foundation for subsequent semantic parsing and knowledge extraction, thereby extracting latent features from the data and improving the robustness and generalization ability of the multimodal fusion representation. Specifically, in the autoencoder selection part, a β-VAE can be used instead of a standard VAE; alternatively, a VQ-VAE can be used for discrete representation learning; preferably, a conditional VAE is used to achieve representation learning for specific scenarios.

[0079] The enhanced multimodal fusion representation is input into a multimodal transformer decoder. The decoder utilizes an enhanced self-attention layer, a hierarchical entity recognition layer, and a relation extraction network layer to progressively decode and refine the input data, obtaining a set of entity triples used to construct a digital spatial knowledge graph. After representing the digital spatial knowledge graph as graph-structured data, an attention enhancement mechanism based on graph neural networks is used to process the graph-structured data to extract node representations. These node representations are then weighted using an attention mechanism to obtain attention-enhanced node representations, which are input into a conditional generative adversarial network (GAN) model to generate inspection decision suggestions for the area to be inspected. Finally, these inspection decision suggestions are input into the UAV control system to guide the UAV in its inspection mission. This invention improves the accuracy and reliability of node representations through an attention enhancement mechanism, providing a foundation for subsequent task prediction. Furthermore, the use of a GAN model to generate task prediction results related to the input data improves the accuracy and diversity of predictions.

[0080] This invention integrates multi-source sensing and monitoring data and external meteorological spatial data, and employs advanced neural network models, MoE architecture, variational autoencoders, multimodal transformer decoders, graph neural network attention enhancement mechanisms, and conditional generative adversarial network models to achieve comprehensive monitoring, accurate decision-making, and autonomous inspection of the area to be inspected. It has significant beneficial effects in improving inspection efficiency, reducing labor costs, and ensuring inspection safety.

[0081] In one embodiment, step S2 includes:

[0082] Geometric features are obtained by extracting features from the unstructured point cloud based on a pre-trained 3D convolutional neural network model and spatial attention mechanism.

[0083] The unstructured image data is used to extract features based on a pre-trained residual neural network model to obtain visual features;

[0084] A cross-modal encoder is used to align and fuse the geometric features and the visual features to generate fused features;

[0085] The fused features are transformed into the structured semantic description by a pre-trained multimodal transformer decoder.

[0086] Specifically, for unstructured multi-source sensing monitoring data, this invention uses a pre-trained 3D convolutional neural network model combined with a spatial attention mechanism to extract features from unstructured point cloud data. The spatial attention mechanism guides the model to focus on important geometric structural information in the point cloud, thereby obtaining geometric features. A pre-trained residual neural network model (ResNet) is used to extract features from unstructured image data. This addresses the gradient vanishing and gradient exploding problems encountered by deep neural networks during training by introducing residual structures, thereby extracting deep visual features from the image. A cross-modal encoder is then used to align and fuse these two types of features to learn the intrinsic connections between different modal features, fusing them into a unified representation, i.e., fused features. Finally, a pre-trained multimodal transformer decoder is used to transform the fused features into a structured semantic description. This means that by pre-training the multimodal transformer decoder on large-scale text data, deep bidirectional representations of language are captured to generate accurate and coherent semantic descriptions. In the multimodal feature extraction part, ViT can be used instead of ResNet, or SwinTransformer can be used for hierarchical feature extraction, such as novel convolutional network architectures like ConvNeXt. For point cloud data processing, PointNet++ can be used instead of 3D convolutional networks. Optionally, DGCNN can be used for dynamic graph convolution processing. Furthermore, Point Transformer can be used to process point cloud data.

[0087] This invention efficiently extracts geometric and visual features from unstructured point clouds and image data using pre-trained 3D convolutional neural networks and residual neural networks, respectively. It leverages the generalization ability of the pre-trained models to improve the accuracy and efficiency of feature extraction. The fused features generated after processing the geometric and visual features through a cross-modal encoder provide a rich information foundation for subsequent semantic description. Finally, a pre-trained multimodal transformer decoder transforms the fused features into structured semantic descriptions, realizing the conversion from unstructured data to structured data, facilitating subsequent analysis and application.

[0088] In one embodiment, step S2 further includes:

[0089] The structured multi-source sensing time-series monitoring data is transformed into a time-series structure sequence using a sliding time window.

[0090] The temporal structure sequence is input into a pre-trained deep learning network model to calculate the similarity between the temporal structure sequence and the historical temporal structure sequence through an attention mechanism, thereby obtaining a similarity distribution.

[0091] Based on the error characteristics of the sensors corresponding to the structured multi-source sensing time-series monitoring data and the similarity distribution, a data deviation correction matrix is ​​generated;

[0092] The data deviation correction matrix is ​​applied to the structured multi-source sensing time-series monitoring data to obtain corrected time-series measurements.

[0093] Specifically, for structured multi-source sensing time-series monitoring data, this invention constructs a deep learning network model—an LSTM (Long Short Term Memory) model—to learn the temporal characteristics and distribution patterns of historical data and establish a normal distribution pattern for sensor data. First, a sliding time window is used to convert the structured multi-source sensing time-series monitoring data into a time-series sequence, which is then input into the LSTM model. This allows the model to calculate the similarity between the current input data and the historical data distribution through an attention mechanism. Simultaneously, the error characteristics of each sensor are analyzed to determine the error range and patterns of each sensor, thereby assessing the degree to which each time-series structure sequence is affected by errors and generating a data deviation correction matrix to correct errors in the time-series measurements. Finally, the correction matrix is ​​applied to the structured multi-source sensing time-series monitoring data, and the time-series measurements are corrected one by one according to the correction coefficients in the correction matrix, outputting the corrected time-series measurements for subsequent analysis. Temporal Convolutional Network can be used instead of LSTM. Optionally, Informer can be used to process long-series time-series data; preferably, Transformer-XL is used to achieve longer-term dependency modeling.

[0094] This invention, through a sliding time window and a deep learning network model, can more effectively capture the dynamic changes of time-series data, improve the accuracy of data similarity calculation, thereby generating a more accurate data bias correction matrix and improving the accuracy of time-series measurements. Using a pre-trained deep learning network model, it can fully utilize the learning results of existing data, quickly adapt to the processing needs of new data, and enhance the model's generalization ability. The entire processing flow is highly automated, reducing manual intervention and improving processing efficiency and accuracy.

[0095] In one embodiment, the digital airspace expert model includes several expert network models and a gating network; wherein, the training process of the digital airspace expert model includes:

[0096] Historical external meteorological spatial data, historical structured semantic descriptions, and historical time-series measurements are used as training set data, and the training set data is labeled with corresponding multi-task labels; the multi-task labels include airspace status, environmental risk level, regional weather classification, and inspection decision suggestions;

[0097] The training data labeled with multiple task tags is aligned and augmented to form an input dataset, which is then passed to the gating network to obtain the output probability.

[0098] The corresponding expert network model is selected based on the output probability to process the input dataset, and the corresponding output results are obtained by weighted summation based on the output probabilities assigned by the gating network to form a multimodal fusion representation prediction result.

[0099] A loss function is used to evaluate the loss between the multimodal fusion representation prediction result and the multi-task label, and the gradient of the loss with respect to the model parameters is calculated using the backpropagation algorithm to update the parameters of each expert network model and the gating network.

[0100] Specifically, for training the digital airspace expert model, it is necessary to initialize it first. The expert network model is the core component of the MoE model. Each expert network model can be an independent small neural network or a more complex structure. During the initialization phase, it is necessary to select an appropriate network structure and parameters for each expert network model. The gating network is responsible for deciding which expert(s) should process the input data. During the initialization phase, it is necessary to set an appropriate structure and parameters for the gating network to ensure that it can accurately select the appropriate expert network model for different inputs.

[0101] Next, the training data is constructed, which includes the following: historical structured semantic descriptions (such as wind speed, air pressure, network information, electromagnetic field information, etc.), for example:

[0102] {

[0103] "sensor_data": {

[0104] Temperature: 25.6

[0105] "humidity": 65.3,

[0106] "wind_speed": 5.2,

[0107] "wind_direction": 180,

[0108] "pressure": 1013.2,

[0109] "electromagnetic_field": 0.5

[0110] },

[0111] "timestamp":"2024-01-20 14:30:00"

[0112] };

[0113] Scene semantic description, such as:

[0114] {

[0115] "scene_description": {

[0116] "obstacles": "There is a 30-meter-high power transmission tower 200 meters ahead, and a grove of trees about 15 meters to the left."

[0117] "environment":"Current visibility is good, with no significant weather obstructions."

[0118] "risk_factors": "Strong electromagnetic interference exists around the transmission line."

[0119] }

[0120] };

[0121] Historical external meteorological spatial data, such as:

[0122] {

[0123] "weather_forecast": {

[0124] "short_term":"Thunderstorms are expected in the next 2 hours",

[0125] "warning": "Severe convective weather warning"

[0126] },

[0127] "airspace_restrictions": {

[0128] "no_fly_zones": ["Region A","Region B"],

[0129] "height_limits":"Maximum flight altitude 300 meters",

[0130] "time_restrictions":"Flight restricted from 14:00 to 16:00"

[0131] }

[0132] };

[0133] There is also historical time-series measurement data, which will not be elaborated here. Multi-task annotation data needs to cover the spatial domain state, such as:

[0134] {

[0135] "flight_recommendations": {

[0136] "route_adjustment":"It is recommended to shift 200 meters eastward to avoid areas with strong electromagnetic interference."

[0137] "height_adjustment":"It is recommended to reduce the flight altitude to 150 meters".

[0138] "timing_suggestion":"It is recommended to complete the task in advance and avoid thunderstorms."

[0139] "risk_level":"Medium risk",

[0140] "priority_actions": [

[0141] "Adjust flight path to avoid power transmission towers"

[0142] Strengthen real-time weather monitoring ]

[0144] }

[0145] };

[0146] Environmental risk early warning and Regional weather forecasts, such as:

[0147] {

[0148] "risk_assessment": {

[0149] "electromagnetic_forecast": {

[0150] "intensity_map":"Prediction of electromagnetic field intensity within 50 meters of power transmission lines",

[0151] "interference_zones": ["Area A","Area B"],

[0152] "safe_corridors": "Recommended flight path coordinates"

[0153] },

[0154] "obstacle_analysis": {

[0155] "dynamic_obstacles":"Mobile tower crane operation early warning",

[0156] "temporary_restrictions":"temporary construction areas",

[0157] "bird_activity": "Bird migration activity prediction"

[0158] },

[0159] "signal_coverage": {

[0160] "weak_signal_areas":"Prediction of communication signal coverage blind spots",

[0161] "backup_channels": "Backup communication solutions"

[0162] }

[0163] }

[0164] };

[0165] Flight decision recommendations, such as:

[0166] {

[0167] "flight_recommendations": {

[0168] "route_adjustment":"It is recommended to shift 200 meters eastward to avoid areas with strong electromagnetic interference."

[0169] "height_adjustment":"It is recommended to reduce the flight altitude to 150 meters".

[0170] "timing_suggestion":"It is recommended to complete the task in advance and avoid thunderstorms."

[0171] "risk_level":"Medium risk",

[0172] "priority_actions": [

[0173] "Adjust flight path to avoid power transmission towers"

[0174] Strengthen real-time weather monitoring ]

[0176] }

[0177] }

[0178] The collected data is labeled using the aforementioned tags to provide the supervision information required for model training. The data labeled with multiple task tags is then cleaned, normalized, aligned, and enhanced (such as grayscale transformation and spatial filtering) to improve data quality and model training efficiency. This forms the input dataset, which is then passed to the gating network. The gating network outputs a probability distribution based on the features of the input data, indicating the importance of each expert network model in processing the current input.

[0179] Based on the output of the gating network, the corresponding expert network model is selected to process the input data. Each expert processes the input data in their area of ​​expertise and produces corresponding output results. Finally, the output results of each expert are weighted and summed according to the probabilities assigned by the gating network to form the final output, which is the multimodal fusion representation prediction result, which integrates the knowledge and information of multiple experts.

[0180] Standard loss functions (such as cross-entropy loss, mean squared error, etc.) are used to evaluate the gap between the model's predictions and the true labels. The gradient of the loss relative to the model parameters is calculated using backpropagation, and the parameters of the expert network and gating network are updated. The training process is iterated until the loss output by the digital spatial domain expert model is less than a preset loss threshold or a preset number of iterations is reached. Furthermore, to reduce computational costs during training, typically only a subset of experts are activated. For example, in a top-k routing strategy, only the k highest-scoring experts are activated. This sparse activation mechanism helps improve the model's training efficiency and generalization ability. To avoid some experts being overloaded while others are idle, load balancing needs to be considered during training. This means that the gating network must not only focus on accuracy but also ensure a relatively balanced workload for each expert. Load balancing can be achieved by introducing an additional load balance loss (LBL). Since MoE contains multiple expert network models, it may face the risk of overfitting. Therefore, appropriate regularization techniques (such as Dropout, L2 regularization, etc.) need to be used during training to prevent overfitting.

[0181] The digital airspace expert model adopts an attention-based routing-based expert selection strategy, enabling each expert to selectively process knowledge from different domains during the training phase. During inference, the most relevant expert output is selected based on the attention score and weighted fusion is performed to obtain a multimodal fusion representation. The digital airspace expert model can use a Switch Transformer to replace the MoE architecture, optionally adopting Dynamic Routing Networks, and preferably using Hierarchical MoE to achieve multi-level expert division of labor. In the routing strategy part, Top-K routing can also be used instead of attention routing, and reinforcement learning can be used to dynamically select experts.

[0182] This invention integrates historical external meteorological spatial data, historical structured semantic descriptions, and historical time-series measurements, along with corresponding multi-task labels, enabling the model to gain a more comprehensive understanding of the spatial state and thus improve prediction accuracy. Employing multi-modal fusion to represent prediction results fully leverages the complementarity between different modalities, enhancing the model's generalization ability and allowing it to better adapt to different scenarios and conditions. By labeling the training set data with corresponding multi-task labels, the model can simultaneously learn multiple related tasks, improving overall performance and efficiency.

[0183] In one embodiment, the multimodal transformer decoder includes an enhanced self-attention layer, a hierarchical entity recognition layer, and a relation extraction network layer; wherein,

[0184] The process of inputting the obtained multimodal fusion enhanced representation into a multimodal transformer decoder for decoding to obtain a set of entity triples for constructing a digital spatial knowledge graph includes:

[0185] The multimodal fusion enhancement representation is input into the enhancement self-attention layer for processing to capture long-range dependencies in the multimodal fusion enhancement representation;

[0186] The hierarchical entity recognition layer performs entity recognition and extraction on the long-range dependency to obtain basic entities; the basic entities include meteorological elements, geographical features, and equipment status.

[0187] Based on the relationship extraction network layer, the relationships between the basic entities are extracted, and a normalized set of entity triples and their corresponding confidence scores are generated; the relationships include spatial relationships, causal relationships and temporal relationships.

[0188] The entity triple set is transformed into a natural language description using a preset prompt template, and query suggestions are constructed by combining domain knowledge.

[0189] The preset prompt template and the query hint are input into the large language model for knowledge reasoning, and the quality control of each reasoning result is performed using the confidence scores as weights.

[0190] The inference results output by the large language model are integrated to form a unified knowledge system for constructing a digital spatial knowledge graph.

[0191] Specifically, the multimodal transformer decoder in this invention can also be considered as a hierarchical structure integrating multiple improved BERT networks; wherein, the enhanced self-attention layer is a RoBERTa network layer, the hierarchical entity recognition layer includes a SpanBERT network layer and a BERT-CRF network layer, and the relation extraction network layer includes an R-BERT network layer and a TDEER network layer. Therefore, the decoding process of the multimodal transformer decoder for the multimodal fusion enhanced representation is essentially a gradual decoding and refinement of the multimodal fusion enhanced representation using multiple improved BERT networks, including: using a multi-head self-attention mechanism through the RoBERTa network layer to calculate different... Attention scores between locations are used to capture long-range dependencies in multimodal fusion-enhanced representations. Then, an abstract feature is gradually transformed into a concrete semantic concept through a multi-layer Transformer structure. Specifically, the SpanBERT network layer is used to identify entity fragments in long-range dependencies and perform entity boundary detection. The BERT-CRF network layer is used to further refine, classify, and extract entities to obtain basic entities. Then, the relationships between basic entities are extracted based on the R-BERT and TDEER network layers, and the extracted relationships are normalized to generate a normalized set of entity triples and their corresponding confidence scores, which are used for quality control during subsequent knowledge graph construction.

[0192] Design specific prompt templates to transform entity triples into natural language descriptions. These templates should be concise and clear, accurately conveying the information within the entity triples. Combine pre-domain knowledge with the natural language descriptions to construct query hints, guiding the large language model in knowledge reasoning in subsequent steps. Input these prompt templates and the constructed query hints into the large language model, which has been fine-tuned for domain adaptation. Based on its powerful knowledge reasoning capabilities, this model supplements missing relations, eliminates semantic conflicts, and unifies conceptual expressions. Specifically, it unifies synonymous entities through entity linking technology and supplements implicit relation edges through relation reasoning. The process involves several steps: first, consistency checks eliminate contradictory relationships; second, during inference, confidence scores of each entity triple are used as weights to weight the inference results for quality control. Then, the inference results from the large language model are integrated to form a unified knowledge system. This integration process requires addressing issues such as information conflicts and duplication to ensure accuracy and consistency. Finally, the integrated knowledge is stored in a knowledge base in an appropriate format. Common storage methods include graph databases (such as Neo4j) and RDF storage systems (such as Jena). The stored knowledge is also optimized to improve query efficiency and accuracy, ultimately forming a digital spatial knowledge graph. Furthermore, rule-based reasoning can replace neural network reasoning in the knowledge graph relation reasoning part, and knowledge distillation can simplify the reasoning process. Moreover, probabilistic graphical model reasoning can be implemented further.

[0193] This invention utilizes a multimodal transformer decoder to more effectively perform entity recognition and relation extraction, thereby improving the accuracy and completeness of the digital spatial domain knowledge graph. The enhanced self-attention layer captures long-range dependencies in the representation, helping to more accurately understand the relationships between entities when constructing the knowledge graph. The introduction of multimodal fusion to enhance representations allows the solution to fully utilize various types of data, improving the richness and practicality of the knowledge graph. By converting entity triple sets into natural language descriptions using a preset prompt template and combining this with domain knowledge to construct query hints, which are then input into a large language model for knowledge reasoning, the efficiency and accuracy of reasoning are improved. Furthermore, the confidence score is used as a weight to perform quality control on the reasoning results, further ensuring the reliability of the knowledge graph.

[0194] In one embodiment, the graph neural network includes a gated attention layer, a temporal attention layer, and a residual connection layer; wherein, the step of processing the digitized spatial knowledge graph using a graph neural network-based attention enhancement mechanism to obtain attention-enhanced node representations includes:

[0195] The edges and nodes in the digital spatial knowledge graph are standardized, and a multi-scale neighborhood feature aggregator is constructed for each node to aggregate the structural feature representations of each node from local to global at multiple scales.

[0196] The gated attention layer calculates the task attention score for each of the structural feature representations, and then performs a weighted summation on each of the structural feature representations based on the task attention scores to obtain the fused scale feature representation.

[0197] The temporal attention score of each structural feature representation is calculated based on the temporal attention layer, and the structural feature representations are weighted and summed according to the temporal attention scores to obtain the fused temporal feature representation.

[0198] The residual connection layer processes the fused scale feature representation and the fused temporal feature representation to achieve online updating of node representations. The processing steps of the gated attention layer, the temporal attention layer and the residual connection layer are iteratively executed to obtain attention-enhanced node representations.

[0199] Specifically, based on knowledge graphs, this invention uses a dynamic graph attention network enhancement mechanism to achieve adaptive representation optimization based on drone scenarios:

[0200] The edges and nodes in the digital spatial knowledge graph are standardized to ensure data consistency and comparability. A multi-scale neighborhood feature aggregator is constructed for each node, namely: 1. Neighbor sampling: For each node, a certain number of neighbor nodes are sampled according to its position in the graph; 2. Feature aggregation: The features of the sampled neighbor nodes are aggregated to form the local feature representation of the node. This operation can be achieved through operations such as summation, averaging, and max pooling; 3. Multi-scale processing: The above neighbor sampling and feature aggregation process is repeated at different scales to capture structural information of different ranges. This allows for the aggregation of the structural feature representations of each node from local to global at multiple scales, capturing multi-level information of the nodes and improving the richness of node representation.

[0201] By employing a gated attention mechanism, the importance weights of features at different scales are dynamically adjusted according to the current task. This involves calculating the task attention score for each structural feature representation through a gated attention layer. Specifically, for each scale feature representation, its attention score related to the current task is calculated. This operation can be achieved by performing a dot product or bilinear transformation on the feature representation and the task-related query vector, followed by applying the softmax function to reflect the importance of different structural features to the current task. Furthermore, a weighted summation is performed on the corresponding structural feature representations based on the attention scores for each task to obtain a fused scale feature representation, which helps to highlight important features and suppress secondary features.

[0202] First, the structural feature representation of each node is transformed into a time series. Then, based on the temporal attention layer, for each time point in the time series, the attention score related to the current time point is calculated. This can be achieved by performing a dot product or bilinear transformation between the state representation of the current time point and the state representation of past time points, and then applying the softmax function to reflect the influence of structural features at different time points on the current task. The corresponding structural feature representations are weighted and summed according to the attention scores of each time point to obtain the fused temporal feature representation. This helps to capture the temporal dependence of the graph data, model the temporal evolution of node states, and improve the model's adaptability to dynamic changes.

[0203] By adding the fused scale feature representation and the fused temporal feature representation through the residual connection layer, the input information is preserved and gradient vanishing or exploding is prevented. The result is then normalized to ensure stable mean and variance, thus ensuring the stability of feature propagation and enabling online updating of node representations. This alleviates the gradient vanishing and oversmoothing problems in deep graph neural networks.

[0204] Finally, the processing of the gated attention layer, temporal attention layer, and residual connection layer is iterated multiple times to gradually optimize the node representation until the predetermined number of iterations or convergence condition is reached, resulting in attention-enhanced node representations. These representations not only contain local and global structural information of the nodes but also incorporate temporal evolution patterns and task-related dynamic adjustment information, which can be used for subsequent UAV scene analysis and decision-making tasks. Furthermore, in each iteration, the importance weights of features at different scales and the temporal evolution feature representations can be dynamically adjusted according to the current task requirements and the structural information of the graph, in order to gradually optimize the node representations and improve the model's accuracy and generalization ability. Moreover, for graph neural networks, this invention can use GraphSAGE instead of GAT, optionally employing GIN for graph isomorphic representation learning, and preferably using a spatiotemporal graph neural network; the attention mechanism can also use multi-head attention instead of gated attention, and sparse attention can be used to improve efficiency.

[0205] This invention utilizes an attention mechanism to focus on important nodes and edges in the graph, thereby generating more representative node representations. A multi-scale neighborhood feature aggregator captures the structural features of nodes at different scales, contributing to a more comprehensive understanding of the role and position of nodes in the graph. A temporal attention layer captures the temporal dependence of structural feature representations, aiding in the processing of dynamically changing graph data. A residual connection layer alleviates the gradient vanishing and oversmoothing problems in deep graph neural networks.

[0206] In one embodiment, the conditional generative adversarial network model includes a generator with an encoding layer, a decoding layer, and a multi-task branching layer, as well as a Transformer-based multi-task discriminator; wherein, the attention-enhanced node representation is input into the conditional generative adversarial network model for task prediction, generating inspection decision suggestions for the area to be inspected, including:

[0207] The node representation is encoded using the encoding layer to obtain an encoding result, and the encoding result and random noise are mapped to the target output space through the decoding layer to obtain a decoded output result;

[0208] The decoding output is processed by the multi-task branch layer to obtain the airspace status prediction result, regional weather prediction result, environmental risk prediction result, and inspection decision suggestion prediction result for the area to be inspected.

[0209] The multi-task discriminator evaluates the airspace state prediction results, the regional weather prediction results, the environmental risk prediction results, and the inspection decision suggestion prediction results, and adjusts the inspection decision suggestion prediction results based on the evaluation results to obtain inspection decision suggestions for the area to be inspected.

[0210] Specifically, this invention utilizes a multi-task prediction and generation framework based on conditional generative adversarial networks to generate high-quality solutions. The generator employs a Transformer-based encoder-decoder structure, including an encoding layer, a decoding layer, and a multi-task branching layer. The encoding layer encodes attention-enhanced node representations to extract global dependencies, and the decoding layer maps the encoded results and random noise to the target output space to obtain the decoded output. The multi-task branching layer then processes the decoded output to obtain the spatial state prediction results for the area to be inspected (i.e., a multi-dimensional state sequence within a future time window, including key indicators such as meteorological conditions, electromagnetic environment, and communication quality) and the regional weather prediction results (combining multi-source data to generate refined local weather forecasts). The system generates weather forecasts (including wind field distribution, precipitation probability, visibility changes, etc.), environmental risk predictions (environmental risk levels of the area to be inspected), and inspection decision-making recommendations (i.e., generating multi-level decision-making recommendations based on the current state and prediction results, including specific instructions such as route planning, altitude adjustment, and speed control). Each task has an independent fully connected header to generate prediction results for that specific task. Finally, a Transformer-based multi-task discriminator is used to evaluate the authenticity and task relevance of these prediction results, outputting authenticity scores and auxiliary classification results. If the authenticity score is lower than the evaluation score threshold, the decision-making recommendations are optimized based on the auxiliary classification results; otherwise, the inspection decision-making recommendation prediction results are used as the inspection decision-making recommendations for the area to be inspected. For example, the input includes: Node representation: Fusion features of region A (meteorological conditions, electromagnetic environment, communication quality); Current conditions: The current time period is the midday peak, and region B is temporarily closed to flights; Output: Airspace status: Meteorological conditions of region A in the next 1 hour (wind speed ↑, precipitation probability ↑), electromagnetic environment (interference ↑), communication quality (↓); Regional weather: Wind field distribution of region A in the next 1 hour (northwest wind 15m / s), precipitation probability (80%), visibility (<1km); Environmental risk: High risk level of drone conflict in region A; Inspection decision: It is recommended to adjust the flight path to avoid region A, reduce the flight altitude to 100m, and control the speed at 10m / s. In addition, the generative architecture can use Diffusion Models instead of GAN, optionally using Flow-based Models, preferably using an autoregressive generative model; In the multi-task learning part, soft parameter sharing can be used instead of hard parameter sharing, a progressive learning strategy can be used, and further, dynamic task weight adjustment can be achieved. It should be noted that all the models used directly in this invention are trained on historical data before they can be applied directly, such as the conditional generative adversarial network model, the multimodal transformer decoder and its layers, etc.

[0211] This invention employs multi-task collaborative prediction, which, through a multi-task branching layer, enables the model to simultaneously generate airspace status, regional weather, environmental risks, and inspection decision suggestions, providing comprehensive airspace management support. It utilizes a Transformer-based encoder-decoder structure and a multi-task discriminator to capture complex dynamic airspace features and generate highly realistic and task-relevant prediction results. Through discriminator evaluation and feedback, the generator's output is dynamically adjusted to improve the rationality and feasibility of decision suggestions. Random noise and adversarial training mechanisms are introduced to enhance the model's robustness to noise and anomalous data, while simultaneously generating diverse decision-making schemes.

[0212] This application addresses the issue of improving the utilization rate of multi-source sensing data used by digital airspace systems for UAV inspections, thereby enhancing UAV inspection efficiency. It proposes an inspection method based on digital airspace systems, combining multimodal learning with data distribution monitoring technology to construct an end-to-end data processing framework. This framework enables intelligent processing of the entire process from multi-source heterogeneous sensing data acquisition, enhancement, knowledge extraction, and decision generation: multimodal feature extraction and temporal data calibration are performed using a pre-trained neural network model to improve data quality; an expert model based on the MoE architecture is developed, employing an attention routing strategy to dynamically fuse knowledge from different domains. This paper introduces variational autoencoders for representation enhancement and combines them with multimodal transformer decoders to achieve accurate semantic parsing. It innovatively applies large language models to knowledge graph construction, improving the completeness of knowledge representation through entity linking and relational reasoning. A dynamic graph attention network is designed to achieve scene-adaptive representation optimization. Finally, high-quality multi-task prediction and decision generation are achieved based on conditional generative adversarial networks. This significantly improves data quality and decision accuracy, realizing end-to-end processing from raw perception data to intelligent decision-making. It provides reliable digital airspace support for UAV power line inspection, significantly enhancing the intelligence level of UAV power line inspection and possessing significant engineering application value.

[0213] It should be noted that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders.

[0214] In another embodiment, such as Figure 2 As shown, a second aspect of the present invention provides an inspection device based on a digital airspace system, comprising:

[0215] Data acquisition module 10 is used to acquire multi-source sensing monitoring data and external meteorological spatial data of the area to be inspected;

[0216] The data processing module 20 is used to process the multi-source sensing and monitoring data using several pre-trained neural network models according to the type of the multi-source sensing and monitoring data, so as to obtain structured semantic descriptions and time-series measurement values.

[0217] The feature fusion module 30 is used to input the external meteorological spatial data, the structured semantic description and the time series measurement values ​​into the digital airspace expert model based on the MoE architecture for fusion processing to obtain a multimodal fusion representation;

[0218] The graph construction module 40 is used to enhance the multimodal fusion representation through a variational autoencoder, and input the obtained multimodal fusion enhanced representation into a multimodal transformer decoder for decoding to obtain a set of entity triples to construct a digital spatial knowledge graph.

[0219] The decision generation module 50 is used to process the digital airspace knowledge graph using an attention enhancement mechanism based on graph neural networks to obtain attention-enhanced node representations, which are then input into a conditional generative adversarial network model for task prediction, generating inspection decision suggestions for the area to be inspected so that the UAV can execute them.

[0220] It should be noted that each module in the aforementioned inspection device based on a digital airspace system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module. For specific limitations regarding the inspection device based on a digital airspace system, please refer to the limitations of the inspection method based on a digital airspace system described above; both have the same function and role, and will not be repeated here.

[0221] A third aspect of the present invention provides an electronic device comprising:

[0222] Processor, memory, and bus;

[0223] The bus is used to connect the processor and the memory;

[0224] The memory is used to store operation instructions;

[0225] The processor is configured to execute operations corresponding to the inspection method based on a digital airspace system as shown in the first aspect of this application by invoking the operation instructions.

[0226] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3The illustrated electronic device 5000 includes a processor 5001 and a memory 5003. The processor 5001 and the memory 5003 are connected, for example, via a bus 5002. Optionally, the electronic device 5000 may also include a transceiver 5004. It should be noted that in practical applications, the transceiver 5004 is not limited to one type, and the structure of this electronic device 5000 does not constitute a limitation on the embodiments of this application.

[0227] Processor 5001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 5001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0228] Bus 5002 may include a path for transmitting information between the aforementioned components. Bus 5002 may be a PCI bus or an EISA bus, etc. Bus 5002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0229] The memory 5003 may be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0230] The memory 5003 is used to store application code that executes the scheme of this application, and its execution is controlled by the processor 5001. The processor 5001 is used to execute the application code stored in the memory 5003 to implement the content shown in any of the foregoing method embodiments.

[0231] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers.

[0232] The fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the inspection method based on a digital airspace system shown in the first aspect of this application.

[0233] Another embodiment of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0234] Furthermore, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0235] In summary, this invention relates to the field of information technology and discloses an inspection method, apparatus, equipment, and medium based on a digital airspace system. It extracts and calibrates features from multi-source sensing and monitoring data of the area to be inspected using a pre-trained neural network model, obtaining structured semantic descriptions and time-series measurements. These are then combined with external meteorological and spatial data and input into a digital airspace expert model based on the MoE architecture for multimodal fusion. The generated multimodal fusion representation is enhanced using a variational autoencoder and input into a multimodal transformer decoder for decoding, obtaining a set of entity triples to construct a digital airspace knowledge graph. An attention enhancement mechanism based on graph neural networks is used to enhance the node representations in the knowledge graph, which are then input into a conditional generative adversarial network model for multi-task prediction, generating inspection decision suggestions for UAV execution. This effectively improves the utilization rate of multi-source sensing data, thereby increasing the inspection efficiency of the UAV.

[0236] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts among the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; refer to the description of the method embodiments for relevant details. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0237] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the technical principles of this invention, and these improvements and substitutions should also be considered within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the scope of the claims.

Claims

1. An inspection method based on a digital airspace system, characterized in that, include: Acquire multi-source sensing and monitoring data and external meteorological spatial data of the area to be inspected; The multi-source sensing monitoring data includes unstructured point cloud data, unstructured image data, and structured multi-source sensing time-series monitoring data. Based on the type of the multi-source sensing and monitoring data, several pre-trained neural network models are used to process the multi-source sensing and monitoring data to obtain structured semantic descriptions and time-series measurement values. The external meteorological spatial data, the structured semantic description, and the time-series measurement values ​​are input into a digital airspace expert model based on the MoE architecture, which includes several expert network models and a gated network, for fusion processing to obtain a multimodal fusion representation. The multimodal fusion representation is enhanced by a variational autoencoder, and the resulting enhanced multimodal fusion representation is input into a multimodal transformer decoder for decoding to obtain a set of entity triples to construct a digital spatial domain knowledge graph. This process includes: inputting the enhanced multimodal fusion representation into an enhanced self-attention layer for processing to obtain long-range dependencies; performing entity recognition and extraction processing on the long-range dependencies through a hierarchical entity recognition layer to obtain basic entities including meteorological elements, geographical features, and equipment status; extracting spatial location relationships, causal relationships, and temporal relationships between the basic entities based on a relation extraction network layer to generate a set of entity triples and their corresponding confidence scores; converting the set of entity triples into natural language descriptions using a preset prompt template, and constructing query prompts by combining domain knowledge, which are then input into a large language model for knowledge reasoning, with each reasoning result weighted by its confidence score; and integrating the reasoning results output by the large language model to construct a digital spatial domain knowledge graph. A graph neural network-based attention enhancement mechanism is used to process the digital spatial knowledge graph to obtain attention-enhanced node representations. This includes: standardizing edges and nodes in the digital spatial knowledge graph and constructing a multi-scale neighborhood feature aggregator for each node to aggregate structural feature representations from local to global at multiple scales; calculating task attention scores for each structural feature representation through a gated attention layer, and performing a weighted summation based on these task attention scores to obtain a fused scale feature representation; calculating temporal attention scores for each structural feature representation based on a temporal attention layer, and performing a weighted summation based on these temporal attention scores to obtain a fused temporal feature representation; processing the fused scale feature representation and the fused temporal feature representation through a residual connection layer to achieve online updating of node representations, and iteratively executing the processing steps of the gated attention layer, the temporal attention layer, and the residual connection layer to obtain attention-enhanced node representations; and inputting these representations into a conditional generative adversarial network model for task prediction, generating inspection decision suggestions for the area to be inspected so that the UAV can execute them.

2. The inspection method based on a digital airspace system according to claim 1, characterized in that, The process involves using several pre-trained neural network models to process the multi-source sensing and monitoring data according to its type, to obtain structured semantic descriptions and time-series measurements, including: Geometric features are obtained by extracting features from the unstructured point cloud based on a pre-trained 3D convolutional neural network model and spatial attention mechanism. The unstructured image data is used to extract features based on a pre-trained residual neural network model to obtain visual features; A cross-modal encoder is used to align and fuse the geometric features and the visual features to generate fused features; The fused features are transformed into the structured semantic description by a pre-trained multimodal transformer decoder.

3. The inspection method based on a digital airspace system according to claim 1, characterized in that, The step of processing the multi-source sensing and monitoring data using several pre-trained neural network models according to the type of the multi-source sensing and monitoring data to obtain structured semantic descriptions and time-series measurement values ​​also includes: The structured multi-source sensing time-series monitoring data is transformed into a time-series structure sequence using a sliding time window. The temporal structure sequence is input into a pre-trained deep learning network model to calculate the similarity between the temporal structure sequence and the historical temporal structure sequence through an attention mechanism, thereby obtaining a similarity distribution. Based on the error characteristics of the sensors corresponding to the structured multi-source sensing time-series monitoring data and the similarity distribution, a data deviation correction matrix is ​​generated; The data deviation correction matrix is ​​applied to the structured multi-source sensing time-series monitoring data to obtain corrected time-series measurements.

4. The inspection method based on a digital airspace system according to claim 1, characterized in that, The training process of the digital airspace expert model includes: Historical external meteorological spatial data, historical structured semantic descriptions, and historical time-series measurements are used as training set data, and the training set data is labeled with corresponding multi-task labels; the multi-task labels include airspace status, environmental risk level, regional weather classification, and inspection decision suggestions; The training data labeled with multiple task tags is aligned and augmented to form an input dataset, which is then passed to the gating network to obtain the output probability. The corresponding expert network model is selected based on the output probability to process the input dataset, and the corresponding output results are obtained by weighted summation based on the output probabilities assigned by the gating network to form a multimodal fusion representation prediction result. A loss function is used to evaluate the loss between the multimodal fusion representation prediction result and the multi-task label, and the gradient of the loss with respect to the model parameters is calculated using the backpropagation algorithm to update the parameters of each expert network model and the gating network.

5. The inspection method based on a digital airspace system according to claim 1, characterized in that, The conditional generative adversarial network model includes a generator with an encoding layer, a decoding layer, and a multi-task branching layer, as well as a Transformer-based multi-task discriminator; wherein... The attention-enhanced node representations are input into a conditional generative adversarial network model for task prediction, generating inspection decision suggestions for the area to be inspected, including: The node representation is encoded using the encoding layer to obtain an encoding result, and the encoding result and random noise are mapped to the target output space through the decoding layer to obtain a decoded output result; The decoding output is processed by the multi-task branch layer to obtain the airspace status prediction result, regional weather prediction result, environmental risk prediction result, and inspection decision suggestion prediction result for the area to be inspected. The multi-task discriminator evaluates the airspace state prediction results, the regional weather prediction results, the environmental risk prediction results, and the inspection decision suggestion prediction results, and adjusts the inspection decision suggestion prediction results based on the evaluation results to obtain inspection decision suggestions for the area to be inspected.

6. An inspection device based on a digital airspace system, characterized in that, include: The data acquisition module is used to acquire multi-source sensing and monitoring data and external meteorological spatial data of the area to be inspected; The multi-source sensing monitoring data includes unstructured point cloud data, unstructured image data, and structured multi-source sensing time-series monitoring data. The data processing module is used to process the multi-source sensing and monitoring data using several pre-trained neural network models according to the type of the multi-source sensing and monitoring data, so as to obtain structured semantic descriptions and time-series measurement values. The feature fusion module is used to input the external meteorological spatial data, the structured semantic description and the time-series measurement values ​​into a digital airspace expert model based on the MoE architecture, which includes several expert network models and a gated network, for fusion processing to obtain a multimodal fusion representation. The knowledge graph construction module is used to enhance the multimodal fusion representation through a variational autoencoder, and input the obtained multimodal fusion enhanced representation into a multimodal transformer decoder for decoding to obtain a set of entity triples to construct a digital spatial domain knowledge graph. This includes: inputting the multimodal fusion enhanced representation into an enhanced self-attention layer for processing to obtain long-range dependencies; performing entity recognition and extraction processing on the long-range dependencies through a hierarchical entity recognition layer to obtain basic entities including meteorological elements, geographical features, and equipment status; extracting spatial location relationships, causal relationships, and temporal relationships between the basic entities based on a relation extraction network layer to generate a set of entity triples and their corresponding confidence scores; converting the set of entity triples into natural language descriptions using a preset prompt template, and constructing query prompts in conjunction with domain knowledge, which are then input into a large language model for knowledge reasoning, with each reasoning result weighted by its confidence score; and integrating the reasoning results output by the large language model to construct a digital spatial domain knowledge graph. The decision generation module is used to process the digital spatial knowledge graph using an attention enhancement mechanism based on graph neural networks to obtain attention-enhanced node representations. This includes: standardizing edges and nodes in the digital spatial knowledge graph and constructing a multi-scale neighborhood feature aggregator for each node to aggregate the structural feature representations of each node from local to global at multiple scales; calculating task attention scores for each structural feature representation through a gated attention layer and performing a weighted summation based on the task attention scores to obtain a fused scale feature representation; calculating temporal attention scores for each structural feature representation based on a temporal attention layer and performing a weighted summation based on the temporal attention scores to obtain a fused temporal feature representation; processing the fused scale feature representation and the fused temporal feature representation through a residual connection layer to achieve online updating of node representations, and iteratively executing the processing steps of the gated attention layer, the temporal attention layer, and the residual connection layer to obtain attention-enhanced node representations; and inputting these representations into a conditional generative adversarial network model for task prediction, generating inspection decision suggestions for the area to be inspected so that the UAV can execute them.

7. An electronic device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the inspection method based on a digital airspace system as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the inspection method based on a digital airspace system as described in any one of claims 1 to 5.

Images