Unmanned aerial vehicle monitoring method and device and electronic equipment

Abstract

The invention provides an unmanned aerial vehicle monitoring method and apparatus, and an electronic device. The method comprises the steps of setting a plurality of Internet of Things base stations in a target area; configuring a regional network of the target region based on the plurality of Internet of Things base stations; acquiring flight data and environment data of the unmanned aerial vehicle in the target area acquired from the plurality of terminals based on the regional network; and monitoring the flight of the unmanned aerial vehicle based on the flight data and the environmental data. According to the application, the exclusive local area network of the target area can be constructed by setting the plurality of Internet of Things base stations in the target area; the flight data and the environment data of the unmanned aerial vehicle in the target area are collected from the plurality of terminals, and the flight of the unmanned aerial vehicle is monitored according to the flight data and the environment data, so that comprehensive flight data and environment data can be obtained, and comprehensive monitoring of the plurality of unmanned aerial vehicles in the target area is realized; air potential safety hazards can be found in time, and monitoring efficiency and monitoring accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and more specifically, to a UAV monitoring method, device, and electronic equipment. Background Technology

[0002] In recent years, unmanned aerial vehicles (UAVs) such as multi-rotor UAVs, fixed-wing UAVs, and vertical takeoff and landing (VTOL) aircraft have been widely used in agriculture, logistics, energy, inspection, and security, leading to increasingly complex flight scenarios. With the rapid increase in the number of UAVs, problems such as illegal flights, abnormal behavior, and potential air safety hazards have become increasingly prominent, especially in low-altitude airspace, posing serious challenges to urban management, airport operations, border security, and the protection of key facilities.

[0003] Therefore, how to monitor unmanned aerial vehicles in target areas has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for monitoring unmanned aerial vehicles (UAVs), which solves the technical problem of how to monitor UAVs in a target area in the prior art.

[0005] Firstly, this application provides a method for monitoring unmanned aerial vehicles, including: Set up multiple smart network base stations in the target area; A regional network for the target area is configured based on multiple Internet of Things (IoT) base stations; Based on the regional network, flight data and environmental data of unmanned aerial vehicles in the target area are acquired from multiple terminals. The flight of the unmanned aerial vehicle is monitored based on the flight data and the environmental data.

[0006] In some embodiments, the terminal includes the Internet base station, the unmanned aerial vehicle (UAV) terminal, and a third-party system; the step of acquiring flight data and environmental data of the UAV in the target area collected from multiple terminals based on the regional network includes: The intelligent network base station is controlled to collect the first flight data of the unmanned aerial vehicle based on radio signals; The control terminal for the aircraft collects second flight data and first environmental data of the unmanned aerial vehicle based on airborne sensors; the aircraft terminal is mounted on the unmanned aerial vehicle. The third-party system is controlled to collect second environmental data from the unmanned aerial vehicle. The flight data and the environmental data are acquired based on the data transmission channel in the regional network; the flight data includes the first flight data and the second flight data; the environmental data includes the first environmental data and the second environmental data.

[0007] In some embodiments, monitoring the flight of the unmanned aerial vehicle based on the flight data and the environmental data includes: The fusion characteristics of the unmanned aerial vehicle are determined based on the flight data and the environmental data; When the flight anomaly of the unmanned aerial vehicle is determined based on the fusion features, the flight anomaly type of the unmanned aerial vehicle is identified based on a classification model; the classification model is obtained by training an initial classification model based on the sample fusion features and the sample flight anomaly types corresponding to the sample fusion features. Predict the flight trajectory of the unmanned aerial vehicle in the future time period based on the flight anomaly type and the fused features; Based on the flight trajectory, the fusion feature prediction, and the risk assessment model, the risk level of the unmanned aerial vehicle is predicted for a future time period. The flight control strategy for the unmanned aerial vehicle is determined based on the risk level. The flight of the unmanned aerial vehicle is controlled based on the flight control strategy.

[0008] In some embodiments, determining the fusion characteristics of the unmanned aerial vehicle based on the flight data and the environmental data includes: The multimodal monitoring data is initially fused based on a multimodal data fusion algorithm to obtain preliminary fusion results; the multimodal monitoring data includes the flight data and the environmental data. Based on the preliminary fusion results, the preliminary fusion features of the unmanned aerial vehicle are obtained; The weights of the monitoring data for each modality are determined based on an attention mechanism; The fusion characteristics of the unmanned aerial vehicle are determined based on the preliminary fusion results and the weights.

[0009] In some embodiments, monitoring the flight of the unmanned aerial vehicle based on the flight data and the environmental data includes: If the flight data and / or the environmental data meet the preset flight anomaly rules, an alarm message is generated based on the flight data and / or the environmental data; The flight of the unmanned aerial vehicle is controlled based on the alarm information.

[0010] In some embodiments, the unmanned aerial vehicle monitoring method further includes: The situation map of the target area is updated based on the flight data and environmental data of the unmanned aerial vehicle in the target area; the situation map marks the flight data and environmental data and shows the flight trajectory of the unmanned aerial vehicle in the future time period; The interface displays the situation map, alarm information, and risk level of the unmanned aerial vehicle.

[0011] In some embodiments, the unmanned aerial vehicle monitoring method further includes: Each module in the unmanned aerial vehicle, the smart network base station, and the unmanned aerial vehicle monitoring device is respectively regarded as an intelligent agent; The tasks in the unmanned aerial vehicle monitoring method steps are assigned to the intelligent agent based on the distributed task allocation rules.

[0012] In some embodiments, the process of assigning tasks to the agent in the unmanned aerial vehicle monitoring method based on distributed task allocation rules includes: The task is broken down into multiple sub-tasks; Each subtask is assigned to a different agent based on the agent's current operational capabilities.

[0013] Secondly, this application provides an unmanned aerial vehicle monitoring device, comprising: The configuration module is used to set up multiple smart network base stations in the target area; A configuration module is used to configure the regional network of the target area based on multiple Internet of Things base stations; The acquisition module is used to acquire flight data and environmental data of unmanned aerial vehicles in the target area collected from multiple terminals based on the regional network; The monitoring module is used to monitor the flight of the unmanned aerial vehicle based on the flight data and the environmental data.

[0014] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the above-described method when executing the program through the computer program.

[0015] The unmanned aerial vehicle (UAV) monitoring method, device, and electronic equipment provided in this application can construct a dedicated local area network for the target area by setting up multiple smart network base stations in the target area and configuring the regional network of the target area through the multiple smart network base stations. By acquiring flight data and environmental data of UAVs in the target area collected from multiple terminals through the regional network, and monitoring the flight of UAVs based on the flight data and environmental data, comprehensive flight data and environmental data can be obtained, realizing comprehensive monitoring of multiple UAVs in the target area. This can promptly detect potential air safety hazards and improve monitoring efficiency and accuracy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the unmanned aerial vehicle monitoring method provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the structure of the unmanned aerial vehicle monitoring device provided in the embodiments of this application.

[0019] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0022] Figure 1 This is a flowchart illustrating the unmanned aerial vehicle monitoring method provided in the embodiments of this application, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130, and 140. These method steps are merely one possible implementation of this application.

[0023] Step 110: Set up multiple smart network base stations in the target area.

[0024] Specifically, the unmanned aerial vehicle monitoring method provided in this application is applicable to ground systems. These ground systems may include various electronic devices with displays and web browsing capabilities, including but not limited to servers, smartphones, tablets, laptops, and desktop computers. The ground system may include multiple internet of things (IoT) base stations, a control subsystem, and a cloud platform.

[0025] The unmanned aerial vehicle (UAV) monitoring method provided in this application is executed by an UAV monitoring device, which can be a hardware device independently set up in the ground system or a software program running in the ground system.

[0026] A smart network base station is a communication device used to connect IoT devices and sensors, enabling data transmission, device management, and network connectivity. It connects terminal devices such as unmanned aerial vehicles to a regional network through wireless communication technology.

[0027] Unmanned aerial vehicles (UAVs) are aircraft that do not have an onboard pilot and have their own power system.

[0028] The target area is a specific geographical region equipped with multiple smart network base stations. A regional network is configured based on these base stations to acquire data related to unmanned aerial vehicles (UAVs), thereby monitoring the flight of these UAVs. The target area can be a low-altitude region within this specific geographical area.

[0029] Multiple intelligent network base stations can be deployed in the target area. These base stations possess integrated sensing and communication (ISAC) capabilities, enabling them to provide communication services and collect sensing data from unmanned aerial vehicles (UAVs) within the target area. For example, intelligent network base stations can not only provide highly reliable, high-bandwidth, and low-latency communication services, but also detect, locate, measure speed, and identify UAVs that are not communicating or are communicating abnormally via radio signals. Furthermore, intelligent network base stations possess edge computing capabilities, allowing for the preprocessing and intelligent analysis of some data.

[0030] Intelligent network base stations can be built based on communication technologies such as 5G-Advanced (5G-A). Utilizing the ISAC function of 5G-A in intelligent network base stations, ground systems possess network-level sensing capabilities, enabling them to detect and monitor uncontrolled or abnormally communicating unmanned aerial vehicles, significantly enhancing monitoring coverage and blind spot elimination performance.

[0031] Step 120: Configure the regional network of the target area based on multiple smart network base stations.

[0032] Specifically, multiple smart network base stations constitute a distributed low-altitude sensing network.

[0033] A regional network is a network system built upon multiple intelligent network base stations to cover a target area, enabling data communication and information exchange between unmanned aerial vehicles (UAVs) and terminal devices within that area. Multiple intelligent network base stations form a distributed regional network covering the target area, working collaboratively to achieve comprehensive perception and monitoring of UAVs within that area.

[0034] Step 130: Acquire flight data and environmental data of unmanned aerial vehicles in the target area collected from multiple terminals based on the regional network.

[0035] Specifically, flight data is data used to characterize the state and behavior of an unmanned aerial vehicle during flight, and may include, for example, flight altitude, speed, heading, and acceleration.

[0036] Environmental data refers to physical, chemical, and biological parameters used to characterize the natural environment and meteorological conditions within the flight area of ​​an unmanned aerial vehicle (UAV). For example, it may include temperature, humidity, wind speed, and rainfall.

[0037] Terminals may include smart network base stations, unmanned aerial vehicle (UAV) terminals, and third-party systems.

[0038] Multiple terminals can collect data from different positions and angles, enabling the acquisition of more comprehensive flight and environmental data. Multiple terminals can be distributed across different locations, covering a larger target area and achieving monitoring of large areas, avoiding blind spots. The target area may include multiple unmanned aerial vehicles (UAVs).

[0039] Step 140: Monitor the flight of the unmanned aerial vehicle based on flight data and environmental data.

[0040] Specifically, the acquired flight data and environmental data can be processed. For example, a multimodal fusion algorithm based on an attention mechanism can be used to fuse and perform deep reasoning on multimodal heterogeneous data such as flight data and environmental data to generate monitoring results. The monitoring results can include the current flight status of the UAV. Based on the monitoring results, it can be determined whether the UAV meets the preset flight plan and safety standards, such as whether it deviates from the flight path or exceeds the flight altitude limit. The flight parameters and flight mission plan of the UAV can be automatically adjusted to ensure the safe flight of the UAV in the target area and to ensure that the flight of the UAV does not cause safety hazards.

[0041] The unmanned aerial vehicle (UAV) monitoring method provided in this application embodiment can construct a dedicated local area network for the target area by setting up multiple smart network base stations in the target area and configuring the regional network of the target area through the multiple smart network base stations. By acquiring flight data and environmental data of UAVs in the target area collected from multiple terminals through the regional network, and monitoring the flight of UAVs based on the flight data and environmental data, comprehensive flight data and environmental data can be obtained, realizing comprehensive monitoring of multiple UAVs in the target area. This can promptly detect potential air safety hazards and improve monitoring efficiency and accuracy.

[0042] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0043] In some embodiments, the terminal includes an Internet of Things base station, an unmanned aerial vehicle (UAV) terminal, and a third-party system; step 130 includes: The control system uses a smart network base station to collect the first flight data of the unmanned aerial vehicle based on radio signals. The control terminal for the unmanned aerial vehicle (UAV) collects second flight data and first environmental data based on onboard sensors; the control terminal is installed on the UAV. Control a third-party system to collect secondary environmental data from the unmanned aerial vehicle; Flight data and environmental data are acquired based on data transmission channels in a regional network; the flight data includes first flight data and second flight data; the environmental data includes first environmental data and second environmental data.

[0044] Specifically, smart network base stations can use radio signals to detect, locate, measure speed, and identify unmanned aerial vehicles (UAVs), and collect perception data, including the UAVs' initial flight data.

[0045] The first flight data may include radar-based sensing information, such as position, velocity, and radar cross section (RCS), as well as signal characteristics.

[0046] The aircraft terminal is an intelligent terminal installed on an unmanned aerial vehicle (UAV) to collect telemetry and payload data such as high-definition video streams, thermal imaging images, millimeter-wave radar data, and gyroscope data from the UAV, thereby obtaining second flight data and first environmental data. The aircraft terminal can acquire the second flight data and first environmental data through the airborne sensors on the UAV.

[0047] Telemetry data from unmanned aerial vehicles can include GPS position, altitude, speed, attitude, battery level, motor status, flight control parameters, and internal sensor readings.

[0048] Airborne payload data can include high-definition video streams, thermal images, audio, airborne radar data, and environmental sensor data, such as temperature, humidity, and atmospheric pressure.

[0049] The aircraft terminal has preliminary edge intelligence processing capabilities and the ability to collaborate with ground systems as an intelligent agent. The aircraft terminal can communicate with ground systems through a regional network.

[0050] Third-party systems refer to external systems that are independent of unmanned aerial vehicles (UAVs) and their onboard sensors and smart network base stations, and are used to collect environmental data from UAVs. For example, third-party systems may include meteorological systems, no-fly zone systems, and third-party sensors.

[0051] Third-party meteorological information, terrain data, electronic fence / no-fly zone database, flight route planning information, and known obstacle information; The data types in the first and second flight data may overlap, but their sources are different. Similarly, the data types in the first and second environmental data may overlap, but their sources are different.

[0052] Flight data and environmental data from multiple terminals can be aggregated through smart network base stations in the ground system or the cloud. The aggregated multimodal heterogeneous data is cleaned, synchronized, converted in format, and initially fused. Multimodal fusion and inference algorithms are then used to perform in-depth analysis on the preprocessed multi-source multimodal data.

[0053] It can also acquire historical data of unmanned aerial vehicles (UAVs), such as historical flight trajectories, abnormal event records, and databases of common behavioral patterns. This data, combined with network-sensing positioning data, the UAV's own GPS data, onboard video recognition results, telemetry status, and historical behavioral data, allows for the monitoring of UAV flight.

[0054] The unmanned aerial vehicle monitoring method provided in this application embodiment can collect data from different positions and angles through multiple terminals, thereby obtaining more comprehensive flight data and environmental data and avoiding monitoring blind spots.

[0055] In some embodiments, step 140 includes: Determine the fusion characteristics of unmanned aerial vehicles based on flight data and environmental data; When unmanned aerial vehicle (UAV) flight anomalies are determined based on fusion features, the type of flight anomaly is identified based on a classification model. The classification model is obtained by training an initial classification model based on the sample fusion features and the sample flight anomaly types corresponding to the sample fusion features. Predict the flight trajectory of unmanned aerial vehicles in the future time period based on flight anomaly type and fusion features; Predict the risk level of unmanned aerial vehicles in the future time period based on flight trajectory, fusion feature prediction and risk assessment model; Determine the flight control strategy for unmanned aerial vehicles based on risk levels; Flight control strategies are used to control the flight of unmanned aerial vehicles.

[0056] Among them, the fusion characteristics of unmanned aerial vehicles (UAVs) determined based on flight data and environmental data include: The multimodal monitoring data is initially fused based on a multimodal data fusion algorithm to obtain preliminary fusion results; the multimodal monitoring data includes flight data and environmental data. Preliminary fusion features of the unmanned aerial vehicle are obtained based on the preliminary fusion results; The weights of the monitoring data for each modality are determined based on an attention mechanism; The fusion characteristics of the unmanned aerial vehicle are determined based on the preliminary fusion results and weights.

[0057] Specifically, the models in this application embodiment can be constructed using convolutional neural network models, fully connected neural network models, recurrent neural network (RNN) models, and long short-term memory (LSTM) neural network models, etc. The ground system can be configured accordingly to enable the training, updating, and management of each model.

[0058] Fusion features are feature representations that integrate the flight data and environmental data of unmanned aerial vehicles (UAVs) to form a more comprehensive and accurate description of the flight status and environmental conditions of UAVs.

[0059] Flight trajectory is the spatial movement path of an unmanned aerial vehicle (UAV) calculated by a prediction model within a future time period based on factors such as its current flight anomaly type, fusion characteristics, and the environment it is in.

[0060] Risk level is a classification of the safety risks that unmanned aerial vehicles may face or cause in the future.

[0061] Unmanned aerial vehicles (UAVs) are methods and rules developed based on the current risk situation of UAVs to adjust parameters such as flight attitude, speed, and heading of UAVs in order to ensure flight safety, stability, and efficient mission completion.

[0062] Multimodal data fusion algorithms can effectively integrate data from different sources, formats, sampling rates, and confidence levels to build a unified understanding of the current low-altitude situation. Before feature extraction, raw or pre-aligned flight and environmental data are concatenated or fused at the input layer of a neural network model to obtain preliminary fusion results. Flight and environmental data can be collectively referred to as monitoring data. Feature extraction and preliminary analysis can be performed independently on monitoring data with strong modal independence, and then fusion can be performed at the decision or output layer of the neural network model.

[0063] Preliminary feature extraction is performed on the monitoring data from each modality to obtain preliminary fusion features. At the intermediate feature level of the neural network model, the importance of different modal information is dynamically measured and the weights of each monitoring data point are determined using an attention mechanism (AM) or cross-modal attention (CMA). The fusion features of the unmanned aerial vehicle (UAV) are determined based on the preliminary fusion results and weights. A heterogeneous graph of the monitoring data can be constructed, and information propagation and feature learning can be performed on the heterogeneous graph using a graph neural network (GNN) to achieve multi-source information fusion and correlation reasoning.

[0064] After obtaining the fused features, it is possible to determine whether there are any abnormalities in the current unmanned aerial vehicle (UAV). If no abnormalities are found, relevant data will continue to be collected for further evaluation. If abnormalities are found, the type of flight anomaly of the UAV will be determined and relevant control measures will be implemented.

[0065] For unknown or rare types of flight anomalies, algorithms such as Isolation Forest, One-Class SVM, Autoencoders, Variational Autoencoders, and GAN-based anomaly detection can be used to learn the distribution of normal flight patterns and identify data points that deviate from this distribution. Identification can be performed based on specific modalities (such as video anomaly detection) or fused features. For known types of flight anomalies (such as spiral descent or rapid yaw), classification models (such as SVM, Random Forest, and deep neural network models) are used for identification.

[0066] RNNs and LSTMs can be used to analyze historical telemetry data and environmental sequence data of unmanned aerial vehicles (UAVs) to capture temporal correlations, predict the flight trajectory of UAVs in future time periods, and detect anomalies or patterns in the sequences. Based on multimodal correlation-based anomaly judgment, the system uses fused features obtained by associating information from different modalities to infer potential power system failures or priority alarms. Analyzing the UAV's current flight trajectory, speed, altitude changes, mission status reports, and onboard payload information, combined with historical data and mission type, identifies its behavioral patterns (such as inspection, spillage, hovering, reconnaissance, illegal crossing, etc.) and attempts to determine its current intent. Behavioral pattern recognition and intent determination can be performed using behavior tree-based, finite state machine-based, or deep learning sequence models.

[0067] It can comprehensively assess collision risk, intrusion risk, and malfunction risk by combining the UAV's flight trajectory, current status, flight environment, behavior patterns, types of flight anomalies, and airspace traffic. Based on the risk assessment model (such as trajectory prediction combined with LSTM and Kalman filtering), it can output the risk level of the UAV in the future.

[0068] Based on the level of risk, and taking into account factors such as flight mission requirements, aircraft performance, and environmental conditions, a corresponding flight control strategy is determined. For example, when the risk level is low, conventional flight control is maintained; when the risk level is high, flight control strategies such as evasive maneuvers, adjustments to flight altitude, or speed are adopted.

[0069] When an anomalies are detected in an unmanned aerial vehicle (UAV), causal graph models or rule-based inference engines can be used to analyze the causal relationship chains between various modal data, tracing the root cause of the anomaly, such as sudden changes in the external environment, internal system failures, or illegal command intervention, thereby enabling appropriate flight control. Intelligent analysis algorithms can be used for real-time situational awareness, abnormal behavior detection and identification, behavior pattern recognition and intent judgment, risk assessment and prediction, and causal reasoning and root cause analysis.

[0070] The unmanned aerial vehicle monitoring method provided in this application embodiment is based on a multimodal data fusion algorithm to fuse and understand data from different data sources, thereby enabling more accurate detection of flight anomaly types and risk assessment, and improving the intelligence and reliability of monitoring.

[0071] In some embodiments, step 140 includes: If the flight data and / or environmental data meet the preset flight anomaly rules, generate alarm information based on the flight data and / or environmental data; Controlling the flight of unmanned aerial vehicles based on alarm information.

[0072] Specifically, the flight anomaly rule is based on airspace management rules and UAV technical parameters. It judges flight data and environmental data, and determines that when the data exceeds preset thresholds for speed, altitude, area and / or battery power, it is considered a flight anomaly.

[0073] Alarm messages are generated based on flight data and / or environmental data when they meet preset flight anomaly rules. They are used to indicate that the unmanned aerial vehicle's flight status is abnormal and that corresponding measures need to be taken.

[0074] When flight data, environmental data, or both flight data and environmental data exceed the corresponding preset thresholds, it can be preliminarily determined that the unmanned aerial vehicle (UAV) has experienced a flight anomaly. Alarm information can be generated based on this abnormal data, and corresponding control commands can be generated based on the alarm information to control the flight of the UAV.

[0075] When an unmanned aerial vehicle (UAV) is initially determined to have a flight anomaly, a neural network model can be used to predict the type and risk level of the flight anomaly, and the flight of the UAV can be controlled based on the prediction results.

[0076] The unmanned aerial vehicle monitoring method provided in this application generates alarm information by pre-setting flight anomaly rules, combining flight data and environmental data, and controls flight based on alarm information and risk assessment. It can effectively identify flight anomalies and provide early warning of risks.

[0077] In some embodiments, the unmanned aerial vehicle monitoring method provided in this application further includes: The situation map of the target area is updated based on the flight data and environmental data of the unmanned aerial vehicles (UAVs) in the target area; the situation map is marked with the flight data and environmental data, and shows the flight trajectory of the UAVs in the future time period; The interface displays a situation map, alarm information and risk level of the unmanned aerial vehicle.

[0078] Specifically, a situation map is a diagram that integrates flight data and environmental data of unmanned aerial vehicles (UAVs), annotates key information and future flight trajectories, and is used to intuitively display the overall status of UAVs in a target area.

[0079] Based on flight data and environmental data, fused data is obtained. The situation map of the target area is constructed and updated in real time according to the fused data. The situation map marks the precise position, speed, status, flight trajectory of all unmanned aerial vehicles in the target area, as well as environmental information.

[0080] It provides a unified display interface that intuitively shows real-time situational maps, UAV locations, flight status, alarm information, risk levels, analysis conclusions, and collaborative command execution status. Users can query, replay, and perform advanced interactive operations under authorization within the display interface.

[0081] The unmanned aerial vehicle (UAV) monitoring method provided in this application embodiment can intuitively display relevant information about UAVs in the target area, improving the user experience.

[0082] In some embodiments, the unmanned aerial vehicle monitoring method provided in this application further includes: Each module in the unmanned aerial vehicle, the smart network base station, and the unmanned aerial vehicle monitoring device is regarded as an intelligent agent; Tasks in the unmanned aerial vehicle monitoring method steps are assigned to the intelligent agent based on distributed task allocation rules.

[0083] The task allocation for the unmanned aerial vehicle monitoring method steps based on distributed task allocation rules includes: Break the task down into multiple subtasks; Each subtask is assigned to a different agent based on the agent's current operational capabilities.

[0084] Specifically, an intelligent agent refers to an independent unit with autonomous decision-making and execution capabilities. In the embodiments of this application, the intelligent agent may include modules in unmanned aerial vehicles, Internet of Things base stations, and unmanned aerial vehicle monitoring devices, which can undertake specific tasks and work together.

[0085] Distributed task allocation rules break down complex tasks into multiple sub-tasks and dynamically allocate them to different agents based on their operational capabilities, thereby achieving efficient collaboration.

[0086] A task refers to a specific step or objective that needs to be completed in an unmanned aerial vehicle (UAV) monitoring method, such as data acquisition, situation updates, or alarm processing.

[0087] Operational capability refers to the ability and resources of an intelligent agent to perform tasks in its current state, including computing power, communication bandwidth, or sensor performance.

[0088] The intelligent agents in this application embodiment can be categorized into central platform intelligent agents, multimodal reasoning intelligent agents, network perception intelligent agents, aircraft terminal intelligent agents, resource management intelligent agents, and task execution intelligent agents based on their functions or the tasks they perform. Multiple intelligent agents can constitute responsive, deliberate, and hybrid intelligent agent architectures. The intelligent agent system uses a Multi-Agent System (MAS) development platform or framework, such as the Java Agent Development Framework (JADE), the Multi-Agent Simulator of Neighborhoods (MASON), NetLogo (primarily used for modeling and simulation), or a combination of deep learning frameworks (such as TensorFlow and PyTorch) and reinforcement learning libraries (such as Ray RLIB OpenAI Gym) to construct learning-based intelligent agents.

[0089] Employ standard or custom Agent Communication Language (ACL) to define the semantics of information exchange between agents, such as message types (notify, query, request, refuse, promise, etc.) and content structure, and configure various agent interaction modes, such as point-to-point communication, broadcast, multicast, and role-based communication.

[0090] Based on distributed task allocation rules, combined with line data and environmental data, the current situation map of the target area and the tasks to be executed, distributed task allocation, resource scheduling, collaborative perception (such as directing multiple smart network base stations or unmanned aerial vehicles with 5G-A perception capabilities to collaboratively track specific targets), collaborative alarms, or generation of control commands can be carried out through communication and negotiation between intelligent agents.

[0091] Complex tasks can be divided into multiple subtasks. Based on the current job capabilities of each agent, the subtasks of each task can be assigned to each agent for execution, thereby improving execution efficiency. Each subtask can be executed in parallel. Agents can interact with each other before and during the subtask assignment process, thus cooperating to complete each task.

[0092] Collaboration among intelligent agents can include distributed task allocation, collaborative perception and tracking, collaborative decision-making and response, and resource scheduling and management.

[0093] Distributed task allocation can take several forms. For example, there's the contract network approach, where agents publish task announcements, other agents bid, and the task publisher selects the best bidder to determine the agent executing the sub-task; the market mechanism approach, which simulates a market economy where agents trade or bid based on their capabilities and task value to determine the agent executing the sub-task; optimization-based methods, which model task allocation as an optimization problem and use linear programming, integer programming, or the Hungarian algorithm to find the agent executing the sub-task; and multi-agent reinforcement learning (MARL)-based allocation, where agents learn through trial and error in a dynamic environment how to collaboratively allocate tasks to maximize overall rewards, thus determining the agent executing the sub-task, and using multi-agent reinforcement learning algorithms to optimize the collaborative strategy.

[0094] Cooperative perception and tracking involves multiple sensing agents (such as 5G-A base station sensing agents at different locations or UAVs equipped with radar) sharing perception data on the same target. Distributed data fusion algorithms (such as distributed Kalman filtering and distributed particle filtering) are used to fuse perception data from different agents to obtain a more accurate target state estimate. If controllable sensing agents (such as UAVs) are involved, cooperative control can instruct them to adjust their position and attitude to optimize target perception or achieve multi-view coverage.

[0095] When anomalies in UAV flight are detected, a collaborative decision-making and response mechanism is activated. Relevant agents negotiate and reach a consensus on a control scheme, such as determining which agent is responsible for continuous tracking, which agent is responsible for notifying external parties, and whether more resources need to be mobilized. Complex anomaly tasks are decomposed into sub-tasks and assigned to different agents for parallel resolution. MARL can be used to train agents to learn how to coordinate actions in various complex scenarios to minimize risks or achieve monitoring objectives. For example, multiple UAV agents can be trained to collaboratively search a designated area, or monitoring platform agents and UAV terminal agents can be trained to collaboratively respond to unauthorized UAV intrusions.

[0096] Intelligent agents can collaborate with resource management agents to dynamically allocate computing resources (such as allocating some multimodal inference tasks to edge computing nodes) and network bandwidth based on current task priorities and system load, thereby performing resource scheduling and management.

[0097] A database can be built to store and manage flight anomaly rules, UAV characteristics, historical data of UAVs, various neural network models, and agent coordination strategies.

[0098] The unmanned aerial vehicle monitoring method of this application embodiment is applicable to areas such as urban airspace, sensitive areas, airport perimeters, military facilities, energy hubs, and forest fire prevention, and can perform tasks such as automatic detection, identification, classification, behavior analysis, and threat assessment of flying targets.

[0099] The unmanned aerial vehicle (UAV) monitoring method provided in this application, through collaboration among intelligent agents, can flexibly coordinate internal resources and external entities (such as the UAV itself) to achieve distributed execution and optimization of monitoring tasks. It is suitable for complex scenarios and large-scale cluster monitoring, improving the response speed and handling efficiency of UAV monitoring. By deeply integrating communication and perception into the intelligent network of the target area, a more functional and intelligent digital infrastructure can be built. It can achieve more comprehensive, intelligent, and collaborative monitoring capabilities, effectively improving the safety margin of the target area, reducing the occurrence of accidents, and optimizing the utilization efficiency of airspace resources. Through multimodal data and intelligent agent collaboration mechanisms, it improves robustness in the face of single data source failure or partial component failure.

[0100] The unmanned aerial vehicle (UAV) monitoring device provided in the embodiments of this application is described below. The UAV monitoring device described below can be referred to in correspondence with the UAV monitoring method described above.

[0101] Figure 2 This is a schematic diagram of the structure of the unmanned aerial vehicle monitoring device provided in the embodiments of this application, as shown below. Figure 2 As shown, the device includes a setting module 210, a configuration module 220, an acquisition module 230, and a monitoring module 240.

[0102] The configuration module is used to set up multiple smart network base stations in the target area; The configuration module is used to configure the regional network of a target area based on multiple smart network base stations; The acquisition module is used to acquire flight data and environmental data of unmanned aerial vehicles in the target area collected from multiple terminals based on the regional network; The monitoring module is used to monitor the flight of unmanned aerial vehicles based on flight data and environmental data.

[0103] Specifically, according to the embodiments of this application, any and multiple modules among the setting module, configuration module, acquisition module and monitoring module can be combined into one module, or any one of the modules can be split into multiple modules.

[0104] Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in a single module.

[0105] According to embodiments of this application, at least one of the setting module, configuration module, acquisition module, and monitoring module can be at least partially implemented as hardware circuitry, such as a Field Programmable Gate Array (FPGA), Programmable Logic Array (PLA), System-on-a-Chip, System-on-a-Substrate, System-on-Package, Application Specific Integrated Circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in hardware or firmware, or in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of them.

[0106] Alternatively, at least one of the setting module, configuration module, acquisition module, and monitoring module can be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0107] The unmanned aerial vehicle (UAV) monitoring device provided in this application embodiment can construct a dedicated local area network for the target area by setting up multiple smart network base stations in the target area and configuring the regional network of the target area through the multiple smart network base stations. By acquiring flight data and environmental data of UAVs in the target area collected from multiple terminals through the regional network, and monitoring the flight of UAVs based on the flight data and environmental data, comprehensive flight data and environmental data can be obtained, realizing comprehensive monitoring of multiple UAVs in the target area. This can promptly detect potential air safety hazards and improve monitoring efficiency and accuracy.

[0108] In some embodiments, the terminal includes an Internet of Things base station, an unmanned aerial vehicle (UAV) terminal, and a third-party system; the acquisition module is specifically used for: The control system uses a smart network base station to collect the first flight data of the unmanned aerial vehicle based on radio signals. The control terminal for the unmanned aerial vehicle (UAV) collects second flight data and first environmental data based on onboard sensors; the control terminal is installed on the UAV. Control a third-party system to collect secondary environmental data from the unmanned aerial vehicle; Flight data and environmental data are acquired based on data transmission channels in a regional network; the flight data includes first flight data and second flight data; the environmental data includes first environmental data and second environmental data.

[0109] In some embodiments, the monitoring module is specifically used for: Determine the fusion characteristics of unmanned aerial vehicles based on flight data and environmental data; When unmanned aerial vehicle (UAV) flight anomalies are determined based on fusion features, the type of flight anomaly is identified based on a classification model. The classification model is obtained by training an initial classification model based on the sample fusion features and the sample flight anomaly types corresponding to the sample fusion features. Predict the flight trajectory of unmanned aerial vehicles in the future time period based on flight anomaly type and fusion features; Predict the risk level of unmanned aerial vehicles in the future time period based on flight trajectory, fusion feature prediction and risk assessment model; Determine the flight control strategy for unmanned aerial vehicles based on risk levels; Flight control strategies are used to control the flight of unmanned aerial vehicles.

[0110] In some embodiments, determining the fused characteristics of an unmanned aerial vehicle based on flight data and environmental data includes: The multimodal monitoring data is initially fused based on a multimodal data fusion algorithm to obtain preliminary fusion results; the multimodal monitoring data includes flight data and environmental data. Preliminary fusion features of the unmanned aerial vehicle are obtained based on the preliminary fusion results; The weights of the monitoring data for each modality are determined based on an attention mechanism; The fusion characteristics of the unmanned aerial vehicle are determined based on the preliminary fusion results and weights.

[0111] In some embodiments, the monitoring module is specifically used for: If the flight data and / or environmental data meet the preset flight anomaly rules, generate alarm information based on the flight data and / or environmental data; Controlling the flight of unmanned aerial vehicles based on alarm information.

[0112] In some embodiments, the unmanned aerial vehicle monitoring device further includes a display module, which is specifically used for: The situation map of the target area is updated based on the flight data and environmental data of the unmanned aerial vehicles (UAVs) in the target area; the situation map is marked with the flight data and environmental data, and shows the flight trajectory of the UAVs in the future time period; The interface displays a situation map, alarm information and risk level of the unmanned aerial vehicle.

[0113] In some embodiments, the unmanned aerial vehicle monitoring device further includes an allocation module, which is specifically used for: Each module in the unmanned aerial vehicle, the smart network base station, and the unmanned aerial vehicle monitoring device is regarded as an intelligent agent; Tasks in the unmanned aerial vehicle monitoring method steps are assigned to the intelligent agent based on distributed task allocation rules.

[0114] In some embodiments, tasks in the unmanned aerial vehicle monitoring method steps are assigned to the agent based on distributed task allocation rules, including: Break the task down into multiple subtasks; Each subtask is assigned to a different agent based on the agent's current operational capabilities.

[0115] It should be noted that the unmanned aerial vehicle monitoring device provided in this application embodiment can implement all the method steps implemented in the above-described unmanned aerial vehicle monitoring method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0116] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 3 As shown, the electronic device may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call the computer program in the memory 330 to execute the above-described method.

[0117] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional modules and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the methods provided in the above embodiments.

[0119] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing the processor to execute the methods provided in the above embodiments.

[0120] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0121] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for monitoring unmanned aerial vehicles, characterized in that, include: Set up multiple smart network base stations in the target area; A regional network for the target area is configured based on multiple Internet of Things (IoT) base stations; Based on the regional network, flight data and environmental data of unmanned aerial vehicles in the target area are acquired from multiple terminals. The flight of the unmanned aerial vehicle is monitored based on the flight data and the environmental data.

2. The unmanned aerial vehicle monitoring method according to claim 1, characterized in that, The terminal includes the smart network base station, the unmanned aerial vehicle's (UAV) terminal, and a third-party system; the acquisition of flight data and environmental data of the UAV in the target area collected from multiple terminals based on the regional network includes: The intelligent network base station is controlled to collect the first flight data of the unmanned aerial vehicle based on radio signals; The control terminal for the aircraft collects second flight data and first environmental data of the unmanned aerial vehicle based on airborne sensors; the aircraft terminal is mounted on the unmanned aerial vehicle. The third-party system is controlled to collect second environmental data from the unmanned aerial vehicle. The flight data and the environmental data are acquired based on the data transmission channel in the regional network; the flight data includes the first flight data and the second flight data; the environmental data includes the first environmental data and the second environmental data.

3. The unmanned aerial vehicle monitoring method according to claim 1, characterized in that, The monitoring of the flight of the unmanned aerial vehicle based on the flight data and the environmental data includes: The fusion characteristics of the unmanned aerial vehicle are determined based on the flight data and the environmental data; When the flight anomaly of the unmanned aerial vehicle is determined based on the fusion features, the flight anomaly type of the unmanned aerial vehicle is identified based on a classification model; the classification model is obtained by training an initial classification model based on the sample fusion features and the sample flight anomaly types corresponding to the sample fusion features. Predict the flight trajectory of the unmanned aerial vehicle in the future time period based on the flight anomaly type and the fused features; Based on the flight trajectory, the fusion feature prediction, and the risk assessment model, the risk level of the unmanned aerial vehicle is predicted for a future time period. The flight control strategy for the unmanned aerial vehicle is determined based on the risk level. The flight of the unmanned aerial vehicle is controlled based on the flight control strategy.

4. The unmanned aerial vehicle monitoring method according to claim 3, characterized in that, The determination of the fusion features of the unmanned aerial vehicle based on the flight data and the environmental data includes: The multimodal monitoring data is initially fused based on a multimodal data fusion algorithm to obtain preliminary fusion results; the multimodal monitoring data includes the flight data and the environmental data. Based on the preliminary fusion results, the preliminary fusion features of the unmanned aerial vehicle are obtained; The weights of the monitoring data for each modality are determined based on an attention mechanism; The fusion characteristics of the unmanned aerial vehicle are determined based on the preliminary fusion results and the weights.

5. The unmanned aerial vehicle monitoring method according to claim 1, characterized in that, The monitoring of the flight of the unmanned aerial vehicle based on the flight data and the environmental data includes: If the flight data and / or the environmental data meet the preset flight anomaly rules, an alarm message is generated based on the flight data and / or the environmental data; The flight of the unmanned aerial vehicle is controlled based on the alarm information.

6. The unmanned aerial vehicle monitoring method according to claim 1, characterized in that, Also includes: The situation map of the target area is updated based on the flight data and environmental data of the unmanned aerial vehicle in the target area; the situation map marks the flight data and environmental data and shows the flight trajectory of the unmanned aerial vehicle in the future time period; The interface displays the situation map, alarm information, and risk level of the unmanned aerial vehicle.

7. The unmanned aerial vehicle monitoring method according to claim 1, characterized in that, Also includes: Each module in the unmanned aerial vehicle, the smart network base station, and the unmanned aerial vehicle monitoring device is respectively regarded as an intelligent agent; The tasks in the unmanned aerial vehicle monitoring method steps are assigned to the intelligent agent based on the distributed task allocation rules.

8. The unmanned aerial vehicle monitoring method according to claim 7, characterized in that, The method of assigning tasks to the agent for monitoring unmanned aerial vehicles based on distributed task allocation rules includes: The task is broken down into multiple sub-tasks; Each subtask is assigned to a different agent based on the agent's current operational capabilities.

9. A monitoring device for unmanned aerial vehicles, characterized in that, include: The configuration module is used to set up multiple smart network base stations in the target area; A configuration module is used to configure the regional network of the target area based on multiple Internet of Things base stations; The acquisition module is used to acquire flight data and environmental data of unmanned aerial vehicles in the target area collected from multiple terminals based on the regional network; The monitoring module is used to monitor the flight of the unmanned aerial vehicle based on the flight data and the environmental data.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the unmanned aerial vehicle monitoring method according to any one of claims 1 to 8 through the computer program.

Images