A method and system for generating visualized early warning information based on sensor data

Abstract

This invention relates to a method for generating visualized early warning information based on sensor data. The method includes: collecting multi-source sensor data from various UAVs, and performing outlier removal, missing value repair, and standardization to obtain preprocessed data; inputting the preprocessed data into a long short-term memory network to obtain risk values, predicted local state values, and early warning levels; generating sparse state features based on risk values, spatial proximity, and task area overlap, and uploading them to a collaborative decision-making center; updating the collaborative graph by the collaborative decision-making center to generate enhanced state representations and global states, further obtaining joint action values; predicting trajectories, calculating collision probabilities, generating visualized early warning information based on enhanced state representations and predicted local state values, and determining and issuing joint actions to relevant UAVs when action triggering conditions are met.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and specifically to a method and system for generating visualized early warning information based on sensor data. Background Technology

[0002] As drones are increasingly used in tasks such as inspection, search, monitoring, reconnaissance, and transportation, monitoring operational status and risk warning based on airborne sensor data have become common technical approaches. Existing solutions typically use data such as position, speed, attitude, battery status, motor status, and environmental perception to determine the flight status of a single drone and output alerts when parameters are abnormal or approach alarm thresholds. This approach is practical for single-drone operation scenarios; however, in multi-drone collaborative operations, the data sources involved in decision-making are more numerous, the status changes are faster, and there are also correlation factors between different drones such as proximity in airspace, overlapping mission areas, and changes in communication conditions. Simply relying on single-drone threshold judgments or manual comprehensive interpretation makes it difficult to balance real-time performance and consistency.

[0003] On the other hand, in existing multi-drone early warning schemes, a common practice is to centrally upload and process the large amounts of raw sensor data collected by each drone. This approach easily leads to increased communication burden, a high proportion of irrelevant state information, and increased central processing latency. Furthermore, if dynamic obstacles, overlapping tasks, or speed differences exist in the mission airspace, the early warning results may lag behind actual risk changes. Simultaneously, while some systems can generate early warning prompts, the content often remains at the level of parameter exceeding limits or simple text alerts, lacking intuitive expression of conflict locations, predicted trajectories, related drone objects, and risk levels, hindering rapid response. For scenarios requiring joint avoidance and subsequent formation recovery, existing solutions also suffer from insufficient integration between individual drone decision-making and global scheduling, inadequate verification of action feasibility, and a disconnect between early warning results and subsequent control processes. Therefore, it is necessary to propose a method for generating visualized early warning information based on sensor data. Summary of the Invention

[0004] This application provides a method and system for generating visualized early warning information based on sensor data, so as to at least partially solve the above-mentioned technical problems.

[0005] This application provides a method for generating visualized early warning information based on sensor data, including the following steps: Collect multi-source sensor data from various UAVs and perform outlier removal, missing value repair, and standardization to obtain preprocessed data; The preprocessed data is input into a long short-term memory network to obtain the risk value corresponding to each state parameter, as well as the local state prediction value and warning level. The importance score is calculated based on the risk value, the spatial proximity between the UAV and neighboring UAVs, and the overlap of the mission area. State parameters are then filtered to generate sparse state features. The sparse state features, the predicted state value of the UAV, and the warning level are then uploaded to the collaborative decision-making center. The collaborative decision-making center updates the collaborative graph, performs graph attention aggregation on the sparse state features, local state prediction values ​​and warning levels to obtain an enhanced state representation, and generates a global state by combining global environmental information. Individual action values ​​are generated based on local observations consisting of the preprocessed data and the local state prediction values, and the individual action values ​​are uploaded to the collaborative decision-making center. The collaborative decision-making center generates joint action values ​​based on the global state and the individual action values. Based on the enhanced state representation and the local state prediction value, the trajectory is predicted and the collision probability is calculated. Visual warning information is generated according to the collision probability. When the collision probability meets the action triggering condition, a joint action is determined according to the joint action value and sent to the relevant UAV.

[0006] In one possible embodiment, the long short-term memory network is a stacked multi-layer network, whose input is a preprocessed data sequence of multiple consecutive time steps, and whose output includes the risk value corresponding to each state parameter, the local state prediction value, and the warning level; the local state prediction value includes at least one of attitude deviation trend, power change trend, and temperature rise trend.

[0007] In one possible embodiment, the step of calculating the importance score based on the risk value, the spatial proximity between the UAV and its neighboring UAVs, and the overlap of the task area includes: for each state parameter, weighting the risk value with the spatial proximity of the UAV relative to its neighboring UAVs and the overlap of the task area to obtain the importance score; retaining state parameters whose importance scores meet a preset screening threshold or are in the top K positions of the sort as the sparsified state features, and discarding the uploading of the remaining state parameters.

[0008] In one possible embodiment, the updated collaborative graph includes: adding, retaining, or deleting edges in the collaborative graph based on the real-time location and communication connectivity status of each UAV; the graph attention aggregation includes: calculating attention coefficients for adjacent UAVs in the collaborative graph based on relative speed, spatial distance, and task priority, and weighting and aggregating the sparse state features, local state prediction values, and warning levels of adjacent UAVs to obtain the enhanced state representation.

[0009] In one possible embodiment, the global environmental information includes airspace density, meteorological information, and data quality status; the individual action value is generated by each UAV based on local observations through a deep Q-network; and the joint action value is generated by a collaborative decision-making center based on the global state and the individual action values ​​through a multi-agent reinforcement learning model.

[0010] In one possible embodiment, predicting the trajectory and calculating the collision probability based on the enhanced state representation and the local state prediction value includes: for any two UAVs, obtaining the predicted trajectory within a preset time window based on their enhanced state representation, current position, current speed and local state prediction value, and obtaining the collision probability within the preset time window based on the position uncertainty corrected by the neighborhood interaction features represented by the enhanced state representation; the visualized early warning information includes at least the early warning level, the relevant UAV number, the conflict location and the predicted trajectory display information.

[0011] In one possible embodiment, when the collision probability meets the action triggering condition, a joint action is determined based on the joint action value and sent to the relevant UAVs, including: comparing the collision probability with a safety conflict probability threshold; when the collision probability reaches or exceeds the safety conflict probability threshold and is determined to be a conflict-type red alert, a joint action is determined based on the joint action value and the conflict cost corresponding to the candidate joint action; the joint action is sent to the relevant UAVs, and the relevant UAVs perform feasibility verification of the joint action based on local observations, and return the verification results to the collaborative decision-making center.

[0012] In one possible embodiment, the collaborative decision-making center performs consensus confirmation based on the verification results. When the number of relevant drones returning feasible identifiers meets the preset consensus conditions, it issues execution instructions to the relevant drones. When the number of confirmed drones does not meet the preset consensus conditions, it issues downgraded collaborative instructions to the relevant drones, which then perform distributed obstacle avoidance actions based on visual perception, neighboring drone location information, and individual action value.

[0013] In one possible embodiment, after the relevant UAVs perform the joint action or the distributed obstacle avoidance action, the collaborative decision center calculates the expected position, expected speed and expected relative position of each UAV based on the updated position and mission requirements. Each UAV generates control input and adjusts its flight state based on the expected position, expected speed and expected relative position, and feeds back the adjusted current position and current speed to the collaborative decision center. The collaborative decision center determines the formation reconfiguration state based on the feedback results.

[0014] In another aspect, this application also provides a system for generating visualized early warning information based on sensor data, comprising: The multi-source sensor data preprocessing module is configured to collect multi-source sensor data from each UAV and perform outlier removal, missing value repair and standardization to obtain preprocessed data. The state assessment module is configured to input the preprocessed data into a long short-term memory network to obtain the risk value corresponding to each state parameter, as well as the local state prediction value and warning level. The sparsity state generation module is configured to calculate an importance score based on the risk value, the spatial proximity between the UAV and neighboring UAVs, and the overlap of the mission area, filter state parameters, generate sparsity state features, and upload the sparsity state features, the predicted state value of the UAV, and the warning level to the collaborative decision-making center. The collaborative graph aggregation module is configured to update the collaborative graph by the collaborative decision center, perform graph attention aggregation on the sparse state features, local state prediction values ​​and warning levels to obtain an enhanced state representation, and generate a global state by combining global environmental information. The action value generation module is configured to generate individual action values ​​based on local observations consisting of the preprocessed data and the local state prediction value, and upload the individual action values ​​to the collaborative decision-making center, which generates joint action values ​​based on the global state and the individual action values. The warning and action delivery module is configured to predict the trajectory and calculate the collision probability based on the enhanced state representation and the local state prediction value, generate visual warning information according to the collision probability, and determine the joint action according to the joint action value and send it to the relevant UAV when the collision probability meets the action triggering condition.

[0015] In another aspect, this application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method for generating visual early warning information based on sensor data as described above.

[0016] In another aspect, this application provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor to implement the method for generating visual early warning information based on sensor data as described above.

[0017] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method for generating visual early warning information based on sensor data as described above.

[0018] Compared with existing technologies, this invention uses multi-source sensor data as a foundation and integrates single-machine state assessment, state parameter screening, multi-machine collaborative graph aggregation, joint action value calculation, and visual early warning generation into a unified processing flow. On the one hand, by comprehensively screening risk values, spatial proximity, and task area overlap, only state features related to the current collaborative decision are uploaded, reducing the burden of communication and centralized processing. On the other hand, by combining enhanced state representation, local state prediction values, and global environmental information to assess collision probability, and synchronously visualizing and outputting early warning levels, related UAV numbers, conflict locations, and predicted trajectories, the early warning information is consistent with the subsequent joint action determination, feasibility verification, and formation reconstruction process, thereby achieving better connectivity and timeliness between risk identification, early warning presentation, and action handling in multi-UAV scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a method for generating visualized early warning information based on sensor data, provided in an embodiment of this application.

[0021] Figure 2 This is a schematic diagram of the structure of a system for generating visual early warning information based on sensor data, provided in an embodiment of this application.

[0022] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] This implementation method is applicable to scenarios where multiple UAVs perform collaborative tasks such as inspection, search, monitoring, reconnaissance, or transportation within the same mission airspace. Each UAV is equipped with position sensors, attitude sensors, power system sensors, and environmental perception sensors, and is configured with an onboard edge computing node; a collaborative decision-making center is deployed in a ground station, edge cloud node, or the lead UAV in the formation. Wireless communication is maintained between the UAVs, and the collaborative decision-making center receives sparsed status information uploaded by each UAV and transmits back early warning information and joint actions. If there are dynamic obstacles in the mission airspace, overlapping mission areas, or significant differences in the operating speeds of the multiple UAVs, this implementation method still follows the same process, only with corresponding configurations in parameter values, update frequency, and action triggering conditions.

[0025] In practical applications, airborne edge computing nodes are responsible for data acquisition, local preprocessing, single-machine status assessment, and individual action value calculation, while the collaborative decision-making center is responsible for collaborative graph updating, graph attention aggregation, joint action value calculation, collision probability assessment, generation of visual early warning information, determination of joint actions, and formation reconfiguration. To ensure consistency between input and output objects in different stages, each UAV completes unified clock synchronization, coordinate system alignment, and communication link establishment before the mission begins. Simultaneously, position sensors, attitude sensors, motor speed sensors, battery voltage and current sensors, and temperature sensors are calibrated. The calibration results are embedded in the driver layer of the airborne edge computing nodes and directly called upon in subsequent preprocessing stages, without being regenerated.

[0026] Airborne edge computing nodes are pre-deployed with Long Short-Term Memory (LSTM) networks and Deep Q-Networks (DQN). The collaborative decision-making center is pre-deployed with Graph Attention Networks (GAT) and Multi-Agent Reinforcement Learning (MARL) models. In this implementation, the MARL model uses the QMIX architecture. The model deployment order is consistent with the invocation order: the airborne edge computing nodes first complete preprocessing and single-machine inference, then the collaborative decision-making center performs multi-machine fusion and action evaluation, and finally, the warning information and action commands are sent to the relevant UAVs.

[0027] For LSTM, the network structure employs a stacked multi-layer sequence network. For example, it can be configured as a three-layer LSTM connected sequentially. The first layer receives preprocessed data sequences from multiple consecutive time steps and extracts short-term variation patterns of various sensor parameters. The second layer performs temporal compression on the latent states output by the first layer. The third layer outputs high-order temporal features related to flight risk. Two fully connected branches are set after the third layer: one branch outputs the risk values ​​corresponding to each state parameter, and the other branch outputs the predicted local state and warning level.

[0028] The predicted state of the aircraft is a prediction of its short-term future operating state, which may include at least one of the following: attitude deviation trend, power change trend, and temperature rise trend. The warning level is output by the classification head and subsequently participates in both the generation of visualized warning information and the graph attention aggregation of the collaborative decision-making center. To ensure stable online inference, the LSTM is trained on historical flight data before task execution. The training data includes historical flight data such as position, velocity, attitude angle, acceleration, motor speed, battery voltage, current, remaining power, and temperature of key components. If both continuous value prediction labels and discrete level labels exist in the training set, the loss function consists of regression loss and classification loss, with their weights determined by offline calibration. This set of weights is only used during the training phase and is not included in the online steps.

[0029] DQN is deployed within an airborne edge computing node. Its input is local observations, which consist of preprocessed data from the current period and the local state prediction. To ensure time consistency between DQN and LSTM, the preprocessed data from the current period is provided to both LSTM and DQN. LSTM first outputs the local state prediction, then concatenates it with the preprocessed data from the same period to form the local observation. For example, DQN can be configured with an input layer, two fully connected hidden layers, and an output layer. The input layer receives a fixed-dimensional local observation vector. The two hidden layers are activated using Rectified Linear Units (ReLU). The output layer corresponds to multiple candidate actions. The set of candidate actions remains fixed and can be categorized into seven types: continue mission, maintain deceleration, ascend avoidance, descend avoidance, heading adjustment, return to base, and diversion. DQN outputs not the final executed action, but rather the individual action value corresponding to each candidate action. This individual action value is then uploaded to the collaborative decision center and enters the QMIX hybrid network, but does not directly replace the joint action of the collaborative decision center locally.

[0030] GAT and QMIX are deployed in the collaborative decision-making center. GAT takes as input a collaborative graph and node features, and outputs an enhanced state representation. QMIX takes as input a global state and the individual action values ​​of each UAV, and outputs a joint action value. Both GAT and QMIX can be jointly trained before the mission begins based on historical flight logs and simulation data. Historical flight logs provide realistic multi-UAV spatial relationships and risk evolution processes, while simulation data supplements high-risk collision samples and formation reconstruction samples. QMIX training uses a centralized training approach, with each UAV's DQN serving as a local action value network. The hybrid network in the collaborative decision-making center generates hybrid weights based on the global state, combining the individual action values ​​into a joint action value. During online execution, the airborne edge computing nodes retain only DQN inference functionality, while the collaborative decision-making center retains both GAT and QMIX inference functionality.

[0031] The following detailed description, in conjunction with specific embodiments, illustrates the implementation process of the method for generating visualized early warning information based on sensor data described in this application. It should be noted that these embodiments are merely for explaining this application and not for limiting its scope of protection. Any conventional adjustments or substitutions made by those skilled in the art to the steps without departing from the concept of this application should be included within the scope of protection of this application.

[0032] like Figure 1 As shown in the figure, this application discloses a method for generating visualized early warning information based on sensor data, including the following method steps: S1: Collect multi-source sensor data from each UAV and perform outlier removal, missing value repair and standardization to obtain preprocessed data; S2, input the preprocessed data into the long short-term memory network to obtain the risk value corresponding to each state parameter, as well as the local state prediction value and warning level; S3. Calculate the importance score based on the risk value, the spatial proximity between the UAV and neighboring UAVs, and the overlap of the mission area. Filter the state parameters, generate sparse state features, and upload the sparse state features, the predicted state value of the UAV, and the warning level to the collaborative decision-making center. S4, the collaborative decision-making center updates the collaborative graph, performs graph attention aggregation on the sparse state features, local state prediction values ​​and warning levels to obtain an enhanced state representation, and generates a global state by combining global environmental information; S5, generate individual action values ​​based on local observations composed of the preprocessed data and local state prediction values, and upload the individual action values ​​to the collaborative decision-making center. The collaborative decision-making center generates joint action values ​​based on the global state and the individual action values. S6. Based on the enhanced state representation and the local state prediction value, predict the trajectory and calculate the collision probability. Generate visual early warning information according to the collision probability. When the collision probability meets the action triggering condition, determine the joint action according to the joint action value and send it to the relevant UAV.

[0033] In some embodiments, for step S1, after the mission begins, each UAV collects multi-source sensor data at a fixed sampling period. The multi-source sensor data includes at least three categories: flight status data, power system data, and environmental perception data. Flight status data includes position, velocity, attitude angle, and acceleration; power system data includes motor speed, battery voltage, current, remaining charge, and temperature of key components; and environmental perception data includes obstacle location, relative distance, or local target information. The airborne edge computing node first performs outlier removal, then missing value repair, and finally standardization on the same type of sensor data to obtain preprocessed data. Outlier removal uses a sliding time window statistical method; for a certain type of sensor data, let the current sampling value be... The mean within the window is The standard deviation is The degree of deviation is calculated using the following formula:

[0034] In the formula, This is a dimensionless deviation. , and Data from the same type of sensor. If... If the sampled value exceeds a preset threshold, the airborne edge computing node will classify it as an outlier and remove it. This threshold can be obtained statistically based on historical noise levels or calibrated in conjunction with the allowable error range given in the sensor specifications. During mission execution, this threshold is generally not updated online to avoid frequent changes in the anomaly judgment criteria within the same flight.

[0035] Missing value imputation is performed after outlier removal because samples deemed outliers are not used as interpolation nodes in the missing value compensation process. For missing values ​​at time points... The sampled values, if their most recent valid sampling times are respectively and The corresponding sample value is and The replacement sampled values ​​are obtained by linear interpolation:

[0036] in, To replace the sampled values, and These are valid sampled values. For cases with long periods of continuous missing data or a high proportion of missing data in a single period, the airborne edge computing node generates not only alternative sampled values ​​but also a data quality status. This data quality status is not a new sensor value but a quantitative description of the completeness and reliability of the data in the current period. It can be configured as a level identifier or a normalized score. This result serves two purposes: firstly, it enters the collaborative decision-making center to form part of the global environmental information; secondly, it participates in location uncertainty estimation.

[0037] After completing the above preprocessing, the airborne edge computing node performs standardization on the effective data, mapping the sensing quantities of different dimensions to a unified numerical range to obtain preprocessed data.

[0038] In some embodiments, for step S2, the preprocessed data is simultaneously fed into LSTM and DQN. Since LSTM requires input sequences of continuous time steps, the airborne edge computing node maintains a time series buffer of fixed length; exemplarily, several past sampling periods can be selected to form an input window, and the preprocessed data for each period is fed into the first-layer LSTM in chronological order. LSTM outputs the risk value, the local state prediction value, and the warning level corresponding to each state parameter. The risk value corresponds to a single state parameter and reflects the strength of that state parameter's contribution to short-term flight risk; the local state prediction value is a continuous state quantity directly used in subsequent trajectory prediction and local observation construction; the warning level can participate in subsequent graph attention aggregation and also enter the process of determining the level of visualized warning information.

[0039] Based on this, airborne edge computing nodes filter state parameters according to risk value, spatial proximity, and task area overlap. To avoid uploading all sensor data to the collaborative decision-making center and causing communication burden, the system only uploads state parameters that are more sensitive to collaborative decision-making in the current period. Spatial proximity is calculated from the geometric distance between the local machine and its neighboring machines. If the local machine number is... The neighboring machine number is The distance between the two is The preset maximum adjacency distance is Spatial proximity is defined as:

[0040] In the formula, The value of is limited to between zero and one; when Increase to At this point, the spatial proximity approaches zero. The task area overlap is obtained from the intersection and union relationship of the current task areas of the two machines, and the ratio of the intersection area to the union area is used in the calculation; when the areas of responsibility of the two machines highly overlap, the task area overlap increases. Both reflect the degree of cooperative correlation between the machine and its neighboring machine.

[0041] In some embodiments, for step S3, the airborne edge computing node calculates an importance score in units of state parameters. The first drone Each state parameter has an importance score that combines the parameter's inherent risk value with the spatial proximity and mission area overlap between the UAV and all neighboring UAVs. Optionally, the importance score is calculated using the following formula:

[0042] in, Indicates importance score, Indicates the first The first drone Risk values ​​for each state parameter This represents the set of neighboring drones. Indicates the number of adjacent machines. Indicates spatial proximity. Indicates the overlap of task regions. and These are weighting coefficients. Their meanings correspond to the contribution ratios of the degree of individual machine state anomaly and the degree of neighborhood collaborative correlation in the screening process, respectively.

[0043] In order to make the contributions of the two parts comparable, and It can be configured to be a non-negative value that sums to one; its specific value is obtained offline based on historical task log statistics, or it can be obtained through simulation sample search. When the task scenario changes significantly, such as a long-term increase in spatial density or a change in task division, the collaborative decision-making center can recalibrate based on recent conflict records. and The new weights are then distributed to each airborne edge computing node, and take effect from the next task cycle. For example, the risk weight for a single machine can be 0.5 to 0.8, and the corresponding neighborhood collaboration weight can be 0.2 to 0.5.

[0044] The airborne edge computing node compares the importance score with the screening criteria, retains the state parameters that meet the screening criteria, generates sparse state features, and discards the uploading of the remaining state parameters. The screening criteria can be set to an importance score not lower than the screening threshold T_s, or to retain the top K state parameters after sorting by importance score in the current period; where T_s can be 0.4 to 0.7, K can be 3 to 8, and is calibrated offline based on a trade-off between historical conflict detection rate and communication bandwidth usage.

[0045] The term "discarding" here refers only to not uploading to the collaborative decision-making center, not to deleting it locally. State parameters not uploaded remain in the local cache and continue to participate in the next cycle of LSTM and DQN inference. In this way, the upload link only contains sparse state features, local state predictions, and warning levels, while the local model can still use the complete preprocessed data for continuous inference. This arrangement reduces uplink traffic and the processing pressure on the collaborative decision-making center simultaneously, without disrupting the continuity of single-machine state assessment.

[0046] After the sparse state features are generated, the airborne edge computing node uploads the sparse state features, the predicted local state value, the warning level, and necessary timestamps and machine identifiers to the collaborative decision-making center. Since the data quality status also needs to be incorporated into the global environment information later, the airborne edge computing node can upload this result along with the data quality status. The uploaded content includes the sparse state features, the predicted local state value, the warning level, the data quality status, the timestamp, and the machine identifier. The sparse state features, the predicted local state value, and the warning level constitute the main body of the node features. The data quality status is incorporated into the global environment information, and the timestamp and machine identifier are used for cross-machine alignment and collaborative graph updates, and are no longer stored as independent decision variables in the long term.

[0047] After receiving the uploaded results from each drone, the collaborative decision-making center first aligns them according to the timestamp for the same period, and then updates the collaborative graph based on location and communication connectivity. Nodes in the collaborative graph represent individual drones, and edges represent pairs of drones that can participate in graph attention calculation as adjacencies within the current period. If communication between two drones is normal and the real-time distance does not exceed the preset adjacency distance, the edge is retained or added; if the distance exceeds this range, or the link is interrupted, the corresponding edge is deleted. After this process, the collaborative graph always reflects the actual multi-drone spatial relationships at the current moment, and does not perpetuate expired adjacency relationships.

[0048] In some embodiments, for step S4, the collaborative decision-making center constructs initial node features for each node. These features are obtained by sequentially concatenating the sparse state features uploaded by the UAV, the predicted local state value, and the warning level. The components of the node features remain fixed in subsequent steps: the sparse state features participate in GAT aggregation, the predicted local state value participates in both GAT aggregation and trajectory prediction, and the warning level participates in GAT aggregation and in the output of visualized warning information.

[0049] GAT performs weighted aggregation on adjacent nodes in the collaboration graph to obtain an enhanced state representation. Specifically, for nodes... and adjacent nodes The collaborative decision-making center calculates the attention coefficient based on the relative speed, spatial distance, and task priority of the two machines. The relative speed is obtained from the difference in their velocity vectors, the spatial distance from the difference in their position vectors, and the task priority from the task urgency indicator provided by the task scheduling module; all three are clearly derived. Optionally, the unnormalized attention calculation formula is:

[0050] in, and Representing nodes respectively With nodes The initial node characteristics, Represents the characteristic transformation matrix, Indicates relative velocity characteristics, Indicates spatial distance characteristics, This indicates features related to task priority. Represents the attention vector, symbol This indicates a splicing operation. The collaborative decision-making center then... Normalization is performed to obtain attention coefficients, and then the features of adjacent nodes are weighted and summed accordingly to generate nodes. The enhanced state representation is then used. If a multi-head graph attention structure is adopted, the outputs of multiple heads are combined by concatenation or averaging before being fed into subsequent layers. This network structure still follows the basic relationship that the input is node features and graph structure, and the output is an enhanced state representation, without affecting object referencing in subsequent steps.

[0051] The enhanced state representation output by GAT constitutes the core representation of the multi-drone collaborative state, but it is insufficient to cover external environment and data integrity information. Therefore, the collaborative decision center also combines airspace density, meteorological information, and data quality status into global environmental information, which, together with all enhanced state representations, constitutes the global state. Here, airspace density can be calculated based on the number of drones and nearest neighbor distance distribution within a unit of airspace; meteorological information can include wind speed, wind direction, and precipitation level; and data quality status comes directly from the results generated by each airborne edge computing node during the preprocessing stage. After the global state is formed, it is only used by QMIX within the collaborative decision center and is not transmitted back to the airborne edge computing nodes to reduce unnecessary data downlink.

[0052] In some embodiments, for step S5, DQN inference on the airborne edge computing node runs in parallel with the collaborative decision-making center. Each UAV generates an individual action value based on local observations, where local observations consist of preprocessed data for the current period and the UAV's predicted state, and obstacle information from the UAV's environmental perception sensors can be added if necessary. The DQN output layer provides an action value for each candidate action, and this set of action values ​​corresponds one-to-one with the action set. Since the action set is consistent across all UAVs, after receiving the individual action values ​​uploaded by each UAV, the collaborative decision-making center can arrange them uniformly by UAV number and action number, and use them as input to QMIX. At this stage, the individual action value does not determine the action independently, but only serves as a component of the joint action value.

[0053] QMIX generates a joint action value based on the global state and the action values ​​of each individual drone. QMIX consists of two parts: a local action value network and a hybrid network. The local action value network is handled by the DQN on each drone, while the hybrid network is deployed in the collaborative decision-making center. The hybrid network generates hybrid weights based on the global state and then combines the individual action values ​​of each drone into a joint action value. Each joint action value corresponds one-to-one with a candidate joint action, which is composed of individual drone actions from each drone's action set. To ensure that the final action selection balances safety and mission execution, the collaborative decision-making center further adds a conflict cost to the joint action value. The conflict cost can be calculated by considering path intersection trends, expected detour costs, and incremental power consumption. For example, candidate joint actions... The rating can be based on

[0054] Calculate, where, This indicates the value of the joint action corresponding to the candidate joint action. This indicates the conflict cost corresponding to the candidate joint action. This represents the conflict cost weight. Here, This refers to the allocation of weight between the value of joint operations and the cost of conflict. Its value can be preset based on the mission safety level or determined based on historical conflict statistics; when the mission airspace is dense or the remaining energy of a single aircraft is low, it can be... The value is appropriately increased to make the action score more biased towards conservative actions. This parameter is a strategy parameter uniformly managed by the collaborative decision-making center and can be updated by the center in batches according to the task, without frequent modification during flight. For example, the conflict cost weight can be set to 0.2 to 0.6.

[0055] In some embodiments, for step S6, after obtaining the enhanced state representation and the local state prediction, the collaborative decision center predicts the future short-term trajectories of any two UAVs. The inputs to the trajectory prediction include the enhanced state representation, current position, current velocity, and the local state prediction, wherein the enhanced state representation is used to characterize neighborhood interaction relationships and correct for position uncertainty estimation. The predicted status of the drone provides short-term trends in attitude, battery level, or temperature rise, which are reflected in the uncertainty of subsequent trajectories and the availability of maneuvers. Specifically, the collaborative decision-making center recursively calculates the position status of each drone within a preset time window to obtain the predicted trajectory; the preset time window ΔT can range from 2s to 10s and is calibrated offline based on the drone's current speed, communication latency, and control refresh cycle. The position uncertainty matrix can be constructed from the position measurement covariance, velocity measurement covariance, data quality status correction factor, and environmental disturbance variance; the worse the data quality status, the larger the corresponding covariance amplification factor. If a significant attitude shift or battery depletion trend appears in the predicted machine status, the position uncertainty within the corresponding time window increases. This uncertainty is characterized by a position uncertainty matrix, whose parameters are affected by sensor accuracy, data quality status, and environmental disturbances. Since the data quality status has already been generated during the preprocessing stage, it is directly used here, and there is no need to construct a separate reliability index.

[0056] The probability of a collision between the two machines is determined by both the relative trajectory and the positional uncertainty. If the number is... and The two drones in time The relative position is denoted as The safe interval distance is denoted as The collision probability within the preset time window can be expressed as:

[0057] in, Indicates the length of the prediction time window. Indicates the time of the two machines The probability of entering within the safe interval. The collision probability is determined by a combination of factors including airframe size, flight speed, and control precision, and can be obtained from historical flight data and flight control test calibration. After the collision probability is calculated, the collaborative decision-making center compares it with the action triggering conditions. These conditions are composed of a safety conflict probability threshold; when the collision probability reaches or exceeds this threshold, the joint action determination process is triggered. The safety conflict probability threshold can be between 0.6 and 0.9, and this threshold is also determined through offline calibration and remains fixed during mission execution unless the entire mission batch switches to a different safety strategy.

[0058] The collaborative decision-making center generates visual early warning information based on collision probability. This visual early warning information includes at least the warning level, the relevant drone number, the conflict location, and the predicted trajectory. If the collision probability is higher than the safety conflict probability threshold, the corresponding warning level is red; this red warning is a conflict-related red warning based on collision probability. Individual drone health status warning levels can be displayed in parallel on the visualization interface, but are not used as a separate trigger for joint actions. If the collision probability is lower than the threshold but still higher than the alert threshold, a lower-level warning can be generated.

[0059] The warning threshold here only affects the visualization and information dissemination range, and does not trigger joint action determination, thus remaining distinct from action triggering conditions. Meanwhile, if a drone's LSTM has already output a high warning level, that level will be overlaid into the visualized warning information, allowing the visualization interface to reflect both multi-drone spatial conflicts and individual drone health status. To facilitate quick identification of conflict locations and risky objects by operators, the visualization interface overlays the relevant drone number and predicted trajectory onto the map layer, and simultaneously displays the corresponding warning level and key status trends in the status panel.

[0060] When the collision probability meets the action triggering condition, the collaborative decision-making center determines the joint action based on the joint action value. The joint action is not selected independently from individual candidate actions, but rather a score is calculated from the candidate joint action set, and the highest score is selected. The candidate joint action set is obtained by a Cartesian combination of the action sets of each UAV; to reduce the combination scale, the collaborative decision-making center can first select several high-value individual actions based on the individual action value of each UAV, and then combine them into candidate joint actions. For each candidate joint action, the collaborative decision-making center takes the joint action value output by QMIX and the corresponding conflict cost, obtains a score according to the aforementioned scoring function, and determines the joint action from it. The determined joint action, along with the relevant UAV number, is written into the action command message and then sent to the relevant UAV.

[0061] After receiving the joint action, the relevant UAV first performs a feasibility verification based on local observations. Local observations include preprocessed data for the current period and predicted UAV status values. Therefore, the feasibility verification at least checks whether the remaining battery power allows the action to be executed, whether there is an excessive deviation in the current attitude, and whether local obstacle information conflicts with the action. If the joint action requires the UAV to ascend to avoid an obstacle, but the UAV's sensors detect a nearby obstacle above, the UAV determines the joint action to be infeasible. Similarly, if the joint action requires yaw adjustment, but the UAV's current attitude deviation trend is too large, it can also be determined to be infeasible. The verification result is encapsulated by the airborne edge computing node into a feasible or infeasible identifier, along with a reason code, and returned to the collaborative decision-making center. The returned result is only valid within the current action decision-making period and is not used as node feature input for the next period.

[0062] After receiving all verification results, the collaborative decision-making center executes consensus confirmation. The number of confirmations represents the number of relevant drones that returned the 'feasible' flag. The preset consensus condition can be set to all relevant drones returning the 'feasible' flag, or the proportion of drones returning the 'feasible' flag to the total number of relevant drones is not less than η, where η can be 0.7 to 1.

[0063] If the number of confirmed cases reaches the preset consensus condition, an execution command is sent to the relevant drones, and the drones enter the control execution phase according to the joint action. If the number of confirmed cases does not reach the preset consensus condition, the collaborative decision-making center issues a downgraded collaborative command to the relevant drones. The downgraded collaborative command does not include a unified joint action, but instead requires each drone to switch to distributed obstacle avoidance actions. After entering this mode, each drone selects an obstacle avoidance action locally based on visual perception, neighbor drone position information, and individual action value. The neighbor drone position information here comes from the most recent valid broadcast message, and the individual action value comes from the local DQN output. The downgrade process is executed based on the most recent valid neighbor drone broadcast information and the local individual action value. During the execution of distributed obstacle avoidance actions, the collaborative decision-making center continues to receive updates on the position and speed of each drone and continuously recalculates the collision probability; when the risk subsides, the system returns to the normal collaborative decision-making process.

[0064] After the joint or distributed obstacle avoidance maneuvers are completed, the collaborative decision-making center calculates the desired position, desired velocity, and desired relative position required for formation reconfiguration based on the updated location and mission requirements. The desired position represents the target spatial location of each UAV in the new formation, the desired velocity represents the target velocity along the new formation trajectory, and the desired relative position represents the geometric relationship that adjacent UAVs should maintain. The generation of these three parameters is based on the current mission area allocation, remaining battery power, airspace status, and mission priority. If a UAV has low battery power or there are still obstacles in the local airspace, the collaborative decision-making center can appropriately adjust its desired position and desired velocity to avoid reassigning it to a high-risk location.

[0065] Each UAV generates control inputs and adjusts its flight state based on its desired position, desired speed, and desired relative position. Optionally, the control inputs are calculated using the following formula:

[0066] in, Indicates the first Control input for the drone Indicates the desired position. Indicates the current position. Indicates the desired speed. Indicates the current speed. Indicates the relationship with neighboring aircraft The expected relative position For the position term coefficient, For the velocity term coefficient, This represents the neighborhood collaboration weight. and The value is obtained by flight control parameter tuning and is generally fixed along with the controller parameters after the aircraft model is determined. This is related to the current communication quality and the stability of neighboring machines, and can be issued by the collaborative decision-making center after each reconstruction cycle.

[0067] The neighborhood cooperation weight can be limited to between 0 and 1, and is determined by the product of the normalized communication quality index and the neighboring drone's state stability index; when communication is interrupted, the neighborhood cooperation weight is set to 0. This setting is because the position and velocity terms mainly determine the convergence speed of a single drone towards the target state, while the neighborhood cooperation weight determines the proportion of geometric constraints between neighboring drones in the control. If the current communication quality deteriorates, then... Lowering the threshold makes single-machine control more reliant on its own state; if the neighboring machines are stable and the link is reliable, then... This improvement makes the formation recovery process more consistent.

[0068] After executing control inputs, the UAVs feed back their adjusted current position and velocity to the collaborative decision-making center. The collaborative decision-making center determines the formation reconfiguration status based on the feedback. Two types of conditions can be used for this determination: one is a geometric condition, namely whether the deviations between each UAV and its desired position, and its desired relative position, are within acceptable limits; the other is a temporal condition, namely whether the aforementioned deviations remain stable over multiple consecutive control cycles. Only when both conditions are met simultaneously will the collaborative decision-making center mark the formation reconfiguration status as complete. If the geometric condition is not yet met, the collaborative decision-making center continues to recalculate the desired quantities of control inputs based on the latest feedback; if the probability of a collision exceeds the action triggering condition again during the reconfiguration process, the current reconfiguration is immediately terminated, and the joint action determination and warning generation process is re-entered. Since the formation reconfiguration status is entirely derived from the updated results of position, velocity, and mission requirements, its determination process is seamlessly integrated with the aforementioned warning and action processes.

[0069] To ensure consistency in the above processes over long-term operation, the Collaborative Decision Center continuously records operational data such as sparse state characteristics, local state predictions, warning levels, enhanced state representations, individual action values, joint action values, collision probabilities, verification results, and formation reconfiguration states. The main functions of this recording include providing data for mission replay and fault localization, and providing samples for parameter recalibration and offline model training. If recent mission logs show too many warnings or triggering too late, the Collaborative Decision Center can recalibrate the safety conflict probability threshold, conflict cost weights, weight coefficients in importance scores, and neighborhood collaboration weights. If the logs show a large prediction error for a certain type of state trend, the LSTM can be retrained. If joint action selection is unstable in a specific scenario, DQN and QMIX can be retrained using new samples. All such updates are completed offline, and new model and policy parameters are loaded before the next mission begins to avoid altering the model during flight and affecting action consistency.

[0070] In summary, this invention integrates multi-source sensor data processing, single-unit state assessment, state parameter filtering, multi-unit graph structure aggregation, joint action assessment, collision probability calculation, visual warning information generation, action feasibility verification, and formation reconstruction into a single processing chain. Each stage sequentially completes multi-source sensor data processing, single-unit state assessment, state filtering, multi-unit aggregation, joint action assessment, collision probability calculation, warning generation, action verification, and formation reconstruction. This not only supports warning generation and action issuance in multi-UAV missions but also allows for continued processing of formation restoration and mission continuation after action execution.

[0071] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a sensor data-based visualization early warning information generation system 2 provided in an embodiment of this application. Figure 2 As shown, the system includes: The multi-source sensor data preprocessing module 201 is configured to collect multi-source sensor data from each UAV and perform outlier removal, missing value repair and standardization to obtain preprocessed data. The state assessment module 202 is configured to input the preprocessed data into a long short-term memory network to obtain the risk value corresponding to each state parameter, as well as the local state prediction value and warning level. The sparsity state generation module 203 is configured to calculate an importance score based on the risk value, the spatial proximity between the UAV and its neighboring UAVs, and the overlap of the mission area, filter state parameters, generate sparsity state features, and upload the sparsity state features, the predicted state value of the UAV, and the warning level to the collaborative decision-making center. The collaborative graph aggregation module 204 is configured to update the collaborative graph by the collaborative decision center, perform graph attention aggregation on the sparse state features, local state prediction values ​​and warning levels to obtain an enhanced state representation, and generate a global state by combining global environmental information. Action value generation module 205 is configured to generate individual action values ​​based on local observations consisting of the preprocessed data and local state prediction values, and upload the individual action values ​​to the collaborative decision-making center, which generates joint action values ​​based on the global state and the individual action values. The warning and action delivery module 206 is configured to predict the trajectory and calculate the collision probability based on the enhanced state representation and the local state prediction value, generate visual warning information according to the collision probability, and determine the joint action according to the joint action value and send it to the relevant UAV when the collision probability meets the action triggering condition.

[0072] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0073] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0074] Based on the same inventive concept, this application also provides an electronic device, the method corresponding to which can be the method in the foregoing embodiments, and its problem-solving principle is similar to that method. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of an electronic device structure provided in an embodiment of the present disclosure. The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the methods and / or technical solutions of the foregoing embodiments of the present application.

[0075] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processor, it performs the functions defined in the methods of this application.

[0076] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0077] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0078] The embodiments disclosed herein are preferred embodiments, but are not limited thereto. Those skilled in the art can readily grasp the spirit of the present invention based on the above embodiments and make different extensions and variations, but as long as they do not depart from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. A method for generating visual warning information based on sensor data, characterized in that, include: Collect multi-source sensor data from various UAVs and perform outlier removal, missing value repair, and standardization to obtain preprocessed data; The preprocessed data is input into a long short-term memory network to obtain the risk value corresponding to each state parameter, as well as the local state prediction value and warning level. The importance score is calculated based on the risk value, the spatial proximity between the UAV and neighboring UAVs, and the overlap of the mission area. State parameters are then filtered to generate sparse state features. The sparse state features, the predicted state value of the UAV, and the warning level are then uploaded to the collaborative decision-making center. The collaborative decision-making center updates the collaborative graph, performs graph attention aggregation on the sparse state features, local state prediction values ​​and warning levels to obtain an enhanced state representation, and generates a global state by combining global environmental information. Individual action values ​​are generated based on local observations consisting of the preprocessed data and the local state prediction values, and the individual action values ​​are uploaded to the collaborative decision-making center. The collaborative decision-making center generates joint action values ​​based on the global state and the individual action values. Based on the enhanced state representation and the local state prediction value, the trajectory is predicted and the collision probability is calculated. Visual warning information is generated according to the collision probability. When the collision probability meets the action triggering condition, a joint action is determined according to the joint action value and sent to the relevant UAV.

2. The method of claim 1, wherein, The Long Short-Term Memory network is a stacked multi-layer network. Its input is a preprocessed data sequence of multiple consecutive time steps, and its output includes the risk value corresponding to each state parameter, the predicted value of the local state, and the warning level. The predicted value of the local state includes at least one of the attitude deviation trend, the power change trend, and the temperature rise trend.

3. The method of claim 1, wherein, The step of calculating the importance score based on the risk value, the spatial proximity between the UAV and its neighboring UAVs, and the overlap of the task area includes: for each state parameter, weighting the risk value with the spatial proximity of the UAV relative to its neighboring UAVs and the overlap of the task area to obtain the importance score; retaining state parameters whose importance scores meet a preset screening threshold or are in the top K positions of the sort as the sparse state features, and discarding the uploading of the remaining state parameters.

4. The method of claim 1, wherein, The updated collaborative graph includes: adding, retaining, or deleting edges in the collaborative graph based on the real-time location and communication connectivity status of each UAV; the graph attention aggregation includes: calculating attention coefficients for adjacent UAVs in the collaborative graph based on relative speed, spatial distance, and task priority, and weighting and aggregating the sparse state features, local state prediction values, and warning levels of adjacent UAVs to obtain the enhanced state representation.

5. The method of claim 1, wherein, The global environmental information includes airspace density, meteorological information, and data quality status; the individual action value is generated by each UAV based on local observations through a deep Q-network; the joint action value is generated by the collaborative decision-making center based on the global state and the individual action values ​​through a multi-agent reinforcement learning model.

6. The method of claim 1, wherein, The process of predicting trajectories and calculating collision probabilities based on the enhanced state representation and the local state prediction value includes: for any two UAVs, obtaining the predicted trajectory within a preset time window based on their enhanced state representation, current position, current speed, and local state prediction value; and obtaining the collision probability within the preset time window based on the position uncertainty corrected by the neighborhood interaction features represented by the enhanced state representation; the visualized early warning information includes at least the early warning level, the relevant UAV number, the conflict location, and the predicted trajectory display information.

7. The method of claim 1, wherein, When the collision probability meets the action triggering condition, a joint action is determined based on the joint action value and sent to the relevant UAVs. This includes: comparing the collision probability with a safety conflict probability threshold; when the collision probability reaches or exceeds the safety conflict probability threshold and is determined to be a red alert for conflict, a joint action is determined based on the joint action value and the conflict cost corresponding to the candidate joint action; the joint action is sent to the relevant UAVs, which then perform feasibility verification of the joint action based on local observations and return the verification results to the collaborative decision-making center.

8. The method of claim 7, wherein, The collaborative decision-making center confirms consensus based on the verification results. When the number of relevant drones returning feasible identifiers meets the preset consensus conditions, it issues execution instructions to the relevant drones. When the number of confirmed drones does not meet the preset consensus conditions, it issues downgraded collaborative instructions to the relevant drones, which then perform distributed obstacle avoidance actions based on visual perception, neighboring drone location information, and individual action value.

9. The method of claim 8, wherein, After the relevant UAVs perform the joint action or the distributed obstacle avoidance action, the collaborative decision center calculates the expected position, expected speed and expected relative position of each UAV based on the updated position and mission requirements. Each UAV generates control input and adjusts its flight state based on the expected position, expected speed and expected relative position, and feeds back the adjusted current position and current speed to the collaborative decision center. The collaborative decision center determines the formation reconfiguration state based on the feedback results.

10. A system for generating visual warning information based on sensory data, characterized in that include: The multi-source sensor data preprocessing module is configured to collect multi-source sensor data from each UAV and perform outlier removal, missing value repair and standardization to obtain preprocessed data. The state assessment module is configured to input the preprocessed data into a long short-term memory network to obtain the risk value corresponding to each state parameter, as well as the local state prediction value and warning level. The sparsity state generation module is configured to calculate an importance score based on the risk value, the spatial proximity between the UAV and neighboring UAVs, and the overlap of the mission area, filter state parameters, generate sparsity state features, and upload the sparsity state features, the predicted state value of the UAV, and the warning level to the collaborative decision-making center. The collaborative graph aggregation module is configured to update the collaborative graph by the collaborative decision center, perform graph attention aggregation on the sparse state features, local state prediction values ​​and warning levels to obtain an enhanced state representation, and generate a global state by combining global environmental information. The action value generation module is configured to generate individual action values ​​based on local observations consisting of the preprocessed data and the local state prediction value, and upload the individual action values ​​to the collaborative decision-making center, which generates joint action values ​​based on the global state and the individual action values. The warning and action delivery module is configured to predict the trajectory and calculate the collision probability based on the enhanced state representation and the local state prediction value, generate visual warning information according to the collision probability, and determine the joint action according to the joint action value and send it to the relevant UAV when the collision probability meets the action triggering condition.

Images