Multimodal data collaborative processing system and method for unmanned aerial vehicles (UAVs) and UAVs

By using a multimodal sensor array and a collaborative processing system, the UAV's perception reliability and adaptability in complex low-altitude environments were improved. This solved the problem of perception mission continuity under sensor anomalies and resource constraints, and enhanced the robustness and mission reliability of the UAV's perception system.

CN123067433APending Publication Date: 2026-11-10CHINA TELECOM CORP LTD TECHNOLOGY INNOVATION CENTER +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611272518.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-20
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

The reliability and robustness of UAV perception systems in complex low-altitude environments are insufficient, making it difficult to maintain the continuity and reliability of perception tasks under sensor anomalies or resource constraints.

Method used

It employs a multimodal sensor array, a health monitoring module, a mode switching module, a confidence assessment module, and a collaborative communication module to achieve refined identification of sensor anomalies and adaptive sensing mode switching, and to collaborate with external nodes for data processing when confidence is low or computing power is insufficient.

Benefits of technology

It improves the perception reliability, adaptability and mission continuity of UAVs in complex low-altitude environments. Through refined sensor anomaly identification and collaborative processing, it ensures the continuity and accuracy of perception missions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123067433A_ABST
    Figure CN123067433A_ABST
Patent Text Reader

Abstract

This application discloses a multimodal data collaborative processing system for unmanned aerial vehicles (UAVs). The system includes: a multimodal sensor group for collecting multimodal environmental data; a health monitoring module for receiving environmental data, assessing the quality of the data to detect anomalies in each sensor, and outputting a health score for each sensor; a mode switching module for identifying anomaly types based on health scores and switching from the current perception mode to a target perception mode among a set of preset perception modes; a perception processing module for processing environmental data in the target perception mode to generate perception results; a confidence assessment module for calculating a confidence score on the perception results; and a collaborative communication module for sending a collaborative request to a collaborative processing node and receiving returned data when the confidence score is below a preset threshold or when local computing power is insufficient, to assist in updating the perception results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV multimodal data collaborative processing system, method, and UAV. Background Technology

[0002] With the rapid development of sensor miniaturization, edge computing, and low-altitude communication technologies, unmanned aerial vehicle (UAV) systems have gradually evolved from simple video acquisition platforms into new intelligent carriers with environmental perception, autonomous decision-making, and multi-UAV collaborative capabilities. Currently, UAVs are widely used in various scenarios such as urban low-altitude monitoring, power line inspection, agricultural remote sensing, emergency search and rescue, and logistics delivery. In practical missions, UAVs typically carry multiple heterogeneous sensors, including visible light cameras, infrared cameras, lidar, and GNSS / IMU integrated navigation systems, to acquire multimodal perception data such as images, point clouds, and position and attitude, thereby supporting core functions such as target detection, obstacle avoidance, path planning, and autonomous navigation.

[0003] However, when performing low-altitude sensing missions, UAVs face complex and ever-changing environmental conditions and resource constraints. The reliability and robustness of their sensing systems still face many challenges, which restricts the further improvement of UAVs' operational capabilities and intelligence level in complex low-altitude environments. Summary of the Invention

[0004] This application provides a multimodal data collaborative processing system, method, and drone for unmanned aerial vehicles (UAVs), which can improve the operational capabilities and intelligence level of UAVs in complex low-altitude environments.

[0005] According to a first aspect of this application, a multimodal data collaborative processing system for unmanned aerial vehicles (UAVs) is disclosed, applied to UAVs, the system comprising: A multimodal sensor array is used to collect multimodal environmental data. A health monitoring module is connected to the multimodal sensor group, used to receive the environmental data, perform quality assessment on the environmental data to detect whether there are any abnormalities in each sensor, and output the health score corresponding to each sensor. The mode switching module, connected to the health monitoring module, is used to identify the abnormality type based on the health score, and switch from the current perception mode to a target perception mode among a set of preset perception modes based on the abnormality type; wherein, the multiple perception modes use different sensors in the multimodal sensor group as the dominant perception source. A perception processing module, connected to the mode switching module, is used to process the environmental data in the target perception mode and generate perception results; The confidence assessment module, connected to the perception processing module, is used to calculate a confidence score on the perception result; The collaborative communication module, connected to the confidence assessment module, is used to send a collaborative request to the collaborative processing node and receive the returned data when the confidence score is lower than a preset score threshold or the local computing power is insufficient, so as to assist in updating the perception results.

[0006] According to a second aspect of this application, a method for collaborative processing of multimodal data from unmanned aerial vehicles (UAVs) is disclosed, applied to UAVs, the method comprising: Multimodal environmental data is collected using a multimodal sensor array; The health monitoring module receives the environmental data, performs a quality assessment on the environmental data to detect whether there are any abnormalities in each sensor, and outputs the corresponding health score for each sensor. The mode switching module identifies the abnormality type based on the health score and switches from the current perception mode to the target perception mode among a set of preset perception modes based on the abnormality type; wherein, the multiple perception modes use different sensors in the multimodal sensor group as the dominant perception source. The perception processing module processes the environmental data under the target perception mode to generate perception results. A confidence score is calculated on the perceived results using a confidence assessment module. When the confidence score is lower than a preset score threshold or the local computing power is insufficient, a collaborative request is sent to the collaborative processing node through the collaborative communication module and the returned data is received to assist in updating the perception results.

[0007] According to a third aspect of this application, a drone is disclosed, comprising: body; A multimodal sensor array, mounted on the device body, is used to collect multimodal environmental data; An edge computing device is disposed inside the fuselage or mounted on the fuselage. The edge computing device includes a processor and a memory. The memory stores a computer program. When the processor executes the computer program, it implements the UAV multimodal data collaborative processing method as described in the second aspect.

[0008] According to a fourth aspect of this application, a computer-readable storage medium is disclosed having a computer program / instructions stored thereon, which, when executed by a processor, implements the UAV multimodal data collaborative processing method as described in the second aspect.

[0009] According to the fifth aspect of this application, a computer program product is disclosed, including a computer program / instructions that, when executed by a processor, implement the UAV multimodal data collaborative processing method as described in the second aspect.

[0010] In this embodiment, the UAV uses a health monitoring module in its multimodal data collaborative processing system to assess the quality of environmental data and output a health score, enabling refined identification of abnormal states of various sensors. The system's mode switching module identifies the anomaly type based on the health score and switches to a target perception mode with different sensors as the primary sensing source, automatically selecting the appropriate perception link when sensors malfunction, ensuring the continuity of the perception task. The system's confidence assessment module calculates a confidence score on the perception results, quantifying the reliability of the perception results. The system's collaborative communication module sends a collaborative request to the collaborative processing node and receives returned data when the confidence score is below a preset threshold or when local computing power is insufficient, leveraging the computing and perception resources of external nodes to improve perception accuracy and system fault tolerance. Thus, the UAV's perception reliability, adaptability, and task continuity are effectively improved in complex low-altitude environments. Attached Figure Description

[0011] Figure 1 This is a structural block diagram of a multimodal data collaborative processing system for unmanned aerial vehicles (UAVs) provided in some embodiments of this application; Figure 2 This is a flowchart of a method for collaborative processing of multimodal data from unmanned aerial vehicles (UAVs) provided in some embodiments of this application; Figure 3 This is a flowchart illustrating one embodiment of step 202 provided in some embodiments of this application; Figure 4 This is a flowchart illustrating one embodiment of step 203 provided in some embodiments of this application; Figure 5 This is a schematic diagram of the architecture of a drone cooperative system provided by some embodiments of this application; Figure 6 This is a signaling interaction flowchart for a cross-device collaboration scenario provided by some embodiments of this application; Figure 7 This is a structural block diagram of a drone provided in some embodiments of this application. Detailed Implementation

[0012] It should be noted that the image data obtained in this application is accessed, collected, stored, and used for subsequent analysis and processing after the user or relevant data owner has been clearly informed of the content of the data collection, the purpose of the data, the processing method, etc., and with the consent and authorization of the user or relevant data owner. Furthermore, the application can provide the user or relevant data owner with the means to access, correct, or delete the data, as well as the method to revoke consent or authorization.

[0013] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0015] To facilitate understanding, some relevant concepts and application scenarios involved in the embodiments of this application will be introduced below.

[0016] I. Related Concepts A Global Navigation Satellite System (GNSS) is a collection of global satellite positioning systems operated by multiple countries or regions. These include the United States' Global Positioning System (GPS), Russia's GLONASS, the European Union's Galileo, and China's BeiDou Navigation Satellite System (BDS). GNSS receives signals from multiple satellites to calculate the precise three-dimensional position and velocity information of terminal equipment on Earth, and is widely used in navigation, positioning, aerospace, and unmanned systems.

[0017] An Inertial Measurement Unit (IMU) is used to measure the acceleration and angular velocity of a device in three-dimensional space. It typically includes a three-axis accelerometer and a three-axis gyroscope. IMUs can be used to calculate attitude, velocity, and displacement, and are an important auxiliary positioning method in scenarios where GNSS is unavailable or has weak signal coverage.

[0018] The Horizontal Dilution of Precision (HDOP) is an important indicator for measuring the quality of satellite geometric distribution in a GNSS system. A lower HDOP value indicates more stable and accurate positioning results; a higher HDOP indicates concentrated satellite distribution and potentially increased positioning errors. In this application's embodiments, HDOP is used as one of the key parameters for assessing GNSS health status.

[0019] Multimodal data refers to heterogeneous sensing information collected from different types of sensors, such as cameras, lidar, and GNSS, including images, point clouds, and location information. Multimodal data fusion can improve the accuracy and robustness of sensing systems in modeling the environment.

[0020] Soft labels refer to the probability distribution output by a model in a classification task, reflecting the model's confidence in predicting different categories. Compared to hard labels (i.e., the final classification result), soft labels contain more information about the model's degree of "hesitation" and have higher information density in knowledge distillation. In federated distillation, each device uploads soft labels generated by its local model instead of the original parameters or data, which are then learned and fused by the central node to train the global model.

[0021] Federated distillation is a distributed model collaborative training method. Each device trains its model locally and outputs soft labels. These soft labels are then reported to an aggregation node, which aggregates and learns a unified teacher model. This teacher model is then distilled into a lightweight model and distributed to each edge node. This method supports collaborative training of heterogeneous models and effectively protects data privacy, as each device only uploads soft labels and not the original data.

[0022] II. Application Scenarios The UAV multimodal data collaborative processing system provided in this application embodiment can be applied to the following scenarios: urban low-altitude monitoring, power line inspection, agricultural remote sensing, and emergency search and rescue.

[0023] In urban low-altitude monitoring scenarios, when UAVs perform inspection missions over urban areas, factors such as building obstruction and electromagnetic interference may cause GNSS signal instability, and camera image quality may degrade in backlit or shadowed areas. The system provided in this application embodiment can monitor the status of each sensor in real time, automatically switch to an appropriate sensing mode when sensor performance degrades, and can call upon the sensing resources of neighboring UAVs for collaboration when the confidence level of a single UAV is low, ensuring the continuity and reliability of urban low-altitude monitoring missions.

[0024] In power line inspection scenarios, power inspection drones need to fly along high-voltage transmission lines and operate for extended periods in complex terrain and weather conditions. Long-term sensor operation can lead to performance drift. The system provided in this application continuously tracks the operating status of each sensor through a health monitoring module, promptly identifies potential performance degradation trends, and ensures high-quality completion of inspection tasks even under harsh conditions through mode switching and cross-device collaboration mechanisms.

[0025] In agricultural remote sensing scenarios, agricultural drones collect farmland image data while flying at low altitudes. However, due to factors such as changes in lighting and haze, a single sensor cannot consistently maintain optimal performance. The system provided in this application can dynamically adjust its sensing strategy based on environmental conditions and sensor status, maintaining the stability and accuracy of sensing results in complex and ever-changing agricultural operating environments.

[0026] In emergency search and rescue scenarios, environmental conditions at disaster sites are typically extremely harsh, posing a severe challenge to the reliability of UAV sensors. The system provided in this application, through a multi-level degradation strategy and a cross-device collaboration mechanism, can effectively cope with sudden sensor failures and maintain the continuous progress of search and rescue missions by leveraging the sensing and computing resources of nearby devices.

[0027] However, UAVs face complex and ever-changing environmental conditions and resource constraints when performing low-altitude sensing missions, and the reliability and robustness of their sensing systems still face many challenges. On the one hand, cameras are susceptible to image quality degradation due to factors such as changes in lighting, rain and fog, and lens damage; lidar has sparse point clouds and is easily interfered with by noise on distant or low-reflectivity targets; GNSS signals are easily affected by obstruction or multipath effects in urban canyons or indoor-outdoor transition areas, leading to a decrease in positioning accuracy. Sensor performance degradation often presents as gradual or sudden, and most existing systems can only determine whether the sensor is online or has data output, making it difficult to finely identify and classify specific degradation forms such as blurriness, overexposure, sparseness, and high noise. On the other hand, when the performance of a certain sensor deteriorates significantly or even fails, the system lacks an adaptive sensing mode switching mechanism for different types of anomalies, and usually can only adopt coarse processing methods such as simple data discarding or overall frequency reduction, making it difficult to promptly activate alternative sensing links to maintain mission continuity. Furthermore, the single-machine perception capability is limited by its own computing resources and observation perspective. When the perception confidence is low or the computing power is insufficient, the existing system is unable to effectively utilize the computing and perception resources of nearby devices or edge nodes for collaborative reasoning. The overall fault tolerance and environmental adaptability of the system still have considerable room for improvement.

[0028] To address the aforementioned technical problems, embodiments of this application provide a multimodal data collaborative processing system, method, and drone for unmanned aerial vehicles (UAVs).

[0029] The following section, with reference to the accompanying drawings, describes the UAV multimodal data collaborative processing system provided in the embodiments of this application.

[0030] Figure 1 This is a structural block diagram of a UAV multimodal data collaborative processing system provided in some embodiments of this application, such as... Figure 1 As shown, the UAV multimodal data collaborative processing system 100 may include: a multimodal sensor group 101, a health monitoring module 102, a mode switching module 103, a perception processing module 104, a confidence assessment module 105, and a collaborative communication module 106.

[0031] The multimodal sensor group 101 is used to collect multimodal environmental data.

[0032] In this embodiment, the multimodal sensor array is mounted on a drone to collect multimodal perception data in the drone's flight environment, providing a raw data foundation for subsequent health monitoring, perception processing, and decision control.

[0033] In this embodiment of the application, the multimodal sensor group includes at least one of the following sensors: a visible light camera, an infrared camera, a lidar, and a GNSS / IMU integrated navigation system.

[0034] In this embodiment, a visible light camera is used to acquire environmental image data, including images and videos in the visible light band. The image data can be used for tasks such as target detection, scene recognition, and visual positioning.

[0035] In this embodiment, a lidar is used to collect environmental point cloud data. By emitting a laser beam and receiving reflected signals, it measures the distance and orientation of target objects, generating three-dimensional point cloud information. The point cloud data can be used for tasks such as obstacle detection, terrain modeling, and geometric feature extraction.

[0036] In this embodiment, the GNSS / IMU integrated navigation system is used to collect the position, velocity, attitude, and acceleration information of the UAV. GNSS provides global positioning coordinates, and the IMU provides inertial measurement data; their combination enables high-precision positioning and attitude estimation.

[0037] In some embodiments, the multimodal sensor group may also include other types of sensors, such as infrared thermal imagers, multispectral cameras, millimeter-wave radars, ultrasonic sensors, etc. The specific type and number of sensors can be configured according to actual task requirements.

[0038] For example, in urban low-altitude inspection missions, a multimodal sensor array can be configured as follows: a high-resolution visible light camera (for capturing high-definition images), a 16-line lidar (for point cloud acquisition), and a GNSS / IMU integrated navigation system (for positioning and attitude estimation). In power line inspection missions, an infrared thermal imager can also be added to detect abnormal heating of power equipment.

[0039] The health monitoring module 102 is connected to the multimodal sensor group 101. It is used to receive environmental data, perform quality assessment on the environmental data to detect whether there are any abnormalities in each sensor, and output the corresponding health score for each sensor.

[0040] In this embodiment, the health monitoring module is a functional module used to evaluate the operating status of each sensor in the multimodal sensor group in real time. The health monitoring module analyzes the quality of the raw data output by each sensor to determine whether the sensor has experienced performance degradation or abnormalities, and quantifies the health status of each sensor in the form of a health score. The health score can be a numerical value, such as 0 to 100, or a level, such as "normal," "warning," or "abnormal," and the specific format can be set according to actual needs.

[0041] In this embodiment, sensor anomaly refers to a decrease in the quality of the data output by the sensor or a deviation of the sensor's operating state from the normal range. Anomalies can manifest in various forms, such as blurred, overexposed, or underexposed camera images; sparse point clouds and excessive noise in lidar; decreased GNSS positioning accuracy (high HDOP value); and excessive data transmission delay.

[0042] In this embodiment, the health monitoring module detects sensor anomalies by performing a quality assessment of environmental data. Quality assessment refers to analyzing the raw data output by the sensor to determine whether the data meets expected quality requirements. Compared to related technologies that only determine whether a sensor is "online" or "outputs data," this embodiment, through multi-dimensional quality assessment, can precisely identify specific sensor anomaly types, providing a more accurate basis for subsequent mode switching and anomaly handling.

[0043] In some embodiments, the health monitoring module 102 includes at least one of the following sub-modules: an image quality assessment sub-module, a point cloud quality assessment sub-module, a positioning quality assessment sub-module, and a communication quality assessment sub-module.

[0044] In this embodiment, the image quality assessment submodule is used to calculate a camera health score based on the image's brightness histogram and / or Laplacian variance.

[0045] In this embodiment, a luminance histogram is used to assess whether an image is overexposed or underexposed. A luminance histogram is a statistical distribution of pixel luminance values ​​in an image, with the horizontal axis representing luminance values ​​(0-255) and the vertical axis representing the number of pixels corresponding to that luminance. When the peak of the luminance histogram is concentrated in high-luminance areas (e.g., grayscale values ​​greater than 200), it indicates that the image is overexposed; when the peak is concentrated in low-luminance areas (e.g., grayscale values ​​less than 50), it indicates that the image is underexposed. Both overexposure and underexposure affect the integrity of target information in the image, leading to a decrease in the accuracy of subsequent sensory processing. The image quality assessment submodule determines whether an image is overexposed or underexposed by calculating the peak distribution position of the luminance histogram and outputs a corresponding health score accordingly.

[0046] In this embodiment, Laplacian variance is used to evaluate image sharpness. The Laplacian operator is a second-order differential operator used to detect edge intensity in an image. Laplacian variance refers to the statistical value of the variance of an image after filtering with the Laplacian operator. A larger variance indicates sharper image edges, while a smaller variance indicates a blurrier image. When the Laplacian variance is lower than a preset variance threshold, it indicates that the image is blurry, and the camera health score is correspondingly reduced.

[0047] For example, in urban low-altitude patrol missions, when a drone takes pictures under backlight conditions, the image may be overexposed, and the peak of the brightness histogram will be concentrated in the high-brightness area. If the image quality assessment submodule identifies that the peak of the brightness histogram exceeds a preset peak threshold, it will determine that the camera has an overexposure anomaly, and the camera health score will be reduced. Under foggy conditions, the image may be blurry, and the Laplacian variance will be lower than a preset variance threshold, which will determine that the camera has a blur anomaly, and the camera health score will be reduced accordingly.

[0048] In this embodiment of the application, the point cloud quality assessment submodule is used to calculate the LiDAR health score based on the point cloud density and / or the proportion of outliers.

[0049] In this embodiment, point cloud density refers to the number of valid points per unit space. Higher point cloud density indicates more refined environmental perception by the lidar; lower point cloud density indicates sparser point cloud data collected by the lidar, which may be due to distant targets, low-reflectivity targets, or a decline in the lidar's own performance. When the point cloud density is below a preset density threshold, the lidar is determined to have a sparsity anomaly, and its health score is correspondingly reduced.

[0050] In this embodiment, outliers refer to point cloud data points that deviate from the normal distribution. Outliers are typically caused by noise, multipath reflections, or sensor anomalies. The outlier ratio refers to the proportion of outliers to the total number of point clouds. When the outlier ratio exceeds a preset threshold, the lidar is determined to have a high noise anomaly, and the lidar health score is correspondingly reduced.

[0051] For example, in long-range target detection scenarios, the lidar may collect sparse point clouds due to the low reflectivity of the target. If the point cloud density is lower than a preset density threshold, the point cloud quality assessment submodule will determine that the lidar has a sparsity anomaly, and the lidar health score will decrease. In environments with strong electromagnetic interference, the lidar may collect a large number of outliers. If the proportion of outliers is higher than a preset proportion threshold, the lidar will be determined to have a high noise anomaly, and the lidar health score will decrease accordingly.

[0052] In this embodiment, the positioning quality assessment submodule is used to calculate the positioning module health score based on the horizontal accuracy factor of the global navigation satellite system.

[0053] In this embodiment, HDOP is an important indicator for measuring the quality of GNSS satellite geometric distribution. A lower HDOP value indicates a better satellite geometric distribution and more stable and accurate positioning results; a higher HDOP value indicates a concentrated satellite distribution, which may increase positioning errors. When the HDOP value exceeds a preset HDOP threshold, it is determined that the GNSS module has an abnormal decrease in positioning accuracy, and the positioning module's health score is correspondingly reduced.

[0054] For example, in an urban canyon environment, building obstruction may reduce the number of visible satellites and cause their distribution to become concentrated, raising the HDOP value to above 5. The positioning quality assessment submodule determines that the GNSS module is abnormal, and the positioning module health score decreases.

[0055] In this embodiment, the communication quality assessment submodule is used to output a communication link health score based on the status parameters of the data transmission link.

[0056] In this embodiment, the status parameters of the data transmission link include communication quality indicators such as data transmission delay, packet loss rate, and signal strength. Data transmission delay refers to the time required for data to travel from the sender to the receiver; when the delay exceeds a preset delay threshold, it indicates an anomaly in the communication link. Packet loss rate refers to the proportion of data packets lost during data transmission out of the total number of data packets sent; a high packet loss rate leads to incomplete data transmission. Signal strength refers to the signal power received by the receiver; a low signal strength may cause communication link instability. The communication quality assessment submodule outputs a communication link health score based on at least one of the above status parameters.

[0057] For example, when a single data transmission delay of more than 100ms is detected, the communication quality assessment submodule determines that there is a "transmission anomaly" and records this status synchronously to the health report, and the communication link health score is reduced accordingly.

[0058] In this embodiment, the health monitoring module integrates the health scores from the aforementioned sub-modules to generate a structured health score report for each sensor. This health score report may include information such as the health score value of each sensor, anomaly type identifier, and anomaly occurrence time, which is then used by the subsequent mode switching module for decision-making.

[0059] As can be seen, in this embodiment, through a multi-dimensional health monitoring system that combines image quality assessment, point cloud quality assessment, positioning quality assessment, and communication quality assessment, the health monitoring module can accurately identify the specific abnormal types of each sensor, such as blurring, overexposure, sparseness, high noise, decreased positioning accuracy, and communication link abnormalities. Compared with the binary detection method in related technologies that can only determine "online / offline", this embodiment can provide more accurate and richer decision-making basis for subsequent mode switching, and improve the system's early warning capability and self-diagnosis level.

[0060] The mode switching module 103 is connected to the health monitoring module 102 and is used to identify the abnormality type according to the health score, and switch from the current perception mode to the target perception mode among the preset multiple perception modes based on the abnormality type; wherein, the multiple perception modes use different sensors in the multimodal sensor group as the dominant perception source.

[0061] In this embodiment, the mode switching module is a functional module used to automatically select and switch sensing modes based on the health scores of the sensors. The mode switching module receives the health score report output by the health monitoring module, identifies the current abnormality type based on the health scores of each sensor, selects a target sensing mode from a variety of preset sensing modes based on the abnormality type, and performs a switching operation from the current sensing mode to the target sensing mode.

[0062] In this embodiment, the perception mode refers to the working configuration of the UAV perception system, defining which sensor is the dominant perception source in perception processing and the corresponding data processing method. Different perception modes correspond to different sensor combinations and processing strategies.

[0063] In this embodiment, the target perception mode refers to the perception mode selected and switched to by the mode switching module from a variety of preset perception modes according to the current anomaly type. The target perception mode is relative to the current perception mode and represents the target state of the switching operation.

[0064] In some embodiments, the preset multiple sensing modes include: camera-dominated mode, lidar-dominated mode, and position compensation mode. In camera-dominated mode, the camera in the multimodal sensor array is the dominant sensing source; in lidar-dominated mode, the lidar in the multimodal sensor array is the dominant sensing source; and in position compensation mode, the GNSS and IMU in the multimodal sensor array are the dominant sensing sources.

[0065] In this embodiment, the camera-dominated mode refers to a working mode in which image data acquired by the camera is the primary data source in the perception processing. In camera-dominated mode, the perception processing module mainly performs perception tasks such as target detection and scene recognition based on image data, with data from the LiDAR and GNSS / IMU serving as auxiliary or backup. This mode is suitable for situations where the LiDAR is malfunctioning but the camera is functioning normally. In this embodiment, the lidar-dominated mode refers to a working mode in which point cloud data acquired by lidar is the primary data source for perception processing. In lidar-dominated mode, the perception processing module mainly performs perception tasks such as obstacle detection and geometric mapping based on point cloud data, while data from the camera and GNSS / IMU serve as auxiliary or backup. This mode is suitable for situations where the camera malfunctions but the lidar is functioning normally.

[0066] In this embodiment, the location compensation mode refers to a perception mode that performs location compensation based on GNSS / IMU integrated navigation data and combined with a geographic database when both the camera and LiDAR are malfunctioning. In location compensation mode, the perception processing module queries the geographic database based on GNSS coordinates to obtain prior geographic information of the current location area and outputs perception results within the area estimation error range. This mode is suitable for extreme cases where both the camera and LiDAR are malfunctioning, providing basic positioning and perception information and ensuring mission continuity to a limited extent.

[0067] In this embodiment, by switching the dominant sensing source among different sensors, the system can automatically activate an alternative sensing link when a certain sensor malfunctions, ensuring the continuity of sensing tasks.

[0068] In some embodiments, the mode switching module 103 is specifically used to switch to camera-dominated mode when the health score indicates that the lidar is abnormal and the camera is normal.

[0069] In this embodiment, when the health score output by the health monitoring module indicates an anomaly in the LiDAR, such as sparse point clouds or high noise, while the health score of the camera is normal, the mode switching module switches the current perception mode to the camera-dominated mode. At this time, the perception processing module uses the image data acquired by the camera as the primary perception source to perform visual perception tasks such as image segmentation, target detection, and material recognition.

[0070] For example, during an inspection mission, when the UAV inspects distant or low-reflectivity targets, the LiDAR may show sparse point clouds or a decrease in the number of effective points. The health monitoring module detects that the LiDAR health score is below the threshold, while the camera's health score is normal. The mode switching module switches the perception mode from LiDAR-dominated mode to camera-dominated mode, and the UAV continues to perform inspections using images collected by the camera.

[0071] In some embodiments, the mode switching module 103 is specifically used to switch to the lidar-dominated mode when the health score indicator camera is abnormal and the lidar is normal.

[0072] In this embodiment, when the health score output by the health monitoring module indicates an anomaly in the camera (e.g., blurry or overexposed), while the health score from the LiDAR is normal, the mode switching module switches the current perception mode to the LiDAR-dominated mode. At this time, the perception processing module uses the point cloud data collected by the LiDAR as the primary perception source, performing perception tasks such as point cloud clustering, obstacle detection, and geometric feature extraction.

[0073] For example, in a low-altitude urban monitoring mission, if the camera of a drone is overexposed under strong backlight conditions, the health monitoring module detects that the camera's health score is below the threshold, while the LiDAR health score is normal. The mode switching module then switches the perception mode from camera-dominated mode to LiDAR-dominated mode, and the drone continues to perform obstacle detection and environmental perception through LiDAR.

[0074] In some embodiments, the mode switching module 103 is specifically used to switch to the position compensation mode when both the health score indicator camera and the lidar are abnormal.

[0075] In this embodiment, when the health score output by the health monitoring module indicates abnormalities in both the camera and the lidar, the mode switching module switches the current perception mode to the location compensation mode. At this time, the perception processing module uses GNSS / IMU integrated navigation data as a basis, combined with a geographic database, to perform location compensation and output perception results within the area estimation error range.

[0076] For example, in severe weather conditions such as sandstorms, the camera may be blurred due to sand and dust obstruction, and the lidar may have excessive point cloud noise due to sand and dust reflection. When both health scores are below the threshold, the mode switching module switches to the position compensation mode. The UAV queries the geographic database based on GNSS coordinates to obtain prior information on the terrain and obstacles in the current area and maintain basic perception capabilities.

[0077] In some embodiments, the mode switching module is specifically used to: switch to camera-dominated mode or lidar-dominated mode when the health score indicates that the global navigation satellite system and inertial measurement unit are abnormal and the camera and lidar are both normal; and the cooperative communication module is also used to request location information and wireless ranging information from the cooperative processing node in order to correct the pose cumulative drift.

[0078] In this embodiment, when the health score output by the health monitoring module indicates an anomaly in the GNSS / IMU, while the health scores of the camera and LiDAR are normal, the mode switching module switches the current perception mode to either camera-dominated or LiDAR-dominated mode. At this time, the perception processing module uses the last reliable GNSS location as the initial anchor point and performs inter-frame registration based on the image sequence acquired by the camera or the point cloud sequence acquired by the LiDAR to calculate the relative pose change of the UAV, maintaining high-frequency pose output to ensure uninterrupted flight control and perception tasks. Simultaneously, the cooperative communication module requests location information and wireless ranging information from nearby cooperative processing nodes to correct for accumulated pose drift.

[0079] In this embodiment, wireless ranging information refers to signal characteristic parameters that reflect the distance between the current UAV and the collaborative processing node, including but not limited to signal round-trip time, ultra-wideband ranging value, angle of arrival, or signal strength. The system requests location information and wireless ranging information from nearby collaborative processing nodes, using wireless ranging information (such as RTT, UWB, or communication link ranging) to construct absolute position observations and periodically correct the cumulative drift of the odometer. If no neighboring UAV is available, pure odometer operation is performed, while the cumulative drift is estimated based on the runtime and loopback detection, and the positional reliability is allowed to decrease monotonically over time. When the positional reliability falls below a threshold, hovering, return-to-home, or ground station intervention is triggered. Furthermore, in scenarios where inter-frame registration reliability decreases, such as weak textures or high-speed maneuvers, existing onboard IMU data can be further fused to improve pose estimation stability.

[0080] For example, in an urban low-altitude monitoring mission, when a UAV is performing an inspection mission in an urban canyon, GNSS signals are lost due to building obstruction. The health monitoring module detects that the GNSS health score is below the threshold (HDOP value exceeds 5), while the health scores of the camera and LiDAR are normal. The mode switching module maintains the current perception mode as camera-dominated mode. The perception processing module uses the last reliable GNSS position as the starting point and performs visual odometry calculation using the image sequence acquired by the camera to maintain high-frequency pose output. Simultaneously, the cooperative communication module requests position information and RTT ranging information from nearby cooperative processing nodes. The nearby cooperative processing nodes return their own positions and RTT measurements between themselves and the current UAV. The current UAV uses this information to construct an absolute position observation and periodically corrects the cumulative drift of the visual odometry. When the UAV flies out of the urban canyon and the GNSS signal is restored, the mode switching module switches back to the normal fusion positioning mode based on the health score.

[0081] In this embodiment, by maintaining the current perception mode and calculating the relative pose using visual or laser odometry when the GNSS / IMU is malfunctioning but the camera and lidar are functioning normally, and simultaneously requesting position information and wireless ranging information from the cooperative processing node to correct accumulated drift, the system can maintain continuous pose output and normal operation of the UAV's perception tasks even in GNSS failure scenarios. Compared with related technologies where GNSS failure triggers a return-to-home or hovering wait, this embodiment fully utilizes visual / laser perception capabilities and wireless ranging cooperative positioning methods, maintaining the UAV's autonomous flight and perception capabilities even during GNSS failure, thus improving the system's robustness and mission completion rate in GNSS-denied environments. Furthermore, by correcting odometry drift through neighboring UAV cross-positioning, the cumulative error problem inherent in long-term pure odometry operation is effectively solved, ensuring the long-term accuracy of pose output.

[0082] As can be seen, compared with the single backup logic of "primary use and backup" in related technologies, the embodiments of this application, for four different scenarios such as camera failure, lidar failure, dual failure, and GNSS failure but visual / radar normal operation, respectively switch to camera-dominated mode, lidar-dominated mode, position compensation mode, and a combination of odometry calculation and neighbor cross-positioning. This achieves a fine-grained switching from single backup-based degradation to failure type-driven switching, ensuring the continuity of perception tasks and improving the autonomous perception capability and mission reliability of UAVs in complex environments.

[0083] The perception processing module 104 and the connection mode switching module 103 are used to process environmental data in the target perception mode and generate perception results.

[0084] In this embodiment, the perception processing module is a functional module used to process multimodal environmental data and generate perception results under the currently selected perception mode. The perception processing module receives environmental data from the multimodal sensor group, processes the environmental data under the target perception mode, and generates perception results.

[0085] In this embodiment, the target perception mode determines the data processing method of the perception processing module. Under different target perception modes, the perception processing module employs different data processing strategies: In camera-dominated mode, the perception processing module primarily processes image data captured by the camera. Specifically, the perception processing module can detect and identify targets in the image using a target detection model, segment the image using an image semantic segmentation model to extract key features, and generate perception results containing information such as target category, location, and size based on the detection and segmentation results.

[0086] In LiDAR-dominated mode, the perception processing module primarily processes the point cloud data acquired by the LiDAR. Specifically, the perception processing module can segment the point cloud using point cloud clustering algorithms to extract the location, shape, and size information of obstacles, generating perception results that include the geometric features of the obstacles.

[0087] In location compensation mode, the perception processing module uses GNSS / IMU integrated navigation data and a geographic database as its processing foundation. Specifically, the perception processing module is used to query the geographic database based on GNSS coordinates to obtain prior geographic information of the current location area and output perception results within the area estimation error range.

[0088] For example, in a scenario where the camera malfunctions and the system switches to LiDAR-dominated mode, the perception processing module receives point cloud data collected by the LiDAR, segments the point cloud using a point cloud clustering algorithm, identifies the location and geometry of obstacles, and generates perception results containing information such as obstacle coordinates, size, and shape for subsequent confidence assessment and collaborative decision-making.

[0089] The confidence assessment module 105 is connected to the perception processing module 104 and is used to calculate the confidence score of the perception results.

[0090] In this embodiment, the confidence assessment module is a functional module used to quantify the reliability of the perception results generated by the perception processing module. The confidence assessment module calculates a confidence score for the perception results, which is used to quantify the reliability of the perception results and guide subsequent processes such as communication decisions, collaborative processing triggering, and model updates.

[0091] In some embodiments, the confidence assessment module 105 is specifically used to: calculate a confidence score for the perception result based on the assessment factors associated with the perception result; wherein the assessment factors include at least one of the following: prediction error of the perception result, environmental impact information, and consistency information of historical regional data; the environmental impact information includes at least one of the following: light intensity, weather conditions, and degree of shading.

[0092] In this embodiment, the prediction error of the perception result refers to the prediction uncertainty parameter synchronously output by the perception processing module when performing target detection or state estimation. In the perception mode based on a deep learning model, the prediction error includes the classification confidence score and the position uncertainty score of the regression box output by the model; in the state estimation mode based on Kalman filtering, the prediction error includes the covariance matrix of the estimated state. A larger prediction error indicates greater uncertainty in the perception result, and the confidence score should be correspondingly lower.

[0093] In this embodiment, environmental impact information refers to factors affecting perception quality, including light intensity, weather conditions, and degree of occlusion. Light intensity can be calculated using the camera's automatic exposure value; perception quality typically decreases under backlight or low-light conditions. Weather conditions can be obtained via a cloud-based weather API or detected locally; perception quality typically decreases under severe weather conditions such as rain, snow, or fog. The degree of occlusion can be calculated using changes in point cloud reflectance or the proportion of occluded areas in image semantic segmentation; perception quality typically decreases under severe occlusion. The more unfavorable the environmental impact information is to perception, the lower the confidence score should be.

[0094] In this embodiment, historical area data consistency information refers to the degree of consistency between the current perception result and the historical perception results for the same area. If a drone has flown over the same area multiple times and accumulated historical perception data, and the current perception result deviates significantly from the historical average result, it indicates that the current perception may be abnormal, and the confidence score should be reduced accordingly. Specifically, the confidence assessment module can establish a map database of perception results, store typical perception features of each area, and calculate the historical area data consistency score by comparing the similarity between the current perception result and the historical records.

[0095] In some embodiments, in GNSS failure scenarios, the confidence assessment module can also use the cumulative positioning drift as an evaluation factor for the confidence score, so that the confidence score can more accurately reflect the reliability of the sensing results during GNSS failure.

[0096] In this embodiment, the confidence assessment module integrates at least one of the above assessment factors and calculates the final confidence score through weighted summation, product fusion, or rule-based comprehensive scoring. The confidence score can be a value between 0 and 1, or it can be a percentage.

[0097] For example, when conducting urban low-altitude monitoring in rainy weather, the confidence assessment module obtains the prediction error of the sensing results (the target detection confidence of the model output is 0.75), environmental impact information (rainy weather causes sparse point clouds, and the environmental correction factor is 0.8), and historical regional data consistency information (the matching degree between the historical sensing results and the current sensing results at the current location is 0.9). Through weighted comprehensive calculation, the confidence score of the current sensing results is obtained as 0.78.

[0098] As can be seen, in this embodiment, by comprehensively considering multiple evaluation factors such as prediction error, environmental impact information, and consistency of historical regional data to calculate the confidence score, the confidence assessment module can dynamically and comprehensively reflect the credibility of the perception results. Compared with the single-factor evaluation method in related technologies that only relies on prediction error or fixed thresholds, the confidence assessment in this embodiment can more accurately reflect the true reliability of the perception results in complex scenarios, providing a more reliable basis for subsequent collaborative decision-making.

[0099] The collaborative communication module 106 is connected to the confidence evaluation module 105. When the confidence score is lower than the preset score threshold or the local computing power is insufficient, it sends a collaborative request to the collaborative processing node and receives the returned data to assist in updating the perception results.

[0100] In this embodiment, the cooperative communication module is a functional module used to initiate a cooperative request to an external cooperative processing node and receive returned data when the UAV's local perception confidence is low or its local computing power is insufficient. The cooperative communication module interacts with the cooperative processing node via a wireless communication link.

[0101] In this embodiment of the application, a collaborative processing node refers to an external node that can respond to collaborative requests and perform auxiliary processing tasks such as collaborative reasoning and model aggregation.

[0102] In some embodiments, the collaborative processing node includes at least one of the following: a nearby drone, an aggregation node, and a ground control station.

[0103] In this embodiment of the application, a neighboring UAV refers to other aircraft in the same network as the UAV currently performing the task, which can respond to collaborative processing requests and complete some computational tasks to achieve inference assistance and model fine-tuning.

[0104] In this embodiment, the aggregation node is a computing node that has a single-hop or low-latency wireless connection with the UAV and is capable of soft tag aggregation and model distillation distribution.

[0105] For example, the aggregation node includes the following two physical forms: one is the cluster head machine elected by the drone swarm to which the drone belongs, such as the cluster head machine with strong computing power and the best link quality. The cluster head machine can be dynamically re-elected according to the mission stage, the load status of each drone and the remaining power; the other is the edge base station / MEC server deployed on the ground or tower near the mission area of ​​the drone.

[0106] It should be noted that the essence of an aggregation node is a functional node rather than a fixed physical entity. Any computing node that can achieve single-hop or low-latency wireless connection with UAV nodes and has the ability to aggregate soft tags and distribute models can be used as an aggregation node.

[0107] In this embodiment, the ground control station is responsible for historical data management, overall task scheduling, strategy adjustment, and final result feedback and display, and is the center of system monitoring and closed-loop control.

[0108] In this embodiment of the application, the collaborative request refers to the request message sent by the UAV to the collaborative processing node, which is used to request external nodes to assist in completing perception reasoning or model training tasks.

[0109] In some embodiments, the collaboration request includes at least one of the following: information about the perception task to be collaborated on, and partial features generated by the local model. The collaboration processing node can perform collaborative inference based on this information without needing to obtain complete perception data or raw input from its own machine, thus reducing communication overhead and protecting local data privacy.

[0110] In some embodiments, the collaborative communication module is specifically used to: send a collaborative request to a neighboring drone when the confidence score is lower than a preset score threshold or the local computing power is insufficient; receive the collaborative reasoning result returned by the neighboring drone after performing collaborative reasoning in response to the collaborative request; the perception processing module is specifically used to: update the perception result based on the collaborative reasoning result.

[0111] In this embodiment, the preset score threshold can be a fixed value, such as 0.8, or it can be a value that is dynamically adjusted according to the scenario. When the confidence score is lower than the preset score threshold, it indicates that the reliability of the current perception result is insufficient, and collaborative processing is required with the help of the perception or computing resources of external nodes.

[0112] In this embodiment, insufficient local computing power refers to the fact that the onboard computing resources of the UAV (such as CPU, GPU, memory, etc.) are insufficient to complete the processing of the current perception task within the real-time requirements. When local computing power is insufficient, even if the confidence score is normal, it is necessary to rely on the computing resources of external nodes for collaborative processing.

[0113] For example, the preset score threshold is 0.8. In an urban low-altitude monitoring task, the confidence assessment module of UAV A calculates the confidence score of the current perception result to be 0.72, which is lower than 0.8. The cooperative communication module sends a cooperative request to the nearby UAV B. The cooperative request includes the perception task information to be cooperated on, "detect obstacles within 50 meters in front of the current position," and some features (image feature vectors) generated by the local model. After receiving the cooperative request, UAV B uses its own perception model to perform cooperative inference, generates a cooperative inference result, "two obstacles were detected, with coordinates of…," and returns it to UAV A. UAV A's perception processing module updates the local perception result based on the cooperative inference result, supplementing the current perception result with obstacle information.

[0114] In some embodiments, the confidence assessment module is further configured to: recalculate the confidence score after the perception processing module updates the perception result based on the collaborative reasoning result; if the recalculated confidence score is still lower than a preset score threshold, and the number of collaborative processing operations performed has not reached a preset number threshold, then send a collaborative request to the neighboring UAV again through the collaborative communication module; if the recalculated confidence score is still lower than a preset score threshold after collaborative processing operations have reached a preset number threshold, then report fault information.

[0115] In this embodiment, if the confidence score is still lower than a preset threshold after a single collaborative processing step, a collaborative request is initiated again. Perception information is gradually accumulated through multiple collaborations to improve perception accuracy. To prevent resource waste due to infinite attempts, a preset threshold for the number of attempts is set, such as 3. If this threshold is not reached, collaborative attempts continue; if the confidence score is still not met after reaching the threshold, the attempt is terminated, fault information is reported, and a request is made to the ground control station for intervention. Through this method, the system strives to improve perception accuracy within a limited number of attempts while avoiding resource waste.

[0116] For example, after the first collaborative processing, the confidence score of UAV A increases from 0.72 to 0.78, but is still below the preset score threshold of 0.8, and the number of collaborative processing operations performed (1 time) has not reached the preset number threshold (3 times). The confidence assessment module triggers a second collaborative request, initiating collaborative inference with UAV C. After the second collaborative processing, the confidence score increases to 0.82, reaching the preset score threshold, and the collaborative processing ends. If the confidence score is still below 0.8 after 3 collaborative processing operations, that is, the preset number threshold has been reached but the confidence score is still not met, then fault information is reported, and the current perception anomaly status of the UAV is reported to the ground control station.

[0117] As can be seen from the above embodiments, in this embodiment, the health monitoring module assesses the quality of environmental data and outputs a health score, enabling refined identification of abnormal states of various sensors; the mode switching module identifies the anomaly type based on the health score and switches to a target perception mode with different sensors as the primary sensing source, automatically selecting the appropriate perception link when sensors malfunction, ensuring the continuity of the perception task; the confidence assessment module calculates a confidence score on the perception results, quantifying the reliability of the perception results; and the collaborative communication module sends a collaborative request to the collaborative processing node and receives returned data when the confidence score is lower than a preset threshold or local computing power is insufficient, leveraging the computing and perception resources of external nodes to improve perception accuracy and system fault tolerance. Thus, the perception reliability, adaptability, and task continuity of the UAV in complex low-altitude environments are effectively improved.

[0118] Currently, in inspection formations containing multiple drones, the computational load of different drones can vary significantly. For example, drones closer to the target area may experience computational resource strain due to high-frequency sensing tasks, while drones farther from the target area may be idle. If a drone initiating a collaboration request randomly selects a neighboring node, it may assign the collaboration task to a drone already under heavy load, leading to response delays or even request failures. To address this, embodiments of this application provide a collaborative load balancing mechanism.

[0119] Accordingly, in some embodiments provided in this application, the collaborative communication module is further configured to: send a resource request to the aggregation node before sending a collaborative request to a neighboring drone; receive a resource scheduling instruction returned by the aggregation node; wherein the resource scheduling instruction is used to indicate the neighboring drone with the best current computing resources; and send a collaborative request to a neighboring drone according to the resource scheduling instruction.

[0120] In this embodiment of the application, resource request refers to the request message from the aggregation node to query the computing resource status (such as CPU utilization, memory utilization, current task load, etc.) of each neighboring drone in the current network before initiating a collaboration request.

[0121] In this embodiment, the resource scheduling instruction refers to the scheduling instruction returned by the aggregation node after comprehensive evaluation based on the computing resource status of each neighboring UAV. The aggregation node schedules tasks to the neighboring UAV with the best computing resources based on the current device load, thereby improving response efficiency.

[0122] For example, drone A needs to initiate a collaboration request, first sending a resource request to the aggregation node. The aggregation node queries the computing resource status of neighboring drones B, C, and D in the current network: drone B's CPU utilization is 85% (high load), drone C's CPU utilization is 30% (low load), and drone D is currently offline. After comprehensive evaluation, the aggregation node returns a resource scheduling instruction, indicating that drone C is the optimal collaboration target with the current computing resources. Drone A sends a collaboration request to drone C according to the resource scheduling instruction, and drone C performs collaborative inference.

[0123] As can be seen, in this embodiment, by sending a resource request to the aggregation node before sending a collaboration request to a neighboring drone, receiving a resource scheduling instruction from the aggregation node indicating the neighboring drone with the best current computing resources, and then sending the collaboration request according to the instruction, the optimal allocation of computing resources for collaborative tasks is achieved. Compared with the random or fixed selection schemes in related technologies, this embodiment assigns collaborative tasks to the neighboring drone with the best current computing resources, avoiding response delays or failures caused by sending requests to overloaded nodes, improving the efficiency and success rate of collaborative processing, and optimizing the overall computing resource utilization of the drone cluster.

[0124] Currently, in urban low-altitude patrol scenarios, multiple drones simultaneously perform patrol tasks in adjacent airspaces. Due to differences in flight paths, lighting conditions, and observation angles, each drone accumulates different perception experiences. For example, drone A mainly patrols the eastern area in the morning, accumulating rich image samples under well-lit conditions; drone B patrols the western area in the afternoon, accumulating image samples under backlight conditions; and drone C patrols the southern area in the evening, accumulating image samples under low-light conditions. If each drone's model is trained solely on its own data, its perception capabilities are limited by the constraints of local data; for example, drone A's target detection accuracy may decrease under backlight conditions. Therefore, this application provides a cross-device collaborative modeling mechanism based on federated distillation.

[0125] Accordingly, in some embodiments provided in this application, the collaborative communication module is further configured to: report the soft tags generated by the local model to the aggregation node, so that the aggregation node can aggregate the soft tags to generate a teacher model and distill the teacher model into a lightweight model; receive the lightweight model returned by the aggregation node; the perception processing module is further configured to: update the parameters of the local model according to the lightweight model; wherein, the parameter update includes: when the local model is a valid model, performing a weighted fusion of the lightweight model and the local model to obtain an updated local model; when the local model is an invalid model, covering the local model with the lightweight model.

[0126] In this embodiment, the local model refers to a perception model deployed on the UAV, used to process environmental data collected by sensors and output perception results containing information such as target category, location, and size. The local model can be a target detection model, a semantic segmentation model, a point cloud classification model, etc., and the specific type is determined according to the UAV's perception task.

[0127] In this embodiment, during the federated distillation process, each UAV reports soft tags generated by its local model to the aggregation node. Upon receiving the soft tags from each UAV, the aggregation node uses an aggregation algorithm to fuse the soft tags from multiple local models, generating a unified teacher model. The teacher model refers to a large-scale, high-performance model trained by the aggregation node using the aggregated soft tags reported by each device, containing the perceptual knowledge of all participating devices.

[0128] In this embodiment, the teacher model distillation is a lightweight model, which means transferring the knowledge of the teacher model to a lightweight model with fewer parameters and faster inference speed through knowledge distillation technology. The lightweight model is suitable for deployment on drones with limited computing resources.

[0129] In this embodiment, a valid local model means that a usable historical model exists locally and the local model has not been determined to be invalid; an invalid local model means that the node is in a cold start or is joining the network for the first time and has no usable historical model, or the confidence score of the local model for a consecutive preset number of perception periods is lower than a preset score threshold and is therefore determined to be invalid. The preset number and preset score threshold can be configured according to the actual task scenario.

[0130] In this embodiment, the local model's parameters are updated using a lightweight model returned by the aggregation node. After the parameter update, the local model incorporates perception knowledge from other drones, improving its generalization ability and perception accuracy. The parameter update of the local model can include the following two methods: If the local model is valid, a weighted fusion of the lightweight model and the local model is performed to obtain the updated local model. The fusion method uses parametric interpolation: θ local ←(1-α)×θ local +α×θ distill , where θ local θ represents the parameters of the local model. distill The parameters represent the lightweight model, where α is the fusion coefficient, dynamically determined by the deviation between the local recent average confidence score and the local data distribution. A higher local confidence score indicates a better fit for the current operational scenario, and a smaller α value retains more local personalization. For example, α can be determined as follows: when the local recent average confidence score is higher than the first threshold, α is less than 0.3; when the local recent average confidence score is lower than the second threshold, α is greater than 0.5.

[0131] It should be noted that the above parameter interpolation update applies to situations where the network structure and corresponding parameter dimensions of the lightweight model and the local model are compatible. Local models between different drone nodes can adopt different model structures; when the parameter structure of the lightweight model issued by the aggregation node is incompatible with the local model of a certain drone node, the above parameter-by-parameter interpolation update is not performed, but knowledge distillation is used to update the local model in order to retain support for heterogeneous model collaborative training.

[0132] If the local model is invalid, a lightweight model will be used to cover it. Direct coverage is only applicable in two situations: first, when a node is cold-starting or joining the network for the first time and there is no valid historical model locally; second, when the local model has been determined to be invalid, i.e., the confidence level is below the threshold for several consecutive periods.

[0133] For example, in an inspection formation consisting of 10 drones, each drone periodically reports the soft labels output by its target detection model to an aggregation node. The aggregation node collects all soft labels and aggregates them to generate a unified teacher model. The aggregation node then distills the teacher model into a lightweight model and sends it back to each drone. Upon receiving the lightweight model, each drone updates its parameters using either weighted fusion or direct overlay based on the effectiveness of its local model, achieving a synergistic improvement in model performance.

[0134] In this embodiment, the model is managed with version numbers to support rollback to historical versions. Each local model is assigned a unique version number upon update, and the aggregation node carries the version number when distributing lightweight models. The current version number is recorded after each local model update. When a rollback is needed, the perception processing module restores the model to the previous version based on the historical version record. Version number management facilitates tracking the model's iteration history, enabling fault localization and version rollback.

[0135] As can be seen, in this embodiment, by reporting the soft tags generated by the local model to the aggregation node, the aggregation node aggregates and generates a teacher model, which is then distilled into a lightweight model and sent back to each drone. Each drone updates the parameters of its local model based on the lightweight model, thus achieving knowledge sharing among multiple devices without transmitting the original perception data. Compared with related technologies that require centralized collection of original data, this embodiment only transmits soft tags, effectively protecting data privacy. Compared with related technologies where all nodes use the same model, this embodiment fuses the lightweight model and the local model through a weighted fusion method, balancing globally consistent knowledge with local personalized scene adaptation, and avoiding the performance degradation of a single global model in specific scenarios.

[0136] In some embodiments provided in this application, the aggregation node updates the teacher model using a continuous incremental update method, rather than retraining after each task. Re-aggregation is triggered when any of the following conditions are met: Quantity conditions trigger the following conditions: the cumulative number of newly received soft tags reaches a preset threshold N; or, the number of drone nodes reporting soft tags reaches a preset threshold K, and the number of newly added samples at each node is not less than a preset threshold M. For example, N can be 500 tags, K can be 5 drones, and M can be 50 tags. The specific values ​​can be configured according to the task scenario, and this application embodiment does not limit them.

[0137] Time-based trigger: The time elapsed since the last aggregation exceeds a preset duration threshold T. For example, T can be 1 hour, and can be configured according to task density; this embodiment does not limit this.

[0138] Event conditions are triggered when: the average confidence level of multiple drones within the area decreases by more than a preset threshold for P consecutive perception cycles; or, the operational scenario changes significantly, such as changes in weather, lighting, or the operational area; or, a new drone node joins the network and its local model version lags behind the current teacher model version by more than a preset generation threshold L. For example, P can be 3 cycles and L can be 2 generations. The specific values ​​can be configured according to the task scenario, and this application embodiment does not limit them.

[0139] In some embodiments, after generating a new teacher model, the aggregation node can also evaluate the performance of the new teacher model on the validation set locally retained by the aggregation node. If the performance of the new teacher model is not better than that of the current version of the teacher model, the new teacher model will not be distributed, and the version number will not be advanced, in order to avoid negative updates polluting the entire network.

[0140] In some embodiments, the teacher model and the lightweight model are managed with version numbers. The aggregation node distributes the lightweight model along with its version number, and each drone node records the currently used model version number. When a rollback is needed, the aggregation node can re-distribute the historical version of the lightweight model to the relevant nodes. Only incremental parameters can be transmitted during distribution to reduce communication overhead.

[0141] To facilitate understanding, an exemplary re-aggregation trigger scenario is given below: In an inspection formation containing 10 drones, the set time trigger threshold T is 1 hour. When more than 1 hour has passed since the last aggregation, the aggregation node automatically triggers re-aggregation. Simultaneously, if two new drones join the formation during the inspection process, and their model versions are more than L generations behind the current version (e.g., 2 generations), the event trigger condition is met, and the aggregation node triggers re-aggregation ahead of schedule, incorporating the new drones into the federated distillation system. The teacher model generated after re-aggregation is evaluated on the aggregation node's local validation set. If the accuracy is not lower than the current version, it is distilled into a lightweight model and deployed, with the version number incremented by 1. If, during testing, a node reports model incompatibility, the aggregation node rolls back to the previous version.

[0142] In actual deployments, while the lightweight model generated by the aggregation node integrates the perception experience of multiple drones, the operational scenarios in different areas vary. For example, the eastern area is dominated by densely built-up urban canyons, the western area by open farmland, and the southern area by waterways and bridges. The lightweight model generated by the aggregation node may perform well in some regional scenarios but experience performance degradation in others. If drones blindly adopt the lightweight model, their local perception capabilities may actually decrease. To address this, this application provides a performance verification and rollback mechanism after model fusion.

[0143] Accordingly, in some embodiments provided in this application, the perception processing module is further configured to: evaluate the performance of the fused model on local verification samples after weighted fusion of the lightweight model and the local model; if the performance of the fused model decreases by more than a preset threshold compared to before fusion, roll back to the local model version before the update, and report to the aggregation node that the current version is not compatible.

[0144] In this embodiment, local validation samples refer to small-scale, labeled, or confirmed datasets stored locally by the drone, used to verify performance changes after model updates. Performance refers to the model's perceptual accuracy, recall, or other evaluation metrics on the local validation samples. The preset threshold refers to the upper limit of tolerance for performance degradation, such as 3% or 5%.

[0145] For example, after receiving the lightweight model from the aggregation node, drone A performs a weighted fusion of the lightweight model and the local model. After fusion, drone A evaluates the detection accuracy of the fused model on local validation samples. If the accuracy before fusion is 87% and the accuracy after fusion is 82%, a decrease of 5%, exceeding the preset 3% threshold, drone A rolls back to the model version before fusion and reports "version incompatibility" feedback to the aggregation node. The aggregation node records this feedback and subsequently adjusts the distillation strategy or excludes the soft label from that node.

[0146] As can be seen, in this embodiment, the parameter update method combining weighted fusion and direct coverage balances global consistency and local personalization; the performance verification and rollback mechanism after fusion avoids negative updates from polluting the local model; and the version number management supports model tracking and rollback, improving the stability and security of model collaborative training.

[0147] In the federated distillation data transmission process between UAVs and aggregation nodes, soft tags, as important information carriers reflecting the characteristics of the UAV's local perception model, may suffer damage to the model aggregation effect or leakage of perception knowledge if intercepted or tampered with during transmission. Therefore, this application provides an end-to-end transmission encryption mechanism.

[0148] Accordingly, in some embodiments provided in this application, the collaborative communication module is also used to: perform end-to-end encrypted transmission of the soft tag before reporting the soft tag to the aggregation node.

[0149] In this embodiment, end-to-end encryption means that the data remains encrypted throughout the entire transmission process from the sender to the receiver, and intermediate nodes cannot decrypt the data content. Specifically, before the UAV reports the soft tag to the aggregation node, it uses an encryption algorithm to encrypt the soft tag data. After receiving the encrypted data, the aggregation node uses the corresponding decryption key to decrypt it and restore the soft tag data.

[0150] In this embodiment, the encryption algorithm can be the Advanced Encryption Standard (AES) symmetric encryption algorithm. AES is a widely used symmetric block cipher algorithm with advantages such as high security, fast encryption / decryption speed, and low resource consumption, making it suitable for deployment on UAV platforms with limited computing resources and energy consumption.

[0151] For example, before reporting the soft tags generated by its local model to the aggregation node, Drone A's collaborative communication module encrypts the soft tag data using a preset AES-256 encryption key and reports the encrypted ciphertext to the aggregation node. Upon receiving the ciphertext, the aggregation node decrypts it using the same AES-256 key to obtain the original soft tag data, which is then used for subsequent teacher model aggregation training. Throughout the transmission process, even if the ciphertext is intercepted by a third party, the specific content of the soft tags cannot be obtained due to the lack of the decryption key.

[0152] In this embodiment of the application, the encryption key can be a pre-configured key, that is, the key is pre-configured when the drone leaves the factory or enters the network; or, a dynamic key negotiation method can be used, that is, the drone and the aggregation node negotiate and generate a session key through a key exchange protocol (such as Diffie-Hellman key exchange) when communication is established.

[0153] In addition to encrypting the soft tag data, the collaborative communication module can also perform end-to-end encrypted transmission of other sensitive data such as perception task information and some features generated by the local model carried in the collaborative request, so as to comprehensively improve communication security.

[0154] As can be seen, in this embodiment, by performing end-to-end encrypted transmission of the soft tags before reporting them to the aggregation node, the interception or tampering of soft tag data during transmission is effectively prevented, protecting the perception knowledge and data privacy of each UAV. Compared with the plaintext transmission scheme in related technologies, this embodiment introduces encryption protection in the knowledge sharing process of federated distillation, ensuring the security of data transmission while realizing model collaborative training, and enhancing the system's security and anti-attack capabilities in open communication environments.

[0155] Currently, in urban low-altitude monitoring scenarios, drones need to transmit sensing results back to the ground control station in real time for ground operators to view and make decisions. However, communication bandwidth resources are limited. Reporting all sensing results would consume a large amount of communication resources; if no results are reported, ground operators cannot understand the situation on site. To address this, this application provides a highly selective reporting mechanism based on confidence level.

[0156] Accordingly, in some embodiments provided in this application, the collaborative communication module is also used to: report the perception results to the ground control station when the confidence score is higher than a preset score threshold and the local computing resources meet the real-time processing requirements.

[0157] In this embodiment of the application, high confidence reporting refers to sending the perception results to the ground control station for summary display or further analysis when the UAV's perception results are highly reliable (confidence score exceeds a preset score threshold) and local computing resources are sufficient.

[0158] In this embodiment, the preset score threshold and the aforementioned collaborative triggering threshold can be the same (e.g., 0.8), or different reporting thresholds and collaborative triggering thresholds can be set. Preferably, the reporting threshold is slightly higher than the collaborative triggering threshold to ensure that the reported perception results have high reliability.

[0159] For example, the preset score threshold is 0.8. After completing a low-altitude urban patrol mission, UAV A generates perception results for each detected target. Target A has a confidence score of 0.91, higher than 0.8, and its local computing resource utilization is 45%, meeting real-time processing requirements. The collaborative communication module reports this perception result to the ground control station. Target B has a confidence score of 0.73, lower than 0.8, triggering a collaborative request from the collaborative communication module. Through this selective reporting method, the ground control station receives only high-confidence, reliable information while saving communication bandwidth.

[0160] As can be seen, in this embodiment, the sensing results are only reported to the ground control station when the confidence score is higher than a preset score threshold and the local computing resources meet the real-time processing requirements, thus achieving selective reporting of sensing results. Compared with related technologies that either report all or none, this embodiment ensures the reliability of the reported information while avoiding the waste of communication bandwidth caused by reporting all sensing results, and also avoids the loss of on-site information caused by not reporting, achieving a balance between communication efficiency and information integrity.

[0161] Currently, in urban low-altitude monitoring scenarios, after UAV A sends a coordination request to a neighboring UAV B, it may encounter various situations such as UAV B being unable to respond due to its own busy tasks, UAV B entering a communication dead zone, or the wireless link being interfered with. If UAV A waits indefinitely, the current sensing task will not be completed in a timely manner, and may even affect the real-time performance of flight control. To address this, this application provides a three-level progressive timeout handling mechanism.

[0162] Accordingly, in some embodiments provided in this application, the collaborative communication module is further configured to: after sending a collaborative request, if no response is received within a preset time period, resend the collaborative request to the backup collaborative processing node; if the number of retries exceeds a preset number threshold or the total collaborative latency exceeds a preset latency threshold, downgrade to local processing and mark the corresponding perception result as a non-collaborative state; if the confidence score corresponding to local processing is still lower than a preset score threshold, report fault information.

[0163] In this embodiment, the preset duration refers to the single-timeout period, such as 500ms. This value can be dynamically adjusted according to the communication link quality to balance response speed and fault tolerance. Backup collaborative processing nodes refer to other available collaborative processing nodes besides the node making the initial request, such as another nearby drone or aggregation node.

[0164] In this embodiment, retry refers to the operation of sending the request again after a timeout in the same collaborative request. The number of retries is preferably no more than two. During retries, a backup neighbor is selected, or the request is resent to the aggregation node for processing, avoiding repeated requests to the same unresponsive node, thus preventing wasted latency budget on failed nodes.

[0165] In this embodiment, the total coordination latency refers to the total time elapsed from the initial sending of the coordination request to the final obtaining of the processing result. The preset latency threshold refers to the maximum latency allowed in the coordination process. The total coordination latency must be less than the preset latency threshold to ensure that the real-time performance of flight control is not blocked by the coordination process. For example, the preset latency threshold can be set to 1.5s; once exceeded, the system unconditionally enters the second-level degradation stage.

[0166] In this embodiment, downgrading to local processing means abandoning collaboration and having the UAV complete the perception processing itself. The corresponding perception results are marked as "non-collaborative state," the confidence level is lowered by a preset percentage, and a conservative task strategy is switched, such as slowing down, hovering, or reducing the task level.

[0167] In this embodiment of the application, reporting fault information means reporting the current abnormal perception status of the UAV to the ground control station, requesting manual intervention or triggering a return to base.

[0168] For example, the preset score threshold is 0.8. After UAV A sends a cooperation request to neighboring UAV B, if it does not receive a response within 500ms, it resends the cooperation request to neighboring UAV C. If the total latency after the second retry reaches 1.6s, exceeding the preset latency threshold of 1.5s, the cooperation attempt is terminated, and the system is downgraded to pure local processing. The perception processing module completes target detection locally, generates a perception result, and marks it as "non-cooperative state," lowering the confidence score by 10 percentage points. If the lowered confidence score is still below 0.8, a fault report is triggered and sent to the ground control station.

[0169] As can be seen, in this embodiment, through a three-level progressive timeout handling mechanism, the collaborative communication module can progressively degrade in the event of communication anomalies, prioritizing the real-time performance of flight control and triggering timely fault reporting in extreme cases. Compared with related technologies that involve infinite waiting or simply discarding requests, this embodiment provides a complete processing chain from retry to degrading to reporting in the event of communication anomalies, triggering timely fault reporting in extreme cases and avoiding flight safety issues caused by collaborative process blockage.

[0170] Currently, when drones perform remote inspection missions, they may pass through areas with signal obstruction, such as behind tall buildings or in valleys, causing temporary interruptions or degraded communication links. If data needs to be transmitted during this period, such as sensing results or soft tags, discarding it directly will result in data loss and affect mission integrity. To address this, this application provides a communication link anomaly handling mechanism.

[0171] Accordingly, in some embodiments provided in this application, the collaborative communication module is further configured to: activate a local caching mechanism to store the data to be sent in the local cache when the communication link health score indicates that the communication link is abnormal; resend the data in the local cache after the communication link is restored; and switch to a low bit rate transmission mode if the communication link fails to be restored within a preset time.

[0172] In this embodiment of the application, communication link anomalies include data transmission delay exceeding a preset delay threshold (e.g., 100ms) or the number of consecutive data transmission failures exceeding a preset number threshold (e.g., 3 times).

[0173] In this embodiment, the local caching mechanism refers to temporarily storing the data to be sent in local storage medium when the communication link is abnormal, and sending it again after the link is restored. The local caching mechanism can prevent data loss during communication failures and ensure the integrity of data transmission.

[0174] In this embodiment, the low bitrate transmission mode refers to reducing the data transmission bitrate to reduce communication pressure when the communication link is continuously abnormal. Low bitrate transmission can be achieved by reducing image resolution, decreasing point cloud sampling density, compressing data, etc.

[0175] For example, when UAV A is performing an inspection task, if it detects a data transmission delay exceeding 100ms with the aggregation node, the collaborative communication module activates a local caching mechanism to store the sensing data to be reported in the local cache. Once the communication link is restored, the cached data is retransmitted. If the communication link fails to be restored within 60 seconds, it switches to a low bitrate transmission mode to continue transmitting critical data at reduced resolution, ensuring uninterrupted data transmission.

[0176] As can be seen, in this embodiment, by combining a local caching mechanism and a low bit-rate transmission mode, the cooperative communication module first stores the data to be sent into the local cache when it detects a communication link anomaly. Once the link is restored, the data is resent. If the link remains abnormal, it switches to the low bit-rate transmission mode. Compared to related technologies that directly discard data when communication is abnormal, this embodiment ensures no data loss through local caching and uninterrupted transmission through the low bit-rate transmission mode, thus improving the system's robustness and data integrity under complex communication conditions.

[0177] Figure 2 This is a flowchart illustrating a multimodal data collaborative processing method for unmanned aerial vehicles (UAVs) according to some embodiments of this application. This method is executed by the UAV, specifically by an edge computing device mounted on the UAV. Figure 2 As shown, the method may include the following steps: step 201, step 202, step 203, step 204, step 205 and step 206.

[0178] In step 201, multimodal environmental data is collected using a multimodal sensor array.

[0179] For details on the specific implementation of step 201, please refer to the aforementioned description of the multimodal sensor group 101, which will not be repeated here.

[0180] In step 202, environmental data is received through the health monitoring module, the quality of the environmental data is assessed to detect whether there are any abnormalities in each sensor, and the health score corresponding to each sensor is output.

[0181] In some embodiments, such as Figure 3 As shown, step 202 above may specifically include the following steps: step 2021, step 2022, step 2023 and step 2024.

[0182] In step 2021, a camera health score is calculated on the camera data in the environmental data based on the brightness histogram and / or Laplacian variance of the image.

[0183] In step 2022, a lidar health score is calculated on the lidar data in the environmental data based on the point cloud density and / or the proportion of outliers.

[0184] In step 2023, a health score for the positioning module is calculated based on the horizontal accuracy factor of the Global Navigation Satellite System (GNSS) for the positioning data in the environmental data.

[0185] In step 2024, a health score report is generated.

[0186] For the specific implementation of step 202, please refer to the aforementioned description of the health monitoring module 102, which will not be repeated here.

[0187] In step 203, the mode switching module identifies the abnormality type based on the health score and switches from the current perception mode to the target perception mode among a set of preset multiple perception modes based on the abnormality type; wherein, the multiple perception modes use different sensors in the multimodal sensor group as the dominant perception source.

[0188] In some embodiments, such as Figure 4 As shown, step 203 can specifically include the following steps: step 2031, step 2032, step 2033, step 2034 and step 2035.

[0189] In step 2031, the abnormality type is identified based on the health score.

[0190] In step 2032, when the health score indicates that the lidar is abnormal but the camera is normal, the system switches to camera-dominated mode.

[0191] In step 2033, when the health score indicates that the camera is abnormal but the LiDAR is normal, the system switches to LiDAR-dominated mode.

[0192] In step 2034, when both the health score indicator camera and the LiDAR are abnormal, the system switches to position compensation mode.

[0193] In step 2035, when the health score indicates that the global navigation satellite system and inertial measurement unit are abnormal, but the camera and lidar are both normal, the system switches to camera-dominated mode or lidar-dominated mode, and requests position information and wireless ranging information from the cooperative processing node to correct the pose cumulative drift.

[0194] For the specific implementation of step 203, please refer to the aforementioned description of the mode switching module 103, which will not be repeated here.

[0195] In step 204, the environmental data is processed by the perception processing module in the target perception mode to generate perception results.

[0196] For the specific implementation of step 204, please refer to the aforementioned description of the perception processing module 104, which will not be repeated here.

[0197] In step 205, a confidence score is calculated on the perception result using the confidence assessment module.

[0198] For the specific implementation of step 205, please refer to the aforementioned description of the confidence assessment module 105, which will not be repeated here.

[0199] In step 206, when the confidence score is lower than the preset score threshold or the local computing power is insufficient, a collaborative request is sent to the collaborative processing node through the collaborative communication module and the returned data is received to assist in updating the perception results.

[0200] For the specific implementation of step 206, please refer to the aforementioned description of the collaborative communication module 106, which will not be repeated here.

[0201] As can be seen from the above embodiments, in this embodiment, the health monitoring module assesses the quality of environmental data and outputs a health score, enabling refined identification of abnormal states of various sensors; the mode switching module identifies the anomaly type based on the health score and switches to a target perception mode with different sensors as the primary sensing source, automatically selecting the appropriate perception link when sensors malfunction, ensuring the continuity of the perception task; the confidence assessment module calculates a confidence score on the perception results, quantifying the reliability of the perception results; and the collaborative communication module sends a collaborative request to the collaborative processing node and receives returned data when the confidence score is lower than a preset threshold or local computing power is insufficient, leveraging the computing and perception resources of external nodes to improve perception accuracy and system fault tolerance. Thus, the perception reliability, adaptability, and task continuity of the UAV in complex low-altitude environments are effectively improved.

[0202] Figure 5 This is a schematic diagram of the architecture of a drone cooperative system provided by some embodiments of this application, such as... Figure 5 As shown, the drone collaboration system 500 may include: a local drone 510, a neighboring drone 520, an aggregation node 530, and a ground control station 540.

[0203] In this embodiment of the application, the local drone 510 refers to the drone performing the current perception task, that is, the local node that initiates the cooperation request.

[0204] In this embodiment of the application, the neighboring UAV 520 refers to other aircraft in the same network as the local UAV 510, which can act as a collaborative processing node to respond to collaborative requests, perform collaborative reasoning, and return collaborative reasoning results.

[0205] In this embodiment, aggregation node 530 refers to a computing node with the ability to aggregate soft tags and distribute model distillation data, including a cluster head dynamically elected by a drone swarm or an edge base station deployed near the mission area.

[0206] In this embodiment, the ground control station 540 is responsible for historical data management, overall task scheduling, strategy adjustment, and final result feedback and display, and is the center of system monitoring and closed-loop control.

[0207] In this embodiment, the local UAV 510 is connected to the neighboring UAV 520 via low-latency wireless communication for transmitting cooperation requests, cooperative inference results, location information, and wireless ranging information. The local UAV 510 is connected to the aggregation node 530 via a wireless relay or backhaul link for reporting soft tags and receiving lightweight models. The local UAV 510 is connected to the ground control station 540 via a wireless backhaul link for reporting sensing results and receiving task scheduling instructions.

[0208] For a detailed description of the functions of the nearby drones, aggregation nodes, and ground control stations, please refer to the description of the collaborative processing nodes in the aforementioned related concepts section, which will not be repeated here.

[0209] Figure 6 This is a signaling interaction flowchart for a cross-device collaboration scenario provided by some embodiments of this application, such as... Figure 6 As shown, the signaling interaction process may include the following steps: step 601, step 602, step 603, step 604, step 605, step 606, step 607, step 608, step 609, step 610 and step 611.

[0210] In step 601, the local drone sends a resource request to the aggregation node.

[0211] In this embodiment of the application, when a local drone needs to initiate a collaboration request, it first sends a resource request to the aggregation node and queries the computing resource status of each neighboring drone in the current network.

[0212] In step 602, the aggregation node returns a resource scheduling instruction to the local drone.

[0213] In this embodiment, the aggregation node returns a resource scheduling instruction after comprehensively evaluating the computing resource status of each neighboring drone, indicating the neighboring drone with the best current computing resources.

[0214] In step 603, the local drone sends a cooperation request to neighboring drones according to the resource scheduling instructions.

[0215] In this embodiment of the application, the collaboration request includes the perception task information to be collaborated with and / or some features generated by the local model.

[0216] In step 604, the neighboring drones perform collaborative reasoning.

[0217] In this embodiment, the neighboring UAV performs collaborative reasoning using its own perception model based on the perception task information or partial features carried in the collaboration request.

[0218] In step 605, the neighboring drone returns the collaborative reasoning result to the local drone.

[0219] In step 606, the local drone updates the perception results based on the collaborative reasoning results.

[0220] In this embodiment of the application, the UAV updates the local perception results based on the collaborative reasoning results and recalculates the confidence score.

[0221] In step 607, the local drone reports the soft tag to the aggregation node.

[0222] In this embodiment, the soft label is a probability distribution generated by the local model, used to reflect the model's confidence in predicting different categories.

[0223] In step 608, the aggregation node aggregates soft tags to generate a teacher model.

[0224] In this embodiment, the aggregation node collects soft tags reported by multiple drones and generates a unified teacher model through an aggregation algorithm.

[0225] In step 609, the aggregation node distills the teacher model into a lightweight model.

[0226] In this embodiment of the application, distillation refers to transferring the knowledge of the teacher model to a lightweight model with fewer parameters and faster reasoning speed.

[0227] In step 610, the aggregation node sends the lightweight model back to the local drone.

[0228] In step 611, the local drone updates the parameters of the local model based on the lightweight model.

[0229] In this embodiment, parameter updates include two methods: weighted fusion or direct overwrite. For specific implementation methods, please refer to the description in the aforementioned federated distillation embodiment, which will not be repeated here.

[0230] For details on the specific implementation methods of resource application, collaboration request, soft tag reporting, teacher model generation and lightweight model return, please refer to the above description of the collaborative communication module and federated distillation mechanism, which will not be repeated here.

[0231] Figure 7 This is a structural block diagram of a drone provided in some embodiments of this application, such as... Figure 7 As shown, the drone 700 may include: a fuselage 701, a multimodal sensor group 702, and an edge computing device 703.

[0232] The fuselage 701 is used to carry the multimodal sensor group and edge computing device, providing a physical mounting platform and structural support for each component.

[0233] The multimodal sensor group 702, mounted on the fuselage 701, is used to collect multimodal environmental data. For the specific composition and function of the multimodal sensor group 702, please refer to the aforementioned description of the multimodal sensor group 101, which will not be repeated here.

[0234] An edge computing device 703 is disposed inside or mounted on the fuselage 701. The edge computing device 703 includes a processor and a memory. The memory stores a computer program. When the processor executes the computer program, it implements the UAV multimodal data collaborative processing method as described in the foregoing method embodiments.

[0235] In this embodiment, the edge computing device is the onboard computing core of the UAV, responsible for performing computing tasks such as sensor data processing, health monitoring, mode switching, confidence assessment, and collaborative communication. The edge computing device can be an embedded AI computing module, and the specific type can be selected according to the computing requirements of the task; this application does not limit this.

[0236] In this embodiment, the processor may be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices. The memory may be Read-Only Memory (ROM), Random Access Memory (RAM), Flash Memory, Hard Disk Drive (HDD), or Solid State Drive (SSD), etc.

[0237] Optionally, the edge computing device also includes a communication interface for data interaction with collaborative processing nodes, including sending collaboration requests, reporting soft tags, receiving lightweight models and collaborative inference results, etc. The communication interface supports wireless communication protocols such as Wi-Fi, 4G / 5G, Dedicated Short Range Communication (DSRC), or dedicated communication links for drones.

[0238] According to another embodiment of this application, this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps in the UAV multimodal data collaborative processing method as described in any of the above embodiments.

[0239] According to another embodiment of this application, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps in the UAV multimodal data collaborative processing method as described in any of the above embodiments.

[0240] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Those skilled in the art will understand that the embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, the embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0241] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0242] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. Although preferred embodiments of the embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0243] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0244] The above provides a detailed description of the UAV multimodal data collaborative processing system, method, and UAV provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal data collaborative processing system for unmanned aerial vehicles (UAVs), applied to UAVs, characterized in that, The system includes: A multimodal sensor array is used to collect multimodal environmental data. A health monitoring module is connected to the multimodal sensor group, used to receive the environmental data, perform quality assessment on the environmental data to detect whether there are any abnormalities in each sensor, and output the health score corresponding to each sensor. The mode switching module, connected to the health monitoring module, is used to identify the abnormality type based on the health score, and switch from the current perception mode to a target perception mode among a set of preset perception modes based on the abnormality type; wherein, the multiple perception modes use different sensors in the multimodal sensor group as the dominant perception source. A perception processing module, connected to the mode switching module, is used to process the environmental data in the target perception mode and generate perception results; The confidence assessment module, connected to the perception processing module, is used to calculate a confidence score on the perception result; The collaborative communication module, connected to the confidence assessment module, is used to send a collaborative request to the collaborative processing node and receive the returned data when the confidence score is lower than a preset score threshold or the local computing power is insufficient, so as to assist in updating the perception results.

2. The system according to claim 1, characterized in that, The preset multiple sensing modes include: camera-dominated mode, lidar-dominated mode, and position compensation mode; wherein, the camera-dominated mode uses the camera in the multimodal sensor group as the dominant sensing source; the lidar-dominated mode uses the lidar in the multimodal sensor group as the dominant sensing source; and the position compensation mode uses the global navigation satellite system and inertial measurement unit in the multimodal sensor group as the dominant sensing source. The mode switching module is specifically used for: When the health score indicates that the lidar is abnormal and the camera is normal, switch to the camera-dominated mode; When the health score indicates that the camera is abnormal and the lidar is normal, switch to the lidar-dominated mode; When the health score indicates that both the camera and the lidar are abnormal, switch to the position compensation mode; When the health score indicates that the global navigation satellite system and inertial measurement unit are abnormal, and the camera and the lidar are both normal, the system switches to the camera-dominated mode or the lidar-dominated mode. The cooperative communication module is also used to request location information and wireless ranging information from the cooperative processing node to correct the pose cumulative drift.

3. The system according to claim 1, characterized in that, The collaborative processing node includes at least one of the following: a neighboring UAV, an aggregation node, and a ground control station; wherein, the aggregation node is a computing node that has a single-hop or low-latency wireless connection with the UAV and has the ability to aggregate soft tags and distribute models distillation.

4. The system according to claim 3, characterized in that, The collaborative communication module is specifically used for: sending a collaborative request to the neighboring drone when the confidence score is lower than a preset score threshold or the local computing power is insufficient; receiving the collaborative reasoning result returned by the neighboring drone after performing collaborative reasoning in response to the collaborative request; wherein the collaborative request includes at least one of the following: perception task information to be collaborated on, and some features generated by the local model; The perception processing module is specifically used to: update the perception result based on the collaborative reasoning result; The confidence assessment module is further configured to: recalculate the confidence score after the perception processing module updates the perception result based on the collaborative reasoning result; if the recalculated confidence score is still lower than the preset score threshold, and the number of collaborative processing operations performed has not reached the preset number threshold, then send a collaborative request to the neighboring UAV again through the collaborative communication module; if the recalculated confidence score is still lower than the preset score threshold after collaborative processing operations have reached the preset number threshold, then report fault information.

5. The system according to claim 4, characterized in that, The collaborative communication module is further configured to: send a resource request to the aggregation node before sending a collaborative request to the neighboring UAVs; wherein the aggregation node is configured to generate resource scheduling instructions based on the computing resource status of each neighboring UAV; Receive the resource scheduling instruction returned by the aggregation node; wherein the resource scheduling instruction is used to indicate the nearest UAV with the best current computing resources; According to the resource scheduling instruction, the cooperation request is sent to the neighboring drone.

6. The system according to claim 3, characterized in that, The collaborative communication module is further configured to: report the soft tags generated by the local model to the aggregation node, so that the aggregation node can aggregate the soft tags to generate a teacher model and distill the teacher model into a lightweight model; and receive the lightweight model returned by the aggregation node. The perception processing module is further configured to: perform parameter update processing on the local model based on the lightweight model; The parameter update process includes: if the local model is a valid model, performing a weighted fusion of the lightweight model and the local model to obtain an updated local model; if the local model is an invalid model, covering the local model with the lightweight model.

7. The system according to claim 3, characterized in that, The collaborative communication module is further configured to: report the perception result to the ground control station when the confidence score is higher than the preset score threshold and the local computing resources meet the real-time processing requirements.

8. The system according to claim 1, characterized in that, The confidence assessment module is specifically used to: calculate a confidence score for the perception result based on the assessment factors associated with the perception result; The evaluation factors include at least one of the following: the prediction error of the perception results, environmental impact information, and historical regional data consistency information; the environmental impact information includes at least one of the following: light intensity, weather conditions, and degree of shading.

9. The system according to claim 1, characterized in that, The health monitoring module includes at least one of the following sub-modules: The image quality assessment submodule is used to calculate the camera health score based on the image's brightness histogram and / or Laplacian variance. The point cloud quality assessment submodule is used to calculate the LiDAR health score based on the point cloud density and / or the proportion of outliers. The positioning quality assessment submodule is used to calculate the positioning module health score based on the horizontal accuracy factor of the Global Navigation Satellite System. The communication quality assessment submodule is used to calculate the communication link health score based on the status parameters of the data transmission link.

10. The system according to claim 9, characterized in that, The collaborative communication module is further configured to: activate a local caching mechanism and store the data to be sent in the local cache when the communication link health score indicates that the communication link is abnormal; wherein, the communication link abnormality includes: the data transmission delay exceeds a preset delay threshold or the number of consecutive data transmission failures exceeds a preset number threshold; After the communication link is restored, the data in the local cache will be retransmitted; If the communication link fails to recover within a preset time period, it will switch to a low bit rate transmission mode.

11. The system according to claim 1, characterized in that, The collaborative communication module is further configured to: if no response is received within a preset time after sending the collaborative request, resend the collaborative request to the backup collaborative processing node; If the number of retries exceeds a preset threshold or the total collaboration latency exceeds a preset latency threshold, the process will be downgraded to local processing, and the corresponding perception result will be marked as non-collaborative. If the confidence score corresponding to the local processing is still lower than the preset score threshold, then a fault information is reported.

12. A method for collaborative processing of multimodal data from unmanned aerial vehicles (UAVs), applied to UAVs, characterized in that, The method includes: Multimodal environmental data is collected using a multimodal sensor array; The health monitoring module receives the environmental data, performs a quality assessment on the environmental data to detect whether there are any abnormalities in each sensor, and outputs the corresponding health score for each sensor. The mode switching module identifies the abnormality type based on the health score and switches from the current perception mode to the target perception mode among a set of preset perception modes based on the abnormality type; wherein, the multiple perception modes use different sensors in the multimodal sensor group as the dominant perception source. The perception processing module processes the environmental data under the target perception mode to generate perception results. A confidence score is calculated on the perceived results using a confidence assessment module. When the confidence score is lower than a preset score threshold or the local computing power is insufficient, a collaborative request is sent to the collaborative processing node through the collaborative communication module and the returned data is received to assist in updating the perception results.

13. A drone, characterized in that, include: body; A multimodal sensor array, mounted on the device body, is used to collect multimodal environmental data; An edge computing device is disposed inside the fuselage or mounted on the fuselage. The edge computing device includes a processor and a memory. The memory stores a computer program. When the processor executes the computer program, it implements the UAV multimodal data collaborative processing method as described in claim 12.