Method, device, equipment and medium for cooperative perception between unmanned aerial vehicles

Abstract

The invention relates to the technical field of unmanned aerial vehicle cluster cooperative control, and discloses a method, a device, equipment and a medium for cooperative sensing between unmanned aerial vehicles, and the method comprises the steps: obtaining the multi-channel heterogeneous sensor data of a target unmanned aerial vehicle, and carrying out the time synchronization of the multi-channel heterogeneous sensor data, and obtaining the first sensor data with the aligned time sequence; based on the first sensor data, generating a structured description of an environment where the target unmanned aerial vehicle is located, and compressing the structured description to obtain compressed sensor data; the compressed sensor data are transmitted to other unmanned aerial vehicles, and the structured data aligned in time sequence can ensure that all the unmanned aerial vehicles make decisions based on the same time reference and environment model, so that collaborative errors caused by data time difference or format difference are avoided, and collaborative consistency is ensured; moreover, the lightweight data transmission delay is lower, the dynamic environment change can be quickly responded, and the real-time cooperative task of the unmanned aerial vehicle cluster is supported.

Description

Technical Field

[0001] This application relates to the field of collaborative control technology for unmanned aerial vehicle (UAV) swarms, specifically to methods, devices, equipment, and media for collaborative perception among UAVs. Background Technology

[0002] In related technologies, the collaborative control of drone swarms often adopts a mode of one sensor per drone and single-point backhaul. By equipping each drone with an independent sensor for environmental perception, each drone transmits the raw data it collects back to the central control node. Control commands are then processed and issued by the central node, forming a centralized control architecture. The perception ranges of multiple drones in the swarm overlap, resulting in the collection of a large amount of duplicate information, which may lead to bandwidth waste. Summary of the Invention

[0003] This application provides a method, apparatus, device, and medium for collaborative sensing among unmanned aerial vehicles (UAVs) to address the problem that overlapping sensing ranges of multiple UAVs in a cluster lead to the collection of a large amount of duplicate information, which may result in bandwidth waste.

[0004] In a first aspect, this application provides a method for collaborative perception among unmanned aerial vehicles (UAVs), the method comprising: acquiring multi-channel heterogeneous sensor data of a target UAV; performing time synchronization on the multi-channel heterogeneous sensor data to obtain time-aligned first sensor data; generating a structured description of the environment in which the target UAV is located based on the first sensor data; compressing the structured description to obtain compressed sensor data; and transmitting the compressed sensor data to other UAVs.

[0005] In one optional implementation, the step of generating a structured description of the environment in which the target UAV is located based on the first sensor data, and compressing the structured description to obtain compressed sensor data, includes: converting the first sensor data into a polar coordinate grid to generate a three-dimensional polar coordinate voxel matrix; determining the probability of the presence of obstacles in the target voxels; performing binarization processing on the target voxels based on the result of comparing the probability with a preset probability threshold to obtain a binarization result of the target voxels; sparsifying the binarization result to obtain a sparsed voxel matrix; and compressing the sparsed voxel matrix to obtain the compressed sensor data.

[0006] In one optional implementation, acquiring multi-channel heterogeneous sensor data of the target UAV and performing time synchronization on the multi-channel heterogeneous sensor data to obtain time-aligned first sensor data includes: marking the multi-channel heterogeneous sensor data with initial timing values ​​when acquiring the multi-channel heterogeneous sensor data; acquiring a timing reference value; calculating a timing deviation based on the initial timing value and the timing reference value; summing the initial timing value and the timing deviation to obtain the time corresponding to the corrected timing reference value of the multi-channel heterogeneous sensor data; and determining the time-aligned first sensor data based on the time.

[0007] In one optional implementation, after sparsifying the binarization result to obtain a sparsified voxel matrix, the method further includes: traversing the sparsified voxel matrix and encoding consecutively repeated values ​​into sequence pairs of values ​​and lengths, wherein the values ​​represent the values ​​of voxels in the sparsified voxel matrix, and the lengths represent the number of repetitions of the values; and compressing the sequence pairs to obtain the compressed sensor data.

[0008] In one optional implementation, transmitting the compressed sensor data to other drones includes: dividing time into periodic frames based on a frame period, dividing the periodic frames into multiple time slots, wherein the multiple time slots do not overlap; determining a first drone corresponding to a target time slot for data transmission; transmitting the compressed sensor data of the first drone to the other drones based on the first drone in the target time slot; and transmitting the compressed sensor data of the other drones based on the first drone in other time slots outside the target time slot.

[0009] In an optional implementation, the method further includes acquiring the multi-channel heterogeneous sensor data based on a field-programmable gate array in the target UAV.

[0010] In one optional implementation, the method further includes: registering multiple heterogeneous sensors to the target UAV, determining sensor identification information and device handles; tagging sensor data based on the sensor identification information; and acquiring the sensor data based on the device handles.

[0011] Secondly, this application provides an apparatus for collaborative perception among unmanned aerial vehicles (UAVs), the apparatus comprising: a synchronization module for acquiring multi-channel heterogeneous sensor data of a target UAV, and performing time synchronization on the multi-channel heterogeneous sensor data to obtain time-aligned first sensor data; a compression module for generating a structured description of the environment in which the target UAV is located based on the first sensor data, and compressing the structured description to obtain compressed sensor data; and a transmission module for transmitting the compressed sensor data to other UAVs.

[0012] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the method for collaborative perception between unmanned aerial vehicles as described in the first aspect or any corresponding embodiment.

[0013] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method for inter-UAV cooperative perception described in the first aspect or any corresponding embodiment.

[0014] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the method for inter-UAV cooperative perception described in the first aspect or any corresponding embodiment.

[0015] The method, apparatus, device, and medium for collaborative perception among UAVs provided in this embodiment can compensate for the blind spots of a single sensor and improve the comprehensiveness of perception by fusing multi-channel heterogeneous sensor data and sharing cluster information, thus building a more complete environmental understanding. At the same time, the reduced data volume of the compressed sensor data can adapt to the bandwidth limitations of the radio frequency link between UAVs, optimize bandwidth utilization, and support large-scale cluster collaboration. In addition, the time-aligned structured data can ensure that all UAVs make decisions based on the same time reference and environmental model, avoiding collaboration errors caused by data time differences or format differences, and ensuring collaboration consistency. Furthermore, the lightweight data transmission has lower latency, enabling rapid response to dynamic environmental changes and supporting real-time collaborative tasks of UAV clusters. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A flowchart of a method for cooperative perception between unmanned aerial vehicles (UAVs) according to an embodiment of this application is shown; Figure 2 A schematic diagram of the count value of the FPGA-based global counter at the rising edge of the corresponding PPS emergency pulse is shown. Figure 3 This illustration shows a schematic diagram of data transmission between drone clusters according to an embodiment of this application; Figure 4 A schematic diagram of the architecture for airborne collaborative control and communication of a drone swarm in an embodiment of this application is shown; Figure 5 A schematic diagram of the structure of the device for collaborative sensing between unmanned aerial vehicles provided in an embodiment of this application is shown; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0020] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0021] The one-sensor-per-machine and single-point feedback modes in related technologies still suffer from significant differences in the output frequencies and interface protocols of different sensors, which cannot be directly parsed by airborne flight controllers. For example, there are increasing differences between Controller Area Network with Flexible Data Rate (CAN-FD), Universal Asynchronous Receiver / Transmitter (UART), Ethernet protocol, and Serial Peripheral Interface (SPI) protocols. In addition, centralized ground station feedback of all raw data can lead to limitations in cluster size and poor real-time performance.

[0022] According to an embodiment of this application, a method embodiment for collaborative perception among unmanned aerial vehicles is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0023] This embodiment provides a method for collaborative perception among unmanned aerial vehicles (UAVs), which can be used in mobile terminals, such as onboard collaborative controllers for UAVs, UAV swarms, or robot swarms. Figure 1 A flowchart of a method for cooperative perception between unmanned aerial vehicles (UAVs) according to an embodiment of this application is shown, such as... Figure 1 As shown, the process includes the following steps: Step S101: Acquire multi-channel heterogeneous sensor data of the target UAV, synchronize the multi-channel heterogeneous sensor data in time, and obtain the first sensor data with time alignment.

[0024] In this step, the "multi-channel heterogeneous sensor" refers to the multi-channel data collected by multiple sensors on the target UAV. These sensors can be lidar sensors, millimeter-wave radar sensors, visual inertial measurement units (IMUs), ultra-wideband (UWB) sensors, and real-time kinematic (RTK) sensors, etc. "Heterogeneous" indicates differences in sensor types and data characteristics. Time synchronization is used to bind the sampling data from different sensors to the same time base through a Precision Time Protocol (PTP), Pulses Per Second (PPS), and Data Distribution Service TimeSync (DDS TimeSync) mechanism, eliminating timing misalignments caused by local clock deviations. The first sensor represents the fused data from all the heterogeneous sensors precisely aligned on the time axis after time synchronization, ensuring that environmental information at the same moment can be correlated and matched.

[0025] PTP and PPS can be used in combination. When using a high-precision timekeeping device (Satellite GPS clock, or GPS) based on satellite navigation system signals, the GPS receiver outputs a high-precision time reference via PPS as the master clock source for PTP. The UAV can achieve sub-microsecond synchronization via the PTP protocol. Then, the system time is calibrated using the PPS signal provided by GPS. By combining PTP and PPS, a time accuracy of less than 1... Clock synchronization.

[0026] Specifically, raw data from various types of sensors on the target UAV can be collected, and the timing deviations of different sensors can be calibrated through time synchronization technology to obtain the first sensor data with consistent timing, so as to solve the problem that data on the same environmental event cannot be correlated due to asynchronous sampling of heterogeneous sensors.

[0027] Step S102: Based on the first sensor data, generate a structured description of the environment in which the target UAV is located, compress the structured description, and obtain compressed sensor data.

[0028] In this step, the structured description is used to characterize the transformation of unstructured raw sensor data into an environment model with a fixed format, such as a 3D polar coordinate voxel matrix or a raster map, allowing the device to directly parse environmental features. The compressed sensor data is used to characterize the structured environment description after processing such as sparsification and encoding, which can significantly reduce data volume while retaining key information such as obstacle and target locations.

[0029] Specifically, based on the time-aligned first sensor data, a structured model that can accurately describe the environment is constructed, and then redundant information is removed through a compression algorithm to obtain compressed sensor data.

[0030] Step S103: Transmit the compressed sensor data to other drones.

[0031] In this step, compressed sensor data can be transmitted to other drones in the cluster via inter-drone communication links, such as optimized link state routing (OLSR), to achieve environmental information sharing and support multi-drone collaborative perception.

[0032] The method for collaborative perception among UAVs provided in this embodiment can compensate for the blind spots of a single sensor and improve the comprehensiveness of perception by fusing multi-channel heterogeneous sensor data and sharing cluster information, thus building a more complete environmental understanding. At the same time, the reduced data volume of the compressed sensor data can adapt to the bandwidth limitations of the radio frequency link between UAVs, optimize bandwidth utilization, and support large-scale cluster collaboration. In addition, the time-aligned structured data can ensure that all UAVs make decisions based on the same time reference and environmental model, avoiding collaboration errors caused by data time differences or format differences, and ensuring collaboration consistency. Furthermore, the lightweight data transmission has lower latency and can quickly respond to dynamic environmental changes, supporting real-time collaborative tasks of UAV clusters.

[0033] In some optional implementations, based on the first sensor data, a structured description of the environment in which the target UAV is located is generated, and the structured description is compressed to obtain compressed sensor data, including: converting the first sensor data into a polar coordinate grid to generate a three-dimensional polar coordinate voxel matrix; determining the probability of the presence of obstacles in the target voxels, and performing binarization processing on the target voxels based on the result of comparing the probability with a preset probability threshold to obtain the binarization processing result of the target voxels; sparsifying the binarization processing result to obtain a sparsified voxel matrix; and compressing the sparsified voxel matrix to obtain compressed sensor data.

[0034] In this embodiment, a polar coordinate grid is used to represent a three-dimensional spatial grid divided by polar coordinates (distance, azimuth, and altitude) with the UAV as the origin. A three-dimensional polar coordinate voxel matrix is ​​used to represent the three-dimensional space discretized into voxels according to the polar coordinate grid. Each voxel corresponds to an element in the matrix, and the matrix value records information such as whether the voxel contains obstacles and environmental features, providing a structured and quantitative representation of the environment. A preset probability threshold is used to represent a manually set probability threshold value used to determine voxel attributes. For example, a value greater than the preset probability threshold indicates the presence of an obstacle in the current voxel, while a value less than or equal to the preset probability threshold indicates the current voxel is idle. Binarization is used to convert the obstacle presence probability value of a voxel into 0 or 1 binary values, with 0 representing idle voxels and 1 representing obstacle voxels, simplifying the data representation dimension. Sparsity is used to remove voxels with 0 values ​​from the voxel matrix, retaining key voxels to form a sparse matrix containing only valid information. The sparsed voxel matrix represents the location and value of non-zero voxels, eliminating redundant idle data from the matrix structure.

[0035] Specifically, the time-aligned first sensor data can be mapped onto a polar coordinate space grid centered on the UAV, discretizing the continuous 3D environment into voxel units. Each voxel corresponds to a matrix element, thus initially constructing a structured voxel model of the environment. Bayesian probability updates can be used to calculate the probability of an obstacle being present in each target voxel. Then, a preset probability threshold is used to binarize the probability values ​​to either 0 or 1, distinguishing between idle voxels and obstacle voxels, simplifying the representation of environmental features. Furthermore, by retaining obstacle voxels and their location information and eliminating redundant data, the voxel matrix is ​​transformed from dense full data to coefficient-effective data. Finally, at least one of Delta encoding or the LZ4 algorithm can be used to compress the sparse voxel matrix, obtaining compressed sensor data.

[0036] In this way, by using a three-dimensional polar coordinate voxel matrix, the raw data from multiple heterogeneous sensors can be transformed into a machine-analyzable structured environment model, which can improve the accuracy of modeling. At the same time, by binarizing the probability and threshold, the environmental characteristics can be clearly defined and the data complexity can be reduced. In addition, by removing redundant and idle data through sparsification and further reducing the volume through compression, the amount of data can be minimized while retaining obstacle information.

[0037] In some optional implementations, acquiring multi-channel heterogeneous sensor data from the target UAV and performing time synchronization on the multi-channel heterogeneous sensor data to obtain time-aligned first sensor data includes: marking initial timing values ​​for the multi-channel heterogeneous sensor data when acquiring multi-channel heterogeneous sensor data; acquiring a timing reference value; calculating a timing deviation based on the initial timing value and the timing reference value; summing the initial timing value and the timing deviation to obtain the time corresponding to the corrected timing reference value of the multi-channel heterogeneous sensor data; and determining the time-aligned first sensor data based on the time.

[0038] In this embodiment, the timing initial value (ue_usec) characterizes the instantaneous reading of the FPGA global counter when the current sensor frame data is marked by the Field Programmable Gate Array (FPGA). It can be obtained from the sensor frame header timestamp and is a 64-bit unsigned integer, measured in FPGA clock ticks. The timing reference value (t_pps) characterizes the instantaneous reading of the FPGA global counter when the most recent rising edge of the PPS is captured by the microcontroller. It can be captured by the timer input in the hardware microcontroller and is a 64-bit unsigned integer, measured in FPGA clock ticks. The microcontroller can be an STM32, and the timer can be a TIM2.

[0039] The timing offset is a correction value used to characterize the time deviation of the current sensor frame relative to the global PPS time reference. It can be calculated and output by software, and the unit is 1 / 2 kilometres. Timing offset can be calculated using the following formula:

[0040] in, The compensation coefficient, used to characterize the coefficient, can be obtained from least squares linear regression and is dimensionless. If , then it indicates no drift.

[0041] The time corresponding to the corrected timing reference value of the multi-channel heterogeneous sensor data is the sum of the initial timing value and the timing deviation. ).

[0042] Figure 2 This diagram illustrates the count value of the FPGA-based global counter at the rising edge of the corresponding PPS emergency pulse. Figure 2 As shown, the FPGA can continuously increment the count at a 48MHz clock. When the rising edge of PPS arrives, the FPGA immediately records the current count value, forming a... Figure 2Timestamp markers such as 0x1234, 0x5678, and 0x90AB are used. The PPS pulse serves as the anchor point for time synchronization. The FPGA records the instantaneous count value of each PPS rising edge using a 48MHz counter, achieving precise binding between the PPS event and the FPGA's local clock. For example... Figure 2 In the table, 0x1234 represents the FPGA counter value at the rising edge of PPS at time t0; 0x5678 represents the FPGA counter value at the rising edge of PPS at time t1; and 0x90AB represents the FPGA counter value at the rising edge of PPS at time t2. These count values ​​are captured by the STM32 in hardware and used to calculate the offset between the FPGA's local clock and the system reference time. Using the offset calculation formula, the FPGA's count values ​​can be converted into time offsets to achieve time synchronization of all sensor data.

[0043] In this way, through explicit deviation calculation and compensation, the timing misalignment problem caused by differences in local clock precision and sampling trigger delays of multiple heterogeneous sensors can be solved, ensuring that multi-source data corresponding to the same actual environmental event are bound to the same time stamp, achieving accurate timing unification. At the same time, after timing alignment, data from different types of sensors can be accurately associated with the same environmental state, providing a spatiotemporally consistent data foundation for subsequent structured environmental description and avoiding fusion errors caused by time differences. In addition, calibration is completed through the steps of marking, calculation and summation, without the need for complex hardware modifications, adapting to the limited computing power of UAV onboard systems and easy to implement in engineering.

[0044] In some optional implementations, after sparsifying the binarization result to obtain a sparsified voxel matrix, the method further includes: traversing the sparsified voxel matrix and encoding consecutively repeated values ​​into sequence pairs of values ​​and lengths, wherein the values ​​represent the values ​​of voxels in the sparsified voxel matrix, and the lengths represent the number of repeated values; and compressing the sequence pairs to obtain compressed sensor data.

[0045] In this embodiment, the sparsed voxel matrix is ​​traversed to identify consecutively repeating numerical segments. Each consecutively repeating segment is encoded as a sequence pair of (value, length). For example, six consecutive 1s are encoded as (1, 6). Replacing lengthy consecutive numerical sequences with values ​​and repetition counts reduces redundancy in data representation. The generated (value, length) sequence pairs can be further compressed to obtain compressed sensor data.

[0046] Specifically, the data from the multi-channel heterogeneous sensors are shown in Table 1.

[0047] Table 1. Data from multiple heterogeneous sensors

[0048] FPGA-based timestamp alignment of multi-channel heterogeneous sensor data can be performed, with an alignment precision configurable to 1. The first sensor data is obtained. In a three-dimensional polar grid, radial, azimuth, and height are used to represent the three dimensions that divide the three-dimensional space. Radial represents the distance dimension along the ray direction with the UAV itself as the origin, with a configurable range of 40 meters. If each grid is divided into 0.2-meter sections, there are a total of 200 grids. Azimuth represents the angular dimension on the horizontal plane, determining the orientation in the horizontal direction. Its range is configurable from 0 degrees to 360 degrees, with each degree divided into 360 grids. Height represents the vertical dimension, distinguishing different height levels. Its range is configurable from -20 meters to +20 meters, with each grid being divided into 0.4-meter sections, for a total of 100 grids. The generated three-dimensional polar voxel matrix can output 360 × 200 × 100 = 7.2M voxels, with an original size of approximately 28.8 MB (float), which can be reduced to 9 kB after bit compression.

[0049] By using the drone itself as the origin and dividing the space with rotational symmetry, we can reduce the number of empty cells in the Cartesian grid at a distance. With the drone spinning, the grid index remains unchanged, which can reduce the computational cost of subsequent Bayesian updates.

[0050] For each voxel in the three-dimensional polar coordinate voxel matrix, the probability (P) of the obstacle's existence can be calculated. This probability can be converted into a logarithmic form of log odds (L), which can be calculated using the following formula:

[0051] Updated log odds It can be calculated using the following formula:

[0052] in, Used to characterize the logarithmic odds of the previous time step. The weights are used to characterize sensor weights. Different sensors have different reliability, and therefore different weights. For example, the weight of a laser sensor... The weighting of the millimeter-wave sensor can be configured to 1.0. The weight of the visual sensor can be configured to 0.7. It can be configured to 0.5. Used to characterize the hit determination of observations If the condition is true, the voxel is detected by the sensor, identified as an obstacle, and then... ; If the result is false, the voxel is not hit and is considered idle. . The logarithmic probability increment for a single observation is used to characterize the probability increment of a single observation. 0.9 is the likelihood probability of a hit, i.e., the confidence level of considering it as an obstacle, and 0.1 is the likelihood probability of a miss, i.e., the confidence level of considering it as an idle object. The logarithm is converted into an increment value for iterative updates.

[0053] This allows for the fusion of probabilities from multiple sensors, rather than simple superposition, through... By retaining information from the previous moment, there is no need to maintain long historical frame data. The probability of obstacle presence changes slowly when the drone is shaking, avoiding instantaneous false alarms and improving the robustness of this solution when the drone is rotating or shaking.

[0054] The preset probability threshold can be configured to 0.6. If the probability of an obstacle exists is greater than the preset probability threshold, it can be determined that there is an obstacle in the target voxel, and the target voxel is configured to 1. If the probability of an obstacle exists is less than or equal to the preset probability threshold, it can be determined that there is no obstacle in the target voxel, and the target voxel is configured to 0.

[0055] In this way, subsequent decisions can be made based on the probability of obstacles existing in the target voxel, which can provide obstacle boundaries for subsequent path planning. This can make bit compression 360×200×100bit≈9kB, which is convenient for UAVs to transmit data between each other in 50ms. At the same time, the probability of obstacles existing in the target voxel corresponds one-to-one with the polar coordinate index, so no coordinate transformation is required.

[0056] The binarization result of the three-dimensional polar coordinate voxel matrix The binary obstacle map is processed as follows:

[0057] in, and Binarization results used to characterize the three-dimensional polar coordinate voxel matrix Elements in adjacent positions in the same row, Used to characterize the XOR operation.

[0058] In this way, by sparsifying the binarization result, the obstacle edges can be made continuous, and most of the data in the resulting sparse voxel matrix is ​​0. At the same time, the average non-zero elements in the binarization result are less than 15%, which can provide a coding basis for subsequent run-length encoding (RLE). In addition, the XOR gate can be completed within one minute. If the binarization result includes 360 rows, each row is 200 bits, the FPGA can run 360 parallel paths.

[0059] A sparse voxel matrix, using RLE encoding, yields a sequence of (value, length) pairs. For example, for a sparse voxel matrix 000011111100, RLE encoding produces a sequence of (0, 4), (1, 6), and (0, 2) pairs. Further sparsity can be achieved, averaging 1.2 kB / frame for a total of 360 rows. Simultaneously, a fixed-length buffer with a maximum run length of 255 can be used. If the length of consecutive identical values ​​exceeds 255, it can be split into multiple run-length pairs, facilitating UDP packet segmentation. Furthermore, for FPGAs, a single clock cycle outputs a (value, length) pair sequence, achieving zero latency.

[0060] Delta and LZ4 compression can be used to compress raw data from 2 MByte / s to 200 kByte / s. Delta and LZ4 are a combination of two data compression techniques. Delta encoding achieves compression by storing only the differences between data points instead of the original values. LZ4 encoding eliminates metadata redundancy through dictionary encoding. Using them in conjunction with Delta encoding can further reduce storage space requirements. The resulting compressed sensor data has a dictionary size of 64 kB and a level of 4.

[0061] This eliminates common headers and fields, such as device identifiers and timestamps, which are mapped by the dictionary, further reducing the data volume by 20% to 30%. Simultaneously, it provides delimited output with a hard upper limit of 1kB; exceeding this limit automatically reduces the raster resolution, for example, from 0.2 meters to 0.4 meters, without requiring recompression of historical frames. Furthermore, the compressed 1kB of sensor data can be 100% transmitted within a 50ms time-division multiple access time slot, achieving a communication link transmission rate of up to 250kbit / s, which is ideal for data transmission between drones.

[0062] In this way, after timestamp alignment, the 28.8MB original three-dimensional polar coordinate voxel matrix data is compressed to ≤1kB / frame through a six-stage pipeline of three-dimensional polar coordinate voxels, Bayesian probabilistic fusion, binarization, Delta sparsity, RLE and LZ4, with a compression rate of ≥90% and a central processing unit (CPU) usage of <5%, which can lay a hard real-time foundation for 50ms low-bandwidth inter-machine distribution.

[0063] In some optional implementations, transmitting compressed sensor data to other drones includes: dividing time into periodic frames based on frame period, dividing the periodic frames into multiple time slots, wherein the multiple time slots do not overlap; determining a first drone for data transmission corresponding to a target time slot; transmitting the compressed sensor data of the first drone to other drones based on the first drone in the target time slot; and transmitting the compressed sensor data of other drones based on the first drone in other time slots outside the target time slot.

[0064] In this implementation, a strategy combining OLSR and Time Division Multiple Access (TDMA) can be employed to ensure that compressed sensor data reaches all neighboring devices within 50ms. TDMA enables multiple users to share the same frequency band resources through time division. Time can be divided into periodic frames, and each frame can be further divided into non-overlapping time slots. Users send or receive data sequentially according to their assigned time slots. A 5.8GHz Wi-Fi mesh can be built in the UAV based on OLSR.

[0065] Figure 3 A schematic diagram illustrating data transmission between drone clusters according to an embodiment of this application is shown. Figure 3 As shown, time can be divided into periodic frames based on a frame period, such as 50ms, and each period can be further divided into multiple time slots. Figure 3 The diagram illustrates dividing periodic frames into five time slots: time slot 0, time slot 1, time slot 2, time slot 3, and time slot 4. Time slot 0 is allocated to UAV #0 for transmitting compressed sensor data, time slot 1 is allocated to UAV #1 for transmitting compressed sensor data, time slot 2 is allocated to UAV #2 for transmitting compressed sensor data, time slot 3 is allocated to UAV #3 for transmitting compressed sensor data, and time slot 4 is used for relaying or acknowledgment (ACK), retransmission / packet loss compensation, i.e., handling data forwarding, acknowledging reception status, or retransmitting lost frames.

[0066] The physical layer can adopt the high-speed wireless communication protocol 5.8 GHz 802.11ac standard, the routing layer can adopt the OLSR protocol, and the MAC layer can adopt TDMA. The UAV prioritizes sending compressed sensor data in its own time slot, and uses other time slots to relay compressed sensor data from neighboring UAVs. This limits the number of compressed sensor data transmissions to less than 3 hops, ensuring wide-range information propagation while avoiding excessive latency caused by multiple hops.

[0067] In this way, by adopting OLSR and TDMA strategies at the network layer, with OLSR responsible for multi-hop routing and TDMA responsible for collision avoidance, it is possible to achieve 3-hop full network arrival within 50ms without the need for a centralized ground server.

[0068] In some alternative implementations, the aforementioned method for collaborative sensing among unmanned aerial vehicles (UAVs) further includes acquiring multi-channel heterogeneous sensor data in the target UAV based on a field-programmable gate array (FPGA).

[0069] In this implementation, an STM32H7 and FPGA coprocessing architecture can be used to connect multiple sensor interfaces on the FPGA side, such as Controller Area Network with Flexible Data Rate (CAN-FD), Low-Voltage Differential Signaling (LVDS), Serial Peripheral Interface (SPI)-IMU, Universal Asynchronous Receiver / Transmitter (UART), and Ethernet. Specifically, CAN-FD has a data segment speed of 8 Mbps and a control segment speed of 1 Mbps; LVDS has a point-to-point speed of 800 Mbit / s and can be used for forwarding via the Mobile Industry Processor Interface (MIPI); the SPI-IMU has a 20MHz SCLK, Direct Memory Access (DMA) double buffering, and zero-copy functionality; and the UART and Ethernet can use standard interfaces.

[0070] In this way, by using an FPGA to simultaneously provide hardware real-time interfaces to devices with multiple electrical standards, data formats, and clock domains on a single silicon chip, and placing the FPGA at the forefront of the sensor, the problems of time uniformity, electrical adaptation, and parallel processing can be solved in one go. The microprocessor's 32-bit TIM counter has a rollover of only 2^32 / 48MHz≈89s, which cannot meet data requirements; the FPGA can build a 64-bit counter with a rollover of approximately 12000 years at 48MHz, ensuring no data overflow.

[0071] Compared to related technologies that use PPS and PTP to trigger MCU branch interpolation, this application implements FPGA 64-bit global tick domain linear regression drift compensation at the hardware level. This enables sensor frames from heterogeneous interfaces such as CAN-FD, LVDS, SPI, UART, and Ethernet to be hard aligned on the same sub-microsecond time base without interruption, back alignment, or floating-point operations. This lays the foundation for a hard real-time clock for subsequent 50ms compressed distribution between UAVs.

[0072] In some optional implementations, the aforementioned method for collaborative perception among unmanned aerial vehicles (UAVs) further includes registering multiple heterogeneous sensors to the target UAV, determining sensor identification information and device handles; tagging sensor data based on sensor identification information; and acquiring sensor data based on device handles.

[0073] In this implementation, a device handle (rt_device) can be registered for each sensor by writing a device tree. The device tree is a tree-like data structure describing the hardware configuration. Device parameters and connection relationships can be defined through nodes and attributes, achieving decoupling between the operating system and the hardware. A sensor_parser_node is written for CAN-FD frames, LVDS video, and SPI-IMU respectively. The sensor_parser_node represents the zero-copy parsing node written for each type of sensor.

[0074] Specifically, during registration, a unique name (such as "sensor1") and device type (such as RT_Device_Class_Miscellaneous) must be specified, and a unified operation interface (such as read, write, and control) must be implemented through the device handle (rt_device). Registration example: rt_device_register(&lidar_dev, "lidar0", RT_Device_Class_Miscellaneous).

[0075] In this way, through the device tree and the pluggable architecture of rt_device at the driver layer, a unified device handle can be registered for various heterogeneous sensors, enabling the modification of new sensor hardware components and plug-and-play functionality.

[0076] At the application layer, a local topic called ObstacleMap can be published for direct use by the PX4 avoidance module; at the same time, it can be transmitted back to the ground station (QGC), where the QGC can display the compressed 3D point cloud contour in real time.

[0077] Figure 4 This illustration shows a schematic diagram of the architecture for airborne collaborative control and communication of a drone swarm, as shown in an embodiment of this application. Figure 4As shown, the airborne collaborative controller 401 can use an STM32H7 and an FPGA. Its interface layers, CAN-FD 402, UART 403, and Ethernet 404, interface with different sensor data 400. After FPGA timestamp synchronization and PPS synchronization, compressed sensor data 405 is obtained. The compressed sensor data is then encapsulated using MAVLink 406 and uses a 5.8GHz Wi-Fi Mesh self-organizing network 407. Multi-hop communication is achieved by combining the OLSR routing protocol and TDMA 408. The UAV #N collaborative node 410 can receive and relay collaborative data from the initiating node UAV #2 collaborative node 409, enabling environmental information sharing among UAVs. The ground station 411 can be used for real-time monitoring of UAV status and receiving cluster collaborative data.

[0078] In this way, the compressed sensor data is published in the airborne operating system as a local topic on ObstacleMap. The PX4 avoidance module obtains obstacle voxels by subscribing to this topic with zero copy, realizing real-time obstacle avoidance without relying on ground links, and the data latency between UAVs is <1ms.

[0079] The following example, using a swarm of 20 drones for tunnel emergency rescue, illustrates the method for collaborative perception among drones provided in this application. In an environment where a 1:1 tunnel CAD file is imported into UE5 with a length of 1.2 km, and 20 drones are used, each carrying a laser sensor, gas sensor, and IMU sensor, the original total bandwidth is 2.1 MB / s. After FPGA timestamp alignment, a 360×200 polar coordinate grid is generated. After compression, each frame is 0.85 kB, with an outgoing bandwidth of 170 kB / s (20 drones × 0.85 kB × 10 Hz). Mesh-TDMA has a 50ms cycle, resulting in an average inter-drone latency of 38ms. The ground station displays 3D obstacle contours in real time, with a virtual-to-real error of 0.18m. This method can save 90% of bandwidth, supports simultaneous operation of 20 or more drones, and achieves an end-to-end latency of 38ms, which is less than the 50ms end-to-end latency.

[0080] This embodiment also provides a device for collaborative perception among unmanned aerial vehicles (UAVs), which implements the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0081] This embodiment provides a device for collaborative sensing among unmanned aerial vehicles (UAVs). Figure 5 A schematic diagram of the structure of the device for collaborative sensing between unmanned aerial vehicles provided in an embodiment of this application is shown, as follows: Figure 5 As shown, it includes: The synchronization module 501 is used to acquire multi-channel heterogeneous sensor data of the target UAV, synchronize the multi-channel heterogeneous sensor data in time, and obtain the first sensor data with time alignment.

[0082] The compression module 502 is used to generate a structured description of the environment in which the target UAV is located based on the first sensor data, and to compress the structured description to obtain compressed sensor data.

[0083] The transmitting module 503 is used to transmit compressed sensor data to other drones.

[0084] In some alternative implementations, the compression module 502 includes: The first unit of the compression module is used to convert the first sensor data into a polar coordinate grid to generate a three-dimensional polar coordinate voxel matrix; determine the probability of the existence of obstacles in the target voxel; based on the comparison between the probability and a preset probability threshold, perform binarization processing on the target voxel to obtain the binarization processing result of the target voxel; perform sparsification on the binarization processing result to obtain a sparsified voxel matrix; and compress the sparsified voxel matrix to obtain the compressed sensor data.

[0085] In some alternative implementations, the synchronization module 501 includes: The first unit of the synchronization module is used to mark the timing initial value for the multi-channel heterogeneous sensor data when acquiring multi-channel heterogeneous sensor data; acquire the timing reference value; calculate the timing deviation based on the timing initial value and the timing reference value; sum the timing initial value and the timing deviation to obtain the time corresponding to the timing reference value after the multi-channel heterogeneous sensor data is corrected; and determine the first sensor data for timing alignment based on the time.

[0086] In some alternative embodiments, the aforementioned apparatus for collaborative sensing among unmanned aerial vehicles further includes: The encoding module is used to traverse the sparse voxel matrix after the binarization result is sparsified to obtain a sparse voxel matrix, and encode the continuous repeating values ​​into sequence pairs of values ​​and lengths. The values ​​represent the values ​​of voxels in the sparse voxel matrix, and the lengths represent the number of repetitions of the values. The sequence pairs are compressed to obtain compressed sensor data.

[0087] In some alternative implementations, the sending module 503 includes: The first unit of the transmission module is used to divide time into periodic frames based on the frame period, divide the periodic frames into multiple time slots, wherein the multiple time slots do not overlap; determine the first UAV corresponding to the target time slot for data transmission; in the target time slot, transmit the compressed sensor data of the first UAV to other UAVs based on the first UAV; in other time slots outside the target time slot, transmit the compressed sensor data of other UAVs based on the first UAV.

[0088] In some alternative embodiments, the aforementioned apparatus for collaborative sensing among unmanned aerial vehicles further includes: The acquisition unit is used to acquire multiple heterogeneous sensor data from the target UAV based on a field-programmable gate array.

[0089] In some alternative embodiments, the aforementioned apparatus for collaborative sensing among unmanned aerial vehicles further includes: The labeling unit is used to register multiple heterogeneous sensors to the target UAV, determine sensor identification information and device handles; label sensor data based on sensor identification information; and acquire sensor data based on device handles.

[0090] The method and apparatus for collaborative perception among UAVs provided in this application complete multi-sensor data parsing, time synchronization, fusion and compression locally on the collaborative controller, and distribute it to the inter-UAV link and ground station through a custom MAVLink extended message, achieving high-frequency, low-bandwidth, and low-latency cluster collaborative perception; the outgoing bandwidth of a single UAV is reduced by 90%, supporting a cluster of ≥20 UAVs; the data latency between UAVs is <80ms, meeting the requirements for high-speed obstacle avoidance; through a unified interface, when adding a new sensor, only the device tree and parser need to be modified, without changing the hardware.

[0091] Through heterogeneous sensor hard timestamp unification (<1μs), 90% compression, and Mesh-TDMA 50ms full network delivery; with a single-unit outbound bandwidth ≤200kB / s, it can still support large-scale clusters of ≥20 drones; adding new sensors only requires modification to the device tree and resolver, without requiring changes to hardware or flight control firmware. Verified in a tunnel rescue field, the inter-drone latency is ≤50ms, the virtual-to-real error is <0.3m, and bandwidth is reduced by 90%, making it widely applicable to collaborative sensing scenarios for drone clusters in emergency firefighting, low-altitude logistics, and other similar applications.

[0092] The apparatus for collaborative perception among unmanned aerial vehicles (UAVs) provided in this application can execute the method for collaborative perception among UAVs provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0093] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0094] The following is a detailed reference. Figure 6 This diagram illustrates a suitable structural schematic for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0095] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0096] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 609, or installed from memory 608, or installed from ROM 602. When the computer program is executed by processor 601, it performs the functions defined in the method for inter-UAV cooperative perception according to embodiments of this application.

[0097] Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0098] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the method for collaborative perception between unmanned aerial vehicles shown in the above embodiments is implemented.

[0099] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0100] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for collaborative sensing among unmanned aerial vehicles (UAVs), characterized in that, The method includes: Acquire multi-channel heterogeneous sensor data of the target UAV, and synchronize the multi-channel heterogeneous sensor data in time to obtain time-aligned first sensor data; Based on the first sensor data, a structured description of the environment in which the target UAV is located is generated, and the structured description is compressed to obtain compressed sensor data; The compressed sensor data is transmitted to other drones.

2. The method according to claim 1, characterized in that, The step of generating a structured description of the environment in which the target UAV is located based on the first sensor data, and compressing the structured description to obtain compressed sensor data includes: The data from the first sensor is converted into a polar coordinate grid to generate a three-dimensional polar coordinate voxel matrix. Determine the probability of the presence of an obstacle in the target voxel, and based on the comparison between the probability and a preset probability threshold, perform binarization processing on the target voxel to obtain the binarization processing result of the target voxel; The binarization result is then sparsified to obtain a sparsified voxel matrix. The sparsed voxel matrix is ​​compressed to obtain the compressed sensor data.

3. The method according to claim 1, characterized in that, The process of acquiring multi-channel heterogeneous sensor data from the target UAV and synchronizing the multi-channel heterogeneous sensor data to obtain time-aligned first sensor data includes: When acquiring the multi-channel heterogeneous sensor data, initial timing values ​​are marked for the multi-channel heterogeneous sensor data; Obtain the timing reference value, and calculate the timing deviation based on the initial timing value and the timing reference value; The initial timing value and the timing deviation are summed to obtain the time corresponding to the timing reference value after the multi-channel heterogeneous sensor data is corrected. Based on the time, the first sensor data of the timing alignment is determined.

4. The method according to claim 2, characterized in that, After sparsifying the binarization result to obtain a sparsified voxel matrix, the process further includes: Traverse the sparsed voxel matrix and encode consecutively repeated values ​​into a sequence of values ​​and lengths, wherein the values ​​represent the values ​​of voxels in the sparsed voxel matrix and the lengths represent the number of repetitions of the values. The sequence pairs are compressed to obtain the compressed sensor data.

5. The method according to claim 1, characterized in that, The step of transmitting the compressed sensor data to other drones includes: The time is divided into periodic frames based on the frame period, and the periodic frames are further divided into multiple time slots, wherein the multiple time slots do not overlap with each other. Identify the first UAV that will transmit data corresponding to the target time slot; During the target time slot, based on the first UAV, the compressed sensor data of the first UAV is transmitted to the other UAVs; In other time slots outside the target time slot, based on the first UAV, compressed sensor data from the other UAVs is transmitted.

6. The method according to claim 1, characterized in that, The method further includes acquiring the multi-channel heterogeneous sensor data based on a field-programmable gate array in the target UAV.

7. The method according to claim 1, characterized in that, The method further includes: Register multiple heterogeneous sensors to the target UAV and determine the sensor identification information and device handle; Based on the sensor identification information, the sensor data is labeled; The sensor data is obtained based on the device handle.

8. A device for collaborative sensing among unmanned aerial vehicles (UAVs), characterized in that, The device includes: The synchronization module is used to acquire multi-channel heterogeneous sensor data of the target UAV, synchronize the multi-channel heterogeneous sensor data in time, and obtain time-aligned first sensor data. The compression module is used to generate a structured description of the environment in which the target UAV is located based on the first sensor data, and to compress the structured description to obtain compressed sensor data. The transmitting module is used to transmit the compressed sensor data to other drones.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the method for collaborative perception between unmanned aerial vehicles as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method for collaborative perception between unmanned aerial vehicles as described in any one of claims 1 to 7.

Images