Unmanned aerial vehicle real-time mapping data fusion method and system of multi-modal edge computing

Abstract

The application discloses a multi-modal edge computing unmanned aerial vehicle real-time surveying and mapping data fusion method and system, comprising: modeling based on sensor physical characteristics, constructing a multi-source sensor dynamic time synchronization model; based on an edge computing device, through a spatial registration algorithm under kinematic constraints, dynamically calibrating the spatial coordinate system of the multi-source sensor; based on a multi-modal data fusion algorithm of adaptive weight distribution, acquiring sensor confidence and environment feature dynamic adjustment fusion weight; based on the edge computing device, performing real-time three-dimensional reconstruction, completing feature-level fusion of point cloud and image data at the unmanned aerial vehicle end; constructing a distributed edge cache mechanism, and performing data life cycle management based on an LRU algorithm.The application has the advantages that: the sensor dynamic synchronization model is innovatively constructed, the real-time three-dimensional reconstruction architecture based on the edge computing is constructed, and the spatiotemporal consistency of surveying and mapping data in a complex environment is significantly improved.

Description

Technical Field

[0001] This invention relates to data fusion technology, and in particular to a method and system for real-time mapping data fusion from unmanned aerial vehicles using multimodal edge computing. Background Technology

[0002] The background technology of UAV real-time mapping data fusion methods and systems mainly involves the combination of UAV technology, remote sensing mapping technology, and data processing technology. With the rapid development of UAV technology, UAVs have been widely used in fields such as geographic information systems, surveying and mapping, agriculture, and environmental monitoring, especially showing significant advantages in real-time data acquisition.

[0003] Current UAV real-time mapping data fusion methods primarily focus on the fusion and real-time processing of multi-source data. UAVs typically carry multiple sensors, such as inertial measurement units (IMUs), lidar (LiDAR), optical cameras, and multispectral cameras. The measurement data provided by these sensors have different characteristics and accuracies. Real-time data fusion technology usually employs algorithms such as Kalman filtering and particle filtering to combine data from different sensors to eliminate various errors and uncertainties. However, traditional UAV real-time mapping data fusion methods still have limitations in computing power and real-time processing capabilities, exhibiting slow data transmission latency and processing speed, making it difficult to cope with complex and dynamic environmental changes. Summary of the Invention

[0004] To improve existing UAV real-time mapping data fusion methods, this paper presents a multimodal edge computing-based UAV real-time mapping data fusion method and system. This method innovatively constructs a sensor dynamic synchronization model by performing real-time data processing on the UAV end and a real-time 3D reconstruction architecture based on edge computing, which significantly improves the spatiotemporal consistency of mapping data in complex environments.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A multimodal edge computing-based method for real-time UAV mapping data fusion includes:

[0007] Based on sensor physical characteristics modeling, a dynamic time synchronization model for multi-source sensors is constructed.

[0008] Based on edge computing devices, a spatial coordinate system of multi-source sensors is dynamically calibrated using a spatial registration algorithm under kinematic constraints.

[0009] A multimodal data fusion algorithm based on adaptive weight allocation is used to dynamically adjust the fusion weights based on sensor confidence and environmental features.

[0010] Real-time 3D reconstruction based on edge computing devices, and feature-level fusion of point cloud and image data at the UAV terminal;

[0011] Build a distributed edge caching mechanism and manage the data lifecycle based on the LRU algorithm.

[0012] Preferably, the step of constructing a multi-source sensor dynamic time synchronization model based on sensor physical characteristics specifically includes:

[0013] Based on the sampling delay characteristics of the inertial measurement unit (IMU), lidar (LiDAR), optical camera, and GNSS module, a timestamp compensation function is established, with the following formula:

[0014]

[0015] Where, τ i α is the inherent delay coefficient of the sensor. i β is the ambient temperature compensation factor. i v is the power supply voltage correction factor. drone Real-time flight speed of the drone;

[0016] Based on the compensated timestamp data from each sensor, the clock offset is recursively estimated at the sensing and network layers using a bidirectional timestamp exchange protocol, as shown in the formula:

[0017]

[0018] Where T1 to T4 are PTP protocol timestamps, and ∈ is the network jitter compensation term, which is corrected by the second derivative of the sensor clock drift rate.

[0019] Preferably, the dynamic calibration of the spatial coordinate system of the multi-source sensors based on the edge computing device, using a spatial registration algorithm under kinematic constraints, specifically includes:

[0020] Data acquired from multiple sensors is transmitted to edge computing devices within the drone.

[0021] Based on the coordinate transformation chain of the UAV's six-DOF pose, the IMU angular velocity ω and acceleration α are used as constraints to construct the registration error function, which is expressed as follows:

[0022]

[0023] Where T is the sensor coordinate transformation matrix, R(ω) k ) is the rotation matrix driven by angular velocity, δα k This is an acceleration compensation term;

[0024] Construct a nonlinear optimization model based on Lie group theory;

[0025] Based on the nonlinear optimization model, the spatial registration problem under kinematic constraints is transformed into least squares optimization on the SE(3) manifold, as shown in the following formula:

[0026]

[0027] Where, ξ i For Lie algebra parameters, Let be the covariance matrix, exp(·) be the exponential mapping, and log(·) be the logarithmic mapping;

[0028] The above steps are iterated until the UAV spatial coordinate error converges.

[0029] Preferably, the multimodal data fusion algorithm based on adaptive weight allocation, which dynamically adjusts the fusion weights based on sensor confidence and environmental features, specifically includes:

[0030] The confidence level c of each sensor is obtained based on historical data, error statistics, environmental conditions, and signal strength factors. i ;

[0031] Based on the confidence level c of each sensor i The functional expression for environmental features with respect to sensor weights is obtained, and the formula is:

[0032]

[0033] Among them, w i (e) represents the sensor weights, and c i (e) represents the confidence level of sensor i under given environmental characteristics;

[0034] Based on the acquired sensor confidence levels and environmental characteristics, the measurement error values ​​for each time period are calculated. The fusion weights are dynamically adjusted using an adaptive adjustment based on particle filtering. If the weight of sensor i in the previous time step was... The formula for updating the weights at the current time is:

[0035]

[0036] in, Let λ be the measurement error of sensor i at time t, and λ be the adjustment parameter.

[0037] Preferably, the real-time 3D reconstruction based on edge computing devices, and the feature-level fusion of point cloud and image data at the UAV end, specifically includes:

[0038] A 3D point cloud model is generated based on multi-view images and point cloud data acquired by sensors.

[0039] A 3D mesh is constructed based on the generated 3D point cloud, and a polygonal mesh is generated to represent the scene surface.

[0040] The FPGA-based edge computing device maps the above-mentioned 3D reconstruction algorithm to the hardware acceleration module of the FPGA to accelerate the key calculations in the 3D mesh generation process.

[0041] Based on the real-time flight altitude of the UAV obtained by the altimeter, the grid resolution is dynamically adjusted. At low altitudes, the grid accuracy is increased to capture details, while at high altitudes, the grid resolution is reduced to decrease the amount of computation.

[0042] A convolutional neural network based on point cloud-image joint feature extraction outputs a fused point cloud with semantic labels. The mesh structure is defined as follows:

[0043] F(P,I)=Conv3D(GraphConv(P)⊕Conv2D(I))

[0044] Where P represents point cloud data, I represents image data, Conv3D represents a 3D convolutional layer, GraphConv(P) represents a kd-tree-based graph convolution operation with scope point cloud data P, Conv2D(I) represents a 2D convolutional layer with scope image data I, and ⊕ represents a feature stitching operation.

[0045] Preferably, the construction of the distributed edge caching mechanism, based on the LRU algorithm for data lifecycle management, specifically includes:

[0046] The drone swarm, ground edge server, and regional edge cloud form a three-level caching architecture;

[0047] The mapping area is divided into three-dimensional spatial grids, and each grid is associated with a unique ID.

[0048] Data blocks are mapped to edge nodes based on the consistent hashing algorithm, and each edge node maintains a doubly linked list and a hash table structure.

[0049] Based on the data lifecycle, expired data is automatically cleaned up, and data that is not expired but has the lowest weight is eliminated. When the local cache is full, low-value data is migrated to the upper-level node first rather than being deleted directly.

[0050] Furthermore, a multimodal edge computing-based UAV real-time mapping data fusion system is proposed, including:

[0051] Multi-source sensor dynamic time synchronization model module: The multi-source sensor dynamic time synchronization model module is mainly used to compensate for timestamps and recursively estimate the clock offset of each sensor;

[0052] Spatial coordinate calibration module: The spatial coordinate calibration module is mainly used to perform registration calculations on the spatial coordinates of the UAV through edge computing devices;

[0053] Fusion weight adjustment module: The fusion weight adjustment module is mainly used to dynamically adjust the fusion weight of multi-modal data based on various influencing factors;

[0054] 3D modeling module: The 3D modeling module is mainly used for real-time 3D modeling of the survey area of ​​the UAV;

[0055] Fusion module: The fusion module is mainly used to perform feature fusion on point cloud data and image data;

[0056] Management Module: The management module is mainly used to manage data according to the data lifecycle based on a distributed edge caching mechanism;

[0057] Processor: The processor is mainly used for the calculation process of each formula and the construction calculation process of each model.

[0058] Compared with the prior art, the advantages of the present invention are:

[0059] By modeling based on sensor physical characteristics and using a dynamic time synchronization model, the time synchronization problem between different sensors is solved, ensuring the spatiotemporal correlation of multi-source data. Simultaneously, a spatial registration algorithm under kinematic constraints accurately calibrates the spatial coordinate systems of different sensors, reducing errors caused by dynamic environmental changes. An adaptive weight allocation multimodal data fusion algorithm dynamically adjusts the fusion weights of data from each sensor, making the fusion results more adaptable to mapping needs in different environments, improving data accuracy and reliability. Real-time 3D reconstruction and feature-level fusion are achieved through edge computing devices, significantly reducing data processing latency and improving processing efficiency. This method enables efficient, accurate, and low-latency real-time data processing in complex environments, ensuring the spatiotemporal consistency of UAV mapping operations and providing strong technical support for multiple application fields. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the UAV real-time mapping data fusion method based on multimodal edge computing proposed in this invention;

[0061] Figure 2 This is a schematic diagram illustrating the construction of the multi-source sensor dynamic time synchronization model proposed in this invention;

[0062] Figure 3 This is a schematic diagram of the dynamic calibration of spatial coordinates proposed in this invention;

[0063] Figure 4 This is a schematic diagram of the fusion weight adjustment proposed in this invention;

[0064] Figure 5 This is a schematic diagram of the feature-level fusion proposed in this invention;

[0065] Figure 6 This is a schematic diagram of the data management proposed in this invention;

[0066] Figure 7 This is an architecture diagram of the electronic devices in this solution;

[0067] Figure 8 This is a schematic diagram of the computer-readable storage medium structure in this scheme. Detailed Implementation

[0068] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0069] A multimodal edge computing-based UAV real-time mapping data fusion system includes:

[0070] Multi-source sensor dynamic time synchronization model module: The multi-source sensor dynamic time synchronization model module is mainly used to compensate for timestamps and recursively estimate the clock offset of each sensor;

[0071] Spatial coordinate calibration module: The spatial coordinate calibration module is mainly used to perform registration calculations on the spatial coordinates of the UAV through edge computing devices;

[0072] Fusion weight adjustment module: The fusion weight adjustment module is mainly used to dynamically adjust the fusion weight of multi-modal data based on various influencing factors;

[0073] 3D modeling module: The 3D modeling module is mainly used for real-time 3D modeling of the survey area of ​​the UAV;

[0074] Fusion module: The fusion module is mainly used to perform feature fusion on point cloud data and image data;

[0075] Management Module: The management module is mainly used to manage data according to the data lifecycle based on a distributed edge caching mechanism;

[0076] Processor: The processor is mainly used for the calculation process of each formula and the construction calculation process of each model.

[0077] See Figure 1 As shown, a multimodal edge computing-based UAV real-time mapping data fusion method includes:

[0078] Step 1: Based on the physical characteristics of the sensors, construct a dynamic time synchronization model for multiple sensor sources;

[0079] Step 2: Based on edge computing devices, dynamically calibrate the spatial coordinate system of multi-source sensors using a spatial registration algorithm under kinematic constraints;

[0080] Step 3: Based on the multimodal data fusion algorithm with adaptive weight allocation, obtain sensor confidence and environmental features to dynamically adjust the fusion weights;

[0081] Step 4: Perform real-time 3D reconstruction based on edge computing devices, and complete feature-level fusion of point cloud and image data on the UAV.

[0082] Step 5: Build a distributed edge caching mechanism and manage the data lifecycle based on the LRU algorithm.

[0083] See Figure 2 As shown, the construction of a multi-source sensor dynamic time synchronization model based on sensor physical characteristics specifically includes:

[0084] Based on the sampling delay characteristics of the inertial measurement unit (IMU), lidar (LiDAR), optical camera, and GNSS module, a timestamp compensation function is established, with the following formula:

[0085]

[0086] Where, τ i α is the inherent delay coefficient of the sensor. i β is the ambient temperature compensation factor. i v is the power supply voltage correction factor. drone Real-time flight speed of the drone;

[0087] Based on the compensated timestamp data from each sensor, the clock offset is recursively estimated at the sensing and network layers using a bidirectional timestamp exchange protocol, as shown in the formula:

[0088]

[0089] Where T1 to T4 are PTP protocol timestamps, and ∈ is the network jitter compensation term, which is corrected by the second derivative of the sensor clock drift rate.

[0090] Specifically, when performing multi-sensor clock synchronization, in addition to timestamp compensation and clock offset estimation, the impact of sensor dynamic characteristics and environmental factors on synchronization accuracy must also be considered. For example, IMUs and LiDARs may produce errors under high acceleration or complex environments, and network fluctuations and data transmission delays can also affect synchronization accuracy. Besides conventional algorithms, dynamic adjustments should be made based on sensor status information to ensure the system has adaptive capabilities, corrects deviations in real time, and improves synchronization stability during long-term operation, thereby achieving accurate data fusion and positioning.

[0091] See Figure 3As shown, based on edge computing devices, the dynamic calibration of the spatial coordinate system of multi-source sensors using a spatial registration algorithm under kinematic constraints specifically includes:

[0092] Data acquired from multiple sensors is transmitted to edge computing devices within the drone.

[0093] Based on the coordinate transformation chain of the UAV's six-DOF pose, the IMU angular velocity ω and acceleration α are used as constraints to construct the registration error function, which is expressed as follows:

[0094]

[0095] Where T is the sensor coordinate transformation matrix, R(ω) k ) is the rotation matrix driven by angular velocity, δα k This is an acceleration compensation term;

[0096] Construct a nonlinear optimization model based on Lie group theory;

[0097] Based on the nonlinear optimization model, the spatial registration problem under kinematic constraints is transformed into least squares optimization on the SE(3) manifold, as shown in the following formula:

[0098]

[0099] Where, ξ i For Lie algebra parameters, Let be the covariance matrix, exp(·) be the exponential mapping, and log(·) be the logarithmic mapping;

[0100] The above steps are iterated until the UAV spatial coordinate error converges.

[0101] Understandably, least squares optimization based on the SE(3) manifold typically requires significant computational resources, especially in high-dimensional spaces and with large-scale data, which can lead to excessive computational overhead and impact real-time performance. Incremental optimization methods can be employed, performing local optimization only at each time step to reduce computational load, or efficient numerical optimization algorithms can be used to improve the convergence speed of the optimization process.

[0102] See Figure 4 As shown, the multimodal data fusion algorithm based on adaptive weight allocation obtains sensor confidence and environmental features and dynamically adjusts the fusion weights, specifically including:

[0103] The confidence level c of each sensor is obtained based on historical data, error statistics, environmental conditions, and signal strength factors. i ;

[0104] Based on the confidence level c of each sensor iThe functional expression for environmental features with respect to sensor weights is obtained, and the formula is:

[0105]

[0106] Among them, w i (e) represents the sensor weights, and c i (e) represents the confidence level of sensor i under given environmental characteristics;

[0107] Based on the acquired sensor confidence levels and environmental characteristics, the measurement error values ​​for each time period are calculated. The fusion weights are dynamically adjusted using an adaptive adjustment based on particle filtering. If the weight of sensor i in the previous time step was... The formula for updating the weights at the current time is:

[0108]

[0109] in, Let λ be the measurement error of sensor i at time t, and λ be the adjustment parameter.

[0110] Specifically, by evaluating historical data, real-time status, and environmental conditions of the sensors, an error model is established to quantify the measurement error of each sensor under specific conditions. The particle filtering method effectively handles these nonlinear errors and dynamically adjusts the data fusion weights based on the sensor confidence level. As the environment changes, the system can adaptively adjust the weights according to real-time error feedback, allowing high-confidence sensors to dominate data fusion while suppressing low-confidence sensors, effectively reducing error accumulation and ensuring high accuracy and reliability of the multi-sensor fusion system during long-term operation.

[0111] See Figure 5 As shown, real-time 3D reconstruction based on edge computing devices, specifically including feature-level fusion of point cloud and image data at the UAV end, includes:

[0112] A 3D point cloud model is generated based on multi-view images and point cloud data acquired by sensors.

[0113] A 3D mesh is constructed based on the generated 3D point cloud, and a polygonal mesh is generated to represent the scene surface.

[0114] The FPGA-based edge computing device maps the above-mentioned 3D reconstruction algorithm to the hardware acceleration module of the FPGA to accelerate the key calculations in the 3D mesh generation process.

[0115] Based on the real-time flight altitude of the UAV obtained by the altimeter, the grid resolution is dynamically adjusted. At low altitudes, the grid accuracy is increased to capture details, while at high altitudes, the grid resolution is reduced to decrease the amount of computation.

[0116] A convolutional neural network based on point cloud-image joint feature extraction outputs a fused point cloud with semantic labels. The mesh structure is defined as follows:

[0117] F(P,I)=Conv3D(GraphConv(P)⊕Conv2D(I))

[0118] Where P represents point cloud data, I represents image data, Conv3D represents a 3D convolutional layer, GraphConv(P) represents a kd-tree-based graph convolution operation with scope point cloud data P, Conv2D(I) represents a 2D convolutional layer with scope image data I, and ⊕ represents a feature stitching operation.

[0119] Understandably, the computational and storage requirements increase dramatically during 3D mesh generation, especially at high resolutions, potentially leading to computational latency and storage bottlenecks, particularly in real-time processing and large-scale scenarios. By mapping the mesh generation process to an FPGA hardware acceleration module, the parallel processing capabilities of the FPGA are leveraged to accelerate critical computations, such as point cloud-to-mesh conversion and triangulation algorithms. Furthermore, a hierarchical meshing technique is used to dynamically adjust mesh accuracy based on the complexity of different scenarios.

[0120] See Figure 6 As shown, the construction of a distributed edge caching mechanism, based on the LRU algorithm for data lifecycle management, specifically includes:

[0121] The drone swarm, ground edge server, and regional edge cloud form a three-level caching architecture;

[0122] The mapping area is divided into three-dimensional spatial grids, and each grid is associated with a unique ID.

[0123] Data blocks are mapped to edge nodes based on the consistent hashing algorithm, and each edge node maintains a doubly linked list and a hash table structure.

[0124] Based on the data lifecycle, expired data is automatically cleaned up, and data that is not expired but has the lowest weight is eliminated. When the local cache is full, low-value data is migrated to the upper-level node first rather than being deleted directly.

[0125] Furthermore, the method according to the embodiments of this application can also be achieved by means of... Figure 7 The architecture of the electronic device shown is used to implement this. For example... Figure 7As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store the multimodal edge computing UAV real-time mapping data fusion method and system provided in this application. The electronic device 500 may also include a terminal interface 508. Of course, Figure 7 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 7 One or more components in the illustrated electronic device.

[0126] Figure 8 This is a schematic diagram of a computer-readable storage medium structure provided in one embodiment of this application. Figure 8 The diagram illustrates a computer-readable storage medium 600 according to one embodiment of this application. The computer-readable storage medium 600 stores computer-readable instructions. When executed by a processor, the computer-readable instructions can perform the UAV real-time mapping data fusion method and system for multimodal edge computing according to an embodiment of this application, as described above with reference to the accompanying drawings. The storage medium 600 includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0127] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0128] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-modal edge computing unmanned aerial vehicle real-time mapping data fusion method, characterized in that, include: Based on the sampling delay characteristics of the inertial measurement unit, lidar, optical camera, and GNSS module, a timestamp compensation function is established, with the following formula: wherein, is a sensor inherent delay coefficient, is an ambient temperature compensation factor, is a power supply voltage correction coefficient, is a real-time flight speed of the UAV; Based on the compensated timestamp data from each sensor, the clock offset is recursively estimated at the sensing and network layers using a bidirectional timestamp exchange protocol, as shown in the formula: wherein, to is a PTP protocol timestamp, is a network jitter compensation term, corrected by the second derivative of the sensor clock drift rate; Based on edge computing devices, a spatial coordinate system of multi-source sensors is dynamically calibrated using a spatial registration algorithm under kinematic constraints. A multimodal data fusion algorithm based on adaptive weight allocation is used to dynamically adjust the fusion weights based on sensor confidence and environmental features. Real-time 3D reconstruction based on edge computing devices, and feature-level fusion of point cloud and image data at the UAV terminal; Build a distributed edge caching mechanism and manage the data lifecycle based on the LRU algorithm.

2. The multi-modal edge computing drone real-time mapping data fusion method of claim 1, wherein, The dynamic calibration of the spatial coordinate system of multi-source sensors based on the edge computing device, using a spatial registration algorithm under kinematic constraints, specifically includes: Data acquired from multiple sensors is transmitted to edge computing devices within the drone. Based on the coordinate transformation chain of the UAV's six-DOF pose, the IMU angular velocity and acceleration As a constraint, a registration error function is constructed, with the following formula: in, The sensor coordinate transformation matrix, A rotation matrix driven by angular velocity. This is an acceleration compensation term; Construct a nonlinear optimization model based on Lie group theory; Based on the nonlinear optimization model, the spatial registration problem under kinematic constraints is transformed into least squares optimization on the SE(3) manifold, as shown in the following formula: wherein, is a Lie algebra parameter, is a covariance matrix, is an exponential map, is a logarithm map; The above steps are iterated until the UAV spatial coordinate error converges. 3.The multi-modal edge computing drone real-time mapping data fusion method of claim 1, wherein, The multimodal data fusion algorithm based on adaptive weight allocation, which obtains sensor confidence and environmental features and dynamically adjusts the fusion weights, specifically includes: obtaining a confidence level of each sensor based on historical data of each sensor, error statistics, environmental conditions, and signal strength factors ; Based on the confidence of each sensor , a function expression of the environmental feature with respect to the sensor weight is obtained, and the formula is: wherein, is the sensor weight, is the confidence of sensor i in a given environmental feature; Based on the acquired sensor confidence and environmental characteristics, the measurement error value in each time period is calculated, the fusion weight is dynamically adjusted based on the adaptive adjustment of the particle filter, and if the weight of the sensor i at the last time is The weight update formula at the current time is: wherein, is the measurement error of sensor i at time instant t, is the tuning parameter.

4. The UAV real-time mapping data fusion method based on multimodal edge computing according to claim 1, characterized in that, The real-time 3D reconstruction based on edge computing devices, which involves feature-level fusion of point cloud and image data on the UAV, specifically includes: A 3D point cloud model is generated based on multi-view images and point cloud data acquired by sensors. A 3D mesh is constructed based on the generated 3D point cloud, and a polygonal mesh is generated to represent the scene surface. The FPGA-based edge computing device maps the above-mentioned 3D reconstruction algorithm to the hardware acceleration module of the FPGA to accelerate the key calculations in the 3D mesh generation process. Based on the real-time flight altitude of the UAV obtained by the altimeter, the grid resolution is dynamically adjusted. At low altitudes, the grid accuracy is increased to capture details, while at high altitudes, the grid resolution is reduced to decrease the amount of computation. A convolutional neural network based on point cloud-image joint feature extraction outputs a fused point cloud with semantic labels. The mesh structure is defined as follows: Where P represents point cloud data and I represents image data. It is a three-dimensional convolutional layer. This is a graph convolution operation based on a kd-tree, with the scope being the point cloud data P. It is a two-dimensional convolutional layer, with the scope being image data I. This is a feature splicing operation.

5. The multi-modal edge computing drone real-time mapping data fusion method of claim 1, wherein, The construction of the distributed edge caching mechanism, based on the LRU algorithm for data lifecycle management, specifically includes: The drone swarm, ground edge server, and regional edge cloud form a three-level caching architecture; The mapping area is divided into three-dimensional spatial grids, and each grid is associated with a unique ID. Data blocks are mapped to edge nodes based on the consistent hashing algorithm, and each edge node maintains a doubly linked list and a hash table structure. Based on the data lifecycle, expired data is automatically cleaned up, and data that is not expired but has the lowest weight is eliminated. When the local cache is full, low-value data is migrated to the upper-level node first rather than being deleted directly.

6. The unmanned aerial vehicle real-time mapping data fusion system combined with a multi-modal edge computing, for implementing the multi-modal edge computing unmanned aerial vehicle real-time mapping data fusion method according to any one of claims 1-5, characterized in that, include: Multi-source sensor dynamic time synchronization model module: The multi-source sensor dynamic time synchronization model module is used to compensate for timestamps and recursively estimate the clock offset of each sensor; Spatial coordinate calibration module: The spatial coordinate calibration module is used to perform registration calculations on the spatial coordinates of the UAV through edge computing devices; Fusion weight adjustment module: The fusion weight adjustment module is used to dynamically adjust the fusion weight of multi-modal data based on various influencing factors; 3D modeling module: The 3D modeling module is used to perform real-time 3D modeling of the survey area of ​​the UAV; Fusion module: The fusion module is used to perform feature fusion on point cloud data and image data; Management Module: The management module is used to manage data based on a distributed edge caching mechanism and according to the data lifecycle; Processor: The processor is mainly used for the calculation process of each formula and the construction calculation process of each model.

7. An electronic device, comprising: include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the UAV real-time mapping data fusion method for multimodal edge computing as described in any one of claims 1-5.

8. A computer-readable storage medium storing computer-readable instructions, the computer-readable instructions comprising: When the computer-readable instructions are executed by the processor, they implement the UAV real-time mapping data fusion method for multimodal edge computing as described in any one of claims 1-5.

Images