Unmanned aerial vehicle real-time surveying and mapping data fusion method and system based on multi-modal edge calculation

Abstract

The invention discloses an unmanned aerial vehicle real-time surveying and mapping data fusion method and system based on multi-modal edge calculation, and the method comprises the steps: carrying out the modeling based on the physical characteristics of a sensor, and constructing a multi-source sensor dynamic time synchronization model; dynamically calibrating a space coordinate system of the multi-source sensor through a space registration algorithm under kinematics constraint on the basis of edge computing equipment; based on a multi-modal data fusion algorithm of adaptive weight distribution, obtaining a sensor confidence coefficient and an environment characteristic dynamic adjustment fusion weight; real-time three-dimensional reconstruction is carried out based on an edge computing device, and feature level fusion of point cloud and image data is completed at an unmanned aerial vehicle end; and constructing a distributed edge caching mechanism, and performing data life cycle management based on an LRU algorithm. The method has the advantages that the dynamic synchronization model of the sensor is innovatively constructed, the real-time three-dimensional reconstruction architecture is based on edge calculation, and the space-time consistency of surveying and mapping data in a complex environment is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to data fusion technology, in particular to a multi-modal edge computing unmanned aerial vehicle real-time mapping data fusion method and system. BACKGROUND

[0002] The background technology of the unmanned aerial vehicle real-time mapping data fusion method and system mainly involves the combination of unmanned aerial vehicle technology, remote sensing mapping technology and data processing technology. With the rapid development of unmanned aerial vehicle technology, unmanned aerial vehicles have been widely used in geographic information systems, mapping, agriculture, environmental monitoring and other fields, especially in real-time data acquisition.

[0003] The current unmanned aerial vehicle real-time mapping data fusion method on the market mainly focuses on the fusion of multi-source data and real-time processing technology. Unmanned aerial vehicles usually carry multiple sensors, such as inertial measurement units (IMU), laser radars (LiDAR), optical cameras and multispectral cameras, etc. The measurement data provided by these sensors has different characteristics and accuracy. Real-time data fusion technology usually uses Kalman filtering particle filtering algorithm to combine data from different sensors to eliminate various errors and uncertainties. However, the traditional unmanned aerial vehicle real-time mapping data fusion method still has problems in computing power and real-time processing capability. Data transmission delay and processing speed are slow, which is difficult to cope with complex and dynamic environmental changes. SUMMARY

[0004] In order to improve the existing unmanned aerial vehicle real-time mapping data fusion method, a multi-modal edge computing unmanned aerial vehicle real-time mapping data fusion method and system is provided. This method innovatively constructs a sensor dynamic synchronization model through real-time data processing on the unmanned aerial vehicle side, based on the real-time three-dimensional reconstruction architecture of edge computing, which significantly improves the spatio-temporal consistency of mapping data in complex environments.

[0005] To achieve the above purpose, the technical solution adopted by the present application is:

[0006] The multi-modal edge computing unmanned aerial vehicle real-time mapping data fusion method comprises:

[0007] Based on the modeling of sensor physical characteristics, a multi-source sensor dynamic time synchronization model is constructed;

[0008] Based on the edge computing device, the spatial coordinate system of the multi-source sensor is dynamically calibrated through the spatial registration algorithm under kinematic constraints;

[0009] Based on the multi-modal data fusion algorithm with adaptive weight distribution, the sensor confidence and environmental feature dynamic adjustment fusion weight are obtained;

[0010] Based on the edge computing device, real-time three-dimensional reconstruction is performed, and feature-level fusion of point cloud and image data is completed on the unmanned aerial vehicle side.

[0011] Build a distributed edge caching mechanism and manage data lifecycle based on the LRU algorithm.

[0012] Preferably, the construction of a multi-source sensor dynamic time synchronization model based on sensor physical characteristics modeling specifically includes:

[0013] Based on the sampling delay characteristics of the inertial measurement unit (IMU), laser radar (LiDAR), optical camera, and GNSS module, a timestamp compensation function is established. The formula is:

[0014]

[0015] Among them, τ i is the sensor’s inherent delay coefficient, α i is the ambient temperature compensation factor, β i is the supply voltage correction factor, v drone The real-time flight speed of the drone;

[0016] Based on the compensated timestamp data of each sensor, the clock offset is recursively estimated at the sensor and network layers through a two-way timestamp exchange protocol. The formula is:

[0017]

[0018] Among them, T1 to T4 are PTP protocol timestamps, and ∈ is the network jitter compensation term, which is corrected by the second derivative of the sensor clock drift rate.

[0019] Preferably, the dynamic calibration of the spatial coordinate system of the multi-source sensor based on the edge computing device through the spatial registration algorithm under kinematic constraints specifically includes:

[0020] Based on the data obtained by multi-source sensors, it is transmitted to the edge computing device in the drone;

[0021] Based on the coordinate transformation chain of the UAV's six-degree-of-freedom posture, the IMU angular velocity ω and acceleration α are used as constraints to construct the registration error function. The formula is:

[0022]

[0023] Among them, T is the sensor coordinate transformation matrix, R(ω k ) is the rotation matrix driven by angular velocity, δα k is the acceleration compensation term;

[0024] Construct a nonlinear optimization model based on Lie group theory;

[0025] Based on the nonlinear optimization model, the spatial registration problem under kinematic constraints is transformed into the least squares optimization on the SE(3) manifold, as follows:

[0026]

[0027] Among them, ξ i is the Lie algebra parameter, is the covariance matrix, exp(·) is the exponential mapping, and log(·) is the logarithmic mapping;

[0028] The above steps are iterated until the UAV spatial coordinate error converges.

[0029] Preferably, the multimodal data fusion algorithm based on adaptive weight allocation, obtaining sensor confidence and environmental characteristics to dynamically adjust fusion weights specifically includes:

[0030] The confidence level c of each sensor is obtained based on the historical data, error statistics, environmental conditions and signal strength factors of each sensor. i ;

[0031] Based on the confidence c of each sensor i , get the function expression of environmental characteristics with respect to sensor weights, the formula is:

[0032]

[0033] Among them, w i (e) is the sensor weight, c i (e) is the confidence of sensor i under given environmental characteristics;

[0034] Based on the obtained sensor confidence and environmental characteristics, the measurement error value in each time period is calculated, and the fusion weight is dynamically adjusted through adaptive adjustment based on particle filtering. If the weight of sensor i is The weight update formula at the current moment is:

[0035]

[0036] in, is the measurement error of sensor i at time t, and λ is the adjustment parameter.

[0037] Preferably, the real-time 3D reconstruction based on the edge computing device and the feature-level fusion of point cloud and image data on the drone end specifically include:

[0038] Generate a 3D point cloud model based on the multi-view images and point cloud data acquired by the sensor;

[0039] Construct a 3D mesh based on the generated 3D point cloud, and generate a polygonal mesh to represent the scene surface;

[0040] FPGA-based edge computing devices map the above-mentioned 3D reconstruction algorithm to the FPGA hardware acceleration module, performing hardware acceleration on key calculations in the 3D mesh generation process.

[0041] Based on the real-time flight altitude of the drone obtained by the altimeter, the grid resolution is dynamically adjusted. At low altitudes, the grid accuracy is improved to capture details; at high altitudes, the grid resolution is reduced to reduce the amount of calculation;

[0042] Based on the convolutional neural network for point cloud-image joint feature extraction, the output is a fused point cloud with semantic labels. The grid structure is defined as:

[0043] F(P,I)=Conv3D(GraphConv(P)⊕Conv2D(I))

[0044] Among them, P is the point cloud data, I is the image data, Conv3D is the three-dimensional convolution layer, GraphConv(P) is the graph convolution operation based on the kd-tree, the scope is the point cloud data P, Conv2D(I) is the two-dimensional convolution layer, the scope is the image data I, and ⊕ is the feature splicing operation.

[0045] Preferably, the construction of a distributed edge cache mechanism and data lifecycle management based on an LRU algorithm specifically includes:

[0046] The drone swarm, ground edge server, and regional edge cloud form a three-level cache architecture;

[0047] Divide the survey area into three-dimensional space grids, and associate each grid with a unique ID;

[0048] Map data blocks to edge nodes based on the consistent hashing algorithm, and each edge node maintains a double-linked list and hash table structure;

[0049] Based on the data life cycle, expired data is automatically cleaned up and non-expired but lowest-weighted data is eliminated. When the local cache is full, low-value data is migrated to the upper-level node instead of being deleted directly.

[0050] Furthermore, a multimodal edge computing UAV real-time mapping data fusion system is proposed, including:

[0051] Multi-source sensor dynamic time synchronization model module: The multi-source sensor dynamic time synchronization model module is mainly used to compensate the timestamp and recursively estimate the clock offset of each sensor;

[0052] Spatial coordinate calibration module: The spatial coordinate calibration module is mainly used to align the spatial coordinates of the drone through the edge computing device;

[0053] Fusion weight adjustment module: The fusion weight adjustment module is mainly used to dynamically adjust the fusion weight of multimodal data based on various influencing factors;

[0054] 3D modeling module: The 3D modeling module is mainly used to perform real-time 3D modeling of the surveying area of ​​the UAV;

[0055] Fusion module: The fusion module is mainly used to characterize and fuse point cloud data and image data;

[0056] Management module: The management module is mainly used to manage data according to the data life cycle based on a distributed edge caching mechanism;

[0057] Processor: The processor is mainly used for the calculation process of each formula and the construction calculation process of each model.

[0058] Compared with the prior art, the advantages of the present invention are:

[0059] By modeling based on the physical characteristics of sensors and a dynamic time synchronization model, the time synchronization problem between different sensors is solved, ensuring the spatiotemporal correlation of multi-source data. At the same time, the spatial registration algorithm under kinematic constraints accurately calibrates the spatial coordinate systems of different sensors, reducing errors caused by dynamic changes in the environment. The multimodal data fusion algorithm with adaptive weight distribution dynamically adjusts the fusion weights of each sensor data, making the fusion results more adaptable to surveying and mapping needs in different environments, and improving the accuracy and reliability of the data. Real-time three-dimensional reconstruction and feature-level fusion are achieved through edge computing devices, significantly reducing data processing delays and improving processing efficiency. This method can achieve efficient, accurate, and low-latency real-time data processing in complex environments, ensuring the spatiotemporal consistency of drone surveying and mapping operations, and providing strong technical support for multiple application fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a schematic diagram of the multimodal edge computing UAV real-time mapping data fusion method proposed in the present invention;

[0061] Figure 2 Schematic diagram of the multi-source sensor dynamic time synchronization model proposed in this invention;

[0062] Figure 3 This is a schematic diagram of the dynamic calibration of spatial coordinates proposed by the present invention;

[0063] Figure 4 This is a schematic diagram of the fusion weight adjustment proposed by the present invention;

[0064] Figure 5 This is a schematic diagram of feature-level fusion proposed in the present invention;

[0065] Figure 6 Data management schematic diagram for the present application;

[0066] Figure 7 Architecture diagram of the electronic device in the present solution;

[0067] Figure 8 Structure schematic diagram of the computer readable storage medium in the present solution. DETAILED DESCRIPTION

[0068] The following description is provided to enable any person skilled in the art to practice the application. The preferred embodiments described in the following description are only examples of the application and the other obvious variants can be conceived by those skilled in the art.

[0069] The unmanned aerial vehicle real-time mapping data fusion system of multi-modal edge computing comprises:

[0070] The multi-source sensor dynamic time synchronization model module is mainly used for compensating the time stamp and recursively estimating the clock offset of each sensor;

[0071] The spatial coordinate calibration module is mainly used for registration calculation of the spatial coordinates of the unmanned aerial vehicle by the edge computing device;

[0072] The fusion weight adjustment module is mainly used for dynamically adjusting the fusion weight of multi-modal data based on various influencing factors;

[0073] The three-dimensional modeling module is mainly used for real-time three-dimensional modeling of the mapping area of the unmanned aerial vehicle;

[0074] The fusion module is mainly used for feature and fusion of point cloud data and image data;

[0075] The management module is mainly used for managing data according to the data life cycle based on the distributed edge caching mechanism;

[0076] The processor is mainly used for the calculation process of each formula and the construction calculation process of each model.

[0077] Referring to Figure 1 The multi-modal edge computing unmanned aerial vehicle real-time mapping data fusion method comprises:

[0078] Step one: based on the physical characteristics modeling of the sensor, a multi-source sensor dynamic time synchronization model is constructed;

[0079] Step two: based on the edge computing device, the spatial coordinate system of the multi-source sensor is dynamically calibrated through the spatial registration algorithm under kinematic constraints;

[0080] Step three: multi-modal data fusion algorithm based on adaptive weight distribution, obtain sensor confidence and environmental feature dynamic adjustment fusion weight;

[0081] Step four: real-time three-dimensional reconstruction based on edge computing device, complete point cloud and image data feature level fusion at the unmanned aerial vehicle end;

[0082] Step five: build a distributed edge cache mechanism, and manage the data life cycle based on the LRU algorithm.

[0083] Referring to Figure 2 Based on the modeling of sensor physical characteristics, the multi-source sensor dynamic time synchronization model is built, which specifically includes:

[0084] The sampling delay characteristics of the inertial measurement unit (IMU), laser radar (LiDAR), optical camera, and GNSS module are modeled respectively, and a timestamp compensation function is established, with the formula being:

[0085]

[0086] Where τ i is the inherent delay coefficient of the sensor, α i is the environmental temperature compensation factor, β i is the power supply voltage correction coefficient, and v drone is the real-time flight speed of the unmanned aerial vehicle.

[0087] Based on the compensated timestamp data of each sensor, the clock offset is recursively estimated at the sensor and network layer through the bidirectional timestamp exchange protocol, with the formula being:

[0088]

[0089] Where T1 to T4 are PTP protocol timestamps, and ∈ is a network jitter compensation term, which is modified by twice differentiation of the sensor clock drift rate.

[0090] Specifically, when performing multi-sensor clock synchronization, in addition to timestamp compensation and clock offset estimation, the influence of sensor dynamic characteristics and environmental factors on synchronization accuracy also needs to be considered. For example, IMU and LiDAR may produce errors under high acceleration or complex environment, and network fluctuations and data transmission delays will also affect synchronization accuracy. In addition to conventional algorithms, sensor state information should also be combined for dynamic adjustment to ensure that the system has adaptive ability, real-time correction of deviation, and improvement of synchronization stability in long-time operation, so as to realize accurate data fusion and positioning.

[0091] Referring to Figure 3As shown in the figure, based on edge computing devices, the spatial coordinate system of multi-source sensors is dynamically calibrated through a spatial registration algorithm under kinematic constraints. Specifically, the following steps are performed:

[0092] Based on the data obtained by multi-source sensors, it is transmitted to the edge computing device in the drone;

[0093] Based on the coordinate transformation chain of the UAV's six-degree-of-freedom posture, the IMU angular velocity ω and acceleration α are used as constraints to construct the registration error function. The formula is:

[0094]

[0095] Among them, T is the sensor coordinate transformation matrix, R(ω k ) is the rotation matrix driven by angular velocity, δα k is the acceleration compensation term;

[0096] Construct a nonlinear optimization model based on Lie group theory;

[0097] Based on the nonlinear optimization model, the spatial registration problem under kinematic constraints is transformed into the least squares optimization on the SE(3) manifold, as follows:

[0098]

[0099] Among them, ξ i is the Lie algebra parameter, is the covariance matrix, exp(·) is the exponential mapping, and log(·) is the logarithmic mapping;

[0100] The above steps are iterated until the UAV spatial coordinate error converges.

[0101] It is understandable that least squares optimization based on SE(3) manifolds usually requires a lot of computing resources, especially in high-dimensional spaces and large-scale data. This may lead to excessive computational overhead and affect real-time performance. Incremental optimization methods can be used to reduce the computational workload by performing only local optimization at each time step, or efficient numerical optimization algorithms can be used to improve the convergence speed of the optimization process.

[0102] See Figure 4 As shown in the figure, the multimodal data fusion algorithm based on adaptive weight allocation obtains sensor confidence and environmental characteristics to dynamically adjust the fusion weight, specifically including:

[0103] The confidence level c of each sensor is obtained based on the historical data, error statistics, environmental conditions and signal strength factors of each sensor. i ;

[0104] Based on the confidence c of each sensor i, get the function expression of environmental characteristics with respect to sensor weights, the formula is:

[0105]

[0106] Among them, w i (e) is the sensor weight, c i (e) is the confidence of sensor i under given environmental characteristics;

[0107] Based on the obtained sensor confidence and environmental characteristics, the measurement error value in each time period is calculated, and the fusion weight is dynamically adjusted through adaptive adjustment based on particle filtering. If the weight of sensor i is The weight update formula at the current moment is:

[0108]

[0109] in, is the measurement error of sensor i at time t, and λ is the adjustment parameter.

[0110] Specifically, by evaluating the sensor's historical data, real-time status, and environmental conditions, an error model is established to quantify the measurement error of each sensor under specific conditions. A particle filter method effectively handles these nonlinear errors and dynamically adjusts data fusion weights based on sensor confidence. As the environment changes, the system adaptively adjusts weights based on real-time error feedback, allowing high-confidence sensors to dominate the data fusion process while low-confidence sensors are suppressed. This effectively reduces error accumulation and ensures the high accuracy and reliability of the multi-sensor fusion system over long periods of operation.

[0111] See Figure 5 As shown in the figure, real-time 3D reconstruction based on edge computing devices and feature-level fusion of point cloud and image data on the drone side specifically include:

[0112] Generate a 3D point cloud model based on the multi-view images and point cloud data acquired by the sensor;

[0113] Construct a 3D mesh based on the generated 3D point cloud, and generate a polygonal mesh to represent the scene surface;

[0114] FPGA-based edge computing devices map the above-mentioned 3D reconstruction algorithm to the FPGA hardware acceleration module, performing hardware acceleration on key calculations in the 3D mesh generation process.

[0115] Based on the real-time flight altitude of the drone obtained by the altimeter, the grid resolution is dynamically adjusted. At low altitudes, the grid accuracy is improved to capture details; at high altitudes, the grid resolution is reduced to reduce the amount of calculation;

[0116] Based on the convolutional neural network for point cloud-image joint feature extraction, the output is a fused point cloud with semantic labels. The grid structure is defined as:

[0117] F(P,I)=Conv3D(GraphConv(P)⊕Conv2D(I))

[0118] Among them, P is the point cloud data, I is the image data, Conv3D is the three-dimensional convolution layer, GraphConv(P) is the graph convolution operation based on the kd-tree, the scope is the point cloud data P, Conv2D(I) is the two-dimensional convolution layer, the scope is the image data I, and ⊕ is the feature splicing operation.

[0119] Understandably, the computational workload and storage requirements increase dramatically during 3D mesh generation, especially at high resolutions. This can lead to computational delays and storage bottlenecks, especially in real-time and large-scale scenarios. By mapping the mesh generation process to an FPGA hardware acceleration module, the FPGA's parallel processing capabilities are leveraged to accelerate key computations, such as point cloud-to-mesh conversion and triangulation algorithms. Furthermore, hierarchical meshing technology is used to dynamically adjust mesh accuracy based on the complexity of different scenarios.

[0120] See Figure 6 As shown in the figure, building a distributed edge cache mechanism and performing data lifecycle management based on the LRU algorithm specifically includes:

[0121] The drone swarm, ground edge server, and regional edge cloud form a three-level cache architecture;

[0122] Divide the survey area into three-dimensional space grids, and associate each grid with a unique ID;

[0123] Map data blocks to edge nodes based on the consistent hashing algorithm, and each edge node maintains a double-linked list and hash table structure;

[0124] Based on the data life cycle, expired data is automatically cleaned up and non-expired but lowest-weighted data is eliminated. When the local cache is full, low-value data is migrated to the upper-level node instead of being deleted directly.

[0125] Furthermore, the method according to the embodiment of the present application can also be used with the aid of Figure 7 The electronic device architecture shown in FIG. Figure 7As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store the multimodal edge computing drone real-time mapping data fusion method and system provided in this application. The electronic device 500 may also include a terminal interface 508. Of course, Figure 7 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 7 One or more components of an electronic device are shown.

[0126] Figure 8 This is a schematic diagram of the computer-readable storage medium structure provided by an embodiment of the present application. Figure 8 As shown, a computer-readable storage medium 600 according to one embodiment of the present application is shown. Computer-readable instructions are stored on the computer-readable storage medium 600. When the computer-readable instructions are executed by the processor, the method and system for real-time mapping data fusion of multimodal edge computing of drones according to the embodiments of the present application described with reference to the above figures can be executed. The storage medium 600 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0127] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0128] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal edge computing UAV real-time mapping data fusion method, characterized by: include: Based on the modeling of sensor physical characteristics, a dynamic time synchronization model of multi-source sensors is constructed; Based on edge computing devices, the spatial coordinate system of multi-source sensors is dynamically calibrated through a spatial registration algorithm under kinematic constraints. A multimodal data fusion algorithm based on adaptive weight allocation obtains sensor confidence and environmental characteristics to dynamically adjust fusion weights; Real-time 3D reconstruction based on edge computing devices, and feature-level fusion of point cloud and image data on the drone side; Build a distributed edge caching mechanism and manage data lifecycle based on the LRU algorithm.

2. The multimodal edge computing UAV real-time mapping data fusion method according to claim 1 is characterized in that: The construction of a multi-source sensor dynamic time synchronization model based on sensor physical characteristics modeling specifically includes: Based on the sampling delay characteristics of the inertial measurement unit (IMU), laser radar (LiDAR), optical camera, and GNSS module, a timestamp compensation function is established. The formula is: Among them, τ i is the sensor’s inherent delay coefficient, α i is the ambient temperature compensation factor, β i is the supply voltage correction factor, v drone The real-time flight speed of the drone; Based on the compensated timestamp data of each sensor, the clock offset is recursively estimated at the sensor and network layers through a two-way timestamp exchange protocol. The formula is: Among them, T1 to T4 are PTP protocol timestamps, and ∈ is the network jitter compensation term, which is corrected by the second derivative of the sensor clock drift rate.

3. The multimodal edge computing UAV real-time mapping data fusion method according to claim 1 is characterized in that: The dynamic calibration of the spatial coordinate system of the multi-source sensor based on the edge computing device through the spatial registration algorithm under kinematic constraints specifically includes: Based on the data obtained by multi-source sensors, it is transmitted to the edge computing device in the drone; Based on the coordinate transformation chain of the UAV's six-degree-of-freedom posture, the IMU angular velocity ω and acceleration α are used as constraints to construct the registration error function. The formula is: Among them, T is the sensor coordinate transformation matrix, R(ω k ) is the rotation matrix driven by angular velocity, δα k is the acceleration compensation term; Construct a nonlinear optimization model based on Lie group theory; Based on the nonlinear optimization model, the spatial registration problem under kinematic constraints is transformed into the least squares optimization on the SE(3) manifold, as follows: Among them, ξ i is the Lie algebra parameter, is the covariance matrix, exp(·) is the exponential mapping, and log(·) is the logarithmic mapping; The above steps are iterated until the UAV spatial coordinate error converges.

4. The multimodal edge computing UAV real-time mapping data fusion method according to claim 1 is characterized in that: The multimodal data fusion algorithm based on adaptive weight allocation, which obtains sensor confidence and environmental characteristics to dynamically adjust fusion weights, specifically includes: The confidence level c of each sensor is obtained based on the historical data, error statistics, environmental conditions and signal strength factors of each sensor. i ; Based on the confidence c of each sensor i , get the function expression of environmental characteristics with respect to sensor weights, the formula is: Among them, w i (e) is the sensor weight, c i (e) is the confidence of sensor i under given environmental characteristics; Based on the obtained sensor confidence and environmental characteristics, the measurement error value in each time period is calculated, and the fusion weight is dynamically adjusted through adaptive adjustment based on particle filtering. If the weight of sensor i is The weight update formula at the current moment is: in, is the measurement error of sensor i at time t, and λ is the adjustment parameter.

5. The multimodal edge computing UAV real-time mapping data fusion method according to claim 1 is characterized in that: The real-time 3D reconstruction based on edge computing devices and the feature-level fusion of point cloud and image data on the drone side specifically include: Generate a 3D point cloud model based on the multi-view images and point cloud data acquired by the sensor; Construct a 3D mesh based on the generated 3D point cloud, and generate a polygonal mesh to represent the scene surface; FPGA-based edge computing devices map the above-mentioned 3D reconstruction algorithm to the FPGA hardware acceleration module, performing hardware acceleration on key calculations in the 3D mesh generation process. Based on the real-time flight altitude of the drone obtained by the altimeter, the grid resolution is dynamically adjusted. At low altitudes, the grid accuracy is improved to capture details; at high altitudes, the grid resolution is reduced to reduce the amount of calculation; Based on the convolutional neural network for point cloud-image joint feature extraction, the output is a fused point cloud with semantic labels. The grid structure is defined as: Among them, P is point cloud data, I is image data, Conv3D is a three-dimensional convolution layer, GraphConv(P) is a graph convolution operation based on a kd tree, and its scope is point cloud data P. Conv2D(I) is a two-dimensional convolution layer, and its scope is image data I. It is a feature splicing operation.

6. The multimodal edge computing UAV real-time mapping data fusion method according to claim 1 is characterized in that: The distributed edge cache mechanism is constructed and data lifecycle management based on the LRU algorithm specifically includes: The drone swarm, ground edge server, and regional edge cloud form a three-level cache architecture; Divide the survey area into three-dimensional space grids, and associate each grid with a unique ID; Map data blocks to edge nodes based on the consistent hashing algorithm, and each edge node maintains a double-linked list and hash table structure; Based on the data life cycle, expired data is automatically cleaned up and non-expired but lowest-weighted data is eliminated. When the local cache is full, low-value data is migrated to the upper-level node instead of being deleted directly.

7. A method for fusion of real-time UAV surveying and mapping data in combination with multimodal edge computing, for realizing a system for fusion of real-time UAV surveying and mapping data in combination with multimodal edge computing as claimed in any one of claims 1 to 7, characterized in that: include: Multi-source sensor dynamic time synchronization model module: The multi-source sensor dynamic time synchronization model module is mainly used to compensate the timestamp and recursively estimate the clock offset of each sensor; Spatial coordinate calibration module: The spatial coordinate calibration module is mainly used to align the spatial coordinates of the drone through the edge computing device; Fusion weight adjustment module: The fusion weight adjustment module is mainly used to dynamically adjust the fusion weight of multimodal data based on various influencing factors; 3D modeling module: The 3D modeling module is mainly used to perform real-time 3D modeling of the surveying area of ​​the UAV; Fusion module: The fusion module is mainly used to characterize and fuse point cloud data and image data; Management module: The management module is mainly used to manage data according to the data life cycle based on a distributed edge caching mechanism; Processor: The processor is mainly used for the calculation process of each formula and the construction calculation process of each model.

8. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal edge computing drone real-time mapping data fusion method as described in any one of claims 1-7.

9. A computer-readable storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, the multimodal edge computing drone real-time mapping data fusion method described in any one of claims 1 to 7 is implemented.

Images