Unmanned aerial vehicle airborne intelligent terminal real-time three-dimensional point cloud processing system based on edge computing

Abstract

The application relates to an unmanned aerial vehicle airborne intelligent terminal real-time three-dimensional point cloud processing system based on edge computing, characterized in that the system comprises a data acquisition module, a data preprocessing and synchronization module, a visual inertial SLAM modeling module, a three-dimensional point cloud generation module, a point cloud processing and optimization module, an edge computing scheduling module, a point cloud data management module and an application output module; high-precision pose information is obtained through a visual inertial fusion SLAM technology, and dense point clouds are generated in combination with a depth estimation and three-dimensional reconstruction algorithm; an edge computing scheduling mechanism is adopted to dynamically adjust the allocation of computing resources according to the flight state, and efficient point cloud data management is realized through a voxel storage and incremental update strategy. The application realizes high-precision and real-time unmanned aerial vehicle three-dimensional point cloud processing.

Description

Technical Field

[0001] This invention relates to the field of point cloud data processing, and in particular to a real-time 3D point cloud processing system for an airborne intelligent terminal of an unmanned aerial vehicle based on edge computing. Background Technology

[0002] With the rapid development of UAV technology, UAV-based 3D environmental perception and modeling have been widely used in fields such as intelligent inspection, environmental monitoring, disaster relief, and surveying. Traditional UAV 3D perception solutions mainly rely on LiDAR or a single vision sensor, but these present numerous technical challenges in complex dynamic environments.

[0003] While lidar can provide relatively accurate distance information, it is expensive, consumes a lot of power, and its performance degrades significantly in adverse weather conditions such as rain and fog. Single-vision sensors, although less expensive, lack depth information and are prone to tracking failures in areas with little texture or in environments with drastic lighting changes. Furthermore, existing solutions typically transmit raw sensor data to a ground station for processing, which, limited by communication bandwidth and transmission latency, makes it difficult to meet real-time requirements.

[0004] In recent years, visual-inertial SLAM (Simultaneous Localization and Mapping) technology has made significant progress in UAV localization and mapping by combining the advantages of camera and IMU sensors. However, existing visual-inertial SLAM schemes are prone to accumulating errors during long-term flights and struggle to maintain stable real-time performance on edge devices with limited computing resources. Particularly in the 3D point cloud generation and processing stages, traditional methods often employ fixed processing strategies, failing to dynamically adjust computing resource allocation based on the UAV's motion state and mission requirements, resulting in insufficient system adaptability in complex environments.

[0005] Furthermore, existing technologies also have shortcomings in point cloud data management. Most solutions employ a global reconstruction strategy, requiring repeated calculations across the entire environment, which not only consumes significant computing resources but also easily introduces noise into dynamic environments. The lack of effective incremental update mechanisms and local optimization strategies makes it difficult for the system to achieve stable operation over long periods with limited edge computing resources. Summary of the Invention

[0006] To address the aforementioned issues, the present invention aims to provide a real-time 3D point cloud processing system for UAV airborne intelligent terminals based on edge computing, enabling high-precision and real-time 3D point cloud processing for UAVs.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A real-time 3D point cloud processing system for an airborne intelligent terminal of an unmanned aerial vehicle (UAV) based on edge computing includes a data acquisition module, a data preprocessing and synchronization module, a visual-inertial SLAM modeling module, a 3D point cloud generation module, a point cloud processing and optimization module, an edge computing scheduling module, a point cloud data management module, and an application output module. The data acquisition module acquires continuous image sequences, angular velocity, and acceleration data transmitted by the UAV; the data preprocessing and synchronization module aligns the image data and inertial data through the time synchronization module to form a unified data input stream; the visual-inertial SLAM module processes the data input stream, constructs constraint relationships between images through feature extraction and matching, and performs state estimation in conjunction with IMU pre-integration results to obtain the UAV's pose information at the current moment. Under the keyframe mechanism, the pose is globally consistently adjusted through the sliding window optimization method, thereby outputting a continuous pose trajectory and sparse map structure. The 3D point cloud generation module performs depth estimation based on pose trajectory and sparse map structure, uses binocular matching to predict the depth value corresponding to each pixel, and back-projects the 2D pixel coordinates to 3D space through the camera intrinsic parameter matrix to generate an initial 3D point cloud. The point cloud processing and optimization module is used to perform global fusion and structural optimization of the 3D point cloud to obtain an optimized 3D point cloud. The edge computing scheduling module is used to coordinate the resource allocation and execution strategies of each computing module, and dynamically schedules the SLAM frequency, point cloud generation density and processing strategy according to the current flight status of the UAV and the task type. The point cloud data management module is used for structured storage and incremental updates of 3D point clouds. It divides the space into multiple voxel blocks and establishes an index structure for each block. When updating the point cloud, it only performs local updates on the areas that have changed.

[0008] Furthermore, constraint relationships between images are constructed through feature extraction and matching, and state estimation is performed by combining the IMU pre-integration results to obtain the UAV's pose information at the current moment, as follows: After receiving synchronized image data and inertial data, the visual-inertial SLAM modeling module performs feature extraction processing on the current image frame. It uses feature operators with scale invariance and rotation invariance to process the image and obtain a set of feature points. ; in, Let represent the i-th feature point in the t-th frame of the image; subsequently, feature matching is performed between adjacent image frames to establish the correspondence between feature points: ; in, Let j be the j-th feature point in the (t-1)-th frame of the image; Based on the above matching relationship, visual constraints are constructed using reprojection errors: ; in, P represents the pixel coordinates of the observed feature points. i Let T be the coordinates of a point in space. t For the current pose, π(·) represents the camera projection model; Pre-integration processing is performed on the synchronously input inertial measurement data. Let the angular velocity and acceleration measured by the IMU between time t-1 and t in two adjacent image frames be ω and a, respectively. Then, the pre-integrated quantity is obtained through integration: ; ; ; Among them, b g With b a These represent the biases of the gyroscope and accelerometer, respectively; ΔR t ω represents the rotation increment from t-1 to t; ∏ represents the chain multiplication operation; exp(·) is the outlier exponential mapping; k b is the angular velocity measured by the IMU at time k; g Δt is the gyroscope bias; Δv is the time interval; Δt is the gyroscope bias. t For speed increment R k Let a be the rotation matrix at time k; k Linear acceleration measured by the IMU; Δp t For position increment; v k R is the velocity at time k; k Let a be a rotation matrix; k This is an acceleration measurement; Based on the above pre-integration results, the inertial constraint error term e is constructed. imu : ; Among them, R t ,R t-1 These are the rotation matrices for times t and t-1, respectively; v t ,v t-1 The velocity vectors at times t and t-1 are respectively; p t ,p t-1 are the position vectors at times t and t-1, respectively; g is the gravitational acceleration vector; Visual constraints and inertial constraints are integrated to construct a unified state estimation optimization model; the system state variables are defined as follows: ; The overall optimization objective function is then expressed as: ; Where, Σ v With Σ i These are the covariance matrices for visual and inertial errors, respectively. This refers to visual reprojection error; By performing nonlinear optimization on the above objective function, the optimal pose estimation result T of the UAV at the current moment is obtained. t This enables tight coupling and fusion positioning of visual and inertial information.

[0009] Furthermore, under the keyframe mechanism, a sliding window optimization method is used to adjust the pose globally for consistency, thereby outputting a continuous pose trajectory and a sparse map structure, as detailed below: During pose estimation, the visual-inertial SLAM modeling module uses a keyframe-based sliding window optimization strategy to continuously estimate and update the system state. It performs keyframe filtering on the continuously input image sequence. When the disparity between adjacent frames exceeds a preset threshold, the current image frame is selected as a keyframe and added to the optimization window; otherwise, it is used as an ordinary frame only for auxiliary tracking and does not participate in global optimization. When constructing a sliding window, let the window contain N keyframes, its state variable be X, and the set of spatial feature points be P; A joint optimization objective function is constructed, which unifies the modeling of visual reprojection error and IMU pre-integration error. The overall optimization objective is expressed as: Where ρ(·) is the robust kernel function; ∑ i,j This represents the summation of observations of all image frames and feature points; ∑ k Accumulate across all IMU time periods; This refers to visual reprojection error; For IMU residuals; It is the square of the L2 norm; Visual reprojection error is defined as: Among them, z i,j Let π(·) represent the pixel coordinates of the j-th feature point observed in the i-th frame, and let π(·) represent the camera projection model. The IMU pre-integration error is defined as: ; in, and For IMU pre-integration terms; Using the above objective function, the Gauss-Newton algorithm is used to iteratively solve the state variables, thereby obtaining the consistent pose estimation results of each keyframe within the current sliding window.

[0010] Furthermore, depth estimation is performed based on the pose trajectory and sparse map structure. Binocular matching is used to predict the depth value corresponding to each pixel, and the two-dimensional pixel coordinates are back-projected to the three-dimensional space through the camera intrinsic parameter matrix to generate initial three-dimensional points, as follows: Adjacent keyframe images are selected as input, and the images are aligned in rows by epipolar constraints. Pixel-level matching is then performed to calculate the disparity map. For each pixel (u,v), the disparity value d(u,v) is obtained by searching for matching points in the corresponding image pair, thereby establishing the correspondence between the two-dimensional image and spatial depth. During depth calculation, the 3D point cloud generation module, based on a binocular geometric model, converts disparity values ​​into depth information. Let the baseline length of the binocular camera be b and the focal length be f, then the depth value Z(u,v) of a pixel is represented as: ; After obtaining the depth map, and combining it with the camera intrinsic parameter matrix K, the two-dimensional pixel coordinates are back-projected into the three-dimensional spatial coordinate system to obtain the corresponding three-dimensional points: ; Simultaneously, the pose T output by visual inertial SLAM is utilized. t The 3D points in the local camera coordinate system are transformed to the global coordinate system to achieve a unified representation of point clouds across multiple frames, resulting in an initial 3D point cloud. Through depth estimation and coordinate back projection, the system can recover continuous 3D point cloud data from 2D images and fuse it with pose trajectories to form an initial 3D point cloud structure.

[0011] Furthermore, the initial 3D point cloud is globally fused, as follows: The initial 3D point cloud P world With historical global point cloud set P global The points are then merged to obtain the updated global point cloud: P global P global ∪P world ; In the point cloud fusion process, a sparse-dense hybrid mapping strategy is introduced, dividing the point cloud into keyframe regions and non-keyframe regions. For keyframe regions, dense reconstruction is performed to preserve detailed information, and its point cloud density function is expressed as: ; Where, N d Let V be the number of dense points and V be the spatial volume; for non-critical regions, only the selected sparse feature points are retained, and their density is expressed as: ; The system defines the density scaling factor λ: And based on the drone's speed v, angular velocity ω, and scene texture intensity This ratio can be dynamically adjusted. ; Where α and β are weighting coefficients.

[0012] Further structural optimizations are as follows: Let the merged point cloud be: ; By constructing an adjacency graph The set of edges Determined by spatial proximity: ; This establishes the local topology of the point cloud; In the structural optimization process, multi-constraint joint modeling is introduced, including smoothing constraints. Normal consistency constraint and structural retention constraints : ; ; ; in, This represents the initial distance between points; Planar constraints are introduced to enhance structural representation if the point set belongs to the same local plane. Then we have: ; Where n is the normal vector of the plane; T denotes transpose; and d is the signed distance from the plane to the origin. The above constraints are unified into a nonlinear least squares optimization problem: ; Where λ1, λ2, and λ3 are the weight coefficients of each constraint; The Gauss-Newton method is used to iteratively update the point positions: .

[0013] Furthermore, based on the current flight status of the UAV and the mission type, the SLAM frequency, point cloud generation density, and processing strategy are dynamically scheduled, as follows: Real-time acquisition of the drone's current flight status information, including linear velocity v t acceleration a t and attitude change ΔR t And construct a system state vector based on the current task type: Among them, C task Indicates task complexity or priority; Based on this state vector, the update frequency, depth estimation density and point cloud processing strategy of the SLAM module are uniformly scheduled to achieve collaborative control among multiple modules. Construct a resource allocation and strategy selection model, and assume the scheduling decision is: ; Among them, f slam Indicates the SLAM update frequency, ρ depth λ represents the point cloud generation density. proc The point cloud processing intensity is represented by the following function; the scheduling strategy is modeled using the following function: ; ; ; Among them, f max With ρ max These are the maximum update frequency and the maximum point cloud density, respectively. These are the weighting coefficients.

[0014] Furthermore, the space is divided into multiple voxel blocks, and an index structure is established for each block. During point cloud updates, only the changed regions are locally updated, as follows: The entire 3D space is divided according to a preset voxel size to form multiple cubic or cuboid voxel blocks, each with a unique spatial index identifier. The system maintains an independent data structure for each voxel block, recording the point cloud information, timestamp, number of observations, and confidence level attributes within that region. At the same time, a multi-level index structure is established, including a coarse-grained spatial index for quickly locating relevant regions and a fine-grained point index for accurately retrieving specific point cloud data. During the point cloud data update process, the system adopts an incremental local update strategy to reduce computational overhead. When new point cloud data arrives, it first quickly determines the set of voxel blocks to which it belongs through spatial indexing, and only performs data update operations on these regions that have changed. For newly added point clouds, they are directly inserted into the corresponding voxel blocks. For points that are repeatedly observed, their positions and confidence are updated using a weighted fusion method. For points that have not been observed for a long time, their weights are adjusted or they are removed according to the time decay strategy.

[0015] The present invention has the following beneficial effects: 1. This invention significantly improves the accuracy and robustness of UAV 3D perception through the multi-source information fusion mechanism of the visual-inertial SLAM modeling module. By combining image feature constraints and IMU pre-integration constraints, the sliding window optimization method is used to adjust the pose globally in a consistent manner, effectively suppressing the cumulative error and drift problem in the traditional single sensor solution. At the same time, through the key frame mechanism and depth estimation technology, a high-quality conversion from sparse map structure to dense 3D point cloud is achieved, effectively improving the geometric accuracy, spatial continuity and detail fidelity of the generated point cloud. 2. The edge computing scheduling module of this invention dynamically adjusts the SLAM frequency, point cloud generation density, and processing strategy according to the UAV's flight status and mission requirements, avoiding the waste of computing power or performance bottlenecks in fixed parameter schemes. At the same time, the point cloud data management module adopts a voxelized storage and incremental update strategy, decomposing the global mapping task into local area processing, and only calculating and updating the voxel blocks that have changed, which greatly reduces memory usage and computational complexity. This enables the system to maintain stable real-time processing capabilities under limited edge computing resources, meeting the rapid response requirements of UAVs in highly dynamic environments. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: refer to Figure 1 In this embodiment, a real-time 3D point cloud processing system for an airborne intelligent terminal of an unmanned aerial vehicle based on edge computing is provided, including a data acquisition module, a data preprocessing and synchronization module, a visual-inertial SLAM modeling module, a 3D point cloud generation module, a point cloud processing and optimization module, an edge computing scheduling module, a point cloud data management module, and an application output module. The data acquisition module acquires continuous image sequences, angular velocity, and acceleration data transmitted by the UAV; the data preprocessing and synchronization module aligns the image data and inertial data through the time synchronization module to form a unified data input stream; the visual-inertial SLAM module processes the data input stream, constructs constraint relationships between images through feature extraction and matching, and performs state estimation in conjunction with IMU pre-integration results to obtain the UAV's pose information at the current moment. Under the keyframe mechanism, the pose is globally consistently adjusted through the sliding window optimization method, thereby outputting a continuous pose trajectory and sparse map structure. The 3D point cloud generation module performs depth estimation based on pose trajectory and sparse map structure, uses binocular matching to predict the depth value corresponding to each pixel, and back-projects the 2D pixel coordinates to 3D space through the camera intrinsic parameter matrix to generate an initial 3D point cloud. The point cloud processing and optimization module is used to perform global fusion and structural optimization of the 3D point cloud to obtain an optimized 3D point cloud. The edge computing scheduling module is used to coordinate the resource allocation and execution strategies of each computing module. It dynamically schedules the SLAM frequency, point cloud generation density and processing strategy according to the current flight status of the UAV (speed, acceleration, attitude change) and the task type. The point cloud data management module is used for structured storage and incremental updates of 3D point clouds. It divides the space into multiple voxel blocks and establishes an index structure for each block. When updating the point cloud, it only performs local updates on the areas that have changed.

[0018] In this embodiment, the application output module is located at the top level of the system and is used to provide the processed point cloud data to specific applications. This module can output local point clouds for real-time obstacle avoidance and achieve path planning by constructing occupancy grids or distance fields; it can also output global point cloud maps for 3D modeling or environment reconstruction. In addition, this module supports outputting point cloud data in a standard format or transmitting it via a wireless link.

[0019] In this embodiment, constraint relationships between images are constructed through feature extraction and matching, and state estimation is performed by combining the IMU pre-integration results to obtain the pose information of the UAV at the current moment, as follows: After receiving synchronized image data and inertial data, the visual-inertial SLAM modeling module performs feature extraction processing on the current image frame. It uses feature operators with scale invariance and rotation invariance to process the image and obtain a set of feature points. ; in, Let represent the i-th feature point in the t-th frame of the image; subsequently, feature matching is performed between adjacent image frames to establish the correspondence between feature points: ; in, Let j be the j-th feature point in the (t-1)-th frame of the image; Based on the above matching relationship, visual constraints are constructed using reprojection errors: ; in, P represents the pixel coordinates of the observed feature points. i Let T be the coordinates of a point in space. t For the current pose, π(·) represents the camera projection model; Pre-integration processing is performed on the synchronously input inertial measurement data. Let the angular velocity and acceleration measured by the IMU between time t-1 and t in two adjacent image frames be ω and a, respectively. Then, the pre-integrated quantity is obtained through integration: ; ; ; Among them, b g With b a These represent the biases of the gyroscope and accelerometer, respectively; ΔR t ω represents the rotation increment from t-1 to t; ∏ represents the chain multiplication operation; exp(·) is the outlier exponential mapping; k b is the angular velocity measured by the IMU at time k; g Δt is the gyroscope bias; Δv is the time interval; Δt is the gyroscope bias. t For speed increment R k Let a be the rotation matrix at time k; k Linear acceleration measured by the IMU; Δp t For position increment; v k R is the velocity at time k; k Let a be a rotation matrix; k This is an acceleration measurement; Based on the above pre-integration results, the inertial constraint error term e is constructed. imu : ; Among them, R t ,R t-1 These are the rotation matrices for times t and t-1, respectively; v t ,v t-1 The velocity vectors at times t and t-1 are respectively; p t ,p t-1are the position vectors at times t and t-1, respectively; g is the gravitational acceleration vector; Visual constraints and inertial constraints are integrated to construct a unified state estimation optimization model; the system state variables are defined as follows: ; The overall optimization objective function is then expressed as: ; Where, Σ v With Σ i These are the covariance matrices for visual and inertial errors, respectively. Visual reprojection error represents the deviation between a 3D point projected onto the image plane and the actual observed pixel. By performing nonlinear optimization on the above objective function (using the Gauss-Newton method), the optimal pose estimation result T of the UAV at the current moment is obtained. t This enables tight coupling and fusion positioning of visual and inertial information.

[0020] In this embodiment, under the keyframe mechanism, a sliding window optimization method is used to adjust the pose globally for consistency, thereby outputting a continuous pose trajectory and a sparse map structure, as detailed below: During pose estimation, the visual-inertial SLAM modeling module uses a keyframe-based sliding window optimization strategy to continuously estimate and update the system state. It performs keyframe filtering on the continuously input image sequence. When the disparity between adjacent frames exceeds a preset threshold, the current image frame is selected as a keyframe and added to the optimization window; otherwise, it is used as an ordinary frame only for auxiliary tracking and does not participate in global optimization. When constructing a sliding window, let the window contain N keyframes, its state variable be X, and the set of spatial feature points be P; A joint optimization objective function is constructed, which unifies the modeling of visual reprojection error and IMU pre-integration error. The overall optimization objective is expressed as: Where ρ(·) is the robust kernel function; ∑ i,j This represents the summation of observations of all image frames and feature points; ∑ k Accumulate across all IMU time periods; This refers to visual reprojection error; For IMU residuals; It is the square of the L2 norm; Visual reprojection error is defined as: Among them, z i,jLet π(·) represent the pixel coordinates of the j-th feature point observed in the i-th frame, and let π(·) represent the camera projection model. The IMU pre-integration error is defined as: ; in, and For IMU pre-integration terms; Using the above objective function, the Gauss-Newton algorithm is used to iteratively solve the state variables, thereby obtaining the consistent pose estimation results of each keyframe within the current sliding window.

[0021] Furthermore, as time progresses and new keyframes are added to the sliding window, the oldest keyframes are marginalized to control computational scale. The marginalization process uses Schur Complement to transform the information of the removed variables into prior constraints, which are then retained in the optimization problem. This maintains the influence of historical information on the current estimate without increasing computational complexity. The marginalized prior terms can be expressed as: ; Among them, H p With b p These are the prior information matrix and information vector obtained after marginalization, respectively.

[0022] By employing keyframe filtering, sliding window optimization, and edge update mechanisms, global consistency adjustment across multiple frames is achieved. This effectively controls computational complexity while ensuring estimation accuracy, enabling the method to run in real time on an UAV-borne edge computing platform and meeting the high-precision pose input requirements of 3D point cloud modeling.

[0023] In this embodiment, depth estimation is performed based on the pose trajectory and sparse map structure. Binocular matching is used to predict the depth value corresponding to each pixel, and the two-dimensional pixel coordinates are back-projected to three-dimensional space using the camera intrinsic parameter matrix to generate initial three-dimensional points, as detailed below: Adjacent keyframe images are selected as input, and the images are aligned in rows by epipolar constraints. Pixel-level matching is then performed to calculate the disparity map. For each pixel (u,v), the disparity value d(u,v) is obtained by searching for matching points in the corresponding image pair, thereby establishing the correspondence between the two-dimensional image and spatial depth. During depth calculation, the 3D point cloud generation module, based on a binocular geometric model, converts disparity values ​​into depth information. Let the baseline length of the binocular camera be b and the focal length be f, then the depth value Z(u,v) of a pixel is represented as: ; After obtaining the depth map, and combining it with the camera intrinsic parameter matrix K, the two-dimensional pixel coordinates are back-projected into the three-dimensional spatial coordinate system to obtain the corresponding three-dimensional points: ; Simultaneously, the pose T output by visual inertial SLAM is utilized. t The 3D points in the local camera coordinate system are transformed to the global coordinate system to achieve a unified representation of point clouds across multiple frames, resulting in an initial 3D point cloud. Through depth estimation and coordinate back projection, the system can recover continuous 3D point cloud data from 2D images and fuse it with pose trajectories to form an initial 3D point cloud structure.

[0024] In this embodiment, the initial 3D point cloud is globally fused, as follows: The initial 3D point cloud P world With historical global point cloud set P global The points are then merged to obtain the updated global point cloud: P global P global ∪P world ; In the point cloud fusion process, a sparse-dense hybrid mapping strategy is introduced, dividing the point cloud into keyframe regions and non-keyframe regions. For keyframe regions, dense reconstruction is performed to preserve detailed information, and its point cloud density function is expressed as: ; Where, N d Let V be the number of dense points and V be the spatial volume; for non-critical regions, only the selected sparse feature points are retained, and their density is expressed as: ; The system defines the density scaling factor λ: And based on the drone's speed v, angular velocity ω, and scene texture intensity This ratio can be dynamically adjusted. ; Here, α and β are weighting coefficients; through this mechanism, the proportion of dense points is reduced during high-speed motion, thereby improving the ability to express local details in complex scenes.

[0025] In this embodiment, the structure is optimized as follows: Let the merged point cloud be: ; By constructing an adjacency graph The set of edges Determined by spatial proximity (k-nearest neighbors): ; This establishes the local topology of the point cloud; In the structural optimization process, multi-constraint joint modeling is introduced, including smoothing constraints. Normal consistency constraint and structural retention constraints : ; ; ; in, This represents the initial distance between points; Planar constraints are introduced to enhance structural representation if the point set belongs to the same local plane. Then we have: ; Where n is the normal vector of the plane; T denotes transpose; and d is the signed distance from the plane to the origin. The above constraints are unified into a nonlinear least squares optimization problem: ; Where λ1, λ2, and λ3 are the weight coefficients of each constraint; The Gauss-Newton method is used to iteratively update the point positions: By employing the aforementioned structural optimization method, noise and discretization errors are effectively suppressed while maintaining the original geometric structure, resulting in a smoother and more continuous point cloud in local regions and a more consistent and stable overall structure. Compared to simple filtering methods, this method significantly improves the surface quality and structural expressiveness of the point cloud through multi-constraint joint optimization, providing a more reliable data foundation for subsequent 3D reconstruction, semantic analysis, and path planning.

[0026] In this embodiment, the SLAM frequency, point cloud generation density, and processing strategy are dynamically scheduled according to the current flight status of the UAV and the mission type, as detailed below: Real-time acquisition of the drone's current flight status information, including linear velocity v t acceleration a t and attitude change ΔR t And construct the system state vector by combining the current task type (such as localization and mapping, environmental perception, or fine reconstruction): Among them, Ctask Indicates task complexity or priority; Based on this state vector, the update frequency, depth estimation density and point cloud processing strategy of the SLAM module are uniformly scheduled to achieve collaborative control among multiple modules. Construct a resource allocation and strategy selection model, and assume the scheduling decision is: ; Among them, f slam Indicates the SLAM update frequency, ρ depth λ represents the point cloud generation density. proc The point cloud processing intensity is represented by the following function; the scheduling strategy is modeled using the following function: ; ; ; Among them, f max With ρ max These are the maximum update frequency and the maximum point cloud density, respectively. The weighting coefficients are used to reduce the computational load during periods of intense motion and improve processing accuracy under stable or high-precision tasks. Through this scheduling mechanism, a dynamic balance is achieved between real-time performance, computational resources, and mapping accuracy. In high-speed flight or resource-constrained conditions, the SLAM update frequency and point cloud density are reduced to ensure stable system operation; in low-speed or high-precision scenarios, mapping resolution and processing intensity are increased, thereby enhancing environmental representation capabilities. This edge computing scheduling module effectively avoids overloading or wasting computational resources, enabling the system to achieve efficient and stable 3D perception and mapping capabilities under the limited computing power of the UAV platform.

[0027] In this embodiment, the space is divided into multiple voxel blocks, and an index structure is established for each block. During point cloud updates, only the changed regions are locally updated, as follows: The entire 3D space is divided according to a preset voxel size to form multiple cubic or cuboid voxel blocks, each with a unique spatial index identifier. The system maintains an independent data structure for each voxel block, recording the point cloud information, timestamp, number of observations, and confidence level attributes within that region. At the same time, a multi-level index structure is established, including a coarse-grained spatial index for quickly locating relevant regions and a fine-grained point index for accurately retrieving specific point cloud data. During the point cloud data update process, the system adopts an incremental local update strategy to reduce computational overhead. When new point cloud data arrives, it first quickly determines the set of voxel blocks to which it belongs through spatial indexing, and only performs data update operations on these regions that have changed. For newly added point clouds, they are directly inserted into the corresponding voxel blocks. For points that are repeatedly observed, their positions and confidence are updated using a weighted fusion method. For points that have not been observed for a long time, their weights are adjusted or they are removed according to the time decay strategy.

[0028] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0029] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0030] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0031] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0032] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A real-time 3D point cloud processing system for an UAV-borne intelligent terminal based on edge computing, characterized in that, It includes a data acquisition module, a data preprocessing and synchronization module, a visual-inertial SLAM modeling module, a 3D point cloud generation module, a point cloud processing and optimization module, an edge computing scheduling module, a point cloud data management module, and an application output module; The data acquisition module acquires continuous image sequences, angular velocity, and acceleration data transmitted by the UAV; the data preprocessing and synchronization module aligns the image data and inertial data through the time synchronization module to form a unified data input stream; the visual-inertial SLAM module processes the data input stream, constructs constraint relationships between images through feature extraction and matching, and performs state estimation in conjunction with IMU pre-integration results to obtain the UAV's pose information at the current moment. Under the keyframe mechanism, the pose is globally consistently adjusted through the sliding window optimization method, thereby outputting a continuous pose trajectory and sparse map structure. The 3D point cloud generation module performs depth estimation based on pose trajectory and sparse map structure, uses binocular matching to predict the depth value corresponding to each pixel, and back-projects the 2D pixel coordinates to 3D space through the camera intrinsic parameter matrix to generate an initial 3D point cloud. The point cloud processing and optimization module is used to perform global fusion and structural optimization of the 3D point cloud to obtain an optimized 3D point cloud. The edge computing scheduling module is used to coordinate the resource allocation and execution strategies of each computing module, and dynamically schedules the SLAM frequency, point cloud generation density and processing strategy according to the current flight status of the UAV and the task type. The point cloud data management module is used for structured storage and incremental updates of 3D point clouds. It divides the space into multiple voxel blocks and establishes an index structure for each block. When updating the point cloud, it only performs local updates on the areas that have changed.

2. The real-time 3D point cloud processing system for an UAV-borne intelligent terminal based on edge computing according to claim 1, characterized in that, The process involves constructing constraint relationships between images through feature extraction and matching, and then combining this with IMU pre-integration results to perform state estimation, thereby obtaining the UAV's pose information at the current moment, as detailed below: After receiving synchronized image data and inertial data, the visual-inertial SLAM modeling module performs feature extraction processing on the current image frame. It uses feature operators with scale invariance and rotation invariance to process the image and obtain a set of feature points. ； in, Let represent the i-th feature point in the t-th frame of the image; subsequently, feature matching is performed between adjacent image frames to establish the correspondence between feature points: ； in, Let j be the j-th feature point in the (t-1)-th frame of the image; Based on the above matching relationship, visual constraints are constructed using reprojection errors: ； in, P represents the pixel coordinates of the observed feature points. i Let T be the coordinates of a point in space. t For the current pose, π(·) represents the camera projection model; Pre-integration is performed on the synchronously input inertial measurement data. Let ω and a be the angular velocities and accelerations measured by the IMU between times t-1 and t in two adjacent image frames. The pre-integrated quantity is then obtained through integration: ； ； ； Among them, b g With b a These represent the biases of the gyroscope and accelerometer, respectively; ΔR t ω represents the rotation increment from t-1 to t; ∏ represents the chain multiplication operation; exp(·) is the outlier exponential mapping; k b is the angular velocity measured by the IMU at time k; g Δt is the gyroscope bias; Δv is the time interval; Δt is the gyroscope bias. t For speed increment R k Let a be the rotation matrix at time k; k Linear acceleration measured by the IMU; Δp t For position increment; v k R is the velocity at time k; k Let a be a rotation matrix; k This is an acceleration measurement; Based on the above pre-integration results, the inertial constraint error term e is constructed. imu : ； Among them, R t ,R t-1 The rotation matrices for times t and t-1 are respectively; v t ,v t-1 The velocity vectors at times t and t-1 are respectively; p t ,p t-1 are the position vectors at times t and t-1, respectively; g is the gravitational acceleration vector; Visual constraints and inertial constraints are integrated to construct a unified state estimation optimization model; the system state variables are defined as follows: ； The overall optimization objective function is then expressed as: ； Where, Σ v With Σ i These are the covariance matrices for visual and inertial errors, respectively. This refers to visual reprojection error; By performing nonlinear optimization on the above objective function, the optimal pose estimation result T of the UAV at the current moment is obtained. t This enables tight coupling and fusion positioning of visual and inertial information.

3. The real-time 3D point cloud processing system for an UAV-borne intelligent terminal based on edge computing according to claim 2, characterized in that, Under the keyframe mechanism, a sliding window optimization method is used to adjust the pose globally for consistency, thereby outputting a continuous pose trajectory and a sparse map structure, as detailed below: During pose estimation, the visual-inertial SLAM modeling module uses a keyframe-based sliding window optimization strategy to continuously estimate and update the system state. It performs keyframe filtering on the continuously input image sequence. When the disparity between adjacent frames exceeds a preset threshold, the current image frame is selected as a keyframe and added to the optimization window; otherwise, it is used as an ordinary frame only for auxiliary tracking and does not participate in global optimization. When constructing a sliding window, let the window contain N keyframes, its state variable be X, and the set of spatial feature points be P; A joint optimization objective function is constructed, which unifies the modeling of visual reprojection error and IMU pre-integration error. The overall optimization objective is expressed as: ； Where ρ(·) is the robust kernel function; ∑ i,j This represents the summation of observations of all image frames and feature points; ∑ k Accumulate across all IMU time periods; This refers to visual reprojection error; For IMU residuals; It is the square of the L2 norm; Visual reprojection error is defined as: ； Among them, z i,j Let π(·) represent the pixel coordinates of the j-th feature point observed in the i-th frame, and let π(·) represent the camera projection model. The IMU pre-integration error is defined as: ； in, and For IMU pre-integration terms; Using the above objective function, the Gauss-Newton algorithm is used to iteratively solve the state variables, thereby obtaining the consistent pose estimation results of each keyframe within the current sliding window.

4. The real-time 3D point cloud processing system for an UAV-borne intelligent terminal based on edge computing according to claim 1, characterized in that, The depth estimation based on pose trajectory and sparse map structure is performed. Binocular matching is used to predict the depth value corresponding to each pixel, and the two-dimensional pixel coordinates are back-projected to three-dimensional space through the camera intrinsic parameter matrix to generate initial three-dimensional points, as follows: Adjacent keyframe images are selected as input, and the images are aligned in rows by epipolar constraints. Pixel-level matching is then performed to calculate the disparity map. For each pixel (u,v), the disparity value d(u,v) is obtained by searching for matching points in the corresponding image pair, thereby establishing the correspondence between the two-dimensional image and spatial depth. During depth calculation, the 3D point cloud generation module, based on a binocular geometric model, converts disparity values ​​into depth information. Let the baseline length of the binocular camera be b and the focal length be f, then the depth value Z(u,v) of a pixel is represented as: ； After obtaining the depth map, and combining it with the camera intrinsic parameter matrix K, the two-dimensional pixel coordinates are back-projected into the three-dimensional spatial coordinate system to obtain the corresponding three-dimensional points: ； Simultaneously, the pose T output by visual inertial SLAM is utilized. t The 3D points in the local camera coordinate system are transformed to the global coordinate system to achieve a unified representation of point clouds across multiple frames, resulting in an initial 3D point cloud. ； Through depth estimation and coordinate back projection, the system can recover continuous 3D point cloud data from 2D images and fuse it with pose trajectory to form an initial 3D point cloud structure.

5. The real-time 3D point cloud processing system for an UAV-borne intelligent terminal based on edge computing according to claim 1, characterized in that, The global fusion of the initial 3D point cloud is performed as follows: The initial 3D point cloud P world With historical global point cloud set P global The points are then merged to obtain the updated global point cloud: P global P global ∪P world ; In the point cloud fusion process, a sparse-dense hybrid mapping strategy is introduced, dividing the point cloud into keyframe regions and non-keyframe regions. For keyframe regions, dense reconstruction is performed to preserve detailed information, and its point cloud density function is expressed as: ； Where, N d Let V be the number of dense points and V be the spatial volume; for non-critical regions, only the selected sparse feature points are retained, and their density is expressed as: ； The system defines the density scaling factor λ: ； And based on the drone's speed v, angular velocity ω, and scene texture intensity This ratio can be dynamically adjusted. ； Where α and β are weighting coefficients.

6. The real-time 3D point cloud processing system for an UAV-borne intelligent terminal based on edge computing according to claim 5, characterized in that, The structural optimization is as follows: Let the merged point cloud be: ； By constructing an adjacency graph The set of edges Determined by spatial proximity: ； This establishes the local topology of the point cloud; In the structural optimization process, multi-constraint joint modeling is introduced, including smoothing constraints. Normal consistency constraint and structural retention constraints : ； ； ； in, This represents the initial distance between points; Furthermore, planar constraints are introduced to enhance structural representation if the point set belongs to the same local plane. Then we have: ； Where n is the normal vector of the plane; T denotes transpose; and d is the signed distance from the plane to the origin. The above constraints are unified into a nonlinear least squares optimization problem: ； Where λ1, λ2, and λ3 are the weight coefficients of each constraint; The Gauss-Newton method is used to iteratively update the point positions: 。 7. The real-time 3D point cloud processing system for an UAV-borne intelligent terminal based on edge computing according to claim 1, characterized in that, The SLAM frequency, point cloud generation density, and processing strategy are dynamically scheduled based on the current flight status of the UAV and the mission type, as detailed below: Real-time acquisition of the drone's current flight status information, including linear velocity v t acceleration a t and attitude change ΔR t And construct a system state vector based on the current task type: ； Among them, C task Indicates task complexity or priority; Based on this state vector, the update frequency, depth estimation density and point cloud processing strategy of the SLAM module are uniformly scheduled to achieve collaborative control among multiple modules. Construct a resource allocation and strategy selection model, and assume the scheduling decision is: ； Among them, f slam Indicates the SLAM update frequency, ρ depth λ represents the point cloud generation density. proc The point cloud processing intensity is represented by the following function; the scheduling strategy is modeled using the following function: ； ； ； Among them, f max With ρ max These are the maximum update frequency and the maximum point cloud density, respectively. These are the weighting coefficients.

8. The real-time 3D point cloud processing system for an UAV-borne intelligent terminal based on edge computing according to claim 1, characterized in that, The process involves dividing the space into multiple voxel blocks and establishing an index structure for each block. During point cloud updates, only the regions that have changed are locally updated, as detailed below: The entire 3D space is divided according to a preset voxel size to form multiple cubic or cuboid voxel blocks, each with a unique spatial index identifier. The system maintains an independent data structure for each voxel block, recording the point cloud information, timestamp, number of observations, and confidence level attributes within that region. At the same time, a multi-level index structure is established, including a coarse-grained spatial index for quickly locating relevant regions and a fine-grained point index for accurately retrieving specific point cloud data. During the point cloud data update process, the system adopts an incremental local update strategy to reduce computational overhead. When new point cloud data arrives, it first quickly determines the set of voxel blocks to which it belongs through spatial indexing, and only performs data update operations on these regions that have changed. For newly added point clouds, they are directly inserted into the corresponding voxel blocks. For points that are repeatedly observed, their positions and confidence are updated using a weighted fusion method. For points that have not been observed for a long time, their weights are adjusted or they are removed according to the time decay strategy.

Images