Unmanned aerial vehicle marshalling dynamic human body real-time three-dimensional reconstruction method based on 3D Gaussian sputtering

Abstract

The invention relates to an unmanned aerial vehicle marshalling real-time human body dynamic three-dimensional reconstruction method based on 3D Gaussian sputtering, and belongs to the technical field of computer vision and unmanned aerial vehicle cooperation. The method comprises the following steps: deploying an unmanned aerial vehicle group carrying an RGB-D camera, constructing a positive chain communication network and carrying out parameter calibration; predicting a human body motion trend through an adaptive Gaussian distribution model, dividing an observation area and optimizing an unmanned aerial vehicle path; incremental parameter fusion is carried out by using a gradient sharing fusion algorithm, and local dense point cloud is generated in real time; constructing an unsupervised space-time segmentation model, and distinguishing a static background and a dynamic human body region; designing a 3D Gaussian reweighting algorithm to optimize the point cloud density and enhance the integrity of the occlusion area; gaussian parameters are optimized through a differentiable rendering framework, and a high-precision surface grid is generated; and in combination with denoising and repairing of a cascaded filter network, real-time three-dimensional reconstruction of the dynamic human body is realized. The method remarkably improves the reconstruction speed and the integrity of the shielding area, and is suitable for the fields of emergency rescue, film and television production and the like.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and unmanned aerial vehicle (UAV) collaborative computing technology, and specifically relates to a real-time three-dimensional reconstruction method for a dynamic human body in a UAV formation based on 3D Gaussian sputtering, which is suitable for motion capture, security monitoring and medical rehabilitation in complex scenarios. Background Art

[0002] Existing 3D reconstruction technologies are mainly divided into two categories: those based on traditional structure from motion (SfM) and deep learning (such as NeRF), but they have significant limitations in dynamic scenes, real-time performance, and multi-device collaboration. Traditional SfM methods (such as COLMAP) rely on static scenes, take a long time to calculate, and are sensitive to occlusion, and cannot meet the needs of real-time reconstruction of moving human bodies. Although deep learning-based methods (such as NeRF) can generate high-quality 3D models, they require extremely high computing resources and are difficult to deploy on UAV platforms with limited computing power. In addition, existing UAV 3D reconstruction solutions (such as LiDAR scanning or multi-UAV SLAM) generally have problems such as high cost, high communication overhead, and poor dynamic adaptability. For example, LiDAR equipment is expensive and affects the endurance of UAVs, while multi-UAV collaborative SLAM systems perform poorly in terms of data fusion efficiency and reconstruction accuracy in low-texture areas.

[0003] In recent years, 3D Gaussian splatting technology has garnered attention for its efficient rendering capabilities. However, its application remains limited to static scenes and single-device environments, making it inadequate for the real-time dynamic reconstruction requirements of drone swarms. Specific challenges include: inefficient Gaussian model computation due to insufficient computing power on mobile devices; model tearing during the reconstruction of moving objects; and the lack of mechanisms for synchronizing and fusing data from multiple drones.

[0004] Therefore, this invention aims to overcome these limitations. By leveraging lightweight Gaussian parameter initialization, a distributed fusion algorithm, and a dynamic Gaussian density adjustment mechanism, it achieves the first real-time, high-precision 3D reconstruction of the human body using a swarm of drones in complex dynamic scenes. Compared to existing solutions, this invention significantly reduces communication bandwidth requirements (reducing the amount of transmitted data by a factor of 40), improves reconstruction speed (≥15 FPS), and supports real-time tracking of moving targets. This approach fills a technological gap in this field and has significant application value in emergency rescue, film and television production, smart city security, and other fields. Summary of the Invention

[0005] The technical problem addressed by this invention is to overcome the problems of poor spatiotemporal synchronization among multiple devices, insufficient real-time 3D reconstruction, low accuracy in capturing dynamic human motion, and extreme imbalance in data distribution between occluded and unoccluded areas in existing drone collaborative data collection. This paper proposes a method for real-time 3D reconstruction of dynamic human bodies using drone formations based on 3D Gaussian sputtering. This method utilizes dynamic multi-drone formation control, 3D Gaussian point cloud representation optimization, and an unsupervised spatiotemporal segmentation algorithm to achieve full-angle synchronized capture of dynamic human targets and millisecond-level high-precision 3D reconstruction, addressing the challenges of motion blur, occlusion, and uneven data distribution in complex scenes.

[0006] In order to solve the above technical problems, the present invention provides a real-time 3D human body reconstruction method based on a 3D Gaussian positive chain drone group, comprising the following steps:

[0007] Step S1: Deploy multiple drones equipped with RGB-D cameras and build a distributed communication framework based on the 5G-Mesh network to achieve clock synchronization and data sharing between drones, providing a real-time data transmission foundation for subsequent steps;

[0008] Step S2: Based on the communication framework constructed in step S1, the motion trend is predicted through the 3D Gaussian distribution parameters of the initial human posture, the observation area is divided, and the obstacle avoidance path and view coverage strategy of the UAV are dynamically planned to ensure the collaborative collection of multi-view data;

[0009] Step S3: Based on the observation path planned in step S2, multi-source RGB-D data are collected synchronously, local dense point clouds are generated through spatiotemporal alignment, and compressed and transmitted to the central server to provide input for subsequent segmentation and optimization;

[0010] Step S4: For the local point cloud generated in step S3, an unsupervised spatiotemporal segmentation model is constructed. The static background and dynamic human body regions are separated by a multi-scale Gaussian mixture model. The human body motion sub-regions are subdivided by temporal coherence analysis to form classification data to support density optimization.

[0011] Step S5: Based on the segmentation results of step S4, a 3D Gaussian reweighting algorithm is designed to adjust the covariance matrix of the dynamic area, enhance the point cloud density of the occluded area, and suppress overfitting in non-critical areas, thereby improving the subsequent rendering accuracy;

[0012] Step S6: Input the point cloud optimized in step S5 into the differentiable rendering framework, jointly optimize the Gaussian parameters to generate a surface mesh, and eliminate motion artifacts through spatiotemporal consistency constraints to provide a basic model for geometric restoration;

[0013] Step S7: For the surface mesh generated in step S6, a cascaded 3D Gaussian filter network is used to fuse the spatial kernel and the temporal LSTM module to repair the geometric loss caused by motion and output a high-integrity reconstruction result;

[0014] Step S8: Generate a semantic heat map based on the restoration result of step S7, identify human joints, and adjust the drone's posture and acquisition frequency in real time to form a closed-loop control to optimize subsequent data acquisition;

[0015] Step S9: Using the time-series point cloud sequence from steps S3 to S8, a Gaussian hidden Markov model is constructed for classification to achieve motion stage segmentation and anomaly detection, providing a basis for semantic module segmentation;

[0016] Step S10: Based on the motion stage labels of step S9, the reconstructed model of step S7 is segmented by semantic modules, and low-latency interaction and multi-view visualization are achieved through parallel transmission to the terminal to complete the dynamic reconstruction closed loop.

[0017] Preferably, step S2 specifically includes: dividing the observation area by the Voronoi-FEM algorithm, quantifying the motion uncertainty according to the joint point covariance matrix, and dynamically allocating the drone coverage density; allocating multiple drones to the high uncertainty area and allocating a single drone to the low uncertainty area.

[0018] Preferably, in step S4, the unsupervised spatiotemporal segmentation model includes: constructing a multi-scale Gaussian mixture model in the coarse segmentation stage, classifying the points into static background, dynamic human body and noise according to the point cloud density and motion speed; and dividing the limb and torso sub-regions by the temporal coherence threshold in the fine segmentation stage.

[0019] Preferably, in step S6, the differentiable rendering framework accelerates Gaussian parameter query through a sparse voxel hash table, and uses stochastic gradient descent to online optimize point cloud position, covariance and opacity.

[0020] Preferably, in step S7, the cascade filtering network smoothes the noise through a spatial domain Gaussian kernel, and combines a temporal LSTM module to enhance motion coherence, focusing on repairing geometric defects in joints.

[0021] Preferably, in step S8, a semantic heat map is generated based on the dynamic Gaussian distribution entropy, and the pitch angle and focal length of the drone are adjusted in real time through a PID controller to optimize the data acquisition frame rate.

[0022] Preferably, in step S9, the Gaussian hidden Markov model uses the spatial Gaussian statistics as a priori, detects abnormal postures through a Mahalanobis distance threshold, and outputs a motion stage label.

[0023] The algorithm innovations and advantages adopted by the present invention are as follows:

[0024] 1. In the second step, to address dynamic path planning and data distribution imbalance, an adaptive formation algorithm based on 3D Gaussian motion prediction is proposed. This algorithm quantifies motion uncertainty using the Gaussian distribution covariance matrix of human joints, driving the drone swarm to dynamically adjust observation density, achieving high-priority coverage of occluded areas and optimal resource allocation in non-critical areas.

[0025] 2. In the fourth step, we designed an unsupervised two-stage segmentation framework based on a 3D Gaussian mixture model (GMM) to address the class imbalance problem in dynamic point cloud segmentation. This framework introduces the Mahalanobis distance weighted KL divergence as a segmentation criterion for the first time. While preserving the geometric accuracy of the point cloud, it also incorporates RGB semantic information, addressing the occlusion sensitivity of traditional clustering algorithms.

[0026] 3. In the sixth step, a differentiable 3D Gaussian optimization architecture was proposed to overcome the computational bottleneck of real-time rendering. This architecture accelerates Gaussian parameter lookups using a sparse voxel hash table and utilizes stochastic gradient descent to optimize point cloud attributes online, achieving a reconstruction rate of 30 frames per second, a threefold improvement over traditional NeRF methods.

[0027] The core features of the present invention are:

[0028] By tightly coupling 3D Gaussian modeling with drone collaborative observation, the problems of single-view occlusion and motion blur are solved.

[0029] Adopting a three-stage processing chain of "Gaussian reweighting - spatiotemporal segmentation - covariance optimization" significantly improves data integrity and reconstruction accuracy in occluded areas;

[0030] Supports real-time high-fidelity reconstruction under markerless and low-light conditions, suitable for dynamic scenarios such as security monitoring and medical rehabilitation;

[0031] The algorithm is implemented through a mixture of C++ / CUDA and Python, integrating the ROS2.0 framework and the PyTorch3D library, and calling OpenGL and PCL for acceleration and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present invention will be further described below with reference to the accompanying drawings.

[0033] Figure 1 These are 2D photos of the human body from different angles taken collaboratively by a group of positive chain drones.

[0034] Figure 2 The 2D semantic segmentation mask and segmentation results of the human body: (a), (d), and (g) are shots of different body postures; (b), (e), and (h) are semantic segmentation masks; (c), (f), and (i) are segmented human body instance result images.

[0035] Figure 3It is a 3D human body white film generated using 3D Gaussian technology.

[0036] Figure 4 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0037] The present invention provides a real-time three-dimensional reconstruction method of a human body based on a 3D Gaussian positive chain drone group, comprising the following steps:

[0038] The first step is to deploy a multi-UAV system, with each UAV equipped with a depth sensor (RGB-D camera), initialize the spatiotemporal synchronization protocol, and build a distributed communication framework based on the 5G-Mesh network to achieve sub-millisecond clock synchronization and data stream sharing.

[0039] In the second step, based on the initial posture of the target human body, the adaptive 3D Gaussian distribution model (AGMM) is used to predict the human body's motion trend. The improved Voronoi-FEM algorithm is combined to divide the drone's observation area and generate a dynamic obstacle avoidance path and optimal view coverage strategy.

[0040] The third step is to start the synchronous collection of multi-source data, perform spatiotemporal alignment of RGB-D images through edge computing nodes, use 3D Gaussian Splatting technology to generate local dense point clouds in real time, and compress and transmit them to the central server.

[0041] The fourth step is to construct an unsupervised spatiotemporal segmentation model based on 3D Gaussian (GUSS-Net). In the first stage, the point cloud is coarsely segmented through a multi-scale Gaussian mixture model (GMM) to distinguish between static background and dynamic human areas; in the second stage, temporal coherence analysis is combined to further separate the human body motion sub-areas (such as limbs and torso).

[0042] Step 5. Aiming at the spatiotemporal non-uniform distribution characteristics of dynamic human body point clouds, a 3D Gaussian reweighting algorithm (GRW) is designed to adaptively adjust the point cloud density through the covariance matrix, enhance the reconstruction integrity of occluded areas, and suppress overfitting in non-critical areas.

[0043] Step 6: Deploy a real-time 3D Gaussian optimization engine on the central server. Based on the differentiable Gaussian rendering framework, jointly optimize the point cloud position, covariance, and opacity parameters to generate a human body surface mesh with millimeter-level accuracy, and eliminate motion artifacts through spatiotemporal consistency constraints.

[0044] Step 7: Use a cascaded 3D Gaussian filter network (CGF-Net) to integrate the spatial Gaussian kernel and the temporal LSTM module to denoise and enhance the details of the reconstructed model, focusing on repairing geometric losses caused by rapid movement in the joints.

[0045] Step 8: Generate a semantic heat map based on the dynamic 3D Gaussian distribution, identify key joints of the human body, and adjust the drone's focal length and posture in real time through the PID controller to optimize the data collection frequency and visual coverage redundancy.

[0046] Step 9. Design a lightweight Gaussian hidden Markov model (GHMM), use the spatial Gaussian statistical parameters as prior information, classify the temporal point cloud subsequences, and achieve motion stage segmentation and abnormal posture detection.

[0047] Step 10: Build a distributed rendering pipeline and use GPU-accelerated 3D Gaussian projection technology to transmit the reconstructed model in parallel to the AR / VR terminal according to semantic modules (head, hands, feet, etc.), supporting low-latency interaction and multi-view visualization.

[0048] In this example, the author uses dynamic formation control of multiple drones to capture 2D images simulating individual motion and occlusion. Then, through 3D Gaussian point cloud representation optimization and unsupervised spatiotemporal segmentation algorithms, they achieve full-angle synchronous capture and millisecond-level high-precision 3D reconstruction of dynamic human targets. The experimental scene is an outdoor natural light environment. The hardware configuration includes a drone cluster consisting of three assembled drones (Intel RealSense D455 RGB-D cameras with 5G modules), an NVIDIA Jetson AGX Orin edge computing unit (32GB of memory), a compute node server (4090 GPU), and the software framework ROS2Humble, PyTorch3D, and CUDA 11.7.

[0049] The present invention will be further explained below with reference to the embodiments and accompanying drawings.

[0050] This embodiment provides a method for real-time 3D reconstruction of a human body based on a 3D Gaussian positive chain drone group. The specific implementation steps are detailed as follows:

[0051] Step 1: Initialize time-space synchronization and achieve sub-microsecond clock synchronization between drones based on IEEE 1588v2 protocol, with synchronization error ≤ 0.2ms. Build a 5G-Mesh ad hoc network, test bandwidth ≥ 800Mbps, and end-to-end delay < 5ms ( Figure 2 network topology).

[0052] Step 2: Adaptive observation path planning, using the improved OpenPose algorithm to detect 17 joint points in the first frame RGB-D point cloud, and output the 3D Gaussian distribution parameters of the joint points (mean Covariance ) to initialize the Gaussian modulus of the human joints. Based on the Voronoi-FEM, the observation area is divided and the workspace is discretized into 5000 tetrahedral units. Then, the drones are dynamically allocated according to the uncertainty of the joints, specifically in the high uncertainty area (such as the hand Σj >0.1) to allocate 3 drones, low uncertainty areas (such as the torso Σ j <0.01) and allocated 1 drone.

[0053] Step 3: Synchronous multi-source data acquisition and preprocessing: RGB-D images (30fps, 640×480 resolution) and IMU poses (100Hz, accuracy ±0.1°) are acquired in real time from the LiDAR point cloud (10Hz, 50,000 points per frame). The ICP algorithm is used to achieve point cloud registration (error <2mm), and spatiotemporal alignment is performed by linear interpolation to compensate for hardware delays.

[0054] Step 4: 3D Gaussian splash real-time reconstruction, for each frame point cloud, fitting 3D Gaussian ellipsoid (position μ I , covariance Σ I , opacity α I ), its initial parameters: Σ j =diag(0.01,0.01,0.01),α I =0.8, use differentiable rendering to optimize Gaussian parameters, formula: Among them I p is the observed RGB value, is the rendering value, and λ = 0.1 is used to integrate the global model.

[0055] Step 5: Unsupervised spatiotemporal segmentation (GUSS-Net), constructing a multi-scale Gaussian mixture model (GMM) (K=3) in the coarse segmentation stage, with a point cloud density >100pts / cm 3 The static background class, the dynamic human class with a speed greater than 0.5 m / s, and the isolated point clusters are classified as occlusion noise. During the fine segmentation phase, temporal coherence analysis is performed on the dynamic human class: if the coherence is less than 0.3, it is classified as a limb sub-region, and vice versa, it is classified as a torso sub-region.

[0056] Step 6: 3D Gaussian reweighting (GRW) optimization is used to enhance the rendering of the occluded area. Its covariance is expanded to Σ1=Σ1+βΙ (β=0.05), and the weight is updated to Also delete the Gaussian ellipsoids with ω1 < 0.1.

[0057] Step 7: Real-time mesh generation and optimization, generate surface mesh from Gaussian point cloud through Marching Cubes algorithm, through spatiotemporal consistency: adjacent frame vertex displacement penalty: Eliminate motion artifacts.

[0058] Step 8: Motion capture and drone control, based on dynamic 3D Gaussian (distribution entropy Η (Σ j )=-log(det(Σ j)))Generate semantic heat maps, identify key nodes of the human body, and adjust the pitch angle of the drone through PID control signals (response time <50ms). Its dynamic frame rate is distributed as 60Hz in the hand area and 30Hz in the torso area.

[0059] Step 9: Temporal point cloud classification and motion segmentation, training lightweight GHMM model, input: subsequence point cloud set S after spatiotemporal segmentation = {S u ,S tv ,S fc}(where S u is a static background, S tv is the dynamic interference, S fc is human motion), the spatial Gaussian statistics is used as a priori (mean μ fc =[0.2,0.5,0.3] T ) covarianceΣ fc =diag(0.1,0.15,0.08)), the state transition probability matrix is ​​iteratively optimized by the EM algorithm (convergence threshold ∈ = 1e -4 ), output: motion stage label (such as "walking" or "jumping") and anomaly detection results. The Mahalanobis distance D M >2.5 is considered abnormal.

[0060] Step 10: Distributed Rendering and Interactive Transmission. The human body is divided into five modules based on joint topology (head, torso, upper limbs × 2, and lower limbs × 2). 3D Gaussian point cloud projections (resolution 4096 × 2160 @ 60 Hz) are generated for each module. Python code is used to implement GPU-accelerated projection. The Draco algorithm is used to compress the Gaussian parameters (compression ratio 18:1). Through the WebRTC protocol, latency is achieved on AR / VR terminals under 20 ms (measured data shown in Table 1).

[0061] Table 1 Measured data

[0062]

[0063] Table 2 Performance Verification

[0064]

[0065] Innovative technical details

[0066] Adaptive formation algorithm: Quantifying motion uncertainty through covariance matrix eigenvalue decomposition, drone coverage density and λ max Directly proportional.

[0067] GUSS-Net segmentation criteria: The Mahalanobis distance weighted KL divergence threshold is set to 0.15, which improves the segmentation accuracy by 23% compared with traditional K-means.

Claims

1. A real-time 3D reconstruction method for a dynamic human body in a UAV formation based on 3D Gaussian sputtering, characterized in that: The following steps are involved: Step S1: Deploy multiple drones equipped with RGB-D cameras and build a distributed communication framework based on the 5G-Mesh network to achieve clock synchronization and data sharing between drones, providing a real-time data transmission foundation for subsequent steps; Step S2: Based on the communication framework constructed in step S1, the motion trend is predicted through the 3D Gaussian distribution parameters of the initial human posture, the observation area is divided, and the obstacle avoidance path and view coverage strategy of the UAV are dynamically planned to ensure the collaborative collection of multi-view data; Step S3: Based on the observation path planned in step S2, multi-source RGB-D data are collected synchronously, local dense point clouds are generated through spatiotemporal alignment, and compressed and transmitted to the central server to provide input for subsequent segmentation and optimization; Step S4: For the local point cloud generated in step S3, an unsupervised spatiotemporal segmentation model is constructed. The static background and dynamic human body regions are separated by a multi-scale Gaussian mixture model. The human body motion sub-regions are subdivided by temporal coherence analysis to form classification data to support density optimization. Step S5: Based on the segmentation results of step S4, a 3D Gaussian reweighting algorithm is designed to adjust the covariance matrix of the dynamic area, enhance the point cloud density of the occluded area, and suppress overfitting in non-critical areas, thereby improving the subsequent rendering accuracy; Step S6: Input the point cloud optimized in step S5 into the differentiable rendering framework, jointly optimize the Gaussian parameters to generate a surface mesh, and eliminate motion artifacts through spatiotemporal consistency constraints to provide a basic model for geometric restoration; Step S7: For the surface mesh generated in step S6, a cascaded 3D Gaussian filter network is used to fuse the spatial kernel and the temporal LSTM module to repair the geometric loss caused by motion and output a high-integrity reconstruction result; Step S8: Generate a semantic heat map based on the restoration result of step S7, identify human joints, and adjust the drone's posture and acquisition frequency in real time to form a closed-loop control to optimize subsequent data acquisition; Step S9: Using the time-series point cloud sequence from steps S3 to S8, a Gaussian hidden Markov model is constructed for classification to achieve motion stage segmentation and anomaly detection, providing a basis for semantic module segmentation; Step S10: Based on the motion stage labels of step S9, the reconstructed model of step S7 is segmented by semantic modules, and low-latency interaction and multi-view visualization are achieved through parallel transmission to the terminal to complete the dynamic reconstruction closed loop.

2. The method according to claim 1, wherein: The step S2 specifically includes: dividing the observation area by the Voronoi-FEM algorithm, quantifying the motion uncertainty according to the joint point covariance matrix, and dynamically allocating the drone coverage density; allocating multiple drones to the high uncertainty area and a single drone to the low uncertainty area.

3. The method according to claim 1, wherein: In step S4, the unsupervised spatiotemporal segmentation model includes: constructing a multi-scale Gaussian mixture model in the coarse segmentation stage, classifying the point cloud into static background, dynamic human body and noise according to the point cloud density and motion speed; and dividing the limb and torso sub-regions by the temporal coherence threshold in the fine segmentation stage.

4. The method according to claim 1, wherein: In step S6, the differentiable rendering framework accelerates Gaussian parameter query through a sparse voxel hash table and uses stochastic gradient descent to online optimize point cloud position, covariance and opacity.

5. The method according to claim 1, wherein In step S7, the cascaded filtering network smoothes the noise through the spatial domain Gaussian kernel and combines the temporal LSTM module to enhance motion coherence, focusing on repairing the geometric loss of joints.

6. The method according to claim 1, characterized in that In step S8, a semantic heat map is generated based on the dynamic Gaussian distribution entropy, and the pitch angle and focal length of the drone are adjusted in real time through a PID controller to optimize the data acquisition frame rate.

7. The method according to claim 1, characterized in that In step S9, the Gaussian hidden Markov model uses the spatial Gaussian statistics as a priori, detects abnormal postures through a Mahalanobis distance threshold, and outputs a motion stage label.

Images