An unmanned aerial vehicle end-to-end navigation model training data generation and closed-loop simulation verification system

Abstract

This invention discloses an end-to-end navigation model training data generation and closed-loop simulation verification system for unmanned aerial vehicles (UAVs), comprising: a data acquisition module; a 3D reconstruction and 3D Gaussian splash modeling module; a coordinate unification module; a trajectory task and expert trajectory generation module, which generates expert trajectories or planned trajectories for corresponding tasks in a unified coordinate system and imposes visibility constraints on the initial point and orientation to ensure that the target is within the field of view; a 3D Gaussian splash rendering module; a command generation and dataset construction module; a strategy model training module; and a closed-loop inference and service-oriented deployment module. This invention can utilize the high-fidelity new perspective rendering capability of 3DGS 3D Gaussian splash to reduce the economic cost, time cost, and flight safety risks of data acquisition. It can also enable training data, navigation models, and simulation trajectories to be directly mapped to real geographical scenes, significantly improving the transferability and adaptability of the model in real-world scenarios such as parks and cities.

Description

Technical Field

[0001] This invention relates to the field of simulation and verification technology for autonomous navigation models of unmanned aerial vehicles (UAVs), specifically to an end-to-end navigation model training data generation and closed-loop simulation and verification system for UAVs. Background Technology

[0002] Autonomous navigation technology for unmanned aerial vehicles (UAVs) is the core of UAV intelligent applications, with wide application needs in civilian fields such as park inspection, building surveying, security monitoring, and emergency rescue, as well as related industrial scenarios. End-to-end navigation models, as the core carrier of autonomous flight for UAVs, can directly use visual observation, status information, and voice commands as inputs and outputs for flight control actions, significantly simplifying the modular design process of traditional navigation systems and becoming a research and application hotspot in the field of UAV intelligent control. However, the performance of end-to-end navigation models depends on large-scale, high-fidelity, and multi-scenario training data, and requires a reliable simulation verification environment to complete the closed-loop iteration and performance evaluation of the model. Therefore, the efficient generation of training data and the construction of a simulation verification system have become key bottlenecks for the implementation of end-to-end navigation technology for UAVs.

[0003] In existing technologies, the acquisition of training data and simulation verification of end-to-end navigation models for UAVs typically employ imitation learning and reinforcement learning methods based on 2D images or videos to train UAV navigation strategy models. The model input consists of 2D images from the camera and UAV state variables, while the output is flight control commands or the state at the next moment. Although this approach avoids the complex process of 3D scene modeling, it relies solely on 2D data and lacks controllable 3D geometric constraints. During trajectory planning, it is difficult to guarantee the visibility and full-view coverage of the target object, easily leading to problems such as target occlusion and missing viewpoints. At the same time, this approach usually uses fixed task category labels (such as inspection, orbiting) or manually written script-based commands in task descriptions, which have limited command expression capabilities and cannot accurately constrain fine-grained visual targets such as the relative orientation of the target object in the image and the target's proportion in the image. This makes it difficult to meet the refined flight requirements of scenarios such as building inspection and precise surveying. Furthermore, the training data and models of this approach lack binding and unified representation with real geographic coordinates, making it difficult to directly transfer the trained model to large-scale geographic scenes such as real parks and cities, resulting in high scene transfer costs and poor adaptability. Summary of the Invention

[0004] The purpose of this invention is to provide an end-to-end navigation model training data generation and closed-loop simulation verification system for unmanned aerial vehicles (UAVs). This system can utilize the high-fidelity new perspective rendering capabilities of 3DGS three-dimensional Gaussian splashing to reduce the economic and time costs of data acquisition and flight safety risks. It can also enable training data, navigation models, and simulation trajectories to be directly mapped to real geographical scenes, significantly improving the transferability and adaptability of the model in real scenes such as parks and cities, thereby solving the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an end-to-end navigation model training data generation and closed-loop simulation verification system for unmanned aerial vehicles, comprising: a data acquisition module, which acquires visible light images and positioning information, and outputs image sequences and corresponding positioning data;

[0006] The 3D reconstruction and 3D Gaussian splash modeling module performs 3D reconstruction on images to obtain 3D models or dense point clouds, camera intrinsic parameters and camera pose, and further transforms or fits to generate a 3D Gaussian splash scene representation.

[0007] The coordinate unification module unifies the positioning data, camera pose, and 3D scene into the same geographic coordinate system corresponding to the metric plane coordinate system, and obtains the landmark / target coordinates and camera pose in the unified coordinate system.

[0008] The trajectory task and expert trajectory generation module generates expert trajectories or planned trajectories for the corresponding tasks in a unified coordinate system, and imposes visibility constraints on the initial point and orientation to ensure that the target is within the field of view.

[0009] The 3D Gaussian splash rendering module renders the 3D Gaussian splash scene from a new perspective based on the camera intrinsic parameters and camera pose, and obtains a visible light image sequence that corresponds one-to-one with the trajectory points.

[0010] The instruction generation and dataset construction module binds image sequences, UAV state sequences, and task instructions to build a multimodal trajectory dataset for supervised learning or reinforcement learning.

[0011] The strategy model training module trains an end-to-end navigation strategy model for UAVs based on a multimodal trajectory dataset and outputs model parameters for executable flight control actions.

[0012] The closed-loop inference and service-oriented deployment module completes the closed-loop verification, evaluation, and engineering deployment of the UAV end-to-end navigation strategy model in a simulation environment, forming an iterative process of training, verification, and retraining.

[0013] Preferably, the data acquisition module, the 3D reconstruction and 3D Gaussian splash modeling module, the coordinate unification module, the trajectory task and expert trajectory generation module, the 3D Gaussian splash rendering module, the instruction generation and dataset construction module, the strategy model training module, and the closed-loop inference and service-oriented deployment module are all equipped with data interface units for transmitting data such as acquired images, positioning information, camera intrinsic parameters, camera pose, trajectory points, instruction text, and training samples.

[0014] Preferably, the coordinate unification module includes a coordinate reference unit for storing and outputting the coordinates of landmarks / buildings, UAV status coordinates, and camera pose coordinates in a unified coordinate system.

[0015] Preferably, the coordinate unification module uses any of the following methods to unify the positioning data, camera pose, and 3D scene into the same geographic coordinate system:

[0016] A1. Directly project latitude, longitude, and altitude onto the UTM plane coordinates and combine them with elevation to form the coordinates.

[0017] A2. First transform to ECEF coordinates, then transform to the local ENU coordinate system, and finally align with UTM coordinates.

[0018] Preferably, the 3D Gaussian splash rendering module includes a rendering service unit for receiving camera intrinsic parameters and camera pose, and generating a visible light image from a corresponding viewpoint based on the 3D Gaussian splash scene.

[0019] Preferably, when generating the visible light image, the three-dimensional model or dense point motion is further converted or fitted into a 3DGS scene representation. In the rendering stage, given the camera intrinsic parameter matrix and camera pose, the pixels are back-projected into camera rays according to the pinhole imaging model, and rasterization or volume rendering-style Gaussian accumulation is performed in the 3DGS scene representation to generate a new perspective RGB image that is close to the actual acquisition distribution.

[0020] Preferably, the original image needs to be preprocessed to remove distortion before the visible light image is generated, or distortion is ignored in the camera model and residuals are absorbed through data enhancement during the training phase. At the same time, exposure, white balance and noise perturbation are introduced at the rendering end.

[0021] Preferably, the strategy model training module includes a strategy reasoning service unit, which adopts a vision, language and state fusion structure to receive rendered images, UAV status and task instructions, and output action control quantities or next state increments.

[0022] Preferably, the closed-loop inference and service-oriented deployment module includes a simulation execution and safety constraint unit and a data playback and evaluation unit. The simulation execution and safety constraint unit is used to limit the speed, limit the altitude, and perform collision detection safety pruning on the actions, and update the UAV state according to the discrete kinematics model, thereby forming the camera pose input for the next moment. The data playback and evaluation unit is used to statistically analyze the indicators of task success rate, minimum safe distance, and surround coverage, and write the closed-loop trajectory and image sequence back to the dataset construction module or training module to support iteration.

[0023] Preferably, the expert trajectory or planned trajectory of the corresponding task generated by the trajectory generation module includes fixed-point landing, fixed-point orbiting, adaptive building inspection orbiting, and obstacle avoidance path planning based on RRT.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] 1. This invention, through the coordinated operation of a data acquisition module, a 3D reconstruction and 3D Gaussian splash modeling module, a coordinate unification module, a trajectory task and expert trajectory generation module, a 3D Gaussian splash rendering module, an instruction generation and dataset construction module, a strategy model training module, and a closed-loop inference and service-oriented deployment module, can leverage the high-fidelity new perspective rendering capabilities of 3DGS 3D Gaussian splash to reduce the economic and time costs of data acquisition and flight safety risks. It also enables training data, navigation models, and simulation trajectories to be directly mapped to real geographical scenes, significantly improving the model's adaptability in real-world scenarios such as parks and cities.

[0026] 2. This invention unifies positioning data, camera pose, and 3D scene into the same geographic coordinate system corresponding to the metric plane coordinate system. It completes the integrated spatial binding of landmark buildings, UAV trajectory points, camera intrinsic / extrinsic parameters, 3D modeling data, and rendering results in the metric plane coordinate system. This allows training data, navigation models, and simulation trajectories to be directly mapped to real geographic scenes, greatly improving the transferability and adaptability of the model in real scenes such as parks and cities, and facilitating the reproduction and verification of multi-scene experiments. Attached Figure Description

[0027] Figure 1 This is a system block diagram of the present invention;

[0028] Figure 2 This is a schematic diagram of the coordinate unification and rendering input elements of the present invention;

[0029] Figure 3 This is a schematic diagram of the four types of trajectory tasks and initial point sampling of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Please see Figure 1-3This invention provides a technical solution: an end-to-end navigation model training data generation and closed-loop simulation verification system for unmanned aerial vehicles (UAVs), comprising a data acquisition module, a 3D reconstruction and 3D Gaussian splash modeling module, a coordinate unification module, a trajectory task and expert trajectory generation module, a 3D Gaussian splash rendering module, a command generation and dataset construction module, a strategy model training module, and a closed-loop inference and service-oriented deployment module. Each of these modules is equipped with a data interface unit for transmitting acquired images, positioning information, camera intrinsic parameters, camera pose, trajectory points, command text, and training samples. The data interface unit includes a geographic coordinate interface and a camera parameter interface. All modules communicate with each other through the unified geographic coordinate interface and camera parameter interface, and achieve integrated control through a closed-loop mechanism of rendering, inference, execution, and updating.

[0032] The data acquisition module acquires visible light images and positioning information, and outputs image sequences and corresponding positioning data.

[0033] The 3D reconstruction and 3D Gaussian splash modeling module performs 3D reconstruction on the image to obtain a 3D model or dense point cloud, camera intrinsic parameters and camera pose, and further transforms or fits to generate a 3D Gaussian splash scene representation.

[0034] For example, a DJI M400 drone equipped with Zenmuse L2 was used to collect visible light images and GNSS information of the park / building complex, which were then imported into DJI Terra for 3D reconstruction. The output included a 3D mesh model, camera intrinsic parameters (without distortion coefficients or with distortion preprocessing), camera pose, and GPS coordinates for each frame.

[0035] The coordinate unification module unifies the positioning data, camera pose, and 3D scene into the same geographic coordinate system corresponding to the metric plane coordinate system, thereby obtaining the landmark / target coordinates and camera pose in the unified coordinate system. The coordinate unification module includes a coordinate reference unit, which is used to store and output the landmark / building target coordinates, UAV status coordinates, and camera pose coordinates in the unified coordinate system.

[0036] The coordinate unification module uses any of the following methods to unify the positioning data, camera pose, and 3D scene into the same geographic coordinate system:

[0037] A1. Directly project latitude, longitude, and altitude onto the UTM plane coordinates and combine them with elevation to form the coordinates.

[0038] A2. First convert to ECEF coordinates, then to the local ENU coordinate system, and finally align with UTM coordinates. This unified process ensures that the same landmark building has a consistent spatial position in different data sources (such as orthophotos, 3D meshes, 3DGS, and trajectory points), which enables subsequent rendering, trajectory planning, and target labeling to be consistent with the real geographic coordinates.

[0039] like Figure 2 As shown, to uniformly describe the mapping relationship between "world coordinates - camera pose - pixel observation", this paper adopts the pinhole imaging model; let the three-dimensional point in the world coordinate system be... Its coordinates in the camera coordinate system Satisfy extrinsic parameter transformation

[0040]

[0041] in, Let be a rotation matrix. It is a translation vector.

[0042] In camera intrinsic parameter matrix With external references Given a world point, the projection relationship from the world point to the homogeneous coordinates of the pixel can be written as:

[0043] ,

[0044] in, These are the homogeneous coordinates of the pixels. The homogeneous coordinates of the world point, "" indicates equality in the homogeneous sense.

[0045] In trajectory and control modeling, the system state is represented by a four-degree-of-freedom form in a unified coordinate system, denoted as:

[0046]

[0047] in, Indicates time positional components This is the yaw angle.

[0048] The trajectory task and expert trajectory generation module generates expert trajectories or planned trajectories for the corresponding tasks in a unified coordinate system, and imposes visibility constraints on the initial point and orientation to ensure that the target is within the field of view.

[0049] The trajectory tasks and the corresponding expert trajectories or planned trajectories generated by the expert trajectory generation module include fixed-point landing, fixed-point orbiting, adaptive building inspection orbiting, and obstacle avoidance path planning based on RRT; each task defines the target point / target building and safety constraints (such as minimum safe height, minimum turning radius, maximum climb / descent rate, etc.) in a unified UTM coordinate system.

[0050] like Figure 3 As shown, in a scene, prominent landmark buildings are first segmented from a 2D top-down view (which can be an orthophoto or a rasterized heightmap), their outlines / center points are extracted, and the corresponding UTM coordinates are recorded. Then, initial points are generated by sampling near the target coordinates. The sampling radius and height bandwidth must be limited to ensure the target building is within the initial field of view. If the pre-render check shows the target is not visible, resampling or orientation adjustment is performed. The drone first rotates in place to face the target, then flies along a straight line or segmented trajectory while descending to above the target or around the entry point. After hovering and completing orientation adjustment, it enters the orbit / inspection or obstacle avoidance planning process.

[0051] To ensure that the generated expert trajectory meets flight safety and executability requirements, this invention introduces basic altitude and kinematic constraints during the trajectory generation process. For example, the flight altitude must meet certain conditions.

[0052]

[0053] in, Indicates time Flight altitude The minimum safe height threshold allowed.

[0054] Meanwhile, to limit the intensity of vertical maneuvering, the descent / climb rate must meet the following requirements.

[0055]

[0056] in, This represents the sampling time interval between adjacent moments. This is the upper limit of the maximum permissible vertical speed (climb / descent rate).

[0057] In addition, to ensure the feasibility of course changes, the yaw rate must meet the following requirements.

[0058]

[0059] in, Indicates time Yaw angle, This is the upper limit of the maximum permissible yaw rate.

[0060] For example, this invention divides expert trajectory generation into four types of tasks and specifies the corresponding input elements and trajectory generation points for each. For the fixed-point landing task, the input includes the target point coordinates, the UAV's initial point, and the camera's field of view / orientation parameters; the expert trajectory typically first performs a nose turn to align with the target, and then descends along a straight line or in segments to above the target, while continuously constraining the target to remain within the camera's field of view during the descent, while also satisfying safety constraints such as the maximum descent rate.

[0061] For fixed-point orbiting tasks, the inputs include the target point coordinates, the orbiting radius r, and the angular velocity or heading control strategy. The expert trajectory can generate a circular or elliptical orbiting path and maintain the heading continuously pointing towards the target point during flight. If necessary, it will sample and stop at several key azimuth angles to obtain more comprehensive multi-view coverage. For adaptive building inspection tasks, the inputs include the outline or edge information of the target building, the minimum safe distance d from the building, and the coverage strategy. The expert trajectory generates a "shape-following" inspection path based on the building's shape (which can be achieved through segmented fitting) and dynamically adjusts the orbiting radius and flight altitude to keep the building edge in the field of view or in the center area. When occlusion occurs or the target visibility decreases, altitude correction or lateral correction can be triggered to restore visibility and coverage.

[0062] For the RRT obstacle avoidance planning task, the input includes the starting point, the ending point, and the environment representation constructed by the obstacle grid or point cloud. During planning, a feasible region is established in three-dimensional or 2.5D space, and a collision-free feasible path is generated using RRT or RRT*. After obtaining the initial path, further smoothing and dynamic feasibility constraints are applied to ensure that it meets flight control limitations such as acceleration and curvature.

[0063] To quantitatively evaluate the quality of obstacle avoidance or inspection paths, this invention employs a path cost function to jointly measure path length and collision risk, the mathematical form of which is as follows:

[0064]

[0065] The first item is used to measure the path length, and the second item is used to penalize collision risks or violations of safe distances related to obstacles on the path. These are the weighting coefficients for the two trade-offs. Indicates the first path The location of each discrete path point Describing the vector norm, This represents a collision penalty function, such as a penalty based on the distance to the nearest obstacle or a penalty based on an occupancy / collision indication.

[0066] The 3D Gaussian splash rendering module renders the 3D Gaussian splash scene from a new perspective based on the camera intrinsic parameters and camera pose, and obtains a visible light image sequence that corresponds one-to-one with the trajectory points.

[0067] The 3D Gaussian splash rendering module includes a rendering service unit, which is used to receive camera intrinsic parameters and camera pose, and generate a visible light image of the corresponding viewpoint based on the 3D Gaussian splash scene.

[0068] During the generation of the visible light image, the 3D model or dense point motion is further transformed or fitted into a 3DGS scene representation. The 3DGS scene representation is a scene representation composed of a 3D Gaussian set. In the rendering stage, given the camera intrinsic parameter matrix K and the camera pose... Based on the pinhole imaging model, pixels are back-projected into camera rays, and rasterization or volume rendering-style Gaussian accumulation is performed in the 3DGS scene representation to generate a new perspective RGB image that is close to the actual acquisition distribution.

[0069] Before generating the visible light image, the original image needs to be preprocessed to remove distortion, or distortion needs to be ignored in the camera model. The residuals are absorbed by data enhancement during the training phase to reduce the projection error caused by the lack of distortion coefficients. At the same time, exposure, white balance and noise perturbation are introduced at the rendering end to improve the robustness to changes in real ambient lighting and imaging noise.

[0070] In terms of rendering representation, this invention views the rendering process as a 3DGS scene. Based on camera intrinsics and pose parameters, an observation generation function is generated, which can represent the rendered image at any given time as...

[0071]

[0072] in, Indicates time Rendering RGB images, This refers to the rendering operators in 3DGS representation. For the camera intrinsic parameter matrix, For a moment Camera pose parameters, This is a 3DGS scene representation.

[0073] The instruction generation and dataset construction module binds image sequences, UAV state sequences, and task instructions to construct a multimodal trajectory dataset for supervised learning or reinforcement learning.

[0074] The training data is based on a sequence of states sampled at uniform intervals along a trajectory. Each sample point contains four degrees of freedom states.

[0075]

[0076] in, Indicates time The system state vector, To unify the position components in a coordinate system.

[0077] And the RGB image rendered by 3DGS in this pose. It is then bound to the task instruction text to form a triple. Or further construct them into supervised learning samples .

[0078] For adaptive building inspection tasks, since the orientation of the target building in the first frame is uncertain, this invention proposes a "geometric annotation-projection-language description" instruction generation process: First, all building edges are manually or semi-automatically annotated in the 3D model of the park; the 3D annotations are projected onto the 2D image through the known camera pose to form prominent color marks, such as red edges; then, a pre-trained language model or rule engine identifies the relative orientation of the marks in the image, such as center, top, bottom, left / right, and four corners, and generates natural language instructions corresponding to the trajectory, such as "keep the red-marked building in the upper right of the image and inspect around the outside of the building".

[0079] To enhance generalization capabilities, instructions can include diverse synonyms and parameterized slots, such as orbit radius, flight altitude range, clockwise / counterclockwise rotation, and desired screen occupancy, and can be generated jointly through templates and language models.

[0080] From the perspective of VLA modeling, the policy model is... As input, output the corresponding action. It can be written as:

[0081] ;

[0082] The strategy model training module trains an end-to-end navigation strategy model for the UAV based on a multimodal trajectory dataset and outputs model parameters for executable flight control actions. The strategy model training module includes a strategy reasoning service unit, which adopts a vision, language and state fusion structure to receive rendered images, UAV state and mission instructions, and output action control quantities or next state increments.

[0083] During model training, an image encoder extracts visual features, a text encoder extracts instruction semantics, and a state encoder processes the four degrees of freedom states; the three are then fused to output the action. (e.g., velocity vector and yaw rate) or the state increment at the next moment. During the training phase, supervised learning (behavioral cloning) loss based on expert trajectories is used, for example... Regression loss, Huber loss, or a hybrid classification-regression loss can be used, and visibility constraint loss (to encourage targets to fall within the desired orientation area) and smoothing loss (to suppress high-frequency jitter) can be added. In another embodiment, offline reinforcement learning or model-based fine-tuning can be used on the basis of behavior cloning, and the reward function consists of terms such as "target visibility", "coverage", "safe distance", and "arrival time" to further improve stability in unseen scenes;

[0084] During the supervised learning phase, the behavioral cloning loss can be expressed as:

[0085]

[0086] in, The parameter is The policy network, with time as input. Observational images System status and text instructions The output is the predicted action; Indicates expert action tags; Represents the L2 norm; Represents the time index of the training trajectory The expected (or empirical average) value.

[0087] To suppress high-frequency jitter in motion output, a smoothing term can be introduced:

[0088]

[0089] in, For strategy at any time The output action, This refers to the action performed at the previous moment.

[0090] Therefore, the total loss can be written as:

[0091]

[0092] in, This represents the visibility constraint loss. These are the weighting coefficients for each loss term.

[0093] In the reinforcement learning or fine-tuning phase, the reward function can be written as:

[0094]

[0095] in, This represents a measure of target visibility. Represents coverage metric, Indicates the costs associated with risk / safe distance. This represents the time cost or arrival time. These are the corresponding weighting coefficients.

[0096] The closed-loop inference and service-oriented deployment module completes the closed-loop verification, evaluation and engineering deployment of the UAV end-to-end navigation strategy model in the simulation environment, forming an iterative process of training, verification and retraining.

[0097] The closed-loop reasoning and service-oriented deployment module includes a simulation execution and safety constraint unit and a data playback and evaluation unit. The simulation execution and safety constraint unit is used to limit the speed and altitude of the actions, perform collision detection and safety pruning, and update the UAV state according to the discrete kinematics model to form the camera pose input for the next moment. The data playback and evaluation unit is used to statistically analyze the indicators of task success rate, minimum safe distance, and surround coverage, and write the closed-loop trajectory and image sequence back to the dataset construction module or training module to support iteration.

[0098] Specifically, after model training is complete, closed-loop rollout inference verification can be performed in a 3DGS scene. This invention provides a lightweight service architecture: the rendererserver receives poses and returns rendered frames; the policy inference service receives... The system outputs actions; the simulator updates the action integral to obtain the next state and performs safety pruning (speed limit / height limit / collision detection). When obstacle avoidance is required, the RRT planning module can act as a safety layer, projecting the policy output into the feasible region or providing alternative paths. The above service-oriented architecture supports parallel sampling, multi-scene playback, and online evaluation metric statistics (such as task success rate, minimum obstacle distance, and surround coverage), thus forming a training-evaluation closed loop that can be rapidly iterated.

[0099] In a closed-loop rollout, state updates can be abstracted as follows:

[0100]

[0101] in, This represents the discrete-time dynamic update function of the system. This represents the sampling time interval.

[0102] For example, in the form of speed control, the discrete integral update can be written as:

[0103]

[0104] in, For the velocity components in a unified coordinate system, This is the yaw rate.

[0105] When working, follow these workflow steps:

[0106] S1. The data acquisition module acquires visible light images and positioning information of the park / building complex, and outputs the image sequence and corresponding positioning data.

[0107] S2. The 3D reconstruction and 3D Gaussian splash modeling module performs 3D reconstruction on the image to obtain the 3D model / dense point cloud, camera intrinsic parameters and camera pose, and further transforms or fits to generate a 3D Gaussian splash scene representation.

[0108] S3. The coordinate unification module unifies the positioning data, camera pose, and 3D scene into the same geographic coordinate system corresponding to the metric plane coordinate system, and obtains the landmark / target coordinates and camera pose in the unified coordinate system.

[0109] S4. The trajectory task and expert trajectory generation module generates the corresponding expert trajectory or planned trajectory in a unified coordinate system, and applies visibility constraints to the initial point and orientation to ensure that the target is within the field of view.

[0110] S5. The 3D Gaussian splash rendering module renders the 3D Gaussian splash scene from a new perspective based on the camera intrinsic parameters and camera pose, and obtains a visible light image sequence that corresponds one-to-one with the trajectory points.

[0111] S6. The instruction generation and dataset construction module binds image sequences, UAV state sequences and task instructions to construct a multimodal trajectory dataset for supervised learning or reinforcement learning.

[0112] S7. The policy model training module trains an end-to-end policy model based on the multimodal trajectory dataset;

[0113] S8. Closed-loop reasoning and service-oriented deployment modules execute closed-loop reasoning in the simulation environment: the rendering service unit outputs the current observed image, the policy reasoning service unit outputs the action, the simulation execution and safety constraint unit updates the state and generates the pose at the next moment, and iterates in a loop until the task is completed or terminated; and the closed-loop results are used for evaluation and retraining, thus forming an integrated closed loop of "acquisition - modeling - rendering - training - playback / verification";

[0114] In summary, leveraging 3DGS to achieve highly realistic new perspective rendering significantly reduces the cost of real-world flight data acquisition and annotation, and improves the effectiveness of end-to-end visual strategy training. By unifying coordinates, landmark buildings, trajectory points, camera parameters, and rendering results are integrated and bound together, facilitating cross-scene transfer and reproduction. A multi-task-oriented expert trajectory generation mechanism and an initial point visibility constraint strategy are proposed to ensure that training data covers effective perspectives and reduces invalid samples. A "geometric annotation-projection-language description" instruction generation method is proposed, enabling the strategy model to learn fine-grained visual orientation constraints, improving the controllability and interpretability of inspection tasks. Furthermore, it supports the fusion of rollout service-based inference and RRT security layer, facilitating online evaluation, simulation verification, and migration to real UAV deployment, improving the model's adaptability in real-world scenarios such as parks and cities.

[0115] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An unmanned aerial vehicle end-to-end navigation model training data generation and closed-loop simulation verification system, characterized in that: include: The data acquisition module acquires visible light images and positioning information, and outputs image sequences and corresponding positioning data. The 3D reconstruction and 3D Gaussian splash modeling module performs 3D reconstruction on images to obtain 3D models or dense point clouds, camera intrinsic parameters and camera pose, and further transforms or fits to generate a 3D Gaussian splash scene representation. The coordinate unification module unifies the positioning data, camera pose, and 3D scene into the same geographic coordinate system corresponding to the metric plane coordinate system, and obtains the landmark / target coordinates and camera pose in the unified coordinate system. The trajectory task and expert trajectory generation module generates expert trajectories or planned trajectories for the corresponding tasks in a unified coordinate system, and imposes visibility constraints on the initial point and orientation to ensure that the target is within the field of view. The 3D Gaussian splash rendering module renders the 3D Gaussian splash scene from a new perspective based on the camera intrinsic parameters and camera pose, and obtains a visible light image sequence that corresponds one-to-one with the trajectory points. The instruction generation and dataset construction module binds image sequences, UAV state sequences, and task instructions to construct a multimodal trajectory dataset for supervised learning or reinforcement learning. The strategy model training module trains an end-to-end navigation strategy model for UAVs based on a multimodal trajectory dataset and outputs model parameters for executable flight control actions. The closed-loop inference and service-oriented deployment module completes the closed-loop verification, evaluation, and engineering deployment of the UAV end-to-end navigation strategy model in a simulation environment, forming an iterative process of training, verification, and retraining.

2. The UAV end-to-end navigation model training data generation and closed-loop simulation verification system according to claim 1, characterized in that: The data acquisition module, 3D reconstruction and 3D Gaussian splash modeling module, coordinate unification module, trajectory task and expert trajectory generation module, 3D Gaussian splash rendering module, instruction generation and dataset construction module, strategy model training module, and closed-loop inference and service-oriented deployment module are all equipped with data interface units for transmitting data such as acquired images, positioning information, camera intrinsic parameters, camera pose, trajectory points, instruction text, and training samples.

3. The UAV end-to-end navigation model training data generation and closed-loop simulation verification system according to claim 2, characterized in that: The coordinate unification module includes a coordinate reference unit, which is used to store and output the coordinates of landmarks / buildings, UAV status coordinates, and camera pose coordinates in a unified coordinate system.

4. The UAV end-to-end navigation model training data generation and closed-loop simulation verification system according to claim 3, characterized in that: The coordinate unification module uses any of the following methods to unify the positioning data, camera pose, and 3D scene into the same geographic coordinate system: A1. Directly project latitude, longitude, and altitude onto the UTM plane coordinates and combine them with elevation to form the coordinates. A2. First transform to ECEF coordinates, then transform to the local ENU coordinate system, and finally align with UTM coordinates.

5. The UAV end-to-end navigation model training data generation and closed-loop simulation verification system according to claim 1, characterized in that: The 3D Gaussian splash rendering module includes a rendering service unit, which receives camera intrinsic parameters and camera pose, and generates a visible light image of the corresponding viewpoint based on the 3D Gaussian splash scene.

6. The UAV end-to-end navigation model training data generation and closed-loop simulation verification system according to claim 1, characterized in that: When generating the visible light image, the 3D model or dense point motion is further transformed or fitted into a 3DGS scene representation. In the rendering stage, given the camera intrinsic parameter matrix and camera pose, the pixels are back-projected into camera rays according to the pinhole imaging model, and rasterization or volume rendering-style Gaussian accumulation is performed in the 3DGS scene representation to generate a new perspective RGB image that is close to the actual acquisition distribution.

7. The UAV end-to-end navigation model training data generation and closed-loop simulation verification system according to claim 1, characterized in that: Before generating the visible light image, the original image needs to be preprocessed to remove distortion, or distortion needs to be ignored in the camera model. The residuals are absorbed by data enhancement during the training phase. At the same time, exposure, white balance and noise perturbation are introduced at the rendering end.

8. The UAV end-to-end navigation model training data generation and closed-loop simulation verification system according to claim 1, characterized in that: The strategy model training module includes a strategy reasoning service unit, which adopts a vision, language and state fusion structure to receive rendered images, UAV status and mission instructions, and output action control quantities or next state increments.

9. The UAV end-to-end navigation model training data generation and closed-loop simulation verification system according to claim 1, characterized in that: The closed-loop inference and service-oriented deployment module includes a simulation execution and safety constraint unit and a data playback and evaluation unit. The simulation execution and safety constraint unit is used to limit the speed and altitude of the actions, perform collision detection and safety pruning, and update the UAV state according to the discrete kinematics model to form the camera pose input for the next moment. The data playback and evaluation unit is used to statistically analyze the indicators of task success rate, minimum safe distance, and surround coverage, and write the closed-loop trajectory and image sequence back to the dataset construction module or training module to support iteration.

10. The UAV end-to-end navigation model training data generation and closed-loop simulation verification system according to claim 1, characterized in that: The trajectory task and the corresponding expert trajectory or planned trajectory generated by the expert trajectory generation module include fixed-point landing, fixed-point orbit, adaptive building inspection orbit, and obstacle avoidance path planning based on RRT.

Images