Unmanned aerial vehicle simulation training method for video guidance and trajectory deviation evaluation based on deep learning

Abstract

The invention relates to an unmanned aerial vehicle simulation training method for video guidance and track deviation evaluation based on deep learning. The method specifically comprises the following steps: S1, data acquisition and input; s2, generating and prompting a path visible area; s3, track deviation evaluation of image and flight control fusion; s4, trajectory behavior modeling and score analysis; s5, training a feedback output mechanism; and S6, carrying out personalized training recommendation and playback analysis. According to the invention, space-level deviation evaluation is realized by fusing the image and the flight control data, and the judgment is more accurate; the real-time visual feedback and voice prompt system can improve the training instantaneity and the interaction experience; a Transform-based trajectory scoring mechanism is close to the nature of an operation behavior, and multi-dimension evaluation is carried out; differentiated task recommendation is supported, and teaching pertinence and training efficiency are improved; the whole process is visual and can be repeated, and the teaching closed-loop design is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and unmanned aerial vehicle (UAV) teaching systems, and specifically relates to a UAV simulation training method based on deep learning video guidance and trajectory deviation assessment. Background Art

[0002] With the rapid expansion of drone applications, the quality and efficiency of pilot training plays a critical role in flight safety and mission execution. Traditional drone operation training, which often relies on manual instructor instruction and assessment, suffers from inconsistent standards, delayed feedback, and subjective assessments. Furthermore, it struggles to effectively simulate real-world operational scenarios in complex mission environments.

[0003] The current training system based on flight control data analysis has the following limitations: First, it relies only on coordinate data and cannot provide visual guidance and evaluation; second, it lacks a real-time feedback mechanism and cannot immediately prompt operational deviations during training; third, it does not combine individual historical training data, making it difficult to achieve differentiated and precise personalized training path design.

[0004] Therefore, there is an urgent need for a drone training system that integrates computer vision, deep learning, and trajectory behavior modeling technologies to achieve a high-standard, full-process, personalized teaching assistance and assessment system. Summary of the Invention

[0005] The purpose of this invention is to provide a UAV simulation training method based on deep learning video guidance and trajectory deviation assessment. Through image semantic segmentation, behavior scoring modeling, flight control data fusion and other means, it can realize UAV flight trajectory guidance, real-time error feedback and personalized teaching recommendations, thereby improving the intelligence and visualization level of the training process.

[0006] To achieve the above objectives, the present invention proposes the following technical solutions: A deep learning-based UAV video guidance and trajectory deviation assessment simulation training method specifically includes the following steps: S1. Data acquisition and input: Multimodal training data during flight is collected in real time through the UAV simulator platform, including flight trajectory coordinates, speed, attitude angle information output by the flight control module, and simulated screen images from the first-person perspective or map perspective; S2. Path Visibility Area Generation and Prompt: A lightweight UNet++ semantic segmentation network is used to perform pixel-level segmentation of the task trajectory in the simulation image, constructing a path visibility area with a preset tolerance width. The guidance channel is then overlaid and displayed in real time on the training interface to provide operational reference for the pilot. S3. Trajectory Deviation Assessment for Image and Flight Control Fusion: This uses the YOLOv8 network to detect the aircraft's position coordinates in the simulated image while simultaneously acquiring real-time flight control data. This fusion mapping of the image and coordinate space is then performed. The deviation between the aircraft's current position and the ideal path is calculated using Euclidean distance, cosine similarity, or the DTW algorithm, and the deviation level and score are output. S4. Trajectory behavior modeling and scoring analysis: The trajectory sequence throughout the flight process is input into the Transformer temporal coding model to extract features such as flight stability, path consistency, and operational volatility, generating multi-dimensional flight scoring indicators for comprehensive evaluation of student operational quality. S5. Feedback output mechanism during training: When the deviation level exceeds the set threshold, the system outputs a prompt message through the TTS voice module and highlights the current aircraft position and trajectory deviation area on the simulator screen, reminding the pilot to correct the behavior in real time; S6. Personalized training recommendations and replay analysis: Based on the trainees' past flight trajectory deviation records and scoring data, the system pushes recommended tasks for the next stage through a rule engine or lightweight reinforcement learning strategy. It also supports animated playback of training trajectories, visualization of behavioral scoring trends, and display of teaching comments.

[0007] Furthermore, the UNet++ network used in step S2 can adapt to different task path lengths and shapes, and achieve flexible expansion and dynamic generation of path fault-tolerant areas; Furthermore, the image detection and flight control data fusion in step S3 adopts a spatial alignment strategy to improve the accuracy of deviation determination, and the error function supports multi-model fusion to adapt to different evaluation criteria; Furthermore, in step S4, the Transformer model uses a multi-head attention mechanism to model trajectory behavior changes and output detailed indicators such as stability, consistency, and completion; Furthermore, the training recommendation mechanism supports outputting recommendations based on different deviation types, such as recommending "edge flight training" for "frequent route drift" and "low-speed control training" for "severe operation jitter", etc., to achieve precise training allocation.

[0008] Based on the UAV simulation training method based on deep learning-based video guidance and trajectory deviation evaluation, the adopted system includes the following modules: (1) Flight data acquisition module: synchronously collects the flight attitude, trajectory coordinates, speed information and real-time images output by the simulator; (2) Path recognition and guidance module: uses semantic segmentation algorithm to extract task trajectory area and constructs a visual guidance channel with tolerance attributes; (3) Trajectory deviation assessment module: This module integrates the image detection results with the flight control data coordinates, calculates the spatial offset between the current position and the mission path, and generates a graded assessment result. (4) Behavior scoring module: Based on the Transformer model, the flight trajectory is modeled in time series and the scoring index is output; (5) Feedback prompt module: outputs voice and visual prompt information according to the evaluation level for real-time operation guidance; (6) Training recommendation module: Combine historical training data and performance to push personalized training tasks; (7) Data playback module: displays the flight trajectory, deviation changes and scoring trends, and visualizes optimization suggestions.

[0009] The beneficial effects of the present invention are: (1) Fusion of images and flight control data enables spatial-level deviation assessment, resulting in more accurate judgment; (2) Real-time visual feedback and voice prompt system improve the immediacy of training and interactive experience; (3) The Transformer-based trajectory scoring mechanism is close to the essence of the operation behavior and provides multidimensional evaluation; (4) Support differentiated task recommendations to improve teaching pertinence and training efficiency; (5) The entire process is visualized and replayable, optimizing the closed-loop teaching design. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a training flow chart of the present invention; Figure 2 It is the system structure module diagram; Figure 3 This is a flowchart of the training playback and personalized recommendation process. DETAILED DESCRIPTION

[0011] The specific implementation of the present invention will be described below with reference to the accompanying drawings and examples for better understanding of the present invention. Example

[0012] like Figure 1 As shown, the UAV simulation training method based on deep learning video guidance and trajectory deviation assessment in this embodiment includes the following steps: S1. Real-time collection of flight behavior data: The real-time trajectory coordinates, speed, attitude angle and flight screen images of the aircraft during training are collected through the UAV simulator platform to form multi-dimensional and multi-modal training input data.

[0013] S2. Path visual area generation and dynamic prompts: The path recognition and guidance module receives mission setting information, segments the trajectory area in the simulator mission screen through a lightweight UNet++ network, constructs a visual path channel, guides the pilot to fly along the mission route, and dynamically displays the area in the interface to indicate the flight tolerance boundary.

[0014] S3. Deviation determination based on image and flight control data fusion: The trajectory deviation assessment module uses the YOLOv8 network to identify the aircraft image position and simultaneously obtains the trajectory coordinate data of the flight control module. It spatially maps the two and uses the Euclidean distance or cosine similarity function to calculate the degree of deviation between the current position and the ideal path in real time, outputting a deviation score.

[0015] S4. Flight behavior sequence modeling and scoring output: The behavior scoring module inputs the trajectory sequence formed during the flight into the Transformer model, extracts temporal features such as path continuity, flight stability, and trajectory deviation frequency, and outputs multi-dimensional results such as flight stability score, path consistency score, and operation standardization score.

[0016] S5. Real-time prompt feedback and correction mechanism: During the flight, the system combines the score and deviation level, and outputs real-time error correction prompts through the TTS speech synthesis module, such as "flight deviates from the route" and "operation is too hasty", and highlights them in the flight picture to guide students to adjust their flight operations in time.

[0017] S6. Personalized training recommendations and trajectory replay analysis: The system combines the student's historical training records with the current deviation situation and generates training suggestions through reinforcement learning or rule-based reasoning algorithms. For example, it recommends personalized projects such as "flying around points and sticking to the line training" and "path correction exercises." At the same time, the data replay module outputs trajectory animations, scoring trend curves, and operational improvement suggestions.

[0018] like Figure 2 As shown, the UAV simulation training system of the present invention includes the following core module structures: Flight data acquisition module 101: connects to the simulator platform interface and collects data such as flight control trajectory, flight attitude, speed change and screen image in real time; Path identification and guidance module 102: uses a semantic segmentation model to identify the task trajectory area, generates a guidance path layer with a tolerance bandwidth, and displays it in real time on the simulation interface; Trajectory Deviation Assessment Module 103: This module integrates the YOLOv8 detection results with the flight control trajectory coordinates, calculates the spatial deviation between the actual position of the aircraft and the ideal path, and outputs the deviation level and score. Behavior scoring module 104: inputs the trajectory time series into the Transformer model and outputs flight stability score, path consistency score and execution accuracy index; Real-time feedback module 105: Prompts flight deviation status through voice and image highlighting, helping students to adjust flight operation behavior immediately; Personalized training recommendation module 106: recommends training programs and automatically assigns training goals based on student rating records, deviation characteristics, and task completion status; Data playback module 107: provides flight process playback, scoring trend analysis and visual improvement suggestion charts after training is completed.

[0019] like Figure 2 and Figure 3 As shown in the figure, when a flight trainee starts a flight training mission in the UAV simulator, the system automatically runs according to the following process: First, the flight data acquisition module 101 starts to acquire the aircraft attitude parameters, speed information, trajectory coordinates and training image frames output by the flight control system in real time. The above data is synchronously transmitted to the path recognition and guidance module 102 and the trajectory deviation assessment module 103.

[0020] Based on the training task settings, the Path Identification and Guidance Module 102 invokes the pre-set path template for the training task. Using a lightweight UNet++ network, it performs semantic image segmentation on the current simulation screen, extracts the task trajectory area, generates a path channel layer with a tolerance bandwidth, and renders the guidance area on the simulator screen in real time to guide flight operations. Simultaneously, the Trajectory Deviation Assessment Module 103 receives the aircraft's on-screen position coordinates from the Image Detection Module and spatially maps them to the trajectory data output by the Flight Control Module. Using Euclidean distance or cosine similarity, it calculates the degree of deviation between the aircraft's current position and the ideal path centerline, and outputs a deviation score based on a preset level.

[0021] The scoring results are synchronously transmitted to the behavior scoring module 104, which uses the Transformer model to perform time series modeling on the trajectory sequence of the entire flight process, identify behavioral characteristics such as path continuity, stability, and offset frequency, and output flight stability scores, path consistency scores, and execution accuracy scores.

[0022] During flight, the real-time feedback module 105 continuously monitors changes in the score and deviation level. When any indicator exceeds a set threshold, a prompt is output through the speech synthesis system (TTS) and the current aircraft deviation area is highlighted on the flight screen, forming a dual visual and auditory guidance mechanism to assist the trainee in correcting flight errors. If continuous deviation from the trajectory or persistently low scores are detected during training, the personalized training recommendation module 106 generates targeted training suggestions based on flight performance and historical records through a rules engine or reinforcement learning model. For example, it can recommend a "side-to-side circumnavigation practice" task for trainees with "poor directional stability" and "S-turn stability training" for trainees with "large path following errors," and automatically schedule the next training task.

[0023] After training, the data playback module 107 generates a trajectory animation playback, a behavior score trend curve, and an improvement suggestion chart based on all the coordinates and scoring data during the flight, and displays them through an interactive interface. Trainees can review their flight performance based on the system feedback, identify problem areas, and determine the next training direction.

[0024] Throughout the system's operation, all flight mission data, scoring reports, training recommendations, and playback recordings are stored in a local server or cloud-based data management system, enabling cross-device access, remote evaluation, and historical comparison. A network communication module ensures high-speed data exchange between the system and the platform, supporting remote updates, instructor intervention, and centralized evaluation.

[0025] The system realizes a closed loop of the entire process from task definition, data collection, real-time evaluation, instant feedback, personalized recommendation to training summary, and has the intelligent teaching capabilities of "perceivable - evaluable - interventionable - optimizable".

[0026] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A UAV simulation training method based on deep learning video guidance and trajectory deviation assessment, characterized by: The following steps are involved: S1. Data acquisition and input: Multimodal training data during flight is collected in real time through the UAV simulator platform, including flight trajectory coordinates, speed, attitude angle information output by the flight control module, and simulated screen images from the first-person perspective or map perspective; S2. Path Visibility Area Generation and Prompt: A lightweight UNet++ semantic segmentation network is used to perform pixel-level segmentation of the task trajectory in the simulation image, constructing a path visibility area with a preset tolerance width. The guidance channel is then overlaid and displayed in real time on the training interface to provide operational reference for the pilot. S3. Trajectory Deviation Assessment for Image and Flight Control Fusion: This uses the YOLOv8 network to detect the aircraft's position coordinates in the simulated image while simultaneously acquiring real-time flight control data. This fusion mapping of the image and coordinate space is then performed. The deviation between the aircraft's current position and the ideal path is calculated using Euclidean distance, cosine similarity, or the DTW algorithm, and the deviation level and score are output. S4. Trajectory behavior modeling and scoring analysis: The trajectory sequence throughout the flight process is input into the Transformer temporal coding model to extract features such as flight stability, path consistency, and operational volatility, generating multi-dimensional flight scoring indicators for comprehensive evaluation of student operational quality. S5. Feedback output mechanism during training: When the deviation level exceeds the set threshold, the system outputs a prompt message through the TTS voice module and highlights the current aircraft position and trajectory deviation area on the simulator screen, reminding the pilot to correct the behavior in real time; S6. Personalized training recommendations and replay analysis: Based on the trainees' past flight trajectory deviation records and scoring data, the system pushes recommended tasks for the next stage through a rule engine or lightweight reinforcement learning strategy. It also supports animated playback of training trajectories, visualization of behavioral scoring trends, and display of teaching comments.

2. The UAV simulation training method based on deep learning video guidance and trajectory deviation assessment according to claim 1 is characterized by: The UNet++ network used in step S2 can adapt to different task path lengths and shapes, and realize flexible expansion and dynamic generation of path fault-tolerant areas.

3. The UAV simulation training method based on deep learning video guidance and trajectory deviation assessment according to claim 1 is characterized by: The image detection and flight control data fusion in step S3 adopts a spatial alignment strategy to improve the accuracy of deviation judgment, and the error function supports multi-model fusion to adapt to different evaluation criteria.

4. The UAV simulation training method based on deep learning video guidance and trajectory deviation assessment according to claim 1 is characterized by: In step S4, the Transformer model uses a multi-head attention mechanism to model trajectory behavior changes and output detailed indicators such as stability, consistency, and completion.

5. The UAV simulation training method based on deep learning video guidance and trajectory deviation assessment according to claim 1 is characterized by: The training recommendation mechanism supports outputting recommendations based on different deviation types, such as recommending "edge flying training" for "frequent route drift" and "low-speed control training" for "severe operation jitter", achieving precise training allocation.

6. The UAV simulation training method based on deep learning video guidance and trajectory deviation assessment according to claim 1 is characterized by: Based on the above training method, the adopted system includes the following modules: (1) Flight data acquisition module: synchronously collects the flight attitude, trajectory coordinates, speed information and real-time images output by the simulator; (2) Path recognition and guidance module: uses semantic segmentation algorithm to extract task trajectory area and constructs a visual guidance channel with tolerance attributes; (3) Trajectory deviation assessment module: This module integrates the image detection results with the flight control data coordinates, calculates the spatial offset between the current position and the mission path, and generates a graded assessment result. (4) Behavior scoring module: Based on the Transformer model, the flight trajectory is modeled in time series and the scoring index is output; (5) Feedback prompt module: outputs voice and visual prompt information according to the evaluation level for real-time operation guidance; (6) Training recommendation module: Combine historical training data and performance to push personalized training tasks; (7) Data playback module: displays the flight trajectory, deviation changes and scoring trends, and visualizes optimization suggestions.

Images