Method for training UAV operators using geo-referenced augmented reality

Abstract

FIELD: unmanned aerial vehicle.SUBSTANCE: method for training an operator of an unmanned aerial vehicle (UAV) generating a video stream on board with assigning to each frame a timestamp synchronized with a single time base; generating telemetry data, including at least the coordinates and orientation of the UAV, with timestamps synchronized with the specified time base; performing frame-by-frame synchronization of the video stream and telemetry data in a certain manner at the ground station; buffering the video stream and performing overlay with a time shift corresponding to the delay in transmitting the video stream; setting the coordinates of the unmanned aerial vehicle and three-dimensional digital objects in a geodetic coordinate system; transforming the specified coordinates into a local Cartesian coordinate system with subsequent transformation into image coordinates based on the parameters of a calibrated camera; forming three-dimensional digital objects tied to the specified coordinates at the ground station; superimposing the specified three-dimensional digital objects onto the video stream based on the spatial position of the UAV corresponding to the moment of formation of each frame, recording the operator’s actions when interacting with the specified three-dimensional digital objects when performing training tasks and evaluate the parameters of completing the training tasks.EFFECT: realism of training is increased by precisely spatially superimposing three-dimensional digital objects onto the UAV video stream through frame-by-frame synchronization of video data and telemetry, taking into account transmission delays.5 cl

Description

[0001] FIELD OF TECHNOLOGY

[0002] The claimed technical solution generally relates to the field of training unmanned aerial vehicle (UAV) operators, namely to methods for training operators using video data processing, telemetry and augmented reality technologies, and can be used in civil, industrial and specialized fields, including infrastructure monitoring, search and rescue operations and other tasks.

[0003] STATE OF THE ART

[0004] There are known methods for training UAV operators using computer simulators, which create a completely virtual flight environment and control the aircraft using simulated real controls. While these solutions ensure training safety, they do not take into account real physical flight conditions, including wind effects, communication delays, aircraft dynamics, and onboard system limitations.

[0005] Augmented reality systems are also known in which virtual objects are superimposed on images of the surrounding environment. In most of these solutions, virtual objects are referenced either to visual markers or based on localization algorithms in a local coordinate system (e.g., SLAM). However, there is no rigid geodetic reference of the objects to real-world coordinates, which limits positioning accuracy.

[0006] Furthermore, systems using digital twins are known, creating virtual models of objects for analysis and modeling. However, such systems do not support real-time use in conjunction with a UAV video stream and do not provide operator interaction with such objects while operating a real aircraft.

[0007] Solutions are also known that transmit video and telemetry from the UAV to the ground station. However, in such solutions, video data and telemetry are typically processed separately, without ensuring precise frame-by-frame synchronization, leading to errors when attempting to combine the data.

[0008] The main disadvantages of known solutions include:

[0009] lack of compatibility between real flight and safe training environment;

[0010] lack of precise georeferencing of virtual objects to real terrain;

[0011] Desynchronization of video stream and telemetry, leading to positioning errors;

[0012] inability to accurately superimpose virtual objects on a video stream in real time;

[0013] lack of objective assessment of operator actions in conditions close to real ones;

[0014] High risk of UAV damage during training on real objects; limited possibilities for creating flexible training scenarios.

[0015] The present invention aims to eliminate these disadvantages of the prior art.

[0016] SUMMARY OF THE INVENTION

[0017] The technical task that the declared technical solution is aimed at solving is the creation of a new method for training UAV operators using geo-referenced augmented reality, ensuring:

[0018] increasing the realism of training;

[0019] providing accurate georeferencing of virtual objects;

[0020] eliminate desynchronization between video stream and telemetry;

[0021] Reducing the risk of equipment damage;

[0022] Ensuring objective assessment of operator actions.

[0023] The technical result of the solution is to ensure precise spatial superposition of three-dimensional digital objects on the video stream from the UAV due to frame-by-frame synchronization of video data and telemetry, taking into account transmission delays.

[0024] Additional technical results are:

[0025] - reducing positioning errors of virtual objects;

[0026] - providing the ability to form geo-referenced training scenarios;

[0027] - reducing the likelihood of accidents and damage to UAVs during training.

[0028] The specified technical result is achieved through a method for training an operator of an unmanned aerial vehicle, which contains the following stages:

[0029] • generate a video stream on board an unmanned aerial vehicle (UAV) with each frame assigned a timestamp synchronized with a single time base;

[0030] • telemetry data including at least the coordinates and orientation of the unmanned aerial vehicle are generated with timestamps synchronized with the specified time base;

[0031] • perform frame-by-frame synchronization of the video stream and telemetry data at the ground station by determining the corresponding telemetry parameters for each frame by interpolating the telemetry data to the time of formation of the specified frame;

[0032] • buffer the video stream and perform overlay with a time shift corresponding to the delay in video stream transmission;

[0033] • set the coordinates of the unmanned aerial vehicle and three-dimensional digital objects in the geodetic coordinate system;

[0034] • perform a transformation of the specified coordinates into a local Cartesian coordinate system, followed by transformation into image coordinates based on the parameters of the calibrated camera;

[0035] • at the ground station, three-dimensional digital objects are formed, linked to the specified coordinates;

[0036] • superimpose the specified three-dimensional digital objects onto the video stream based on the spatial position of the unmanned aerial vehicle corresponding to the moment of formation of each frame;

[0037] • the operator interacts with the specified three-dimensional digital objects while performing educational tasks;

[0038] • At the same time, the operator’s actions are recorded with a time reference and the parameters for completing training tasks are assessed.

[0039] In a particular embodiment of the method, a signal from a global navigation satellite system with the generation of PPS pulses is used as a single time base.

[0040] In a particular embodiment of the method, the telemetry data includes data from inertial sensors with a frequency exceeding the frequency of the video stream.

[0041] In a particular embodiment of the method, interpolation of telemetry data is performed by linear or spline interpolation.

[0042] In a particular embodiment of the method, the local coordinate system is formed in the east-north-up (ENU) system with the origin at the starting point or the center of the scenario.

[0043] DETAILED DESCRIPTION OF THE INVENTION

[0044] The following detailed description of the invention includes numerous implementation details to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention may be used with or without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid obscuring the features of the present invention.

[0045] Furthermore, it will be clear from the foregoing description that the invention is not limited to the embodiment described. Numerous possible modifications, changes, variations, and substitutions, while preserving the spirit and form of the present invention, will be apparent to those skilled in the art.

[0046] The claimed technical solution proposes a method for training an unmanned aerial vehicle operator.

[0047] The method includes the following main steps:

[0048] The video stream is generated on board the UAV, with each frame assigned a timestamp synchronized with a single time base;

[0049] Telemetry data is generated with timestamps synchronized with the specified time base;

[0050] At the ground station, frame-by-frame synchronization of the video stream and telemetry data is performed by determining the corresponding telemetry parameters for each frame by interpolating the telemetry data to the time of frame formation;

[0051] Buffer the video stream and perform overlay taking into account the transmission delay;

[0052] The coordinates of UAVs and digital objects are specified in the geodetic coordinate system;

[0053] perform coordinate transformation to local Cartesian coordinate system and then transform to image coordinates using calibrated camera parameters;

[0054] form three-dimensional digital objects linked to coordinates;

[0055] overlay the specified objects on the video stream, taking into account the position of the UAV at the time the frame is formed;

[0056] The operator interacts with digital objects while completing educational tasks;

[0057] Record operator actions and evaluate task completion.

[0058] Compared with known solutions, the proposed method has the following advantages:

[0059] Combination of real flight with virtual training environment;

[0060] high precision georeferencing of virtual objects;

[0061] Eliminating desynchronization of video stream and telemetry;

[0062] the ability to create dynamic and adaptive learning scenarios;

[0063] Reducing the risk of damage to UAVs;

[0064] Improving the efficiency and speed of learning;

[0065] the ability to objectively control and analyze operator actions.

[0066] The method is implemented as follows.

[0067] Before the flight, a training scenario, including parameters of 3D digital objects, is loaded at the ground station. Time synchronization is performed between the UAV's onboard system and the ground station.

[0068] During flight, the UAV generates a video stream, each frame of which is assigned a timestamp. Simultaneously, telemetry data is generated, including the UAV's coordinates and orientation, also provided with timestamps.

[0069] The video stream and telemetry data are transmitted to the ground station, where they are synchronized frame by frame. For each frame, the corresponding telemetry parameters are determined by interpolating the data to the time the frame was generated.

[0070] Given the coordinates and orientation of the UAV, it transforms the coordinates of digital objects from the geodetic system into the local coordinate system, and then into image coordinates using the camera parameters.

[0071] After this, the digital objects are rendered and overlaid onto the video stream, taking into account the data transmission delay. The overlay is performed using the camera's internal calibration parameters, including focal length and distortion coefficients.

[0072] The operator observes the combined image and performs training tasks by interacting with digital objects. All operator actions are recorded and used to evaluate training results. Task performance is assessed based on trajectory deviation, reaction time, and targeting accuracy.

[0073] Three-dimensional digital objects include dynamic objects with a given function of movement over time.

[0074] These application materials present a preferred implementation of the claimed technical solution, which should not be used as limiting other particular embodiments of its implementation that do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims

1. A method for training an operator of an unmanned aerial vehicle, comprising the following stages: • generate a video stream on board an unmanned aerial vehicle (UAV) with each frame assigned a timestamp synchronized with a single time base; • telemetry data, including at least the coordinates and orientation of the unmanned aerial vehicle, are generated with time stamps synchronized with the specified time base; • perform frame-by-frame synchronization of the video stream and telemetry data at the ground station by determining the corresponding telemetry parameters for each frame through interpolation of the telemetry data to the time of formation of the specified frame; • buffer the video stream and perform overlay with a time shift corresponding to the delay in video stream transmission; • set the coordinates of the unmanned aerial vehicle and three-dimensional digital objects in the geodetic coordinate system; • perform the transformation of the specified coordinates into a local Cartesian coordinate system, followed by transformation into image coordinates based on the parameters of the calibrated camera; • at the ground station, three-dimensional digital objects are formed, linked to the specified coordinates; • superimpose the specified three-dimensional digital objects onto the video stream based on the spatial position of the unmanned aerial vehicle corresponding to the moment of formation of each frame; • the operator interacts with the specified three-dimensional digital objects when performing training tasks; • at the same time, the operator’s actions are recorded with a time reference and the parameters for completing training tasks are assessed.

2. The method according to paragraph 1, characterized in that a signal from a global navigation satellite system with the formation of PPS pulses is used as a single time base.

3. The method according to paragraph 1, characterized in that the telemetry data includes data from inertial sensors with a frequency exceeding the frequency of the video stream.

4. The method according to paragraph 1, characterized in that the interpolation of telemetry data is performed by linear or spline interpolation.

5. The method according to paragraph 1, characterized in that the local coordinate system is formed in the east-north-up (ENU) system with the origin at the starting point or the center of the scenario.