A pilot training method, system and device based on VR equipment

Abstract

This invention relates to a pilot training method, system, and device based on VR equipment. The VR equipment is installed in a simulated aircraft cockpit. The method includes: S1. Establishing the coordinate system and transformation relationships of all objects within the VR equipment's field of view; S2. Establishing the coordinates of the points of the objects being viewed by the user of the VR equipment in the corresponding coordinate system; S3. Establishing hotspot regions based on the coordinates of the viewed points; S4. Determining the corresponding modeled objects based on the hotspot regions; S5. Storing the data related to the hotspot regions and the modeled objects for use during training. This invention's method, by spatially labeling the user's gaze points in the VR headset and recording the objects the user focuses on at different times, can efficiently and completely transfer experienced personnel's knowledge to new trainees, reducing problems encountered during training and improving learning efficiency.

Description

Technical Field

[0001] This invention belongs to the field of pilot training simulation technology, specifically relating to a pilot training method, system, and device based on VR equipment. Background Technology

[0002] Pilots are a nation's most valuable asset, and their training advances national defense and enhances military strength. For civil airlines, training more pilots increases company revenue. Pilot training has always been a crucial issue in the aviation field. Because real aircraft piloting requires extremely high levels of professional knowledge, operational skills, and emergency response capabilities, training inexperienced trainees is inherently dangerous. Considering factors such as personnel safety and efficiency, flight training for new trainees is conducted on ground-based simulators.

[0003] In actual teaching, instructors inform trainees of the functions and operating methods of various instruments and devices, the entire flight mission process, and the tasks to be performed, either in writing or verbally. However, existing training methods only convey the main mission information, missing the details and points to note at each moment. This results in trainees, even if they have a very clear grasp of the process and knowledge, still experiencing considerable confusion when on the simulator, unsure of what to pay attention to next.

[0004] For an experienced pilot, different phases of a flight mission often require attention to different instruments and environments. A particular action is often the result of perceiving multiple data points and environmental conditions, and even a seemingly unintentional glance may be the culmination of years of trial and error in flight experience. This sub-second-scale attention process is difficult to teach new trainees through verbal instruction or written documentation; new trainees typically require a considerable amount of training and trial and error to gradually accumulate experience. Summary of the Invention

[0005] In order to efficiently and completely impart experience to new trainees, reduce problems encountered in trainee training, and improve learning efficiency, this invention proposes a pilot training method, system, and device based on VR equipment. By spatially annotating the gaze points of the person in the VR headset, the object that the person pays attention to at different times is recorded, and the method supports the reproduction of the person's operation according to the chronological order and the generation of global gaze time allocation ratio.

[0006] A pilot training method based on a VR device, wherein the VR device is installed in a simulated aircraft cockpit, the method comprising:

[0007] S1. Establish the coordinate system and transformation relationships for all objects within the VR device's field of view;

[0008] S2. Establish the coordinates of the points of the object that the user is looking at in the VR device within the corresponding coordinate system;

[0009] S3. Establish hotspot regions based on the coordinates of the viewed point;

[0010] S4. Determine the corresponding modeling objects based on the hotspot areas;

[0011] S5. Store the data related to the hotspot area and the modeled object for use during training.

[0012] In addition to the aspects described above and any possible implementations, a further implementation is provided in which all objects within the field of view of the VR device include: objects on the ground, objects moving in the air, and clouds in the air.

[0013] In addition to the aspects described above and any possible implementation, a further implementation is provided in which the coordinate systems of all objects within the field of view of the VR device include: the VR device image depth coordinate system, the world coordinate system, the flight coordinate system, the cloud coordinate system, and the viewpoint coordinate system.

[0014] In addition to the aspects described above and any possible implementation, a further implementation is provided in which the transformation relationship of the coordinate systems of all objects within the field of view of the VR device includes: the transformation between the VR device image depth coordinate system and the viewpoint coordinate system, the transformation between the viewpoint coordinate system and the flight coordinate system, the transformation between the flight coordinate system and the world coordinate system, and the transformation between the world coordinate system and the cloud coordinate system.

[0015] In addition to the aspects described above and any possible implementation, a further implementation is provided in which the flight coordinate system includes the local coordinate system of the simulated aircraft where the VR device is located and the coordinate systems of other aircraft within the field of view.

[0016] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein S3 is the process of establishing the hotspot region as follows: S31. For The distance to the gaze point in the viewpoint coordinate system is determined at any given time: when the distance between the point and the origin is less than a set threshold, the point is considered an object within the VR device and is processed in the viewpoint coordinate system; when the distance between the point and the origin is less than the set threshold, the point is considered an object outside the VR device and its host machine, and is converted to the world coordinate system for processing.

[0017] S32. On the coordinate system corresponding to this point, establish a side with a length of [missing information] centered on the coordinates of the gaze point. Given a cube, set the initial weights of its internal space to be... ;

[0018] S33. The Next Moment Establish a side with length as the center of the updated gaze point coordinates. The cube, the initial weights of the interior space of the cube are... ;

[0019] S34. Find the overlapping region of the two cubes. The weight of the overlapping region is the sum of the weights of the two cubes before they overlap.

[0020] S35. Set decay parameters ,repeat This process involves eliminating individual non-overlapping cubes by subtracting the weights of the non-negative weights in the entire coordinate system. , Round up to the nearest integer of the ratio k / n;

[0021] S36. Set a hotspot threshold. When the weight obtained after processing in S35 is higher than the hotspot threshold, record the hotspot area.

[0022] S37. Continue updating the time until the time expires and all recorded hotspot areas are obtained.

[0023] In addition to the aspects and any possible implementations described above, an implementation is further provided in which S4 includes the following steps:

[0024] S41. Calculate the sum of the bounding area of ​​the current hotspot region and all nearby objects. If the sum of the bounding area with a certain object is significantly greater than the sum of the bounding areas with other objects, then mark this object as the modeling object corresponding to the current hotspot region.

[0025] S42. If there is no significant difference in the surrounding area with other objects, or if there is no surrounding area, then a sphere is built with the spatial center point of the hot spot area as the center. The radius of the sphere is gradually increased within a certain range until it contacts the first object inside the sphere. This object is the modeling object corresponding to the current hot spot area.

[0026] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the storage format is as follows:

[0027] Hotspot area: {task execution time, applied coordinate system, coordinate system, weights};

[0028] Modeling objects: {task execution time, model scene, object index number}.

[0029] The present invention also provides a pilot training system based on VR devices for implementing the method, the system comprising:

[0030] The first module is used to establish the coordinate system and transformation relationships of all objects within the VR device's field of view;

[0031] The second module is used to establish the coordinates of the points of the object being viewed by the user of the VR device in the corresponding coordinate system.

[0032] The third module is used to establish hotspot regions based on the coordinates of the viewed point;

[0033] The determination module is used to determine the corresponding modeling object based on the hotspot area;

[0034] The storage module is used to store data related to the hotspot area and the modeled object for use during training.

[0035] The present invention also provides a computer device including a processor and a memory, wherein the memory stores a computer program, the computer program being loaded and executed by the processor to implement the method as described in the present invention.

[0036] Beneficial effects of the present invention

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The VR device of the present invention is implemented using a VR headset. Based on the coordinates and depth of the current user’s gaze calculated by the VR headset, the acquired object depth information is converted into the form of object coordinates in any Cartesian coordinate system, which greatly facilitates the subsequent data processing.

[0039] (2) The method of the present invention defines five coordinate systems in the modeling scene: VR device image depth coordinate system, world coordinate system, flight coordinate system, cloud coordinate system and viewpoint coordinate system. Establishing coordinate systems can conveniently represent any hotspot area in the world, any object in the cockpit of the aircraft, and any moving object in the air, thus annotating all objects in any state in the modeling scene, including all objects that the pilot may pay attention to in real flight.

[0040] (3) Due to errors in the calculation of gaze points by VR headsets, the calculated coordinates in the Cartesian coordinate system differ significantly from normal data. The method of this invention cleans this dirty data, making the calculation results for hotspot areas and objects of interest more accurate and stable. The method of this invention processes gaze points to form hotspot areas, and infers objects in the model based on the enclosing structure of the hotspot areas. Objects of interest are labeled. For different types of objects, such as stationary objects on the ground, objects in the cockpit, and moving objects in the air, labeling is performed in different coordinate systems, realizing dynamic labeling in dynamic scenes.

[0041] (4) The method of the present invention stores different objects and hot spots at different times while preserving the temporal information, thus meeting the requirements of temporal display of dynamic scenes.

[0042] (5) The method of the present invention also supports the reproduction of personnel operations according to the time sequence and the generation of global gaze time allocation ratio. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the device image depth coordinate system of the present invention;

[0044] Figure 2 This is a schematic diagram of the world coordinate system of the present invention;

[0045] Figure 3 This is a schematic diagram of the flight (machine) coordinate system of the present invention;

[0046] Figure 4 This is a schematic diagram of the flight (other aircraft) coordinate system of the present invention;

[0047] Figure 5 This is a schematic diagram of the cloud coordinate system of the present invention;

[0048] Figure 6 This is a schematic diagram of the viewpoint coordinate system of the present invention;

[0049] Figures 7(a)-(b) are schematic diagrams of the helmet pose angle of the present invention;

[0050] Figure 8 This is a schematic diagram of the depth percentage to distance model of the present invention;

[0051] Figure 9 This is a schematic diagram showing the conversion of depth distance to coordinate system position in the viewpoint coordinate system of the present invention;

[0052] Figure 10 This is a schematic diagram illustrating the transformation of the viewpoint coordinate system to the flight aircraft coordinate system according to the present invention;

[0053] Figure 11 This is a schematic diagram illustrating the transformation of the aircraft's coordinate system to the world coordinate system according to the present invention;

[0054] Figure 12 This is a schematic diagram illustrating the transformation from the world coordinate system to the coordinate system of another aircraft in flight according to the present invention;

[0055] Figures 13(a)-(b) are schematic diagrams illustrating that the gaze point on another machine is independent of relative motion according to the present invention;

[0056] Figure 14 This is a schematic diagram of the gaze point distribution at 60 frames per second according to the present invention;

[0057] Figure 15This is a schematic diagram (planar view) of the hot spot area of ​​the present invention;

[0058] Figure 16 Flow chart of the method of the present invention. Detailed Implementation

[0059] To better understand the technical solution of this invention, the content of this invention includes, but is not limited to, the specific embodiments described below. Similar technologies and methods should be considered within the scope of protection of this invention. To make the technical problems to be solved, the technical solutions, and advantages of this invention clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0060] It should be understood that the embodiments described in this invention are merely some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0061] like Figure 16 As shown, the present invention provides a pilot training method based on a VR device, wherein the VR device is installed in a simulated aircraft cockpit, and the method includes:

[0062] S1. Establish the coordinate system and transformation relationships for all objects within the VR device's field of view;

[0063] S2. Establish the coordinates of the points of the object that the user is looking at in the VR device within the corresponding coordinate system;

[0064] S3. Establish hotspot regions based on the coordinates of the viewed point;

[0065] S4. Determine the corresponding modeling objects based on the hotspot areas;

[0066] S5. Store the data related to the hotspot area and the modeled object for use during training.

[0067] The VR device in this invention is implemented using a VR headset. The method of this invention spatially labels the gaze points of the person within the VR headset, recording the objects the person focuses on at different times. The person is not limited to experienced pilots but can also be a novice. This invention records, and analyzes, the objects gazed upon by experienced pilots throughout a flight mission, chronologically. Then, when a new pilot performs a flight mission again within the VR headset, this invention provides prompts based on the gaze information previously recorded by the experienced pilot, allowing the new pilot to follow the experienced pilot's rhythm. Furthermore, this method integrates the gaze information from experienced pilots' training missions, forming a chart to teach new trainees, reducing problems encountered during training and improving learning efficiency. This invention also supports reproducing personnel operations chronologically and generating a global gaze time allocation ratio.

[0068] This invention is a method for evaluating pilot training effectiveness based on spatial annotation of the user's gaze point in an existing 3D simulated flight scenario displayed in a VR headset. The 3D simulated flight scenario has been constructed, featuring object models and a scale identical to reality. Within this scenario, a large-scale 3D terrain map and weather scene have been built, along with precise models of the flight operator's instruments and moving objects such as clouds and aircraft. The method of this invention primarily involves spatial annotation and processing of all objects that the flight operator might gaze at within the scenario. The relevant technologies used in this invention will be explained below.

[0069] I. Definition of Coordinate System

[0070] First, let's define the coordinate systems used in the method of this invention: As mentioned earlier, this invention uses five coordinate systems: VR device image depth coordinate system, world coordinate system, flight coordinate system, cloud coordinate system, and viewpoint coordinate system. The coordinate systems used in this invention are: the world coordinate system, aircraft coordinate system, and viewpoint coordinate system are all Cartesian coordinate systems. There are many definitions of coordinate systems; this invention uses a coordinate system construction method named after Cartesian coordinates. Specifically, the construction method is as follows: Through a fixed point O in space, construct three mutually perpendicular number axes, all with O as the origin and generally having the same unit of length. These three axes are called the x-axis (horizontal axis), y-axis (vertical axis), and z-axis (vertical axis); collectively referred to as coordinate axes. Usually, the x-axis and y-axis are placed on a horizontal plane, while the z-axis is a vertical line. Their positive directions follow the right-hand rule; that is, if you grasp the z-axis with your right hand, when the four fingers of your right hand turn from the positive x-axis to the positive y-axis at an angle of π / 2, the thumb points in the positive direction of the z-axis. These three coordinate axes form a spatial rectangular coordinate system, with point O as the origin. This forms a Cartesian coordinate system.

[0071] The details are as follows:

[0072] 1. VR device image depth coordinate system

[0073] Because VR headsets display three-dimensional scenes with depth, the angle between the lines of sight of the two eyes varies depending on the depth of the object viewed. The VR headset can determine the depth of the gaze point—the distance between that point and the viewpoint—based on the different positions and angles of sight of the two eyes. In the scene, all objects on the ground, in the air, and on the same machine and other machines, according to the imaging principle of a pinhole camera, will be imaged in the human eye, thus creating an imaginary imaging plane. All visible objects in the scene will be imaged on this plane. For example... Figure 1 As shown, this is a schematic diagram of a point on a mountain as seen by a person in a 3D scene, displayed on the image imaging plane. Based on the aforementioned imaging principle, analyzing any point in the scene, the imaging coordinates of that point on the imaging plane are (xd, yd). In this coordinate system, the current image presented by the VR headset in the direction directly opposite the person (the center of the viewer) is defined. This is the aforementioned imaging plane, which will be represented as an image in the VR headset and subsequent processing. This image includes stationary objects on the ground and flying objects in the air: the aircraft where the VR headset is located, other aircraft, and clouds. The origin is the pixel in this image, and the horizontal direction to the right is the coordinate system. The positive axis direction, with the image vertically downwards. Positive direction of the axis. Assume the coordinates of the human eye's gaze are... The depth calculated by the VR headset based on the line of sight of both eyes is Then the device image depth coordinate system is defined as follows: The coordinate system is in units of (pixels, pixels, percentages). Depth is also considered. The data is represented as percentages. For example, the depth of viewing the nearest object is 0%, and the depth of viewing the farthest object is 100%. The VR headset can then derive the following coordinates based on the user's current line of sight. This coordinate can represent an object at any depth within the scene as seen by the user from any perspective.

[0074] 2. World Coordinate System

[0075] In large-scale 3D scene modeling, the entire ground scene is modeled statically. To represent the position of all objects within the large scene, a world coordinate system is established. The units are (meters, meters, meters). The origin is a fixed point on the ground of the scene. Establish a Cartesian coordinate system with the axis perpendicular to the horizontal plane and pointing upwards, such as... Figure 2 As shown, the midpoint It represents any point in a 3D scene.

[0076] 3. Flight coordinate system

[0077] Based on the world coordinate system In this world scenario, the method of the present invention takes into account the need to process the relative motion between objects moving relative to the ground (e.g., an airplane relative to an airport) and moving objects (e.g., an airplane moving in the air relative to other airplanes or clouds) in practical applications. Based on the world coordinate system, the method of the present invention defines a flight coordinate system for the objects that need to move. The unit is (meter, meter, meter), such as Figure 3 The interior of the aircraft cabin is shown (all objects in this invention are virtual and do not involve real-world objects; the aircraft interior refers to the objects inside the cabin as seen by a person wearing a VR headset). A suitable point in the modeled cabin scene is defined as the origin, typically the center point of the cabin, with the aircraft's forward direction (nose) as the reference point. The positive direction of the axis, the vertically upward direction is If a Cartesian coordinate system is established along the positive axis, then... It can represent any point in the cabin within a user scenario. For example... Figure 4 The image shown is an exterior view of another aircraft, with the center point of that other aircraft as the origin and the direction of its flight (nose) as the reference point. The positive direction of the axis, the vertically upward direction is Establish a coordinate system for the other aircraft in flight along the positive axis. Then... It can represent any point in the aircraft or other aircraft. The coordinate system of the local aircraft and the coordinate system of other aircraft are the same in function and meaning. Therefore, they are represented by the same symbols in calculation. Separating them separately is to facilitate the distinction between the local aircraft and other aircraft.

[0078] 4. Cloud Coordinate System

[0079] like Figure 5 As shown in the image of clouds in the sky, since all clouds in the scene are generated uniformly without individual cloud modeling, a cloud coordinate system is defined. .in Coinciding with the world coordinate system and defining The currently set cloud motion velocity vector .

[0080] 5. Viewpoint Coordinate System

[0081] This coordinate system refers to the coordinate system set by the VR headset within the cabin scene of the aircraft. It is a flight coordinate system based on the aircraft cabin interior. Based on this, inside the aircraft cabin where the user is located, the user sees all objects within the VR headset's field of view by wearing a VR headset, such as... Figure 6 The diagram shows the VR headset's initial position in this scene defined as the origin, with the aircraft's forward direction (nose) as the coordinate axis. The positive direction, perpendicular to The upward direction is the coordinate axis. Establish a Cartesian coordinate system in the positive direction, denoted as the viewpoint coordinate system. In this viewpoint coordinate system, all six degrees of freedom of the user's viewpoint can be determined. The coordinates of the user's viewpoint position relative to the initial position of the origin are: The direction of the line of sight relative to the coordinate axes The positive direction exists The pitch, yaw, and roll angles.

[0082] II. Transformation between coordinate systems

[0083] 1. Conversion between device image depth coordinate system and viewpoint coordinate system

[0084] After a user wearing a VR headset gazes at an object in a 3D simulated flight scene, the VR device can obtain the following data:

[0085] The VR headset image depth coordinate system uses the coordinates of the person's gaze. VR headset's inherent field of view (FOV) and image resolution (horizontal FOV and resolution are denoted as...). The vertical horizontal field of view (FOV) and resolution are denoted as follows: .

[0086] Position of the person's viewpoint in the viewpoint coordinate system and posture

[0087] Let the current roll angle be The coordinates of the back gaze are The current pose VR headset image depth coordinate system uses the coordinates of the person's gaze. VR headset's inherent field of view (FOV) and image resolution (horizontal FOV and resolution are denoted as...). The vertical horizontal field of view (FOV) and resolution are denoted as follows: .

[0088] Position of the person's viewpoint in the viewpoint coordinate system and posture

[0089] Let the current roll angle be The coordinates of the back gaze are Current pose (Without roll angle) The coordinates of the gaze are As shown in Figures 7(a)-(b), it can be expressed by the following formula: (1)

[0090] The relative position of the gaze point in the viewpoint coordinate system can be obtained. The angle in the positive direction. Then, the position of the gaze point in the viewpoint coordinate system can be determined by... The data format is temporarily saved. and This is an intermediate variable in the formula, representing the pixel distance from the viewpoint to the imaging plane. Because... Since depth information cannot be represented in the viewpoint coordinate system, it needs to be transformed. Based on the principle of VR headset depth calculation, we can obtain... That is, in formula (2) below Partial. Assuming we are looking at the nearest object, the depth is... At that time, the distance between the viewpoint and the nearest object is At this time, the angle between the lines of sight of the left and right eyes is The depth at the farthest point is defined using the same method. Time distance is Depth is Distance at time .like Figure 8 As shown, the following formula can be used to calculate... . (2)

[0091] , Indicates the distance between the left and right eyes and the viewpoint. , The sum represents the distance between the two eyes. The above formula only calculates for two points at depths of 0% and 100%. Due to the small amount of data, the sampling error is large. Based on the calculation in formula (2), the method of this invention can also take several objects at different depths from objects with depths of 0% to 100% for calculation, and use the least squares method to obtain a more accurate result. Value, thus based on a more precise The calculated value has a small error. After completing the conversion between depth percentage and actual distance, record... , The following representation of gaze point position can be obtained. Then, based on the pose of the viewpoint and the line of sight, as well as the depth distance, the coordinates of the marked gaze point in the viewpoint coordinate system are obtained according to the following formula (3). ,like Figure 9 As shown: (3)

[0092] , , This represents the difference in distances along the x, y, and z directions between the coordinates of the viewpoint and the coordinates of the point of gaze in the viewpoint coordinate system.

[0093] This coordinate system transformation allows the objects that the user is looking at, including stationary objects on the ground and flying objects in the air (such as the aircraft where the VR headset is located, other aircraft, and clouds), to be transformed into coordinates in the viewpoint coordinate system, eliminating factors such as depth and posture during viewing.

[0094] 2. Conversion between viewpoint coordinate system and flight (machine) coordinate system

[0095] Since a VR headset represents its pose relative to an initially set origin in the current scene, and the origin of the VR headset may not necessarily coincide with the origin of the flight (native) coordinate system, it is necessary to convert the gaze point coordinates in the viewpoint coordinate system to coordinates in the flight (native) coordinate system. For ease of calculation, the method of this invention sets these two coordinate systems to coincide.

[0096] When the two coordinate systems do not coincide, such as Figure 10 As shown, the flight coordinate system has a displacement vector relative to the viewpoint coordinate system. and position The viewpoint coordinate system can be adjusted using the following formula. coordinates below Transform into flight (machine) coordinate system coordinates .

[0097] (4)

[0098] in, coordinate system Transform to coordinate system The rotation matrix is ​​expressed as shown in (5). These represent pitch, yaw, and roll angles, respectively. (5)

[0099] 3. Conversion between the flight (machine) coordinate system and the world coordinate system

[0100] The previous step involved modeling all objects that the user could look at, including all objects within the cockpit, the ground scene, clouds, and other aircraft. Since the aircraft is in motion, it's necessary to transform the objects outside the cockpit calculated in the viewpoint coordinate system to the world coordinate system. Currently, the following information can be obtained from the system modeling the scene:

[0101] World coordinate system Flight (machine) coordinate system The displacement vector of the origin of the world coordinate system relative to the viewpoint coordinate system of this aircraft at the current moment. and position

[0102] Any point on any object within the marked 3D simulated flight scene is the gaze point, that is, the coordinates of any point in space in the flight (machine) coordinate system. Let the coordinates in the world coordinate system be... .according to Figure 11 As shown, the following formula can be obtained based on the Euler angle transformation relationship and position transformation relationship between the two coordinate systems. (6)

[0103] in, coordinate system Transform to coordinate system The rotation matrix, These represent pitch, yaw, and roll angles, respectively. (7)

[0104] After the above formula transformation, it can be... Transformed into the world coordinate system It can represent stationary objects in a large-scale scene and serve as a transition coordinate system for other coordinate systems.

[0105] Each aircraft (not the user's own) is modeled separately, whether it is moving in the air or stationary on the ground, for the coordinate system of each other aircraft. Currently, the following information can be obtained using the method of this invention and previous calculations: such as... Figure 12 As shown:

[0106] World coordinate system Other machine coordinate system . The position vector of another machine relative to the world coordinate system at any given moment and attitude vector , Coordinates of the gaze point of the time marker in the world coordinate system ,

[0107] Assumption The gaze point of the time marker is in the world coordinate system. Then, based on the positional relationship of the other aircraft relative to the world coordinate system, the coordinates in the flight (other aircraft) coordinate system can be solved according to formula (8). .

[0108] Based on the Euler angle transformation relationship and position transformation relationship between the two coordinate systems, the following formula can be obtained.

[0109] (8)

[0110] in, coordinate system Transform to coordinate system The rotation matrix.

[0111] (9).

[0112] The advantage of establishing a coordinate system for another aircraft is that it makes the point of observation on the other aircraft independent of the movement of the user. As shown in Figures 13(a)-(b), if the user observes the left wing of another aircraft at a certain moment, and the pose of the other aircraft relative to the user changes at the next moment, the coordinates of the observation point will change with the change of the other aircraft.

[0113] Currently, all clouds are not modeled individually; that is, the clouds in the sky are not isolated entities, but rather a unified whole comprised of all connected clouds, all moving at a certain speed. Based on previous calculations, the following information can be obtained: Figure 5 As shown.

[0114] The constant velocity vector of the clouds at all times ;

[0115] The coordinates of the gaze point marked on the time marker are in the air and not near other machines. ;

[0116] Suppose for a certain The point on the cloud at any given moment, spatially labeled, is located in the world coordinate system as follows: Then, the coordinates in the cloud coordinate system at this time are It's over. The coordinates of the marked point after the specified time are: The cloud speed updates to , recorded as .

[0117] III. Generating Hotspot Areas

[0118] Step two yields the coordinates of the user's gaze point in various coordinate systems at a given moment. This step includes two aspects:

[0119] The action of "completion of gazing at an object" is defined as discrete in time on electronic devices such as computers. 60 calculation tasks are performed per second. A frame is a moment in 1 / 60th of a second. In this frame, part of the calculation task is for the VR headset to form an image frame. That is, the coordinates calculated in a frame are not necessarily the user's gaze at the object. It may also be the intermediate process of moving from one gazed object to another.

[0120] Eliminating dirty data during the calculation process is crucial. Dirty data refers to data where the calculated coordinates in adjacent frames have significant positional deviations. Failure to eliminate dirty data will increase the instability of the annotation of the gaze object.

[0121] The method of this invention uses a set of data from an actual testing process for example analysis, such as... Figure 14 As shown. In a simulated scene lasting one second and 60 frames, the user completed the operation of not looking at the dashboard - looking at the dashboard - not looking at the dashboard. The scene data reveals that within a short period, a large number of gaze points are concentrated in a certain area on the viewpoint coordinate system, denoted as a hotspot area. Based on these characteristics, the method of this invention proposes a hotspot area identification method based on spatial annotation. The method flow is as follows:

[0122] Step 1. For Distance is determined at the fixation point in the viewpoint coordinate system at any given time. If the distance between this point and the origin is less than a threshold... When an object is identified as belonging to the user's local machine, it is processed in the viewpoint coordinate system; if the distance from the origin is greater than a threshold... If an object is identified as not belonging to the local machine, it is converted to the world coordinate system for processing.

[0123] In the world coordinate system, there are three possible objects: stationary objects on the ground, clouds, and other moving objects (whether in the sky or on the ground). Therefore, a height threshold is set. If the value is above this threshold, the calculation will be performed in the cloud coordinate system; otherwise, it will continue to be performed in the world coordinate system. Simultaneously, all gaze points in the world coordinate system need to be transferred to the other machine's coordinate system for calculation.

[0124] Step 2. On the coordinate system corresponding to this point, use the coordinates of the gaze point. Centered on, establish a side length of A cube. Set the initial weights of the interior space of this cube to be... , The initial weights of the entire coordinate system space are integers. The significance of setting the weight k: When a spatial point is viewed, the weight of the cube containing that point is increased by k, meaning the probability of that region being viewed increases by k. When this weight exceeds a certain threshold... A single glance is considered a valid gaze.

[0125] Step 3. Next Moment The updated gaze point coordinates are Establish a side length with it as the center. The cube, the initial weights of the interior space of the cube are also... Each gaze is an independent action, producing the same side length and cube size;

[0126] Step 4. Find the overlapping area of ​​the two cubes. The weight of the overlapping area is the sum of the weights of the two cubes before they overlap. That is, the two cubes will overlap in some places in space. The weight of the overlapping area is k+k, and the weight of the non-overlapping area is still k.

[0127] Step 5. Set the decay parameters That is, at most need This can eliminate a single non-overlapping cube. If non-overlapping cubes are deleted directly, then the decay parameter n=k. The purpose of setting the decay parameter here is to delay the deletion of this non-overlapping cube. This is because it's possible that there is a cube at time t1, a cube at time t2 doesn't overlap with it (assumed to be dirty data), but at time t3 it's identical to the cube at time t1. This way, the original normal data can be preserved. After this, dirty data is completely eliminated. Since dirty data is far from normal data, its weights will not overlap with the cubes generated in subsequent time steps. Therefore, a decay parameter can be used, decreasing the value by one point at each time step until it reaches 0. Here, k / n is not an integer, and a decay parameter is used... Round it up. The non-negative weight space refers to the three-dimensional simulated flight scene, that is, the entire virtual world. All objects in this scene have a set weight in each defined coordinate system. Steps 2-4 are constantly processing spatial weights. The weights will change at different times and are not fixed values.

[0128] Step 6. Set hotspot threshold The spatial region whose weight is higher than the threshold after processing in step 5 is the hot spot region corresponding to the gaze point.

[0129] Step 7. Jump back to step 3, update the operation at the next time step t3, and obtain the hotspot region corresponding to the next gaze point. Repeat the operation to obtain an infinite number of gaze points corresponding to an infinite number of hotspot regions.

[0130] In step 1 and subsequent steps, the points and spaces are processed according to the classification in step 1.

[0131] This method can accurately and quickly determine the spatial extent of hotspot regions in space based on the sequence of time and the position of the gaze point. Different hotspot regions are not represented in a single coordinate system, but in their respective corresponding coordinate systems. For example, a stationary object on the ground can be represented in the world coordinate system, while a hotspot region on another machine can only be represented in that machine's coordinate system and cannot be formed in the world coordinate system.

[0132] At the same time, this method can eliminate the influence of dirty data. The decay parameter set earlier is to eliminate dirty data. Even if there are several large errors in a continuous fixation point, it will not affect the overall judgment.

[0133] IV. Calculate the modeling objects corresponding to hotspot regions

[0134] The previous step three recorded the hotspot areas. This step uses the hotspot areas to find the modeled objects, which will facilitate subsequent statistics and playback.

[0135] Step three records the weights of a hotspot region and its vicinity after it has been identified. For example... Figure 15 As shown, the deeper the region, the higher the weight. The method of this invention uses the following steps to determine the modeled object corresponding to the hotspot region:

[0136] 1) Prioritize calculation. Calculate the sum of the bounding sum of the current hotspot region and all nearby objects. If the bounding sum with a certain object is significantly greater than the bounding sum with all other objects, then mark this object as the corresponding modeling object;

[0137] 2) If the area is not significantly different from other objects or has no enclosing element, calculate the spatial center point of the hotspot region. Using this point as the center, establish a radius of... A sphere. The radius is gradually increased within a certain range until an object in the scene just enters the radius of this sphere and makes contact with the first object within that sphere; this object is the modeled object corresponding to the hotspot area.

[0138] V. Time-series data storage

[0139] The method of this invention retains information in the temporal dimension when storing data about hotspot areas and corresponding modeled objects. For a given frame, the specific data storage formats are as follows:

[0140] Hotspot area: {Task execution time, applied coordinate system, coordinates, weights}

[0141] Modeling Object: {Task Duration, Model Scene, Object Index Number}

[0142] The explanation is as follows: Mission duration: The time is counted from the moment the flight training mission begins;

[0143] Weight: A parameter used to determine whether an area is a hotspot;

[0144] Model Scenarios: Different application coordinate systems correspond to different model scenarios. The world coordinate system and cloud coordinate system correspond to the entire 3D simulation flight scenario, while the local and other aircraft coordinate systems correspond to the aircraft scenario.

[0145] Object Index Number: A unique identifier for the modeled object; recording this number is equivalent to recording the object's identity.

[0146] The applied coordinate systems are the various coordinate systems mentioned above: VR device image depth coordinate system, world coordinate system, flight coordinate system, cloud coordinate system, and viewpoint coordinate system.

[0147] The method of this invention supports displaying hotspot areas and corresponding modeled objects, and can reproduce the objects gazed at by pilots at different times throughout the flight mission. When a new pilot performs a flight mission again using a VR headset, this invention provides prompts based on the gaze information previously recorded by experienced pilots, manifested as highlighting the modeled objects corresponding to hotspot areas, allowing the new pilot to learn and train, thus enabling them to follow the rhythm of experienced pilots in mission execution. Furthermore, the gaze information from experienced pilots' training missions can be integrated into charts for new pilots to learn and train from.

[0148] When hotspot areas are set, in each frame (a calculation task performed every 1 / 60th of a second), the rendering of the VR image will be based on the highlighted hotspot areas in the corresponding coordinate system; the higher the weight, the brighter the area. When a modeled object is set, the object being viewed will be highlighted. The entire display process supports playback and pause.

[0149] In this invention, the modeled object and the object of interest are actually the same object. The only difference is the terminology used at different stages. The explanation is as follows: First, there is a set of gaze points in a three-dimensional space. These gaze points are then processed and calculated to determine the regions they represent in space; these regions are called hotspot regions. These hotspot regions are represented on a coordinate system, and coincidentally, objects within the scene are also modeled on this coordinate system. Therefore, the hotspot regions and the modeled scene objects can be bounded together, and the modeled object in the bounding sum is the object of interest / the object being gazed upon.

[0150] The present invention also provides a pilot training system based on VR devices for implementing the method, the system comprising:

[0151] The first module is used to establish the coordinate system and transformation relationships of all objects within the VR device's field of view;

[0152] The second module is used to establish the coordinates of the points of the object being viewed by the user of the VR device in the corresponding coordinate system.

[0153] The third module is used to establish hotspot regions based on the coordinates of the viewed point;

[0154] The determination module is used to determine the corresponding modeling object based on the hotspot area;

[0155] The storage module is used to store data related to the hotspot area and the modeled object for use during training.

[0156] The present invention also provides a computer device including a processor and a memory, wherein the memory stores a computer program, the computer program being loaded and executed by the processor to implement the method.

[0157] The terminology used is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in the embodiments of the invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0158] The foregoing description illustrates and describes several preferred embodiments of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A pilot training method based on VR equipment, wherein the VR equipment is installed in a simulated aircraft cockpit, characterized in that, The method includes: S1. Establish the coordinate system and transformation relationships for all objects within the VR device's field of view; S2. Establish the coordinates of the points of the object that the user is looking at in the VR device within the corresponding coordinate system; S3. Establish a hotspot region based on the coordinates of the viewed point. The process for establishing the hotspot region is as follows: S31. To The distance to the gaze point in the viewpoint coordinate system is determined at any given time. When the distance between the gaze point and the origin is less than a set threshold, the point is considered an object within the VR device and is processed in the viewpoint coordinate system. When the distance between the gaze point and the origin is greater than the set threshold, the point is considered an object outside the VR device and its host machine, and is converted to the world coordinate system for processing. S32. On the coordinate system corresponding to this point, establish a side with a length of [missing information] centered on the coordinates of the gaze point. Given a cube, set the initial weights of its internal space to be... ; S33. The Next Moment Establish a side with length as the center of the updated gaze point coordinates. The cube has initial weights of its interior space. ; S34. Find the overlapping region of the two cubes. The weight of the overlapping region is the sum of the weights of the two cubes before they overlap. S35. Set decay parameters ,repeat This process involves eliminating individual non-overlapping cubes by subtracting the weights of the non-negative weights in the entire coordinate system. , Round up to the nearest integer of the ratio k / n; S36. Set a hotspot threshold. When the weight obtained after processing in S35 is higher than the hotspot threshold, record the hotspot area. S37. Continue updating the time until the time expires and all recorded hotspot areas are obtained; S4. Determine the corresponding modeling objects based on the hotspot areas; S5. Store the data related to the hotspot area and the modeled object for use during training.

2. The pilot training method based on VR equipment according to claim 1, characterized in that, All objects within the VR device's field of view include: objects on the ground, moving objects in the air, and clouds in the sky.

3. The pilot training method based on VR equipment according to claim 2, characterized in that, The coordinate systems of all objects within the field of view of the VR device include: the VR device image depth coordinate system, the world coordinate system, the flight coordinate system, the cloud coordinate system, and the viewpoint coordinate system.

4. The pilot training method based on VR equipment according to claim 3, characterized in that, The coordinate system transformation relationships of all objects within the VR device's field of view include: the transformation between the VR device's image depth coordinate system and the viewpoint coordinate system, the transformation between the viewpoint coordinate system and the flight coordinate system, the transformation between the flight coordinate system and the world coordinate system, and the transformation between the world coordinate system and the cloud coordinate system.

5. The pilot training method based on VR equipment according to claim 4, characterized in that, The flight coordinate system includes the coordinate system of the simulated aircraft where the VR device is located and the coordinate systems of other aircraft within the field of view.

6. The pilot training method based on VR equipment according to claim 1, characterized in that, S4 includes the following steps: S41. Calculate the sum of the bounding area of ​​the current hotspot region and all nearby objects. If the sum of the bounding area with a certain object is significantly greater than the sum of the bounding areas with other objects, then mark this object as the modeling object corresponding to the current hotspot region. S42. If there is no significant difference in the surrounding area with other objects, or if there is no surrounding area, then a sphere is built with the spatial center point of the hot spot area as the center. The radius of the sphere is gradually increased within a certain range until it contacts the first object inside the sphere. This object is the modeling object corresponding to the current hot spot area.

7. The pilot training method based on VR equipment according to claim 1, characterized in that, The storage format is as follows: Hotspot area: {task execution time, applied coordinate system, coordinate system, weights}; Modeling objects: {task execution time, model scene, object index number}.

8. A pilot training system based on VR equipment, used to implement the method according to any one of claims 1-7, characterized in that, The system includes: The first module is used to establish the coordinate system and transformation relationships of all objects within the VR device's field of view; The second module is used to establish the coordinates of the points of the object being viewed by the user of the VR device in the corresponding coordinate system. The third module is used to establish hotspot regions based on the coordinates of the viewed point; The determination module is used to determine the corresponding modeling object based on the hotspot area; The storage module is used to store data related to the hotspot area and the modeled object for use during training.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 7.

Images