Real-time flight information display method based on augmented reality

Abstract

The invention relates to an aircraft display technology, and introduces a real-time flight information display method based on augmented reality, which comprises the following steps: (1) acquiring real-time flight attitude data, flight position data and surrounding environment data, and carrying out data preprocessing and fusion on the real-time flight attitude data, flight position data and surrounding environment data to obtain a real-time flight attitude data, flight position data and surrounding environment data; (2) acquiring real-time geographic coordinate data of flight according to a GPS (Global Positioning System), constructing a corresponding terrain model by utilizing a preset GIS (Geographic Information System) database and the real-time geographic coordinate data of the flight, and embedding the flight attitude data and the flight position data into the corresponding terrain model to obtain a three-dimensional scene; and (3) processing the three-dimensional scene through a graphic rendering technology to obtain saliency information of the three-dimensional scene, constructing an augmented reality model through a mode recognition algorithm in combination with the saliency information of the three-dimensional scene, superposing the augmented reality model into the three-dimensional scene, and outputting the three-dimensional scene in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft display, and particularly to a real-time flight information display method based on augmented reality. Background Art

[0002] In the field of aviation flight, manned aircraft are increasingly developed. During the flight of an aircraft, it is necessary to monitor the environmental conditions in real time to improve flight safety. At present, although the head-mounted display system can display data in real time, however, the slight delay in the displayed image after the aggregation of different data tables in response to the head movement of the pilot is very likely to cause the pilot to faint and hinder their perception of the surrounding environment. Therefore, ensuring low latency of the helmet display is crucial.

[0003] Secondly, although the current head-mounted display system can display data in real time, it cannot enhance the real-time display of the augmented reality scene and flight parameters to the pilot, which is not conducive to flight observation. Summary of the Invention

[0004] The present invention provides a real-time flight information display method based on augmented reality, and its main purpose is to improve the interactivity and immersion of the flight experience.

[0005] To achieve the above object, a real-time flight information display method based on augmented reality provided by the present invention includes the following steps:

[0006] Step 1: Obtain real-time flight attitude data, flight position data, and surrounding environment data, perform data preprocessing and fusion on the real-time flight attitude data, the flight position data, and the surrounding environment data to obtain flight attitude data and flight position data;

[0007] Step 2: Obtain real-time geographic coordinate data of the flight according to GPS, construct a corresponding terrain model by using a preset GIS database and the real-time geographic coordinate data of the flight, and embed the flight attitude data and the flight position data into the corresponding terrain model to obtain a three-dimensional scene;

[0008] Step 3: Process the three-dimensional scene through graphics rendering technology to obtain the saliency information of the three-dimensional scene, construct an augmented reality model by using a pattern recognition algorithm in combination with the saliency information of the three-dimensional scene, superimpose the augmented display model on the three-dimensional scene, generate navigation data according to a preset path planning algorithm, superimpose the navigation data on the three-dimensional scene, and fix the navigation data on the HUD area of the user and output the three-dimensional scene in real time.

[0009] Optionally, the Step 1 includes:

[0010] Step A1: Align the timestamps of the flight attitude data, the flight position data, and the surrounding environment data, and perform noise reduction processing on the aligned flight attitude data, flight position data, and surrounding environment data;

[0011] Step A2: Use a preset Kalman filter to fuse the noise-reduced flight attitude data, flight position data, and surrounding environment data to obtain flight attitude data and flight position data.

[0012] Optionally, step 3 includes:

[0013] Step C1: The navigation data includes data such as waypoints, direction indicators, distance information, speed suggestions, turning angles, etc.

[0014] Optionally, the flight attitude data is displayed in the HUD area.

[0015] Optionally, the flight attitude data includes flight pitch data, roll data, and yaw angle data.

[0016] In the embodiments of the present invention, by acquiring relevant flight data in real time, and preprocessing and fusing the relevant data in real time, flight attitude data and flight position data are obtained. The flight attitude data and flight position data are used for terrain model construction and three-dimensional scene rendering, combined with augmented reality technology, to achieve dynamic visualization and real-time output of the flight environment, providing intuitive visual support for navigation, obstacle avoidance, and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic flowchart of a method for displaying real-time flight information based on augmented reality provided by an embodiment of the present invention;

[0018] Figure 2 is a detailed schematic flowchart of a step of a method for displaying real-time flight information based on augmented reality provided by an embodiment of the present invention;

[0019] Figure 3 is a detailed schematic flowchart of a step of a method for displaying real-time flight information based on augmented reality provided by an embodiment of the present invention;

[0020] The realization, functional features, and advantages of the object of the present invention will be further described in conjunction with the embodiments with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] The embodiments of the present application provide a method for displaying real-time flight information based on augmented reality. In the embodiments of the present application, the execution subject of the method for displaying real-time flight information based on augmented reality includes, but is not limited to, at least one of electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiments of the present application. In other words, the method for displaying real-time flight information based on augmented reality can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster, etc. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0023] Referring to Figure 1 As shown, it is a schematic flowchart of a method for displaying real-time flight information based on augmented reality provided by an embodiment of the present invention. In this embodiment, the method for displaying real-time flight information based on augmented reality includes:

[0024] S1. Obtain real-time flight attitude data, flight position data, and surrounding environment data, perform data preprocessing and fusion on the real-time flight attitude data, the flight position data, and the surrounding environment data to obtain flight attitude data and flight position data;

[0025] In the embodiments of the present invention, relevant data of a flight device are usually obtained by using devices commonly used in flight devices such as an Electronic Flight Instrument System (EFIS), a radar detector, etc.

[0026] In the embodiments of the present invention, the flight attitude data includes flight pitch data, roll data, and yaw angle data.

[0027] Among them, including but not limited to, using an IMU to measure flight pitch data, roll data, and yaw angle data, using GPS to collect flight position data, including but not limited to position data such as longitude, latitude, and altitude, and using a vision sensor (such as a camera, etc.) to collect environmental images as a basis for subsequent augmented reality information.

[0028] Specifically, the S1 includes:

[0029] SA1. Align the time stamps of the flight attitude data, the flight position data, and the surrounding environment data, and perform noise reduction processing on the aligned flight attitude data, flight position data, and surrounding environment data;

[0030] In the embodiments of the present invention, the flight attitude data, the flight position data, and the surrounding environment data are time-stamped and aligned to ensure that the above data are strictly aligned in time, and the system clock is used as the reference for all times.

[0031] Further, the noise reduction process includes: performing low-pass filtering on the flight attitude data to remove high-frequency noise; performing median or Kalman filtering on the flight position data to ensure the smoothness of the position data.

[0032] Among them, outliers (such as sudden jumps) in the flight attitude data and the flight position data are detected regularly, and a redundancy mechanism is set to repair the outliers.

[0033] SA2. Use a preset Kalman filter to fuse the flight attitude data, the flight position data, and the surrounding environment data after noise reduction to obtain the flight attitude data and the flight position data.

[0034] In the embodiments of the present invention, the Kalman filter is initialized to fuse different data and predict the flight attitude data and the flight position data at the next moment. Among them, data such as GPS are continuously referenced to correct and optimize the predicted data until the flight attitude data and the flight position data are output.

[0035] In the embodiments of the present invention, the real-time flight information display method further includes a head-mounted display, which can also be set as a device such as glasses.

[0036] In the embodiments of the present invention, a camera is provided in the head-mounted display to capture the range visible to the human eye for subsequent image acquisition and image information processing. The display is located close to the human eye, and the picture is fixedly presented in the HUD area.

[0037] S2. Obtain the real-time geographical coordinate data of the flight according to GPS, construct a corresponding terrain model by using the preset GIS database and the real-time geographical coordinate data of the flight, and embed the flight attitude data and the flight position data into the corresponding terrain model to obtain a three-dimensional scene;

[0038] In the embodiments of the present invention, the global positioning system (GPS) is used to obtain the geographical coordinate data of the aircraft (including but not limited to unmanned aircraft, airplanes, etc.) in real time, including information such as longitude, latitude, and altitude. These data provide the precise position of the aircraft in three-dimensional space.

[0039] Among them, the geographical coordinate data obtained by using GPS are updated in real time to ensure that the dynamic changes of the aircraft position can be captured in time, providing a basis for subsequent modeling and scene construction.

[0040] In the embodiments of the present invention, the GIS database stores geographical information data, including terrain elevation, landform features, buildings, roads, rivers, etc. These data are the basis for constructing a three-dimensional terrain model.

[0041] In the embodiments of the present invention, based on the geographical information data in the GIS database and combined with the flight coordinate data obtained in real time, a three-dimensional terrain model of the flight area is constructed.

[0042] Among them, the three-dimensional terrain model not only includes the undulations of the terrain, but may also include the distribution of ground objects (including but not limited to buildings, vegetation, etc.).

[0043] In the embodiments of the present invention, embedding the flight attitude data and flight position data into the terrain model means dynamically associating the real-time state of the aircraft with the terrain model. For example, the position and attitude of the aircraft will be updated in real time in the terrain model, reflecting the actual motion state of the aircraft in three-dimensional space.

[0044] In the embodiments of the present invention, by embedding the flight attitude and position data into the terrain model, a dynamic three-dimensional scene is generated. This scene not only shows the terrain features of the area where the aircraft is located, but also reflects the motion state of the aircraft in real time.

[0045] Among them, the three-dimensional scene can be used to visualize the flight path, attitude changes of the aircraft, and the terrain features of the surrounding environment, which is of great significance for application scenarios such as flight monitoring, path planning, and obstacle avoidance.

[0046] In another embodiment of the present invention, if there are N (N>1) aircraft in the flight area, other aircraft flying in coordination are also loaded and superimposed on the three-dimensional scene. The specific implementation method is the same as the above implementation method and will not be elaborated in detail here.

[0047] S3. Process the three-dimensional scene through graphics rendering technology to obtain the saliency information of the three-dimensional scene. Through a pattern recognition algorithm, combined with the saliency information of the three-dimensional scene, an augmented reality model is constructed. The augmented display model is superimposed on the three-dimensional scene. Among them, navigation data is generated according to a preset path planning algorithm, the navigation data is superimposed on the three-dimensional scene, and the navigation data is fixedly presented in the HUD area of the user and the three-dimensional scene is output in real time.

[0048] In the embodiments of the present invention, graphics rendering technology refers to the process of converting a three-dimensional model into a two-dimensional image using computer graphics algorithms. Common rendering technologies include rasterization, ray tracing, shadow processing, texture mapping, etc.

[0049] In the embodiments of the present invention, the three-dimensional scene is processed through graphics rendering technology, aiming to enhance the visual effect of the scene, extract key information in the three-dimensional scene (such as lighting, materials, object outlines, etc.), and provide a basis for subsequent extraction of saliency information and construction of an augmented reality model.

[0050] Among them, saliency information refers to the regions or objects in the scene that can attract human visual attention. Such information usually includes color contrast, texture difference, moving objects, lighting changes, etc.

[0051] Among them, through graphics rendering technology, saliency information in the three-dimensional scene can be extracted. For example, the changes in lighting and shadows are used to highlight key objects in the scene, or the significant regions in the scene are identified through color and texture contrast.

[0052] In the embodiments of the present invention, a pattern recognition algorithm refers to an algorithm used to recognize and classify specific patterns in data. Common algorithms include machine learning, deep learning, feature extraction, etc.

[0053] Among them, combining the saliency information of the three-dimensional scene, a pattern recognition algorithm is used to construct an augmented reality model. This model can include virtual objects, information labels, interactive interfaces, etc., aiming to enhance the user's understanding and interactive experience of the three-dimensional scene.

[0054] In the embodiments of the present invention, a path planning algorithm is used to calculate the optimal path from the starting point to the ending point. Commonly used algorithms include, but are not limited to, the A* algorithm, Dijkstra algorithm, RRT (Rapidly-exploring Random Tree), etc.

[0055] Specifically, S3 includes:

[0056] SC1: The navigation data includes data such as path points, direction indications, distance information, speed suggestions, steering angles, etc.

[0057] Among them, the navigation data includes data such as path points, direction indications, distance information, speed suggestions, steering angles, etc.

[0058] Among them, the navigation data will be superimposed on the pilot's field of vision in a virtual manner, usually fixed in the HUD (Head-Up Display) area. The display positions of these parameters should ensure that the pilot's gaze can turn naturally without excessive head movement, thereby minimizing the perception burden and ensuring that the pilot can obtain these key data quickly and accurately.

[0059] In the embodiments of the present invention, through graphic rendering technology and pattern recognition algorithms, a dynamic three-dimensional scene has been constructed, which includes information such as terrain, the attitude and position of the aircraft, and the distribution of ground objects. The navigation data generated by path planning (such as path lines, target points, direction arrows, etc.) is superimposed on the three-dimensional scene, and this superimposition is achieved through augmented reality (AR) technology, that is, seamlessly integrating virtual navigation information with the real three-dimensional scene.

[0060] Among them, HUD is a display technology that directly superimposes key information onto the user's field of view. Users can obtain important information without having to look down at the instrument panel.

[0061] Among them, the navigation data (such as path lines, target points, direction indications, etc.) is fixed in a specific area of the HUD to ensure the consistency and readability of the information. For example, the navigation path can be displayed in the HUD in the form of a blue dotted line, the target point can be marked with a green circle, and at important navigation turning points or target positions, prominent marks can be set to remind the pilot of the upcoming key nodes. At the same time, the arrow indicates the future heading of the aircraft. According to the real-time changing meteorological conditions (such as wind speed, air pressure, etc.) and flight status (such as fuel consumption, load), the system can intelligently adjust the color or thickness of the flight path to emphasize the current environmental limitations. For example, when the wind speed increases, the flight path can be displayed in a more obvious color to remind the pilot to adjust the flight strategy.

[0062] In the embodiments of the present invention, the constructed augmented reality model is integrated with the original three-dimensional scene, that is, virtual objects or information labels are accurately placed at the corresponding positions in the three-dimensional scene to ensure the spatial alignment of the augmented reality model and the three-dimensional scene, making the virtual objects look as if they really exist in the scene.

[0063] In the embodiments of the present invention, by obtaining relevant flight data in real time and preprocessing and fusing the relevant data in real time, flight attitude data and flight position data are obtained. The flight attitude data and flight position data are used for terrain model construction and three-dimensional scene rendering, combined with augmented reality technology, to achieve dynamic visualization and real-time output of the flight environment, providing intuitive visual support for navigation, obstacle avoidance, and decision-making.

[0064] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0065] Therefore, from any perspective, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Accordingly, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced by the present invention. Any reference signs in the claims should not be construed as limiting the claims concerned.

[0066] The blockchain referred to in the present invention is an application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, in essence, is a decentralized database, a series of data blocks generated by using cryptographic methods. Each data block contains information on a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block. The blockchain can include the blockchain underlying platform, the platform product service layer, and the application service layer, etc.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use the knowledge to obtain the best results.

[0068] In addition, it is obvious that the term "including" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices stated in the system claims can also be implemented by one unit or device through software or hardware. Words such as first, second, etc. are used to denote names and do not denote any particular order.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A real-time flight information display method based on augmented reality, characterized in that: The method comprises the following steps: Step 1: Acquire real-time flight attitude data, flight position data, and surrounding environment data, perform data preprocessing and fusion on the real-time flight attitude data, the flight position data, and surrounding environment data, and obtain flight attitude data and flight position data; Step 2: Acquire the real-time geographic coordinate data of the flight according to the GPS, build a corresponding terrain model using a preset GIS database and the real-time geographic coordinate data of the flight, embed the flight attitude data and the flight position data into the corresponding terrain model, and obtain a three-dimensional scene; Step 3: Process the three-dimensional scene through graphics rendering technology to obtain saliency information of the three-dimensional scene, and construct an augmented reality model through a pattern recognition algorithm in combination with the saliency information of the three-dimensional scene. Superimpose the augmented display model on the three-dimensional scene, generate navigation data according to a preset path planning algorithm, superimpose the navigation data on the three-dimensional scene, and present the navigation data fixedly in the user's HUD area, and output the three-dimensional scene in real time.

2. A method for displaying real-time flight information based on augmented reality as claimed in claim 1, characterized in that: The step 1 comprises: Step A1: performing time stamp alignment on the flight attitude data, the flight position data and the surrounding environment data, and performing noise reduction processing on the aligned flight attitude data, the flight position data and the surrounding environment data; Step A2: Using a preset Kalman filter to fuse the flight attitude data, flight position data and surrounding environment data after noise reduction processing, to obtain flight attitude data and flight position data.

3. The method for displaying real-time flight information based on augmented reality according to claim 1, characterized in that: The step 3 comprises: Step C1: The navigation data includes path points, direction indications, distance information, speed recommendations, steering angles and other data.

4. The method for displaying real-time flight information based on augmented reality according to claim 1, characterized in that: The flight attitude data is displayed in the HUD area.

5. The method for displaying real-time flight information based on augmented reality according to claim 1, characterized in that: The flight attitude data includes flight pitch data, roll data, and yaw angle data.

Images